vania-janet/multiturn-rag-retrieval-data huggingface dataset:
<-- </section "id="section-sum-syntax" "> --><-- <section "id="section-sum-example" "> --> Example <-- <table> -->Table 6. Sample product prices| Product | Price || -------- | ----- || I Series | 200 || J Series | 450 || X Series | 325 |<-- </table ""> -->When the sum aggregation type is applied to a set of documents in which the price field contains the values that are shown in Table 6, the result is 975. sum(price)=975
This aggregation type performs a terminal operation. When combined with other aggregations, the output is not used as input for the next aggregation. The output is returned in a discrete group.<-- </section "id="section-sum-example" "> --><-- </section "id="section-sum" "> --><-- <section "id="section-term" "> --> term Indicates the frequency of a term or set of terms in a set of queried documents.<-- <section "id="section-term-syntax" "> --> Syntax term(field:{field_name})
You can optionally specify the following parameters:<-- <ul> --> * count: Specifies the maximum number of terms to return. * name: You can optionally include a custom name. Not returned if relevancy information is included in the request. * relevancy: Boolean value that indicates whether to include relevancy information in the result. You can use relevancy to get a score that indicates the level of relevancy between the term and keywords in the query. This parameter is false by default. If set to true, the following fields are returned also:
<-- <ul> -->

* total_matching_documents: Number of documents in the collection where the term is mentioned in the specified field.
* estimated_matching_results: Number of documents that are estimated to have the term in the specified field in the set of documents that are returned by the query.
Query history 

A query history audits all the current and past queries across the existing engines in IBM® watsonx.data.

The query history page in watsonx.data provides the following details that are related to the queries that are run:



*  Query ID
*  Query
*  State
*  Engine
*  User
*  Created



In the Query history page, you can search, refresh, filter, and customize the queries. You can select a Query from the page, view or copy the details of query statement, logical execution plan, and distributed execution plan. You can open the queries directly in a workspace, and also get the explain details of a query from the overflow menu of each query listed.

For more information about exporting and importing query history, see [Exporting and importing the query history](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-eximp-q-hist).
--resource-query Optional Enter the ID of a resource query that you created. A resource query helps to dynamically build your resource inventory by using the IBM Cloud resources that you created with a Schematics Workspace. To create a resource query, see the ibmcloud schematics resource-query create[command](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-create-rq). To enter multiple resource query IDs, use --resource-query id1 --resource-query id2. 
 --file or -f Optional The path to the JSON file where you specified the resource inventory that you want to create. 
 --output or -o Optional Specify the output format. Only JSON format is supported. 
 --no-prompt Optional Set this flag to create an inventory without an interactive command-line session. 





 Using the payload file 

You can provide a payload file to specify certain parameters for the inventory create command. Then, you pass the file name to the command by using the --file command option.

You need to replace the <...> placeholders with the actual values. For example, "<INVENTORY_NAME>" as "myinventory".

Syntax

{
"name": "<INVENTORY_NAME>",
"description": "<DESCRIPTION",
"location": "<GEOGRAPHY>",
"resource_group": "<RESOURCE_GROUP>",
"resource_queries": [
"<RESOURCE_QUERY_ID>"
]
}

Example

{
"name": "myinventory",
"description": "This is the resource inventory for production",
"location": "us-east",
"resource_group": "default",
"resource_queries": [
"default.RESOURCEQUERY.string.df3d8a47"
]

}

ibmcloud schematics inventory create --file inventory.json





 Using the interactive mode
For more information, see [Confidence scores](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-toolingconfidence).
* Discovery returns query results that include special characters for the following languages: English, German, French, Dutch, Italian, and Portuguese. For example, if you query for aqui, you receive results for both aqui and aquí.







 Options for querying 



* You have the option to turn on passage retrieval. Passages are short, relevant excerpts extracted from the full documents returned by your query. By default, up to 10 passages of around 400 characters each are returned for a query. A maximum of three passages are extracted from a single result. The passages parameter is only available for private collections; it is not available in the Discovery News collection. For more information about how passages are identified, see [Passages](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parameterspassages).
* You can create longer, more complex queries that include multiple filters and complex aggregations. This option is available only in the API and increases the character limit of a query to 10,000 characters. For more information, see [Long collection queries](https://cloud.ibm.com/apidocs/discoverylong-collection-queries) and [Long environment queries](https://cloud.ibm.com/apidocs/discoverylong-environment-queries).

For more information about writing queries, see:



* [Getting started with querying tutorial](https://cloud.ibm.com/docs/discovery?topic=discovery-getting-started-with-querying)
* [Query reference](https://cloud.ibm.com/docs/discovery?topic=discovery-query-reference) (includes the list of parameters, operators, and aggregations available in the Discovery Query Language)









 The Discovery data schema 

Start out by getting to know the Discovery JSON. To understand how to build a query using the Discovery Query Language, you need to be familiar with the JSON produced by Discovery after it enriches the documents in your collection.
* find_answers is optional and defaults to false. If it is set to true (and the natural_language_query parameter is set to a query string), the answer-finding feature is enabled.
* max_answers_per_passage is optional and defaults to 1. In this case, the answer-finding feature finds the number of answers that are specified at most from any one passage.



A section is also added to the return value within each passage object. That section is called answers, and it is a list of answer objects. The list can be up to max_answers_per_passage in length. Each answer object contains the following fields:



* answer_text is the text of the concise answer to the query.
* confidence is a number between 0 and 1 that is an estimate of the probability that the answer is correct. Some answers have low confidence and are unlikely to be correct. Be selective about what you do with answers based on this value. The confidence and order of documents in the search results are adjusted based on this combination if the per_document parameter of passage retrieval is set to true (which is the default).
* start_offset is the start character offset (the index of the first character) of the answer within the field that the passage came from. It is greater than or equal to the start offset of the passage (since the answer must be within the passage).
* end_offset is the end character offset (the index of the last character, plus one) of the answer within the field that the passage came from. It is less than or equal to the end offset of the passage.



To find answers across the entire project:



* Set passages.enabled to true
* Set passages.find_answers to true



To find answers within a single known document (for example, a document review application with long, complex documents):



* Set passages.enabled to true
* Set passages.find_answers to true
* Set filter to select the document_id for the document



The following example shows a query that uses this API:

POST /v2/projects/{project_id}/query{
"natural_language_query": "Why did Nixon resign?",
Specifying a value for cross-reference does not link one document to another; rather, it ensures that the service retains the document's pertinent information in case the document gets renamed or overwritten.
* relevance: An integer from 0 to 100, inclusive, that indicates the relative relevance of the query to the training data. Higher values indicate higher relevance. A value of 0 indicates no relevance to the query, whereas a value of 100 indicates absolute relevance to the query.

Although the range of the relevance parameter is 0 to 100 for maximum flexibility, you can use a smaller range to simplify scoring examples. A standard range might be 0 to 4, or you could use the entire range but only ever use increments of 20 (0,20,40,60,80, and 100). The Discovery tooling uses the relevance scores of 0 to represent not relevant and 10 to represent relevant. If you plan to score your documents, using both the Discovery tooling and the API or if you plan to begin with the API and move to the tooling, use the 0 and 10 relevancy scores.





The following object shows a populated training-data query.

{
"natural_language_query": "why is the sky blue",
"filter": "text:meteorology",
"examples": [
{
"document_id": "9afcf06e-2ef7-438e-b2a5-e4d9432dc877",
"relevance": 0
},
{
"document_id": "54f95ac0-3e4f-4756-bea6-7a67b2713c81",
"relevance": 1
},
{
"document_id": "01bcca32-7300-4c9f-8d32-33ed7ea643da",
"cross_reference": "my_id_field:1463",
"relevance": 4
}
]
}
Show more
Note the : operator in place of a dot (intents:intent)

Use the same pattern to match on any field of a detected intent or entity in the response. For example, this query matches any logged event where the response includes a detected entity with the value soda:

response.output.entities:value::soda

Similarly, you can filter on intents or entities sent as part of the request, as in this example:

request.input.intents:intent::hello





 Filtering by other fields 

To filter on another field in the log data, specify the location as a path identifying the levels of nested objects in the JSON response from the /logs API. Use dots (.) to specify successive levels of nesting in the JSON data. For example, the location request.input.text idenfities the user input text field as shown in the following JSON fragment:

"request": {
"input": {
"text": "Good morning"
}
}

Filtering is not available for all fields. You can filter on the following fields:



* assistant_id
* customer_id
* language
* request.context.global.system.user_id
* request.input.text
* request_timestamp
* response.context.global.system.user_id
* response.output.entities
* response.output.intents
* response_timestamp
* session_id
* skill_id
* snapshot



Filtering on other fields is not currently supported.







 Examples 

The following examples illustrate various types of queries using this syntax.



 Description Query 

 The date of the response is in the month of July 2020. response_timestamp>=2020-07-01,response_timestamp<2020-08-01 
 The timestamp of the response is earlier than 2019-11-01T04:00:00.000Z. response_timestamp<2019-11-01T04:00:00.000Z 
 The message is labeled with the customer ID my_id. customer_id::my_id 
 The message was sent to a specific assistant.
Filter search results are not returned in order of relevance.When you write a query that includes both a filter, and an aggregation, query, or natural_language_query parameter, the filter parameter runs first, and then any aggregation, query, or natural_language_query parameters run in parallel.With a simple query, especially on a small data set, the filter and query parameters often return the exact same (or similar) results. If the filter and query calls return similar results, and you don't need the responses to be returned in order of relevance, use the filter parameter. Filter calls are faster and are cached. Caching means that the next time you make the same call, you get a much quicker response, particularly in a big data set.<-- </section "id="section-filter" "> --><-- <section "id="section-query-parameters-structure" "> --> Structure parameters Structure parameters define the content and organization of the documents in the returned JSON. Structure parameters don't affect which documents are part of the entire results set.<-- </section "id="section-query-parameters-structure" "> --><-- <section "id="section-return" "> --> return A comma-separated list of the portion of the document hierarchy to return. Any of the document hierarchies are valid values. If this parameter is an empty list, then all fields are returned.<-- </section "id="section-return" "> --><-- <section "id="section-count" "> --> count The number of documents that you want to return in the response. The default is 10. The maximum for the count and offset values together in any one query is 10000.<-- </section "id="section-count" "> --><-- <section "id="section-offset" "> --> offset Index value of the position of the search result where the set of results to return begins. For example, if the total number of results that are returned is 10, and the offset is 8, it returns the last two results. The default is 0.
You can use relevancy to get a score that indicates the level of relevancy between the term and keywords in the query. This parameter is false by default. If set to true, the following fields are returned also:
<-- <ul> -->

* total_matching_documents: Number of documents in the collection where the term is mentioned in the specified field.
* estimated_matching_results: Number of documents that are estimated to have the term in the specified field in the set of documents that are returned by the query.

<-- </ul> -->

<-- </ul> --><-- </section "id="section-term-syntax" "> --><-- <section "id="section-term-example" "> --> Example The following example returns the text from the recognized entities in the document, and specifies to return a maximum of 10 terms.For example: term(enriched_text.entities.text,count:10)
When relevancy is set to true, a relevancy score is shown in the results. Relevancy measures the level of uniqueness of the frequency count compared to other documents that match your query. If the relevancy shows 2.0, it means that the number of times that the two data points intersect is 2 times larger than expected.For more examples, see Grouping documents]] and Combining aggregation types]] ! ! ! ! ! .<-- </section "id="section-term-example" "> --><-- </section "id="section-term" "> --><-- <section "id="section-timeslice" "> --> timeslice A specialized histogram that uses dates to create interval segments.<-- <section "id="section-timeslice-syntax" "> --> Syntax The syntax is timeslice({field},{interval},{time_zone}).<-- <ul> --> * The field that you specify must have a date data type. For more information about date field, see How dates are handled]] ! ! ! ! ! !
discovery.training-query.read Read the contents of a query 
 discovery.training-example.create Add an example to a query in the training data 
 discovery.training-example.update Update an example of a query in the training data 
 discovery.training-example.delete Delete an example from a query in the training data 
 discovery.training-example.read Read the contents of a training example 
 discovery.training-examples.read Read the contents of all examples in a query 
 discovery.collection-training-status.read Get the training status of a single-collection training 
 discovery.multi-collection-training-status.read Get the training status of a multi-collection training 
 discovery.metric.read Request a metric 
 discovery.event.create Add a click event to a query 







 Viewing events 

Events that are generated by an instance of the Discovery service are automatically forwarded to the IBM Cloud Activity Tracker service instance that is available in the same location.

IBM Cloud Activity Tracker can have only one instance per location. To view events, you must access the web UI of the IBM Cloud Activity Tracker service in the same location where your service instance is available. For more information, see [Navigating to the UI](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-launch).
Query aggregations 

Aggregations return a set of data values. For the complete list of available aggregations, see the [Query reference](https://cloud.ibm.com/docs/discovery?topic=discovery-query-referenceaggregations).



 term 

Returns the top values (by score and by frequency) for the selected enrichments. All enrichments are valid values. You can optionally use count to specify the number of terms to return. The count parameter has a default value of 10. This example returns the full text and enrichments of the top values with the concept enrichment, and specifies to return 10 terms.

For example:

term(enriched_text.concepts.text,count:10)





 filter 

A modifier that narrows the document set of the aggregation query it precedes. This example filters down to the set of documents that include the concept Cloud computing.

For example:

filter(enriched_text.concepts.text:"cloud computing")





 nested 

Applying nested before an aggregation query restricts the aggregation to the area of the results specified. For example: nested(enriched_text.entities) means that only the enriched_text.entities components of any result are used to aggregate against.

For example:

nested(enriched_text.entities)





 histogram 

Creates numeric interval segments to categorize documents. Uses field values from a single numeric field to describe the category. The field used to create the histogram must be of number (integer, float, double, or date) type. Non-number types such as string are not supported. For example, "price": 1.30 is a number value that works, and "price": "1.30" is a string, so it wouldn’t work. Use the interval argument to define the size of the sections the results are split into. Interval values must be whole, non-negative numbers, and are set to make sense for segmenting your possible field values.
"field": "CITY",
"count": 2,
"results":
{
"key": "FONTANA",
"matching_results": 6
},
{
"key": "ALTA LOMA",
"matching_results": 5
}
]
}
] ! ! ! !
}
Show more
<-- </section "id="section-aggfilter-example" "> --><-- </section "id="section-aggfilter" "> --><-- <section "id="section-group_by" "> --> group_by Separates results into groups that you define.<-- <section "id="section-group_by-syntax" "> --> Syntax group_by(condition:(condition 1),(condition 2)...] ! ! )
Each condition must be specified as a valid Discovery Query Language expression surrounded by parentheses. For example, (age<20) or (flavor:chocolate). The maximum number of conditions that you can define is 50.You can optionally include the relevancy parameter and set it to true to return the relevancy value of the set of documents that meet the specified condition. When true, the results are sorted by relevance. When false, the results are sorted by the highest number of matching_results.<-- </section "id="section-group_by-syntax" "> --><-- <section "id="section-group_by-example" "> --> Example The following request looks for documents that mention the term engine, and groups them by car manufacturing year. The documents are sorted into 3 groups, one group of traffic incident reports involving cars that were manufactured before 2000, one group for cars manufactured in 2000, and one group for cars manufactured after 2000. {
"query":"engine",
"aggregation": "group_by(condition:(YEARTXT<2000),(YEARTXT=2000),(YEARTXT>2000)] ,relevancy:true)"
This documentation is for the classic Watson Assistant experience. To see the documentation for the new Watson Assistant, please go [here](https://cloud.ibm.com/docs/watson-assistant).



 Defining information to look for in customer input 

Entities represent information in the user input that is relevant to the user's purpose.

If intents represent verbs (the action a user wants to do), entities represent nouns (the object of, or the context for, that action). For example, when the intent is to get a weather forecast, the relevant location and date entities are required before the application can return an accurate forecast.

Recognizing entities in the user's input helps you to craft more useful, targeted responses. For example, you might have a buy_something intent. When a user makes a request that triggers the buy_something intent, the assistant's response should reflect an understanding of what the something is that the customer wants to buy. You can add a @product entity, and then use it to extract information from the user input about the product that the customer is interested in. (The @ prepended to the entity name helps to clearly identify it as an entity.)

Finally, you can add multiple responses to your dialog tree with wording that differs based on the @product value that is detected in the user's request.



 Entity evaluation overview 

Your assistant detects entities in the user input by using one of the following evaluation methods:



 Dictionary-based method 

Your assistant looks for terms in the user input that match the values, synonyms, or patterns you define for the entity.



* Synonym entity: You define a category of terms as an entity (color), and then one or more values in that category (blue). For each value you specify a bunch of synonyms (aqua, navy). You can also pick synonyms to add from recommendations made to you by Watson.

At run time, your assistant recognizes terms in the user input that exactly match the values or synonyms that you defined for the entity as mentions of that entity.
* Pattern entity: You define a category of terms as an entity (contact_info), and then one or more values in that category (email).
A boolean that specifies whether the service returns a set of the most relevant passages from the documents returned by a query that uses the natural_language_query parameter. The passages are generated by sophisticated Watson algorithms to determine the best passages of text from all of the documents returned by the query. The default is false.

The passages parameter can only be used on private collections. It cannot be used on the IBM Watson™ Discovery News collection.

Discovery attempts to return passages that start at the beginning of a sentence and stop at the end using sentence boundary detection. To do so, it first searches for passages approximately the length specified in the [passages.characters parameter](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parameterspassages_characters). The default is 400. It then expands each passage to the limit of twice the specified length to return full sentences. If your passages.characters parameter is short and/or the sentences in your documents are very long, there might be no sentence boundaries close enough to return the full sentence without going over twice the requested length. In that case, Discovery stays within the limit of twice the passages.characters parameter, so the passage returned does not include the entire sentence and omits the beginning, end, or both.

Because sentence boundary adjustments expand passage size, there is a substantial increase in average passage length. If your application has limited screen space, you might want to set a smaller value for passages.characters and/or truncate the passages that are returned by Discovery. Sentence boundary detection works for all supported languages and uses language-specific logic.

You can adjust the fields in the documents over which passage retrieval searches with the [passages.fields](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parameterspassages_fields) parameter.

The passages parameter returns matching passages (passage_text), as well as the score, document_id, the name of the field the passage was extracted from (field), and the starting and ending characters of the passage text within the field (start_offset and end_offset), as shown in the following example. The query is shown at the top of the example.