vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* [min](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationsmin)
* [nested](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationsnested)
* [pair](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationspair)
* [sum](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationssum)
* [term](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationsterm)
* [timeslice](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationstimeslice)
* [top_hits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationstop_hits)
* [trend](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationstrend)
* [topic](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationstopic)
* [unique_count](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregationsunique_count)



For Document Retrieval project types, when you don't include an aggregation parameter in a query request, a default aggregation request is applied. For more information, see [Document Retrieval project aggregations](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-defaultsaggregations-dr).

For more information about how to submit a query, see the Discovery [API reference](https://cloud.ibm.com/apidocs/discovery-dataquery).





 average 

Returns the mean of values of the specified field across all matching documents.
For example, you can use an aggregation to search a set of traffic incident reports for documents that mention the term brake. And from the returned documents, find reports from the US states with the most relevant mentions of the term.

In the following example request, the count parameter that returns only 3 aggregation results is included to make the example easier to follow.

{
"query":"brake",
"aggregation": "term(field:STATE,count:3,relevancy:true)"
}

The output of the aggregation parameter is returned in an aggregations object that is displayed before the results object, which contains the query results. A maximum of 50,000 values can be returned in the aggregations object for a single query.

The resulting aggregations object contains summary information about the query results. In this example, for instance, it shows that traffic incident reports from New York, California, and Florida have the most relevant mentions of the term brake.

{
"matching_results": 9064,
"retrieval_details": {
"document_retrieval_strategy": "untrained"
},
"aggregations": [
{
"type": "term",
"field": "STATE",
"results":
{
"key": "NY",
"matching_results": 693,
"relevancy": 1.1649531567631084,
"total_matching_documents": 2156,
"estimated_matching_results": 542
},
{
"key": "CA",
"matching_results": 1210,
"relevancy": 1.1170819184294765,
"total_matching_documents": 4017,
"estimated_matching_results": 1011
},
{
"key": "FL",
"matching_results": 511,
"relevancy": 0.828014956418841,
"total_matching_documents": 2199,
"estimated_matching_results": 553
If you test the same search term on a small data set, you might find that the filter and query parameters return very similar (if not identical) results. However, there is a difference between the two parameters.



* Using a filter parameter alone returns search results in no specific order.
* Using a query parameter alone returns search results in order of relevance.



In large data sets, if you need results returned in order of relevance, it is advisable that you combine the filter and query parameters because using them together improves performance. The reason is because the filter parameter runs first and caches results, and then the query parameter ranks them. For an example of using filters and queries together, see [Building combined queries](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsbuilding-combined-queries). Filters can also be used in aggregations.

When you write a query that includes both a filter, and an aggregation, query, or natural_language_query parameter; the filter parameters run first, after which any aggregation, query, or natural_language_query parameters run in parallel.

With a simple query, especially on a small data set, filter and query often return the exact same (or similar) results. If a filter and query call return similar results, and getting a response in order of relevance does not matter, it is better to use filter because filter calls are faster and are cached. Caching means that the next time you make that call, you get a much quicker response, particularly in a big data set.







 aggregation 

Aggregation queries return a count of documents matching a set of data values; for example, top keywords, overall sentiment of entities, and more. For the full list of aggregation options, see the [Aggregations table](https://cloud.ibm.com/docs/discovery?topic=discovery-query-aggregations). These aggregations are written using the [Discovery Query Language](https://cloud.ibm.com/docs/discovery?topic=discovery-query-operators).





 natural_language_query
The maximum for the count and offset values together in any one query is 10000.<-- </section "id="section-count" "> --><-- <section "id="section-offset" "> --> offset Index value of the position of the search result where the set of results to return begins. For example, if the total number of results that are returned is 10, and the offset is 8, it returns the last two results. The default is 0. The maximum allowed value for the count and offset together in any one query is 10000.<-- </section "id="section-offset" "> --><-- <section "id="section-spell" "> --> spell correction In natural language queries, checks the query that is submitted for misspelled terms. The query is processed as-is. However, likely corrections to the original query, if any exist, are returned in the suggested_query field of the response. The suggestions are not used automatically, but your application can make use of them.<-- </section "id="section-spell" "> --><-- <section "id="section-sort" "> --> sort A comma-separated list of fields in the document to sort by. You can optionally specify a sort direction by prefixing the field with - for descending order or + for ascending order. Ascending order is the default sort direction.<-- </section "id="section-sort" "> --><-- <section "id="section-highlight" "> --> highlight A Boolean value that specifies whether to include a highlight object in the returned output. When included, the highlight returns keys that are field names and values that are arrays. The arrays contain segments of query-matching text that is highlighted by using the HTML emphasis (<em>) tag.This parameter is ignored if passages.enabled and passages.per_document are true, in which case passages are returned for each document instead of highlights.Currently, if the query searches for an exact match of an enrichment mention, only lowercase matches are highlighted.
,
"field": "text",
"start_offset": 281,
"end_offset": 553,
"answers":
{
"answer_text": "his complicity in the cover-up made public and his political support completely eroded",
"start_offset": 286, "end_offset": 373, "confidence": 0.78214
}
]
}
] ! ! ! !
}
Show more
<-- </section "id="section-answer-finding-api" "> --><-- </section "id="section-answer-finding" "> --><-- <section "id="section-nlq" "> --> natural_language_query Use a natural language query to enter queries that are expressed in natural language, as might be received from a user in a conversational or free-text interface, such as IBM Watson Assistant. The parameter uses the entire input as the query text. It does not recognize operators.The maximum query string length for a natural language query is 2048.<-- <section "id="section-nlq-confidence" "> --> Result confidence scores When the query type is a natural language query, each result has a confidence score. The confidence score is a measure of the relevancy of the result. Each query result is evaluated and scored independently.A variety of techniques are used to evaluate confidence. One important factor is the frequency of word matches between the query and the document.Because a variety of techniques are used in different contexts to evaluate the result, the number range of result scores can vary widely from query to query. This variability means that comparing the confidence score to a static threshold value is an unsuitable method by which to delimit the results that are returned by your application. Results are ordered from highest to lowest confidence. You can find the best candidate answers by taking the top results, regardless of their confidence score values.The natural_language_query parameter enables capabilities such as relevancy training. For more information, see Improving result relevance with training]] ! ! ! ! !
windowClusterBy 





 windowPartitionBy 





 windowOrderBy 

The window specification consists of the following clauses:



* The PARTITION BY clause defines which rows belong to the same window partition. DISTRIBUTE BY can be used as a synonym for PARTITION BY.
* The ORDER BY clause defines the ordering of rows within a window partition. SORT BY can be used as a synonym for ORDER BY.
* The two ways to define a window size are:



* RANGE: The window size is defined as a value range.
* ROW: The window size is defined as the number of rows before and/or after the current row.



* The following keywords can be used to define range boundaries:



* CURRENT ROW: Specifies to use the current row as a bound.
* UNBOUNDED: Specifies to use negative or positive infinity as a lower or upper bound.
* Depending on the context, PRECEDING specifies:



* RANGE: The lower bound of the value range.
* ROWS: The number of rows before the current row.



* Depending on the context, FOLLOWING specifies:



* RANGE: The upper bound of the value range.
* ROWS: The number of rows after the current row.











 windowFrame 





 frameBound 





 Examples - window function 

The following window function examples use values clauses to define result sets for group by operations. For more information about the values clause, see [valuesClause](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencevaluesClause).



 Ranking function example 

This example uses a table that contains information about employee social media activity (posts):



* Column 1: employee ID
* Column 2: department ID
* Column 3: number of social media posts



The example shows how to retrieve the ID of the two most active employees in each department.

-- derive posts ranking by using a named window specification
SELECT 
FROM (
SELECT
posts.col1 AS emp_id,
posts.col2 AS dept_id,
posts.col3 AS posts,
DENSE_RANK() OVER post_ranking AS rank
FROM VALUES
(1, 1 ,100),
: When you include the string variation operator with query input that contains a phrase, the variation is applied to each word in the phrase. For example, "tom cat"1 matches top hat in addition to tom cat. For more information about Discovery Query Language operators, see [Query operators](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operators).





 10 Febuary 2023 

Entity extractor is generally available
: The Extract entities enrichment brings the powerful ability to build a custom type system into Discovery. Use the tool to label entity examples within your industry data to build a machine learning model that Discovery can use to recognize meaningful terms for your business. Already built an entity type system in Knowledge Studio? You can use the corpus from Knowledge Studio as a starting point for your Discovery entity extractor training data. For more information, see [Entity extractor](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractor).

If you created an entity extractor enrichment for testing purposes when the feature was in beta release, now that it is generally available, it will count toward your custom model limit. The entity extractor enrichment incurs charges whether or not it is applied to a collection.





 7 Febuary 2023 

Support for hourly crawls was removed
: You can no longer choose to crawl a data source every hour. If an existing collection is configured to crawl hourly, you will be prompted to change the scheduled crawl the next time you edit the connector settings.

You can no longer enable FAQ extraction for a collection
: The checkbox to enable or disable the beta FAQ extraction feature was removed. FAQ extraction was a beta feature that captured question-and-answer pairs from the data source as it was crawled. FAQ extraction generated a new subdocument for each pair and stored the question in the title field and the answer in the text field.

You cannot apply FAQ extraction to new collections.

Any existing collections with FAQ extraction enabled retain FAQ documents in their indexes until the collection is reprocessed. At that time, most of the question-and-answer pair subdocuments are deleted.
Improving result relevance with training 

The relevance of natural language query results can be improved in IBM Watson® Discovery with training.

A relevancy model determines the most relevant documents to return in search results. Without relevancy training, a standard mechanism is used to determine relevance based on common factors. When you train a relevancy model, you help Discovery to use features that are unique to your documents as it determines relevance.

The relevancy training model that is associated with a project is used at run time only when natural language queries are submitted. The model is not applied to Discovery Query Language (DQL) queries.

You cannot apply relevancy training to Content Mining project types.

To train a relevancy model, you provide sample natural language queries, submit them to get results from your documents, and then rate those results. As you add more examples, the information you provide about result relevance for each query is used to learn about your project. The system uses your assessments to assign importance to different types of structural information within the documents. For example, it learns the importance of when a keyword from the search query appears in the title versus the header, body, or in the metadata of the document. It also learns from the importance of the distance between one matching keyword and another. After a successful relevancy training session, a ranker model is created. The model is used automatically by Discovery with the next natural language query. Discovery reorders the document results so that the most relevant results according to the relevancy training model are displayed first.

Training applies to an entire project. It cannot be skipped for one collection and applied to other collections in the same project. You do not enable use of the training model by specifying a query parameter. If present, the model is used for every natural language query that is submitted for the project. The model is used whether you limit the search to one collection or all of the collections. For this reason, it is important that your training data represents queries that are likely to be answered by all of the collections in your project. To stop a project from using the relevancy training model, you can delete the model by using the API.

Relevancy training does not run continuously.
"key": "CA",
"matching_results": 77
}
]
}
]
},
{
"key": "SAN DIEGO",
"matching_results": 66,
"aggregations":
{
"type": "term",
"field": "STATE",
"count": 1,
"results":
{
"key": "CA",
"matching_results": 66
}
]
}
]
},
{
"key": "CHICAGO",
"matching_results": 59,
"aggregations":
{
"type": "term",
"field": "STATE",
"count": 1,
"results":
{
"key": "IL",
"matching_results": 59
}
]
}
]
}
]
}
],
"results": []
Show more





 Using aggregations to explore enrichments 

The term() aggregation is especially useful for analyzing results to find out how many enrichments are recognized in the documents. For example, to count how many times each entity type is recognized in the filtered documents, you can submit the following query parameters:

{
"filter": "enriched_text.entities:(text::Gilroy,type::Location)",
"aggregation": "term(enriched_text.entities.type)"
}

The query first selects the documents that have at least one entity of type Location and whose text is Gilroy. This action returns 3 documents. From the returned documents, the aggregation then counts the number of documents in which each entity type appears.

{
"matching_results": 3,
"retrieval_details": {
"document_retrieval_strategy": "untrained"
},
"aggregations": [
{
"type": "term",
Filter example: Find documents that mention an entity value 

The query looks for documents that mention the entity Gilroy and finds 4 matching documents.

Query body:

{
"filter": "enriched_text.entities.text::Gilroy"
}

Response:

{
"matching_results": 4
}







 Filtering nested values 

You can nest one filter inside another to ensure that the documents that are returned match more than one condition.

In the documents used for these examples, the entity "Gilroy" appears as both a "Location" (a town in California) and as a "Person" (a surname) entity type. To find documents where "Gilroy" appears as a location, write a query that filters on two nested fields at the same time: the entity text must be "Gilroy" and the entity type must be "Location".

One way to write the query is as follows:

{
"filter": "enriched_text.entities.text::Gilroy,enriched_text.entities.type::Location"
}

This query matches documents where some path enriched_text.entities.text is Gilroy and some path enriched_text.entities.type::Location is Location. However, there is no guarantee that those two paths will be under the same entities object. For example, the query matches documents that have Gilroy as a Person entity type and, at the same time, have some other Location entity type object.

To accurately capture the nested semantics of this query, nest the filter values by using the following syntax:

Query body:

{
"filter": "enriched_text.entities:(text::Gilroy,type::Location)"
}

This stricter query matches only those documents in which there is an entities object with text equal to Gilroy and type equal to Location.

As another example, if you want to match documents that contain an entities object with text equal to Gilroy but typenot equal to Location, you can use the not equal operator in the query, for example:

{
"filter": "enriched_text.entities:(text::Gilroy,type::!Location)"
}
This example queries for a keyword within a specific date range. The time zone information is not required:



* enriched_text.keywords.text:"olympics", publication_date<=2018-02-15T00:00:00Z, publication_date>=2018-02-01T00:00:00Z



News articles might be syndicated to several news outlets, and IBM Watson™ Discovery News picks up each of them, resulting in duplicate articles. This means that a query to IBM Watson™ Discovery News might potentially return several identical or nearly identical articles in query results. You can manage this, using deduplication. To learn more about this beta capability, see [Excluding duplicate documents from query results](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersdeduplication).





 Querying multiple collections 

If you have multiple collections in your environment, you might want to view results across those collections. The query methods (query, fields, and notices) at the environments level allow you to query multiple specified collections. Querying across collections is not currently available in the Discovery tooling.

You can query multiple collections in the same environment by using the environments/{environment_id}/query API method. When querying across multiple collections, take into consideration the following items.



* The collection_ids parameter must be specified when using this method. collection_ids is a comma-separated list of collections in the environment to query.
* passages are supported when querying multiple collections.
* collection_id is returned as part of each result object. This field specifies the collection where the result was found.
* The Discovery News is part of the system environment and cannot be included in multiple collection queries.
* Individual collection relevancy training does not affect ranking of results when querying multiple collections. To rerank results returned when querying multiple collections implement [Continuous Relevancy Training](https://cloud.ibm.com/docs/discovery?topic=discovery-crt).
* Reranking is not performed on any part of a multiple collection query, even if all collections in the query are trained.
* The characters _, +, and - in the prefix of the field_name. For example, +extracted-content.
* The characters ., ,, and : in the field_name. For example, new:extracted-content.









 Operators 

Operators are the separators between different parts of a query. The following table lists the available operators.



Query operators

 Operator Description Example 

 [.](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsdelimiter) JSON delimiter enriched_text.concepts.text 
 [:](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsincludes) Includes text:computer 
 [::](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsmatch) Exact match title::"Query building" 
 [!:](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsnotinclude) Does not include text:!computer 
 [!::](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsnotamatch) Not an exact match text::!winter 
 [](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsescape) Escape character title::"Dorothy said: "There's no place like home."" 
 [""](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsphrase) Phrase query enriched_text.concepts.text:"IBM Watson" 
 [(), ]](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsnestedquery) Nested groups filter-entities:(text:Turkey,type:Location)
[Learn more](https://cloud.ibm.com/docs/services/activity-tracker?topic=activity-tracker-endpointsendpoints).
* QUERY_PARAMETERS are parameters that define the filtering criteria that is applied to the export request.
* SERVICE_KEY is the service key that you created in the previous step.









 Samples 

To query events for a service in us-south, you can run the following command:

curl "https://api.us-south.logging.cloud.ibm.com/v1/export?to=1592337600&from=1592179200&hosts=kms&size=10&query=(target.id:crn:v1:bluemix:public:kms:us-south:a/xxxxxxx:xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx%20(action:kms.secrets.create%20%20OR%20action:kms.secrets.delete%20))" -u $TOKEN:

To write events into the terminal, you can run the following command:

curl "https://api.us-south.logging.cloud.ibm.com/v1/export?to=$(date +%s)000&from=$(($(date +%s)-86400))000" -u $TOKEN:

To send an email with the link to download the events that are specified on the export, you can run the following command:

curl "https://api.us-south.logging.cloud.ibm.com/v1/export?to=$(date +%s)000&from=$(($(date +%s)-86400))000&email=joe@ibm.com" -u $TOKEN:

To send an email with a custom subject, you can run the following command: