vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Filter query reference 

The Watson Assistant service REST API offers powerful log search capabilities through filter queries. You can use the v2 /logs API filter parameter to search your skill log for events that match a specified query.

The filter parameter is a cacheable query that limits the results to those matching the specified filter. You can filter on various objects that are part of the JSON response model (for example, the user input text, the detected intents and entities, or the confidence score).

To see examples of filter queries, see [Examples](https://cloud.ibm.com/docs/assistant?topic=assistant-filter-referencefilter-reference-examples).

For more information about the /logs GET method and its response model, refer to the [API Reference](https://cloud.ibm.com/apidocs/assistant/assistant-v2listlogs).



 Filter query syntax 

The following example shows the general form of a filter query:



 Location Query operator Term 

 request.input.text :: "IBM Watson" 





* The location identifies the field that you want to filter on (in this example, request.input.text).
* The query operator, which specifies the type of matching you want to use (fuzzy matching or exact matching).
* The term specifies the expression or value you want to use to evaluate the field for matching. The term can contain literal text and operators, as described in the [next section](https://cloud.ibm.com/docs/assistant?topic=assistant-filter-referencefilter-reference-operators).



Filtering by intent or entity requires slightly different syntax from filtering on other fields. For more information, see [Filtering by intent or entity](https://cloud.ibm.com/docs/assistant?topic=assistant-filter-referencefilter-reference-intent-entity).

Note: The filter query syntax uses some characters that are not allowed in HTTP queries. Make sure that all special characters, including spaces and quotation marks, are URL encoded when sent as part of an HTTP query.
Filter query reference 

The Watson Assistant service REST API offers powerful log search capabilities through filter queries. You can use the v2 /logs API filter parameter to search your skill log for events that match a specified query.

The filter parameter is a cacheable query that limits the results to those matching the specified filter. You can filter on various objects that are part of the JSON response model (for example, the user input text, the detected intents and entities, or the confidence score).

To see examples of filter queries, see [Examples](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-filter-referencefilter-reference-examples).

The ability to access logs was introduced with version 1.5.0.

For more information about the /logs GET method and its response model, refer to the [API Reference](https://cloud.ibm.com/apidocs/assistant-data-v2listlogs).



 Filter query syntax 

The following example shows the general form of a filter query:



 Location Query operator Term 

 request.input.text :: "IBM Watson" 





* The location identifies the field that you want to filter on (in this example, request.input.text).
* The query operator, which specifies the type of matching you want to use (fuzzy matching or exact matching).
* The term specifies the expression or value you want to use to evaluate the field for matching. The term can contain literal text and operators, as described in the [next section](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-filter-referencefilter-reference-operators).



Filtering by intent or entity requires slightly different syntax from filtering on other fields. For more information, see [Filtering by intent or entity](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-filter-referencefilter-reference-intent-entity).

Note: The filter query syntax uses some characters that are not allowed in HTTP queries. Make sure that all special characters, including spaces and quotation marks, are URL encoded when sent as part of an HTTP query.
Such operations include queries that are submitted by using the API. It does not include queries that are submitted from the search bar on the Improve and customize page of the product user interface.

A query is counted only if the request is successful, meaning it returns a response (with message code 200).

The number of search queries that you can submit per month per service instance depends on your Discovery plan type.



Number of queries per month

 Plan Queries per month per service instance 

 Cloud Pak for Data Unlimited 
 Premium Unlimited 
 Enterprise Unlimited 
 Plus (includes Trial) 500,000 



For Enterprise plans only, your bill labels requests that are generated from both query searches and analyze API calls as "Queries". For more information about Analyze API calls, see [Analyze API limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-analyzeapianalyzeapi-limits).

The number of queries that can be processed per second per service instance depends on your Discovery plan type.



Number of concurrent queries

 Plan Concurrent queries per service instance 

 Cloud Pak for Data Unlimited 
 Premium 50 
 Enterprise 5 
 Plus (includes Trial) 5 



For information about pricing, see [Discovery pricing plans](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-pricing-plans).





 Estimating query usage 

How to estimate the number of queries your application will use per month depends on your use case.



* For use cases that focus more on data enrichment and analysis or where the output from the document processing is not heavily searched, you can estimate query numbers based on the total number of documents.
* For use cases where many users interact with the application that uses Discovery, you can estimate by calculating the number of searches per user times the number of expected users. For example, 50% of the questions that are submitted by users to a virtual assistant are likely to be answered by Discovery. With 100,000 users per month and an average of 3 questions per user, you can expect 15,000 queries per month. (10,000 users/mo * 3 queries/user * 50% to Discovery = 15,000)







 Querying with document-level security enabled
"field": "DATEA",
"interval": "1d",
"results":
{
"key": 1262304000000,
"key_as_string": "2010-01-01T00:00:00.000Z",
"matching_results": 5
},
{
"key": 1262390400000,
"key_as_string": "2010-01-02T00:00:00.000Z",
"matching_results": 18
},
{
"key": 1262476800000,
"key_as_string": "2010-01-03T00:00:00.000Z",
"matching_results": 38
},
{
"key": 1262563200000,
"key_as_string": "2010-01-04T00:00:00.000Z",
"matching_results": 66
}
Show more
<-- </section "id="section-timeslice-example" "> --><-- </section "id="section-timeslice" "> --><-- <section "id="section-top_hits" "> --> top_hits Returns the documents ranked by the score of the query or enrichment. Can be used with any query parameter or aggregation.<-- <section "id="section-top_hits-syntax" "> --> Syntax {aggregation}.top_hits({n})
<-- </section "id="section-top_hits-syntax" "> --><-- <section "id="section-top_hits-example" "> --> Example The following example returns the top hit for the term halt per city. {
"query":"halt",
"aggregation": "term(CITY).top_hits(1)"
}
The response contains the top query results for the term halt grouped by cities mentioned in documents where the term is most mentioned.
For example, this query matches any logged event where the response includes a detected entity with the value soda:

response.output.entities:value::soda

Similarly, you can filter on intents or entities sent as part of the request, as in this example:

request.input.intents:intent::hello





 Filtering by other fields 

To filter on another field in the log data, specify the location as a path identifying the levels of nested objects in the JSON response from the /logs API. Use dots (.) to specify successive levels of nesting in the JSON data. For example, the location request.input.text idenfities the user input text field as shown in the following JSON fragment:

"request": {
"input": {
"text": "Good morning"
}
}

Filtering is not available for all fields. You can filter on the following fields:



* assistant_id
* customer_id
* language
* request.context.global.system.user_id
* request.input.text
* request_timestamp
* response.context.global.system.user_id
* response.output.entities
* response.output.intents
* response_timestamp
* session_id
* skill_id
* snapshot



Filtering on other fields is not currently supported.







 Examples 

The following examples illustrate various types of queries using this syntax.



 Description Query 

 The date of the response is in the month of July 2020. response_timestamp>=2020-07-01,response_timestamp<2020-08-01 
 The timestamp of the response is earlier than 2019-11-01T04:00:00.000Z. response_timestamp<2019-11-01T04:00:00.000Z 
 The message is labeled with the customer ID my_id. customer_id::my_id 
 The message was sent to a specific assistant. assistant_id::dcd5c5ad-f3a1-4345-89c5-708b0b5ff4f7
For example, the following syntax searches for documents in which Google is identified as an entity or the string IBM is present:

{
"query":"enriched_text.entities.text:Google|IBM"
}

It is treated as follows:

(enriched_text.entities.text:Google) OR IBM





 , (and) 

Boolean operator for "and".

In the following example, documents in which Google and IBM both are identified as entities are returned:

{
"query":"enriched_text.entities.text:Google,enriched_text.entities.text:IBM"
}

The includes (:,:!) and match (::, ::!) operators have precedence over the AND operator.

For example, the following syntax searches for documents in which Google is identified as an entity and the string IBM is present:

{
"query":"enriched_text.entities.text:Google,IBM"
}

It is treated as follows:

(enriched_text.entities.text:Google) AND IBM





 <=, >=, >, < (Numerical comparisons) 

Creates numerical comparisons of less than or equal to, greater than or equal to, greater than, and less than.

Only use numerical comparison operators when the value is a number or date.

Any value that is surrounded by quotations is a String. Therefore, score>=0.5 is a valid query and score>="0.5" is not.

For example:

{
"query":"invoice.total>100.50"
}





 ^x (Score multiplier) 

Increases the score value of a search term.

For example:

{
"query":"enriched_text.entities.text:IBM^3"
}





 * (Wildcard) 

Matches unknown characters in a search expression. Do not use capital letters with wildcards.

For example:

{
"query":"enriched_text.entities.text:ib"
}





 n (String variation) 

The number of character differences that are allowed when matching a string. The maximum variation number that can be used is 2.
As noted, every example must include a label that indicates the "goodness" or "badness" of the document as it pertains to the query.

Examples can optionally specify a cross-reference query. The cross-reference query needs to return only the example document and must be independent of the unique Watson Discovery document ID. Cross-reference queries are not currently used automatically but can be used to repair training data in the event that new IDs are assigned to documents during an ingestion event.





 Training data requirements 

Training data must meet the following minimal quality criteria to effectively improve the relevance of answers that are returned by the Discovery service. Note that minimal does not mean optimal. The service checks the training data periodically to determine if these requirements are met, and automatically updates itself based on any changes.



* The collection's training-data set must contain at least 49 unique training queries (that is, sets of queries and examples). Depending on the size and complexity of your collection, the number of training queries in your set might need to be higher than 49 before Watson is able to apply relevancy training to the collection. Watson provides feedback if it needs more queries so that it can train.
* When assigning relevance scores via the API: The relevance score for each training query must be a non-negative integer, for example 0 to represent not relevant, 1 to represent somewhat relevant, and 2 to represent highly relevant. However, for maximum flexibility, the service accepts non-negative integers between 0 and 100 for advanced users experimenting with different scoring schemes. Regardless of the range you use, the largest integer in the set of training queries indicates maximum relevance.
* When assigning relevance ratings, using the tooling, the Discovery tooling uses the relevance scores of 0 to represent not relevant, and 10 to represent relevant. We recommend that you apply both available ratings to your results: Relevant and Not relevant. Only rating the Relevant documents does not provide the data needed. If you plan to score your documents, using both the Discovery tooling and the API, or if you plan to begin with the API and move to the tooling, use the 0 and 10 relevancy scores.
* A random sample of documents that are not explicitly given a relevance rating is assigned a relevance score of 0.
Using the parameters together improves performance because the filter parameter is applied first. It filters the documents and caches the results. The query parameter then ranks the cached results.



 Filter example: Get a document by its ID 

Query body:

{
"filter": "document_id::b6d8c6e3-1097-421b-9e39-75717d2554aa"
}

If the document exists, the query returns 1 matching result. If it doesn't, the query returns no matching results.





 Filter example: Find a document ID by its file name 

If you don't know the document_id of a document, but you know the original filename of the document, you can use the filter and return parameters together to discover the document_id.

Query body:

{
"filter": "extracted_metadata.filename::100674.txt",
"return": [ "document_id", "extracted_metadata" ]
}

Response:

{
"matching_results": 1,
"results": [
{
"document_id": "b6d8c6e3-1097-421b-9e39-75717d2554aa",
"extracted_metadata": {
"sha1": "AD447F7592A17CDCBF0A589C4E6EC2087AF7H35F",
"filename": "100674.txt",
"file_type": "text"
}
}
]
}





 Filter example: Find documents that mention an entity value 

The query looks for documents that mention the entity Gilroy and finds 4 matching documents.

Query body:

{
"filter": "enriched_text.entities.text::Gilroy"
}

Response:

{
"matching_results": 4
}







 Filtering nested values 

You can nest one filter inside another to ensure that the documents that are returned match more than one condition.
Each region has a different URL. [Learn more](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-endpoints).
* QUERY_PARAMETERS are parameters that define the filtering criteria that is applied to the export request.
* SERVICE_KEY is the service key that you created in the previous step.









 Samples 

For example, to write event lines into the terminal, you can run the following command:

curl "https://api.us-south.logging.cloud.ibm.com/v2/export?to=$(date +%s)000&from=$(($(date +%s)-86400))000<=vels=info" -u e08c0c759663491880b0d61712346789:

To use the query parameter to find all event lines with a level of info, you can run the following command:

curl -s "https://api.us-south.logging.cloud.ibm.com/v2/export?query=test_query<=vels=info" -u :
A query facility that allows a user to find a book by a known publisher and year.
2. A general-purpose search engine that allows a user to find books by a combination of one or more of the following descriptors: author, title, year, and publisher.
3. A report that details the number of books that are published by year.









 Step 2. Querying books by publisher and year - IBM Cloudant Query 

IBM Cloudant Query is a query language that allows small slices of a total database to be located. The following query finds 10 books that are published by Penguin in the year 2000:

{
"selector": {
"$and": [
{ "publisher": "Penguin" },
{ "year": { "$gt": 2000 } }
]
},
"limit": 10
}

The query contains a selector object, which uses operators and text fields to define the slice of data you need:



* $and means both of the query clauses must be satisfied for a document to make it to the result set.
* { "publisher": "Penguin" }- the publisher must be "Penguin".
* { "year": { "$gt": 2000 } } - the year must be greater than 2000. $gt means "greater than".



We can try the query by choosing "Query" when you view our books database in the IBM Cloudant Dashboard. You can paste in the query JSON and click Run Query.



 Running a query 

To try the query, do the following steps:



1. Go to IBM Cloudant Dashboard.
2. Open the service instance that you created in the prerequisite section.
3. Open the database that you created.
4. Go to the Query tab.
5. Paste the query JSON from the previous section into the Cloudant Query window.
6. Click Run Query. See the results in the following screen capture:

Zoom

![Run the query, and the results show the _id, author, pages, publisher, and year.]
Being able to explicitly specify the location and the type of the result object enables a user to take the following actions:



* Read raw data stored in one format, for example, CSV.
* Cleanse the data by using SQL constructs.
* Store the cleansed data in a format that offers benefits regarding query performance and storage consumption on Cloud Object Storage, for example, a column-based object format, such as Parquet.



Moreover, a user can explicitly define the way a query result is stored physically on Cloud Object Storage by using the result partitioned clause.

As shown in the syntax diagrams, three main use cases exist to define the physical layout of a query's result on Cloud Object Storage:



* Partition by columns, which is the so-called Hive-style partitioning.
* Partition into buckets or objects (both terms can be used synonymously), which generates the query result into objects, with or without specifying columns.
* Partition by number of rows.



A partition is an object on Cloud Object Storage that is potentially a part of an aggregated object. The presence of multiple partitions allows for parallel input/output (I/O) during query execution. If no result partitioned clause is specified, the query result is stored in a single partition on Cloud Object Storage.





 partitionedClause 

You can use the result partitioned clause to control the layout of the SQL query result set being stored. The default behavior is to store the result into one single partition, which is a single object in Cloud Object Storage.





 sortClause 

This clause can be used to sort in many ways. When specified in combination with PARTITIONED BY, it sorts the rows within each partition by the sort order that is specified in the SORT BY clause. When specified in combination with PARTITIONED INTO, the same is done, which is often referred to as clustering the rows by the specified columns into the fixed number of partitions specified by PARTITIONED INTO. When specified without the PARTITIONED clause, it is equivalent to an ORDER BY clause specified at the top level of the SQL SELECT statement. If PARTITIONED INTO is specified, the ORDER BY clause is ignored.



 Partition by columns
This variability means that comparing the confidence score to a static threshold value is an unsuitable method by which to delimit the results that are returned by your application. Results are ordered from highest to lowest confidence. You can find the best candidate answers by taking the top results, regardless of their confidence score values.The natural_language_query parameter enables capabilities such as relevancy training. For more information, see Improving result relevance with training]] ! ! ! ! ! .<-- </section "id="section-nlq-confidence" "> --><-- </section "id="section-nlq" "> --><-- <section "id="section-query" "> --> query A query search returns all documents in your data set with full enrichments and full text in order of relevance. A query also excludes any documents that don't mention the query content.<-- </section "id="section-query" "> --><-- <section "id="section-aggregation" "> --> aggregation Aggregation queries return a count of documents that match a set of data values. For the full list of aggregation options, see Query aggregations]] ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! .<-- </section "id="section-aggregation" "> --><-- <section "id="section-filter" "> --> filter A cacheable query that excludes any documents that don't mention the query content. Filter search results are not returned in order of relevance.When you write a query that includes both a filter, and an aggregation, query, or natural_language_query parameter, the filter parameter runs first, and then any aggregation, query, or natural_language_query parameters run in parallel.With a simple query, especially on a small data set, the filter and query parameters often return the exact same (or similar) results. If the filter and query calls return similar results, and you don't need the responses to be returned in order of relevance, use the filter parameter.
Viewing metrics and improving query results with the Performance dashboard 

The Performance dashboard in the Discovery tooling can be used to view query metrics, as well as improve query results, including query relevance.

You can access the Performance dashboard by clicking the View data metrics icon. The dashboard is not available in Premium or Dedicated environments.

There are two options to improve natural language query results:



* [Fix queries with no results by adding more data](https://cloud.ibm.com/docs/discovery?topic=discovery-performance-dashboardaddmore)
* [Bring relevant results to the top by training your data](https://cloud.ibm.com/docs/discovery?topic=discovery-performance-dashboardtraindata)



You can view the data metrics in the [query overview](https://cloud.ibm.com/docs/discovery?topic=discovery-performance-dashboardoverview).



 Fix queries with no results by adding more data 

In this section of the dashboard, you can review queries that returned zero results and add more data so that the query returns results in the future. Click the View all and add data button to get started.





 Bring relevant results to the top by training your data 

In this section, you can train your collections to improve the relevance of natural language query results. Click the View all and perform relevancy training button to get started. Then see [Adding queries and rating the relevancy of results](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-toolingresults) for instructions.

For more about training requirements and options, see [Improving result relevance with the tooling](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-tooling).





 Query overview 

The Query overview section displays:



* The total number of queries made by users
* The percentage of queries with one or more results clicked
* The percentage of queries with no results clicked
* The percentage of queries with no results returned