vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Annotation Query Language reference 

Annotation Query Language (AQL) is the primary language that is used to create IBM Watson® Knowledge Studio advanced rules extractors.



* Data model: The data model for AQL is similar to the standard relational model that is used by an SQL database such as DB2®. All data in AQL is stored in tuples, data records of one or more columns, or fields. A collection of tuples forms a view. All tuples in a view must have the same schema that is the names and types of the fields across all tuples.
* Execution model: The runtime component has a document-at-a-time execution model. The runtime component receives a collection of documents and runs the extractor on each document to extract information from that document.
* AQL statements: By using AQL statements, you can create and then use modules, views, tables, dictionaries, and functions.
* Built-in functions: AQL has a collection of built-in functions for use in extraction rules.
* The create function statement: To perform operations on extracted values that are not supported by AQL, you can define custom functions to use in extraction rules called user-defined functions (UDFs).





 Annotation Query Language (AQL) syntax 

Like many programming languages, AQL is based on common syntax and grammar.

The lexical structure of a programming language is the set of elementary rules that define the tokens or basic components of that language such as its reserved words, identifiers, constants, and more.

The syntax of AQL is similar to that of SQL, but several important differences exist:



* AQL is case-sensitive.
* AQL currently does not support advanced SQL features such as correlated subqueries and recursive queries.
* AQL has a new statement type, extract, which is not present in SQL.
* AQL does not allow keywords (reserved words) as view, column, or function names.
* AQL allows, but does not require, regular expressions to be expressed in Perl syntax. Regular expressions begin with a slash (/) and end with a slash (/), as in Perl syntax.
PostViewOptions viewOptions = new PostViewOptions.Builder()
.db("users")
.ddoc("allusers")
.view("getVerifiedEmails")
.descending(true)
.startKey("alpha")
.endKey("beta")
.build();

ViewResult response =
service.postView(viewOptions).execute()
.getResult();

System.out.println(response);

const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

service.postView({
db: 'users',
ddoc: 'allusers',
view: 'getVerifiedEmails',
descending: true,
startKey: 'alpha',
endKey: 'beta'
}).then(response => {
console.log(response.result);
});

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()

response = service.post_view(
db='users',
ddoc='allusers',
view='getVerifiedEmails',
descending=True,
start_key='alpha',
end_key='beta'
).get_result()

print(response)

postViewOptions := service.NewPostViewOptions(
"users",
"allusers",
"getVerifiedEmails",
)

postViewOptions.SetDescending(true)
postViewOptions.StartKey = "alpha"
postViewOptions.EndKey = "beta"

viewResult, response, err := service.PostView(postViewOptions)
if err != nil {
panic(err)
}
Specify whether Not null is true or false.
A column that allows NULL values also allows rows to be inserted with no value in that column. A column that does not allow NULL values does not accept rows with no value.
4. Specify the default value to be used if no value is specified when a row is inserted.
5. In the Distribute on and Organize on sections, specify the distribution key for the table by selecting up to four columns.

To add another column, click the plus sign.



9. Click Create.





 Column and table constraints 

When you create a table, you can specify constraints for a column, table, or both.

For example, a table_constraint can be:

{ PRIMARY KEY ( column_name [, ... ] ) | FOREIGN KEY ( column_name [,
... ] ) REFERENCES reftable (refcolumn ) [ MATCH matchtype ] [ ON
DELETE action ] [ ON UPDATE action ] [ NOT ] DEFERRABLE ] [ INITIALLY
checktime ] } [, ...]

The system permits and maintains primary key, default, foreign key, unique, and references. Because Netezza Performance Server does not support constraint checking and referential integrity, you must ensure your own constraint checking and referential integrity.

If you have permission to create a table, you can specify a constraint.

If you have permission to alter a table, you can add or drop a table constraint.

You cannot change constraint names or directly change the owner of the constraint. The owner of the constraint is always the owner of the table. Thus, if you change the owner of the table, the system changes the owner of all associated constraints.







 Grooming tables 

GROOM TABLES processes and reorganizes table records in each data slice in a series of steps. You can still run operations such as SELECT, UPDATE, DELETE, and INSERT while data grooming is happening.
SELECT operations run in parallel with the groom operations.
INSERT, UPDATE, and DELETE operations run serially between the groom steps.
For cluster based tables (CBTs), GRROM TABLE takes longer. Tthe INSERT, UPDATE, and DELETE operations might be pending longer until the current step completes.
(3, 42, -200),
(4, 42, -200),
(5, 4711, 1000),
(6, 4711, -300),
(7, 4711, -300),
(8, 42, 1000),
(9, 4711, -400) AS txn
WINDOW current_balance AS (ORDER BY txn.col3)
ORDER BY txn.col3
)
GROUP BY txn_amount
Show more

The result of the example query is shown in the following table.



Table 28. Query result for example: cumulative distribution of transaction amounts.

 TXN_AMOUNT MAX(BALANCE_DIST) 

 -400 0.1111111111111111 
 -300 0.3333333333333333 
 -200 0.5555555555555556 
 1000 0.8888888888888888 
 2000 1.0 







 Aggregation Function Example 

This example uses a table that contains transaction information. The layout is as follows:



* Column 1: transaction ID
* Column 2: account number
* Column 3: transaction amount



The example shows how to retrieve the total balance of each account at the time of each transaction.

--- total balance of each account at the time of each transaction
SELECT
txn.col1 AS txn_id,
txn.col2 AS account,
txn.col3 AS txn_amount,
SUM(txn.col3) OVER current_balance AS balance
FROM VALUES
(1, 42, 100),
(2, 4711, 300),
(3, 42, 50),
(4, 42, -250),
(5, 4711, 1000),
(6, 4711, -300),
(7, 4711, 100),
(8, 42, 200),
(9, 4711, -400) AS txn
WINDOW current_balance AS (
PARTITION BY txn.col2
ORDER BY txn.col1
> SELECT decode(unhex('537061726B2053514C'), 'UTF-8')
: Result value

Spark SQL





 unix_timestamp 

unix_timestamp([expr[, pattern]])
: Returns the UNIX timestamp of current or specified time.
: Example of an SQL function usage fragment

> SELECT unix_timestamp()
: Result value

1476884637
: Example of an SQL function usage fragment

> SELECT unix_timestamp('2016-04-08', 'yyyy-MM-dd')
: Result value

1460041200





 upper 

upper(str)
: Returns str with all characters that are changed to uppercase.
: Example of an SQL function usage fragment

> SELECT upper('SparkSql')
: Result value

SPARKSQL





 uuid 

uuid()
: Returns a universally unique identifier (UUID) string. The value is returned as a canonical UUID 36-character string.
: Example of an SQL function usage fragment

> SELECT uuid()
: Result value

46707d92-02f4-4817-8116-





 var_pop 

var_pop(expr)
: Returns the population variance that is calculated from values of a group.





 var_samp 

var_samp(expr)
: Returns the sample variance that is calculated from values of a group.





 variance 

variance(expr)
: Returns the sample variance that is calculated from values of a group.





 weekofyear 

weekofyear(date)
: Returns the week of the year of the indicated date. A week is considered to start on a Monday and week 1 is the first week with >3 days.
: Example of an SQL function usage fragment

> SELECT weekofyear('2008-02-20')
: Result value

8

when
"system_job": {}
},
"updated_at": "0001-01-01T00:00:00Z"
}

For more information, see [troubleshooting section](https://cloud.ibm.com/docs/schematics?topic=schematics-bp-apply-fails).







 Next steps 



* Looking for additional blueprint samples to work with? Check out the [IBM Cloud Schematics GitHub repository](https://github.com/orgs/Cloud-Schematics/repositories/?q=topic:blueprint). Check the Readme files of the examples for further customization and usage for each sample.
"Aduki and orange casserole - microwave"
]
},
{
"id" : "Aioli-garlicmayonnaise",
"key" : "Aioli - garlic mayonnaise",
"value" :
null,
"Aioli - garlic mayonnaise"
]
},
{
"id" : "Alabamapeanutchicken",
"key" : "Alabama peanut chicken",
"value" :
null,
"Alabama peanut chicken"
]
}
],
"total_rows" : 2667
}
]
}
Show more

Multiple queries are also supported in /$DATABASE/_design_docs/queries, which is similar to /$DATABASE/_all_docs/queries.





 Sending multiple view queries to a database by using _view 

To send multiple view queries to a specific database, send a POST request to https://$ACCOUNT.cloudant.com/$DATABASE/_design/$DDOC/_view/$VIEW/queries.

See the following example that uses HTTP to send multiple queries to a database:

POST /_view/$VIEW/queries HTTP/1.1

See the following example that runs multiple specified view queries against the view function from the specified design document:



* Curl
* Java
* Python
* Go
* Node



curl -H "Authorization: Bearer $API_BEARER_TOKEN" -X POST "$SERVICE_URL/users/_design/allusers/_view/getVerifiedEmails/queries" -H "Content-Type: application/json" --data '{ "queries": [ { "include_docs": true, "limit": 5 },{ "descending": true, "skip": 1 } ]}'

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.PostViewQueriesOptions;
import com.ibm.cloud.cloudant.v1.model.ViewQueriesResult;
If two or more candidate indexes still exist, the index with the first alphabetical name is chosen.
* If a json type index and a text type index might both satisfy a selector, the json index is chosen by default.
* The text type index is chosen when the following conditions are met:



* A json type index and a text type index exist in the same field (for example fieldone).
* The selector can be satisfied only by using a text type index.





For example, assume that you have a text type index and a json type index for the field foo, and you want to use a selector similar to the following sample:

{
"foo": {
"$in": ["red","blue","green"]
}
}

IBM Cloudant Query uses the text type index because a json type index can't satisfy the selector.

However, you might use a different selector with the same indexes:

{
"foo": {
"$gt": 2
}
}

In this example, IBM Cloudant Query uses the json type index because both types of indexes can satisfy the selector.

To identify which index is being used by a particular query, send a POST to the _explain endpoint for the database, with the query as data. The details of the index in use are shown in the index object within the result.

See the following example that uses HTTP to show how to identify the index that was used to answer a query:

POST /movies/_explain HTTP/1.1
Host: $SERVICE_URL
Content-Type: application/json
{
"selector": {
"$text": "Pacino",
"year": 2010
}
}

See the following example that uses the command line to show how to identify the index that was used to answer a query:



* Curl
* Java
* Node
* Python
* Go



curl "$SERVICE_URL/movies/_explain" 	-X POST 	-H "Content-Type: application/json" 	-d '{
"selector": {
: Parses the timestamp expression with the fmt expression to a timestamp. Returns null with invalid input. By default, it follows casting rules to a timestamp if the fmt is omitted.
: Example of an SQL function usage fragment

> SELECT to_timestamp('2016-12-31 00:12:00')
: Result value

2016-12-31 00:12:00
: Example of an SQL function usage fragment

> SELECT to_timestamp('2016-12-31', 'yyyy-MM-dd')
: Result value

2016-12-31 00:00:00





 to_unix_timestamp 

to_unix_timestamp(expr[, pattern])
: Returns the UNIX timestamp of the indicated time.
: Example of an SQL function usage fragment

> SELECT to_unix_timestamp('2016-04-08', 'yyyy-MM-dd')
: Result value

1460041200





 to_utc_timestamp 

to_utc_timestamp(timestamp, timezone)
: Given a timestamp, such as '2017-07-14 02:40:00.0', this function interprets it as a time in the indicated time zone, and renders that time as a timestamp in UTC. For example, 'GMT+1' would yield '2017-07-14 01:40:00.0'.
: Example of an SQL function usage fragment

> SELECT to_utc_timestamp('2016-08-31', 'Asia/Seoul')
: Result value

2016-08-30 15:00:00





 translate 

translate(input, from, to)
: Converts the input string by replacing the characters present in the from string with the corresponding characters in the to string.
: Example of an SQL function usage fragment

> SELECT translate('AaBbCc', 'abc', '123')
: Result value

A1B2C3





 trim 

trim(str)
: Removes the leading and trailing space characters from str.
Using VCAP_SERVICES is more convenient than hardcoding the database details in your PHP App.)
4. In your local working directory, update the index.php file to connect to the BLUDB database by using the [db2_connect function](http://php.net/manual/en/function.db2-connect.php).

Example:

<!DOCTYPE html>
<html>
<head>
<title>PHP Starter Application</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<link rel="stylesheet" href="style.css" />
</head>
<body>
<?php
if( getenv( "VCAP_SERVICES" ) )
{
 Get database details from the VCAP_SERVICES environment variable

 This can only work if you have used the Bluemix dashboard to
 create a connection from your dashDB service to your PHP App.

$details = json_decode( getenv( "VCAP_SERVICES" ), true );
$dsn = $details [ "dashDB" ][ "credentials" ];
$ssl_dsn = $details [ "dashDB" ][ "credentials" ];
 Build the connection string

$driver = "DRIVER={IBM DB2 ODBC DRIVER};";
$conn_string = $driver . $dsn; Non-SSL
$conn_string = $driver . $ssl_dsn; SSL
$conn = db2_connect( $conn_string, "", "" );
if( $conn )
{
echo "<p>Connection succeeded.</p>";
db2_close( $conn );
}
else
{
echo "<p>Connection failed.</p>";
}
}
else
{
echo "<p>No credentials.</p>";
}
?>
> SELECT xpath('<a><b>b1</b><b>b2</b><b>b3</b><c>c1</c><c>c2</c></a>','a/b/text()')
: Result value

['b1','b2','b3']





 xpath_boolean 

xpath_boolean(xml, xpath)
: Returns true if the XPath expression evaluates to true, or if a matching node is found.
: Example of an SQL function usage fragment

> SELECT xpath_boolean('<a><b>1</b></a>','a/b')
: Result value

true





 xpath_double 

xpath_double(xml, xpath)
: Returns a double value, the value zero if no match is found, or NaN if a match is found but the value is nonnumeric.
: Example of an SQL function usage fragment

> SELECT xpath_double('<a><b>1</b><b>2</b></a>', 'sum(a/b)')
: Result value

3.0





 xpath_float 

xpath_float(xml, xpath)
: Returns a float value, the value zero if no match is found, or NaN if a match is found but the value is nonnumeric.
: Example of an SQL function usage fragment

> SELECT xpath_float('<a><b>1</b><b>2</b></a>', 'sum(a/b)')
: Result value

3.0





 xpath_int 

xpath_int(xml, xpath)
: Returns an integer value, or the value zero if no match is found, or a match is found but the value is nonnumeric.
: Example of an SQL function usage fragment
Expanding the meaning of queries 

You can improve the quality of search results by expanding the meaning of the queries that are submitted by customers.

To expand the scope of a query beyond exact matches, add a synonyms list to your collection. When synonyms are defined, the customer does not need to submit an exact phrase or keyword that your project is trained to understand. Even variations of the term are recognized and used to find the best results. For example, you can expand a query for ibm to include international business machines and big blue. Query expansion terms are typically synonyms, antonyms, or common misspellings for terms.

Synonyms that you add to improve the search results function differently from synonyms that you add to a dictionary. Dictionary synonyms are recognized and tagged at the time that a document is ingested. The synonyms that you define are recognized and tagged as occurrences of the associated dictionary term, so that they can be retrieved later by search. For more information about adding synonyms that are recognized when documents are processed, see [Dictionaries](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-dictionary).

You can define two types of expansions:

Bidirectional
: Each entry in the expanded_terms list expands to include all expanded terms. For example, a query for ibm expands to ibm OR international business machines OR big blue.

Bidirectional example:

{
"expansions": [
{
"expanded_terms":
"ibm",
"international business machines",
"big blue"
]
}
]
}

Unidirectional
: The input_terms in the query is replaced by the expanded_terms. For example, a query for banana is converted to plantain OR fruit and does not contain the original term, banana. If you want an input term to be included in the query, then repeat the input term in the expanded terms list.

Unidirectional example:

{
"expansions": [
{
"input_terms":
"banana"
],
"expanded_terms":
Query parameters 

You can use these parameters when you write queries with the Discovery Query Language. For more information, see the Discovery [API reference](https://cloud.ibm.com/apidocs/discovery-dataquery). For an overview of query concepts, see the [Query overview](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-concepts).

Queries that are written in the Discovery Query Language can include both search and structure parameters.

The default values for query parameters can differ by project type. For more information about default values, see [Default query settings](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-defaults).



 Search parameters 

Use search parameters to search your collection, identify a result set, and analyze the result set.

The results set is the group of documents that are identified by the combined searches of the search parameters. The results set might be significantly larger than the returned results. If an empty query is submitted, the results set is equal to all the documents in the collection.

Documents that you do not have permissions to access are not returned in query results.





 Answer finding 

IBM Cloud

The find_answers parameter is supported in managed deployments only.

By default, Discovery provides answers by returning the entire [passage](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parameterspassages) that contains the answer to a natural language query. When the answer-finding feature is enabled, Discovery also provides a "short answer" within the passage, and a confidence score to show whether the "short answer" answers the question that is explicit or implicit in the user query. Applications that use the answer-finding feature can display the short answer alone or can display the short answer emphasized in the context of the full passage. For most applications, displaying the short answer emphasized within the full passage is preferable, because answers generally make more sense in context.

The answer finding feature behaves in the following ways:

In the passage examples that follow, the short answers are shown in bold font.



* Finds answers.
"filter": "string",
"examples": [
{
"document_id": "string",
"cross_reference": "string",
"relevance": 0
}
]
}

The values in this object are as follows:



* query_id: A unique ID for the query. If you do not specify this field, the service automatically generates an ID.
* natural_language_query: A Discovery natural-language query that applies to the training set.
* filter: An optional filter for the query, as described in the [Query reference](https://cloud.ibm.com/docs/discovery?topic=discovery-query-referenceparameter-descriptions).

If you include filters in your training-data queries, be sure to use the same filters, when you use natural language queries in your trained collection. If you train the collection with filtered data but do not use the same types of filters when you query the collection, the results can be unpredictable.
* examples: An array that contains the following values:



* document_id: The unique ID of the document that applies to the training set. This is the example described previously.
* cross_reference: An optional label, typically consisting of a field in the referenced document, that "pins" the document and the field's information in place in the event the document's ID changes, for example if a newly ingested document is assigned the same ID. Specifying a value for cross-reference does not link one document to another; rather, it ensures that the service retains the document's pertinent information in case the document gets renamed or overwritten.
* relevance: An integer from 0 to 100, inclusive, that indicates the relative relevance of the query to the training data. Higher values indicate higher relevance. A value of 0 indicates no relevance to the query, whereas a value of 100 indicates absolute relevance to the query.

Although the range of the relevance parameter is 0 to 100 for maximum flexibility, you can use a smaller range to simplify scoring examples.
For more about training requirements and options, see [Improving result relevance with the tooling](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-tooling).





 Query overview 

The Query overview section displays:



* The total number of queries made by users
* The percentage of queries with one or more results clicked
* The percentage of queries with no results clicked
* The percentage of queries with no results returned
* A graph that displays these results over time, so that you can track how adding more data and relevancy training are improving performance



These results are gathered using the Events and Feedback API. See the [API reference](https://cloud.ibm.com/apidocs/discoverycreate-event) for more information.
"query":"enriched_text.entities.text::"IBM Cloud""
}





 :! (Does not include) 

This operator specifies that the results do not contain a match for the query term.

For example:

{
"query":"enriched_text.entities.text:!"cloud computing""
}





 ::! (Not an exact match) 

This operator specifies that the results do not exactly match the query term.

For example:

{
"query":"enriched_text.entities.text::!"Cloud computing""
}

Exact matches are case-sensitive.





 (Escape character) 

Escape character that preserves the literal value of the character that follows it.

For example, you can place an escape character before a quotation mark in query text to include the quotation mark in the query string.

{
"query":"title::Dorothy said: "There's no place like home""
}





 (),[] (Nested grouping) 

Logical groupings can be formed to specify more specific information.

For example:

{
"query":"enriched_text.entities:(text:IBM,type:Company)"
}





 | (or) 

Boolean operator for "or".

In the following example, documents in which Google or IBM are identified as entities are returned:

{
"query":"enriched_text.entities.text:Google|enriched_text.entities.text:IBM"
}

The includes (:,:!) and match (::, ::!) operators have precedence over the OR operator.

For example, the following syntax searches for documents in which Google is identified as an entity or the string IBM is present:

{
"query":"enriched_text.entities.text:Google|IBM"
}

It is treated as follows:

(enriched_text.entities.text:Google) OR IBM





 , (and) 

Boolean operator for "and".

In the following example, documents in which Google and IBM both are identified as entities are returned:

{