vania-janet/multiturn-rag-retrieval-data huggingface dataset:
{1,20}/, last);

create view FirstName as
extract dictionary 'FirstNamesDict'
on D.text as first
from Document D
having MatchesRegex(/p{Lu}p{M}.{1,20}/, first);

create view PersonName as
extract pattern <F.first> <L.last>? as fullName
from FirstName F, LastName L;
Show more







 The select statement 

The select statement in AQL provides a powerful mechanism for using various specifications to construct and combine sets of tuples.



 Syntax 

The select statement is similar in structure to an SQL SELECT statement:

select <select list>
from <from list>
[where <where clause>]
[consolidate on <column>
using '<policy>' with priority
from <column> priority order]]]]
[group by <group by list>]
[order by <order by list>]
[limit <maximum number of output tuples for each document>];





 Description 



* <select list>

A comma-delimited list of output expressions.
* <from list>

A comma-delimited list that is the source of the tuples to be selected.
* [where <where clause>]

Defines a predicate to be applied on each tuple that is generated from the Cartesian product of all of the tuples in the relations in the from clause. This clause is optional.
* [consolidate on<column>using '<policy>' with priority from <column> priority order]]]

Defines a consolidation policy to manage overlapping spans. This clause is optional.
* [group by<group by list>]

Groups the tuples that are produced from the same document by common values of a specified field. This clause is optional.
* [order by<order by list>]

Orders the output tuples that are produced by the select statement from each document. The order is based on the values of the order-by list, a comma-delimited list of expressions. This clause is optional.
Person P;

Example 3: Select list expression from a dictionary extraction

The following example illustrates how the select list expression can pick values of type Span from a result of dictionary extraction.

create view PersonNames as
select N.match as name
from
(extract
dictionary 'firstNames.dict'
on D.text
as match
from Document D
)N;

Example 4: Examples of case expressions

This first example shows you how to specify null handling on specific fields:

create view School as
select
case
when Not(NotNull(P.education)) then 'Unknown'
else GetString(P.education)
as name
from Person P;

This example explains how to classify data:

create view Company as
select
PM.name as productname,
case
when ContainsRegex (/IBM/,PM.name) then 'IBM'
when ContainsDict ('OpenSourceDict',PM.name) then 'OSS'
else 'Unknown'
as name
from ProductMatches PM;







 The from list 

The second part of a select or an extract statement in AQL is the from list. The from list is a comma-separated list that is the source of the tuples to be selected or extracted.



 Syntax 

from <from list item> <name> [, <from list item> <name>]





 Description 



* <from list item>

A view, table, table function reference or nested AQL statement. All nested statements in AQL must be surrounded by parentheses.
* <name>

Local name of the <from list item>, that is scoped within the select statement or the extract statement. A local name can be a simple identifier or a double-quoted identifier. Local names that contain spaces, punctuation characters, or AQL keywords must be enclosed in double quotation marks.







 Examples 

Example 1: A from list with a view and a nested statement

This example shows a from list that references a view and a nested extract statement.
> SELECT 2L / 2L
: Result value

1.0





 < 

expr1 < expr2
: Returns true if expr1 is less than expr2.
: Arguments

expr1, expr2 - The two expressions must be same type or can be cast to a common type, and must be a type that can be ordered. For example, map type is not orderable, so it is not supported. For complex types such array and struct, the data types of fields must be orderable.
: Example of an SQL function usage fragment

> SELECT 1 < 2
: Result value

true
: Example of an SQL function usage fragment

> SELECT 1.1 < '1'
: Result value

false
: Example of an SQL function usage fragment

> SELECT to_date('2009-07-30 04:17:52') < to_date('2009-07-30 04:17:52')
: Result value

false
: Example of an SQL function usage fragment

> SELECT to_date('2009-07-30 04:17:52') < to_date('2009-08-01 04:17:52')
: Result value

true
: Example of an SQL function usage fragment

> SELECT 1 < NULL
: Result value

NULL





 <= 

expr1 >= expr2
: Returns true if expr1 is less than or equal to expr2.
: Arguments

expr1, expr2 - The two expressions must be same type or can be cast to a common type, and must be a type that can be ordered. For example, map type is not orderable, so it is not supported. For complex types such array and struct, the data types of fields must be orderable.
: Example of an SQL function usage fragment

> SELECT 2 <= 2
: Result value

true
: Example of an SQL function usage fragment

> SELECT 1.0 <= '1'
: Result value

true
: Example of an SQL function usage fragment
An INTO clause is implicitly added by the driver based on the Cloud Object Storage location that is given with the targetcosurl connection property.

Results in the targetcosurl location are never deleted by the driver. Use [expiration rules](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-expiry) to clean up your results on the IBM Cloud automatically.





 Basic limitations 

The following limitations are implied by the use of Data Engine, which is not a full-featured database:



* The driver is based on an asynchronous REST API and does not maintain persistent connections to the backend.
* Data Engine is not designed for interactive performance on small data. Even tiny queries usually take several seconds to run.
* Query results are returned through results in Cloud Object Storage. A SELECT query creates a full copy of the selected table (or Cloud Object Storage) in the targetcosurl location.
* Streaming of query results cannot start until the execution fully completed and results were written to Cloud Object Storage.
* Data Engine works on read-only data in Cloud Object Storage, so the following functions that are related to data updates is not supported:



* Transactions are not supported. commit() and rollback() are no-ops.
* Result sets that can be updated are not supported.



* SQL statements cannot be canceled.







 Driver-specific limitations 

See the following technical limitations of the driver:



* Only primitive SQL types are supported in the result. Types like STRUCT, ARRAY, LOB, or ROWID cannot be retrieved from the result.
* When you access the result, use the getter method, matching the result column type, for example, getInt() to retrieve an INTEGER column. Some implicit conversions are supported, for example, accessing an INTEGER column with getLong(), but not all conversions are implemented.
* All result sets are FORWARD_ONLY and can be iterated with the ResultSet.next() method only. Determining the cursor position and explicitly moving the cursor is not supported.
* Prepared statements are not supported.
* Use [cleancols](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-reference) to avoid transformation problems into PARQUET format when the name of the columns include special characters or blanks.



For example, the following query is used to transform an archive file:

SELECT * FROM cleancols(cos://ams03/at-eu-de/999999d8f1f.2019-06-03.62.json.gz STORED AS JSON)
INTO cos://eu-de/results-at STORED AS PARQUET
2. Click Run.

You can see the query result in the Result area of the UI.

You can see the target COS URL in the Result location area of the UI. This URL points to the object storage bucket used by Data Engine to store the results.

With the lite plan, you can run only 1 query. You can run up to 5 queries simultaneously with a paid plan.
3. Copy the Result location URL. You need it to run other queries to analyze the events that are included in that archive file.



After you run the query, three objects are written as a result set in your COS results bucket:



1. jobid=<job_id>
2. jobid=<job_id>/_SUCCESS
3. jobid=<job_id>/<part-number>



Only one object contains the result set (jobid=<job_id>/<part-number>), and the other two are empty and don't contain any data.

It is important not to delete any of the files if you want to use the result set.

Each result is stored with an own job ID prefix that allows you to use the result directly in a query.

When you want to specify a result as input in your SQL query, specify the first (jobid=<job_id>) or the third one (jobid=<job_id>/<part-number>).
The following example uses the information from your connection string and the Java driver [jdbc](https://dev.mysql.com/doc/connector-j/8.0/en/) to connect to your database.

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;

public class App {
private final String STATUS_COMMAND = "SHOW VARIABLES LIKE '%version%';";

private Connection connect = null;
private Statement stmt = null;
private ResultSet rs = null;

private final String url = "mysql://127.0.0.1:30799";
private final String username = "";
private final String password = "";
private final Boolean useSSL = true;

public static void main(String args[]) throws Exception {
App app = new App();
final byte maxConnectionAttempt = 5;
byte currentConnectionAttempt = 0;

while (!app.connectDatabase() && currentConnectionAttempt < maxConnectionAttempt)
++currentConnectionAttempt;

if (currentConnectionAttempt >= maxConnectionAttempt) {
System.out.println(currentConnectionAttempt + " weren't successfull!");
} else {
app.printStatus();
app.closeConnection();
}
}

public void printStatus() {
try {
stmt = connect.createStatement();
rs = stmt.executeQuery(STATUS_COMMAND);
while (rs.next())
System.out.println(rs.getString(1) + ": " + rs.getString(2));
} catch (SQLException ex) {
: Example of an SQL function usage fragment

> SELECT to_json(named_struct('time', to_timestamp('2015-08-26', 'yyyy-MM-dd')), map('timestampFormat', 'dd/MM/yyyy'))
: Result value

{"time":"26/08/2015"}
: Example of an SQL function usage fragment

> SELECT to_json(array(named_struct('a', 1, 'b', 2))
: Result value

[{"a":1,"b":2}]
: Example of an SQL function usage fragment

> SELECT to_json(map('a', named_struct('b', 1)))
: Result value

{"a":{"b":1}}
: Example of an SQL function usage fragment

> SELECT to_json(map(named_struct('a', 1),named_struct('b', 2)))
: Result value

{"[1]":{"b":2}}
: Example of an SQL function usage fragment

> SELECT to_json(map('a', 1))
: Result value

{"a":1}
: Example of an SQL function usage fragment

> SELECT to_json(array((map('a', 1))))
: Result value

[{"a":1}]





 to_timestamp 

to_timestamp(timestamp[, fmt])
: Parses the timestamp expression with the fmt expression to a timestamp. Returns null with invalid input. By default, it follows casting rules to a timestamp if the fmt is omitted.
: Example of an SQL function usage fragment

> SELECT to_timestamp('2016-12-31 00:12:00')
: Result value

2016-12-31 00:12:00
If your result set includes the warning, No matching index found. Create an index to optimize query time, then you need to revisit your indexing strategy. Use the [explain](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-queryexplain-plans) feature to see which index is being selected for each query.
* With several object types in the same database, many use cases can be serviced by a few indexes on fixed attributes. For more information, see [Optimal IBM Cloudant Indexing](https://blog.cloudant.com/2019/05/10/Optimal-Cloudant-Indexing.html).
* Give your indexes meaningful names, and specify the index name at query-time, so that it's obvious which index corresponds to which of your application's queries.



For more information, see the following blog posts:



* [Generating sample data](https://blog.cloudant.com/2018/09/14/Generating-sample-JSON-data.html)
* [Optimal IBM Cloudant Indexing](https://blog.cloudant.com/2019/05/10/Optimal-Cloudant-Indexing.html)
Using Views 

Use views to search for content within a database that matches specific criteria. The criteria are specified within the view definition.

Criteria can also be supplied as arguments when you use the view.



 Querying a view 

To query a view, submit a GET request with the following format:

Method
: Issue a partition query by using the following command, GET $SERVICE_URL/$DATABASE/_partition/$PARTITION_KEY/_design/$DDOC/_view/$VIEW_NAME. Or issue a global query by using the following command, GET $SERVICE_URL/$DATABASE/_design/$DDOC/_view/$VIEW_NAME.

Request
: None

Response
: JSON of the documents that are returned by the view.

Roles permitted
: _reader

The request runs either:



* The specified $VIEW_NAME from the specified $DDOC design document within the $DATABASE database, which is constrained to results within the specified $PARTITION_KEY data partition.
* The specified $VIEW_NAME from the specified $DDOC design document within the $DATABASE database.



The examples in this document vary between partition and global queries for illustrative purposes. Unless otherwise noted, modifying the path to embed or remove the partition name works for any view query type.



 Query and JSON Body Arguments 

Global queries can use all query and JSON body arguments. Partition queries can use only the subset that is indicated in the table.



Table 1. Subset of query and JSON body arguments available for partitioned queries

 Argument Description Optional Type Default Supported values Partition query 

 conflicts Specify whether to include a list of conflicted revisions in the _conflicts property of the returned document. Ignored if include_docs isn't set to true. Yes Boolean False Yes 
 descending Return the documents in descending by key order. Yes Boolean False Yes 
 end_key Stop returning records when the specified key is reached. Yes String or JSON array Yes 
 end_key_docid Stop returning records when the specified document ID is reached. Yes String Yes 
 group Specify whether to group reduced results by key. Valid only if a reduce function is defined in the view.
* Fill in cos://us-south/<your-bucket-name> as Target. Replace us-south and <your-bucket-name> similar to how you did it earlier.
* As Password under Credentials use the API key which you created earlier. The value is from the field apikey.
* Finally, click Create.



4. Now create the notebook. Click New asset and Jupyter notebook editor.



* From the Blank tab, enter a Data lake notebook as Name.
* Leave the Runtime and Language as default and click Create.





Once the notebook is available, follow these steps.



1. On the top right of the notebook click on the three dot menu and pick Insert project token. It inserts a code snippet into the first cell. Run that cell.
2. Install ibmcloudsql and folium by adding the following commands to the next In [ ]: input prompt (cell) and then Run.

!pip install ibmcloudsql
!pip install folium
3. In the next cell, import folium and ibmcloudsql by adding the following commands to the In [ ]: input prompt and then Run

import folium
from folium.plugins import HeatMap
import ibmcloudsql
4. Enter the following commands into the next cell and click Run to view the result set. You will also have new accidents/jobid=<id>/<part>.csv file added to your bucket that includes the result of the SELECT.

sqlquery_config=project.get_connection(name="SQLQuery")

sqlClient = ibmcloudsql.SQLQuery(sqlquery_config['password'], sqlquery_config['crn'], sqlquery_config['target_cos_url'] + '/accidents')

data_source = sqlquery_config['target_cos_url'] + "/Traffic_Collision_Data_from_2010_to_Present.csv"
As explained in Example 2, a view is compatible with a table locator argument of a function, as long as the number of columns and the column types in the schema of the input view and table locator match. However, it is not necessary that the column names match.

Finally, the select clause discards column outStr from the output table of MyTableFunc, keeping only the first two columns outSpan1 and outSpan2.

Example 4: Using a table UDF implemented in PMML

This example illustrates the usage of the IrisDecisionTree table UDF function that is declared in example 4 of [Declaring user-defined functions](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-annotation-query-language-referenceaql-user-def-dec).

-- External view to hold the input records
create external view IrisData(
sepal_length Float,
sepal_width Float,
petal_length Float,
petal_width Float
)
external_name 'IrisData';
--Invoke the function on the input data records
create view IrisDecisionTreeOutput as
select * from IrisDecisionTree(IrisData);

output view IrisDecisionTreeOutput;
Use one of the following workarounds:



* For a nested structure, use the [FLATTEN](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencetableTransformer) table transformation function. Alternatively, you can specify the fully nested column names instead of the wildcard, for example, SELECT address.city, address.street, ... from cos://....
* For an array, use the Spark SQL explode() function, for example, select explode(contact_names) from cos://....



* If you receive a corrupted result, verify that the source URI is correct and that the correct input format is specified, by using 'STORED AS' in the SQL statement.
* If you receive an error message that states that some columns are not found in the input columns, but the columns do exist in the input, check if the input format that is specified as 'STORED AS' in the SQL statement is the actual format of your input.
* To process CSV input with Data Engine, each row must be contained within one line. Multi-line values are not supported.

If you use Data Engine to generate CSV results from other data formats like Parquet that support newlines within values and these CSV results are queried again, newlines must explicitly be removed before you write the results. To do so, use the SQL function regexp_replace. For example, a Parquet object data has an attribute multi_line containing values that span multiple lines. To select a subset of rows based on a condition and store it on Cloud Object Storage for further processing, a skeleton SQL statement looks like the following example:

SELECT regexp_replace(multi_line, '[\r\n]', ' ') as multi_line FROM data STORED AS parquet WHERE condition
Under File selection, click on Drag a file here or browse files to locate and pick the file "cities1000.txt" you downloaded in the first section of this guide.
4. Click Next to get to the schema overview. Choose the schema starting with BLUADMIN, then the table CITIES. Click on Next again.

Because the table is empty it does not make a difference to either append to or overwrite existing data.
5. Now customize how the data from the file "cities1000.txt" is interpreted during the load process. First, disable Header in first row because the file contains data only.
6. Next, type in 0x09 as separator. It means that values within the file are delimited by tab(ulator).
7. Last, pick "YYYY-MM-DD" as date format. Now, everything should look similar to what is shown in this screen capture.

Zoom

![Screen capture showing the sampled data](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution5/LoadTabSeparator.png)

Screen capture showing the sampled data
8. Click Next and you are offered to review the load settings. Agree and click Begin Load to start loading the data into the CITIES table. The progress is displayed. Once the data is uploaded it should only take few seconds until the load is finished and some statistics are presented.
9. Click on View Table to browse the data. You may scroll down or click on column names to change the sort order.







 Step 4: Verify Loaded Data Using SQL 

The data has been loaded into the relational database. There were no errors, but you should run some quick tests anyway. Use the built-in SQL editor to type in and execute some SQL statements.



1. In the left navigation click on Run SQL to get back to the SQL editor.
: Result value

true
: Example of an SQL function usage fragment

> SELECT 1 == '1'
: Result value

true
: Example of an SQL function usage fragment

> SELECT true == NULL
: Result value

NULL
: Example of an SQL function usage fragment

> SELECT NULL == NULL
: Result value

NULL





 > 

expr1 > expr2
: Returns true if expr1 is greater than expr2.
: Arguments

expr1, expr2 - the two expressions must be same type or can be cast to a common type, and must be a type that can be ordered. For example, map type is not orderable, so it is not supported. For complex types such array and struct, the data types of fields must be orderable.
: Example of an SQL function usage fragment

> SELECT 2 > 1
: Result value

true
: Example of an SQL function usage fragment

> SELECT 2 > '1.1'
: Result value

true
: Example of an SQL function usage fragment

> SELECT to_date('2009-07-30 04:17:52') > to_date('2009-07-30 04:17:52')
: Result value

false
: Example of an SQL function usage fragment

> SELECT to_date('2009-07-30 04:17:52') > to_date('2009-08-01 04:17:52')
: Result value

false
: Example of an SQL function usage fragment

> SELECT 1 > NULL
: Result value

NULL





 >= 

expr1 >= expr2
: Returns true if expr1 is greater than or equal to expr2.
: Arguments

expr1, expr2the two expressions must be same type or can be cast to a common type, and must be a type that can be ordered. For example, map type is not orderable, so it is not supported. For complex types such array and struct, the data types of fields must be orderable.
: Example of an SQL function usage fragment
For more information, see [catalog management](https://cloud.ibm.com/docs/sql-query?topic=sql-query-getting_started_catalog).



 Create table 



 createTable 





 columnDefinition 

Create a table definition in the catalog based on the objects in the specified Object Storage location. The LOCATION option is mandatory. If a table or view with the same name exists in the same Data Engine instance, you receive an error, unless the IF NOT EXISTS clause is specified.

The column and partition definitions are optional. If they are not provided, the table schema and partitioning is detected from the structure of the data at the indicated location. If you explicitly provide these definitions, ensure that they match the objects that are stored in Object Storage. See [data types](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencedataType) for details on the supported column types.

-- create a definition for the table customer
CREATE TABLE customers (
customerID string,
companyName string,
contactName string,
contactTitle string,
address string,
city string,
region string,
postalCode string,
country string,
phone string,
fax string
)
USING CSV
location cos://us-geo/sql/customers.csv
Show more

Before you can use a newly created partitioned table, you must call ALTER TABLE tablename RECOVER PARTITIONS. Otherwise, querying the table returns an empty result.

-- create a definition for the table customers_partitioned
CREATE TABLE customers_partitioned (
customerID string,
companyName string,
contactName string,
contactTitle string,
address string,
city string,
region string,
postalCode string,
country string,
phone string,
fax string
)
USING CSV
PARTITIONED BY (COUNTRY)
location cos://us-geo/sql/customers_partitioned.csv

-- attach table partitions by scanning the location of the table