vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* The following SELECT statement returns null because although all members of the specified array occur in ts4 in the correct order, the array does not match ts4 exactly.

SELECT TS_MATCH(ts4, TS_MATCHER_SEQ(), ARRAY('a','b','c'))

Result:

NaN
* The following SELECT statement returns the matching observations. The array [b,c,d] is a match because the following conditions are satisfied:



* The three values of the pattern occur in the input time series contiguously.
* The matcher threshold is 0.6, and the three input values represent at least 60% of the entire time series. If the threshold is set to a number greater than 0.6, this function returns null.



SELECT TS_MATCH(ts4, TS_MATCHER_SUBLIST(0.6), ARRAY('b','c','d'))

Result:

[(1,'b'), (3,'c'), (7,'d')]
* The following SELECT statement returns the matching observations. The array [a,b,d] is a match because the following conditions are satisfied:



* The three values of the pattern occur in the input time series in the specified order. (They do not have to be contiguous.)
* The matcher threshold is 0.6, and the three input values represent at least 60% of the entire time series. If the threshold is set to a number greater than 0.6, this function returns null.



SELECT TS_MATCH(ts4, TS_MATCHER_SUBSET_PS(0.6), ARRAY('a','b','d'))

Result:

[(0,'a'), (1,'b'), (7,'d')]
* The following SELECT statement returns the matching observations. The array [a,b,d] is a match because the following conditions are satisfied:



* The three values of the pattern occur in the input time series in the specified order. (They do not have to be contiguous.)
while (resultSet.next())
{
String phone = resultSet.getString("phone");
String name = resultSet.getString("name");
System.out.println("phone = " + phone + ", name = " + name);
}
}
catch (Exception e)
{
e.printStackTrace();
}
finally
{
/ clean up at the end always /
if (resultSet != null)
{
resultSet.close();
}
if (statement != null)
{
statement.close();
}
if (connection != null)
{
connection.close();
}
}
}
}
Show more

Replace the parameters in the command with the following: <PRESTO_URL> Identifies the jdbc URL to the Presto server. <EMAIL_ID> with your email ID <API_KEY> with the API key that you downloaded from IBM Cloud. If you are using IBM IAM token, replace ibmapikey with ibmiamtoken and pass the token.
4. Compile and run the command.







 Connecting to Presto engine using Python scripts 



1. Install python 3.x (3.10 or later recommended) and pip3 on your client workstation.
2. Use the DBAPI interface to query Presto. Following is a sample python script.

import os
import prestodb
username=os.environ["ENG_USERNAME"]
password=os.environ["ENG_PASSWORD"]
hostname=os.environ["ENG_HOST"]
portnumber=os.environ["ENG_PORT"]
cert_location='./ibm-lh-client/localstorage/volumes/infra/tls/aliases/' + hostname + ':' +
portnumber + '.crt'
with prestodb.dbapi.connect(
[Data Engine console](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution31/sql_query_console.png)

Data Engine console

The Select statement would looks like

SELECT * FROM <EVENT_STREAMS_CRN>/webserver
STORED AS JSON EMIT cos://<REGION>/<BUCKET_NAME>/logs-stream-landing/topic=webserver
STORED AS PARQUET EXECUTE AS <KEY_PROTECT_CRN_WITH_KEY>

It is a SELECT statement from your Event Streams instance and topic (identified via the unique CRN) and the selected data is emitted (EMIT) to your Object Storage bucket AS PARQUET format. The operation is executed (EXECUTE) with the service ID's API key that is stored in the Key Protect instance.
4. Click on the link in the Result location field, which opens the Object Storage UI with a filter set to the objects that are being written by that job.

Zoom

![COS object view](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution31/cos_object_view.png)

COS object view

In the COS UI, switch to object view by clicking on the icon next to Upload, You should see that there are a couple of metadata objects to track, such as the latest offset that has been consumed and landed. But, in addition, you can find the Parquet files with the actual payload data.
5. Return to the Data Engine UI and in the Details tab click on Query the result and then click Run to execute a Batch job. You should see the query in the panel pointing to the Object Storage file (under FROM) with the log message(s) you sent above. Wait for the job to change to Completed.
6.
* [fullselect](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencefullselect)
* [functionOrAggregate](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencefunctionOrAggregate)
* [groupByClause](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencegroupByClause)
* [groupingSet](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencegroupingSet)
* [lateralView](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencelateralView)
* [predicate](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencepredicate)
* [primaryExpression](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceprimaryExpression)
* [resultColumn](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceresultColumn)
* [sample](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencesample)
* [sortItem](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencesortItem)
* [tableValuedFunction](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencetableValuedFunction)
* [valuesClause](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencevaluesClause)
System.out.printf("%sn", e);
LOGGER.info("Not connected, retying ...");
}
}

return conn;

}

public static void main(String[] args) {

PGConnect icd = new PGConnect();

try {
Connection connection = icd.connect();
Statement stmt = connection.createStatement();

ResultSet rs = stmt.executeQuery("SELECT * from pg_database");
while (rs.next()) {
System.out.println("DB Name: " + rs.getString(1));
}

} catch (SQLException e) {
System.out.println(e.getMessage());
}
}
}
Show more

The following example uses the information from your connection string and the Python driver [Psycopg2](https://wiki.postgresql.org/wiki/Psycopg2_Tutorial) to connect to your database. This is just a simple connection example, without error handling or retry logic and might not be suitable for production.

import psycopg2

try:
conn = psycopg2.connect(
host="hostname.databases.appdomain.cloud",
port= 31525,
user="username",
password="password",
sslmode="verify-full",
sslrootcert="/path/to/cert/ca-certificate.crt",
database="ibmclouddb")
except:
print("Unable to connect to database")

cur = conn.cursor()
cur.execute("SELECT datname FROM pg_database")
rows = cur.fetchall()

print("List of databases:")
for row in rows:
print(" ",row[0])
Show more
SELECT get_json_object(value, "$._source._host") as host,
get_json_object(value, "$._source._ts") / 1000 as unix_timestamp,
from_unixtime(get_json_object(value, "$._source._ts") / 1000, 'yyyy-MM-dd HH:mm:ss') as timestamp,
get_json_object(value, "$._source._file") as file,
get_json_object(value, "$._source.request_method") as request_method,
get_json_object(value, "$._source.request_uri") as request_uri
FROM cos://us-geo/sql/LogDNA/_year=2019/_dayofyear=226/_hour=13 STORED AS TEXT
)
SELECT 
FROM logs
WHERE minute(from_unixtime(unix_timestamp)) >= 40 AND minute(from_unixtime(unix_timestamp)) <= 44
ORDER BY timestamp
Once you have Data Engine running on IBM Cloud, you can immediately start querying your data by using the Data Engine UI, or programmatically by using either [the Data Engine REST API](https://cloud.ibm.com/apidocs/sql-query), or the Python ibmcloudsql library.





 Step 2. Granting user permissions to run a query 

To run queries with the SQL query service, a user needs a platform role and a service role. The following tables show the roles and actions that you can assign a user to run SQL queries:



Table 1. Platform roles

 Platform actions Administrator Editor Operator Viewer 

 View details of the Data Engine service instance ![Checkmark icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/images/checkmark-icon.svg) ![Checkmark icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/images/checkmark-icon.svg) ![Checkmark icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/images/checkmark-icon.svg) ![Checkmark icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/images/checkmark-icon.svg) 





Table 2. Service roles

 Service actions Manager Writer Reader 

 Run an SQL query !
The select statement

The require document with columns statement

The set default dictionary language statement

The create function statement



The syntax specifications in the AQL statements contain brackets ([ ]). These specifications indicate that the brackets and the constructs that they hold are optional when the corresponding syntax is used in writing a statement. In addition, they stand as place holders to define how to specify additional instances of a construct or argument.





 The module statement 

Use the module statement to create a module that has self-contained required resources. You can export and import these resources as AQL objects to and from other modules.



 Syntax 

module <module-name>;





 Description 



* <module-name>

Declares that the current file is part of the module that is named <module-name>. The module name must be a simple identifier. Double-quoted identifiers are not allowed as module names.

Each AQL file inside the module must have exactly one module declaration, and this declaration must be the first statement in each file. This declaration establishes a namespace identical to the module-name. All views (and other objects such as dictionaries, tables, and functions) declared in AQL files inside this module are located inside this single namespace. They are accessible to all AQL files in this namespace.

All of the files that are declared as part of the module <module-name> must be inside a folder that is called <module-name> to enforce this namespace. No ordering of the AQL files within this module folder exists. The AQL compiler examines all AQL files of the module and determines the right order to compile all views, dictionaries, tables, and functions declared in the module.







 Usage notes 



* When the underlying AQL file is not located as part of a module (modular AQL) and is compiled in compatibility mode, this statement is not supported.
* Circular dependencies between views are not allowed.
* AQL modules do not support submodules.
* AQL files within subfolders of the top-level module folder are ignored.







 Examples 

Example 1: Sample module
For more information, see [Filtering by intent or entity](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-filter-referencefilter-reference-intent-entity).

Note: The filter query syntax uses some characters that are not allowed in HTTP queries. Make sure that all special characters, including spaces and quotation marks, are URL encoded when sent as part of an HTTP query. For example, the filter response_timestamp<2020-01-01 would be specified as response_timestamp%3C2020-01-01.





 Operators 

You can use the following operators in your filter query.



 Operator Description 

 : Fuzzy match query operator. Prefix the query term with : if you want to match any value that contains the query term, or a grammatical variant of the query term. Fuzzy matching is available for user input text, response output text, and entity values. 
 :: Exact match query operator. Prefix the query term with :: if you want to match only values that exactly equal the query term. 
 :! Negative fuzzy match query operator. Prefix the query term with :! if you want to match only values that do not contain the query term or a grammatical variant of the query term. 
 ::! Negative exact match query operator. Prefix the query term with ::! if you want to match only values that do not exactly match the query term. 
 <=, >=, >, < Comparison operators. Prefix the query term with these operators to match based on arithmetic comparison. 
 `` Escape operator. Use in queries that include control characters that would otherwise be parsed as operators. For example, !hello would match the text !hello. 
 "" Literal phrase. Use to encloses a query term that contains spaces or other special characters. No special characters within the quotation marks are parsed by the API. 
 Approximate match. Append this operator followed by a 1 or 2 to the end of the query term to specify the allowed number of single-character differences between the query string and a match in the response object.
2022-06-29 04:13:09.061443 PDT [27743]  DEBUG:  QUERY: SELECT * FROM "DB1"."USER1"."TEST_TABLE"
ANALYZE
2022-06-29 04:13:09.064209 PDT [27743]  DEBUG:  QUERY: SELECT * FROM "DB1"."USER1"."TEST_TABLE"
* For the sink connector:

2022-06-29 09:08:32.379442 PDT [18976]  DEBUG:  QUERY: CREATE EXTERNAL TABLE bulkETL_18976_0 ( c0 nvarchar(8),c1 nvarchar(10),c2 nvarchar(4),c3 nvarchar(6),c4 nvarchar(7),c5 nvarchar(9),c6 nvarchar(7),c7 nvarchar(9),c8 nvarchar(12),c9 nvarchar(12),c10 nvarchar(25),c11 nvarchar(13),c12 nvarchar(1) )  USING (  DATAOBJECT('/tmp/junk')  REMOTESOURCE 'jdbc'  DELIMITER ' '  ESCAPECHAR ''  CTRLCHARS 'YES'  CRINSTRING 'YES'  ENCODING 'INTERNAL'  MAXERRORS 1 QUOTEDVALUE 'YES' );
When you open the Console for the first time, the Data Engine service automatically creates a bucket for you in your Cloud Object Storage instance. It uses this bucket as the default target for your query results.







 Run one of the samples to see how to use the service 



1. Select a sample query. The query automatically gets loaded into the editor. The input data that is used by the sample query is available in a publicly accessible bucket.
2. The Target field is automatically completed with the unique resource identifier (URI) of your default bucket. You can use this bucket or specify an INTO clause in your query.
3. Click Run to run the query. The query result is displayed.
4. Each sample data set is available in each of the supported input data formats (CSV, JSON, ORC, and Parquet). If you want to experiment with different formats, edit the selected sample query and change the specified file name and format. For example, change orders.parquet STORED AS PARQUET to orders.orc STORED AS ORC.







 Next steps 

To analyze your own data (CSV, JSON, ORC, or Parquet), upload it to a Cloud Object Storage instance and [run SQL queries](https://cloud.ibm.com/docs/services/sql-query?topic=sql-query-overviewrunning).

For more advanced capabilities, check out the following video tutorials.



* [IBM Cloud Data Engine introduction](https://cloud.ibm.com/docs/sql-query?topic=sql-query-videovideo_samples_notebooks_api).
* [How to connect to IBM Cloud Object Storage through the Command Line](https://cloud.ibm.com/docs/sql-query?topic=sql-query-videovideo_command_line).
* [How to use the Data Engine REST API](https://cloud.ibm.com/docs/sql-query?topic=sql-query-videovideo_rest_api).
: Example of an SQL function usage fragment

> SELECT asin(2)
: Result value

NaN





 assert_true 

assert_true(expr1 [, expr2])
: Returns null if column expr1 is true; throws an exception with the provided error message expr2 otherwise. Useful to check if a cast for a column is successful for every row.
: Example of an SQL function usage fragment

> SELECT ifnull(assert_true(cast(col as int) is not null), cast(col as int)) as col_int from mytable
: Alternatively, using it in the WHERE clause

SELECT cast(col as int) as col_int
FROM mytable
WHERE assert_true(cast(col as int) is not null) is null
: Result value when expr1 is false

SQL execution failed
An assert_true function failed: ''NOT (cast(col14393 as int) = cast(null as int))' is not true!'. Check your data to see which rows don't pass the function's condition and fix the issue.
: Example with a custom error message

> SELECT ifnull(assert_true(cast(col as int) is not null, "col must be a valid integer and not null!"), cast(col as int)) as col_int from mytable
: Result value when expr1 is false

SQL execution failed
An assert_true function failed: 'col must be a valid integer and not null!'. Check your data to see which rows don't pass the function's condition and fix the issue.





 atan 

atan(expr)
: Returns the inverse tangent (also known as arctangent).
: Example of an SQL function usage fragment

> SELECT atan(0)
: Result value

0.0





 atan2 

atan2(expr1, expr2)
: Returns the angle in radians between the positive x-axis of a plane and the point that is indicated by the coordinates (expr1, expr2).
: Example of an SQL function usage fragment
> SELECT 1 in(2, 3, 4)
: Result value

false
: Example of an SQL function usage fragment

> SELECT named_struct('a', 1, 'b', 2) in(named_struct('a', 1, 'b', 1), named_struct('a', 1, 'b', 3))
: Result value

false
: Example of an SQL function usage fragment

> SELECT named_struct('a', 1, 'b', 2) in(named_struct('a', 1, 'b', 2), named_struct('a', 1, 'b', 3))
: Result value

true





 initcap 

initcap(str)
: Returns str with the first letter of each word in uppercase. All other letters are in lowercase. Words are delimited by white space.
: Example of an SQL function usage fragment

> SELECT initcap('sPark sql')
: Result value

Spark Sql





 inline 

inline(expr)
: Explodes an array of structs into a table.
: Example of an SQL function usage fragment

> SELECT inline(array(struct(1, 'a'), struct(2, 'b')))
: Result value

1 a 2 b





 inline_outer 

inline_outer(expr)
: Explodes an array of structs into a table.
: Example of an SQL function usage fragment

> SELECT inline_outer(array(struct(1, 'a'), struct(2, 'b')))
: Result value

1 a 2 b





 input_file_name 

input_file_name()
: Returns the name of the file that is being read, or empty string if not available.





 instr 

instr(str, substr)
: Returns the (1-based) index of the first occurrence of substr in str.
: Example of an SQL function usage fragment
Figure 3 shows the panel that opens when the card is selected. This panel give you control over the location and costs regarding your new instance of IBM Cloud Data Engine. Select the region appropriate for your buckets and the plan suitable for your projects. If you want more information you can use the documentation to [get started](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overview).

Zoom

![Create SQL Instance](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/create-sql-instance-cos.jpg)

Figure 3. Fill out form fields to configure your instance



 Querying Object Storage with SQL Query 

You can use SQL Query to create SELECT statements only; actions such as CREATE, DELETE, INSERT, and UPDATE are not possible.

Input data for your queries are read from ORC, CSV, JSON, or Parquet files located in one or more IBM Cloud Object Storage instances. Each query result is written by default to a CSV file in a Cloud Object Storage instance where you created the integration. But you can freely override and customise the format and Object Storage location as part of the SQL statement that you run.

You can use a custom INTO clause of a SELECT statement to control where and how result data from a SELECT statement is written to IBM Cloud Object Storage.

Getting started using SQL Query SELECT statements from inside your instance is as easy as creating an integration. Objects of queryable data formats, as well as folders with multiple objects of a consistent queryable format (when shown in the "folders" view) are labeled as shown in Figure 4.

Zoom

![Object with SQL label](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/accessible-using-sql.jpg)

Figure 4.
.query("GST-sdk-")
.fields(new java.util.ArrayList<String>(java.util.Arrays.asList("crn")))
.searchCursor(searchCursor)
.build();

Response<ScanResult> response = service.search(searchOptions).execute();
ScanResult scanResult = response.getResult();

System.out.println(scanResult);

response = global_search_service.search(query='GST-sdk-',
fields=['crn'])
scan_result = response.get_result()

print(json.dumps(scan_result, indent=2))

searchOptions := globalSearchService.NewSearchOptions()
searchOptions.SetLimit(10)
searchOptions.SetQuery("GST-sdk-")
searchOptions.SetFields([]string{"crn"})

scanResult, response, err := globalSearchService.Search(searchOptions)
if err != nil {
panic(err)
}
b, _ := json.MarshalIndent(scanResult, "", " ")
fmt.Println(string(b))

const params = {
query: 'GST-sdk-',
fields: ['crn'],
searchCursor: searchCursor,
};

globalSearchService.search(params)
.then(res => {
console.log(JSON.stringify(res.result, null, 2));
})
.catch(err => {
console.warn(err)
});
2. Extract the value of the CRN field from the response.

{
"items": [
{
Initiating DROP SCHEMA statement against a non-empty schema may result in Db2 SQL Error SQLCODE=-478 and SQLSTATE=42893.





 Issue: CREATE VIEW statement that is partially supported by Db2 

Db2 connector partially supports CREATE VIEW statement. The Presto supported SQL syntax does not include creating views with custom column names (different than the table column names).





 Issue: CREATE VIEW statement that is partially supported by Netezza 

Netezza connector partially supports CREATE VIEW statement. The Presto Supported SQL syntax does not include creating views with custom column names (different than the table column names).





 Issue: Data ingestion through CLI 

Following are the limitations when ingesting data through the CLI:



* Schema evolution is not supported.
* Partitioning is not supported.
* Iceberg target table is the only supported output format.
* Source CSV file containing TAB or space as delimiter is not supported.
* pathStyleAccess property for object storage is not supported.
* Source files must be of the same format type and only Parquet and csv file formats are supported.







 Issue: Data ingestion through web console 

Following are the limitations when ingesting data through the web console:



* Iceberg target table is the only supported format.
* Partitioning is not supported.
* Source CSV file containing TAB or space as delimiter is not supported.
* Configure options are disabled for GA.
* Target table output format is Iceberg and the target data format is Parquet.
* Target storage path is default and cannot be changed.







 Issue: Presto does not recognize the path as a directory 

When you create a new table with a Presto Hive connector that uses an S3 folder from an external location, Presto does not recognize the path as a directory and an error might occur.

For example, when creating a customer table in the target directory DBCERT/tbint in a bucket that is called dqmdbcertpq by using the IBM Cloud UX and Aspera S3 console, the following error is encountered: External location must be a directory.

CREATE TABLE "hive-beta"."dbcert"."
If you don't use Hive-style partitioning or catalog tables, all objects in the source location must be listed as part of the query execution, even if only a single row from a single object is required for the query.

To successfully query the source location, use the following best practices.



* Create a table with the correct schema specified. If you don't specify the schema, schema inference causes the same error to be returned.
* If you use more than 10 UNION/JOIN constructs for different source URIs, try to lower the number of different sources.
* Depending on the number of table partitions, you can either add the partition manually one by one, or use the ALTER TABLE … RECOVER PARTITION command. Although the limit is 20,000 partitions per table by default, stay below 10,000 partitions for a single table.
* Lay your data out by using Hive-style partitioning and aim for an object size of 128 MB if possible.