vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* When you access the result, use the getter method, matching the result column type, for example, getInt() to retrieve an INTEGER column. Some implicit conversions are supported, for example, accessing an INTEGER column with getLong(), but not all conversions are implemented.
* All result sets are FORWARD_ONLY and can be iterated with the ResultSet.next() method only. Determining the cursor position and explicitly moving the cursor is not supported.
* Prepared statements are not supported.
* Batch execution with the Statement.addBatch() and executeBatch() methods is not supported.
* Various other uncommon JDBC methods are not implemented.







 ETL-type statements 

You can use the JDBC driver to run transform-type statements on your data in Cloud Object Storage, where you do not want to retrieve the data in the JDBC client itself. See the following example of such a statement:

SELECT ...
FROM cos://source1 JOIN cos://source2 ON ... JOIN ...
INTO cos://target STORED AS ... PARTITIONED BY ...

You can run this type of statement (with a user-defined INTO clause) by the Statement.executeUpdate method. This statement does not give you a result set in the JDBC client (and avoids all overhead for result fetching), but creates a target object with the specific format and partitioning that you need.

You cannot run statements with an INTO clause by using the generic Statement.execute() method because the driver cannot detect to treat these statements as an update, not a query. Therefore, the driver generates a second INTO clause, leading to a syntax error.





 JDBC driver logging 

JDBC driver logging uses the java.util.logging framework, similar to the [postgresql JDBC driver](https://jdbc.postgresql.org/documentation/logging/). However, since driver version 2.5.30, logging is turned off by default to avoid unexpected console output in a default logging configuration. To turn on logging, use the connection property loggerLevel:
Query editor 

With the query editor, you can run SQL queries on a specific host and database. You can also save any of the queries that you create.



  Creating queries 



1.  Go to Query editor.
2.  From Data objects, select the database and schema in which you want to run the query.
If you do not pick a schema, the default database schema is selected.
3.  Type the SQL query that you want to run.
If your query is a select statement, a Set Limit option shows up to allow you to specify how many rows of data you would like to retrieve. The default is No limit.

In the Worksheet settings you can specify your Default maximum number of rows limit in result. If you decide, however, to add a limit clause in a select statement that is greater than your Default maximum number of rows limit in result, for example: select * from table1 limit 10;, the Results field shows the smaller value of these two parameters.

In the Worksheet settings you can also specify the Statement separator you want to use. A semicolon (";") is the default Statement separator and you must change it to an ampersand ("&") when your queries contain semicolons (";") to avoid errors.
4.  When you input the information, you can do one of the following:



*  Click Run to run the query.
The results of the query are displayed in the panel.
*  Click the floppy disk icon that is in the SQLworksheet toolbar to save the query as a template.
The saved query is added to Saved queries and Queries > Recent Queries.
*  Click Clear to clear the query.
John : 433-999-1000
Martha Mob 433-999-1001



* The select list The select list in an AQL select or extract statement consists of a comma-delimited list of output expressions.
* The from list The second part of a select or an extract statement in AQL is the from list. The from list is a comma-separated list that is the source of the tuples to be selected or extracted.
* The where clause The optional where clause defines a predicate to be applied on each tuple generated from the Cartesian product of all tuples in the relations in the from clause.
* The consolidate on clause The optional consolidate on clause specifies how overlapping spans are resolved across tuples that are output by a select or extract statement. Tuples with non-overlapping spans are not affected when this clause is used.
* The group by clause The optional group by clause of a select statement directs the runtime component to group tuples that are produced from the same document by common values of a specified field.
* The order by clause The optional order by clause directs the runtime component to order the output tuples that are produced by the select statement from each document based on the values of the order by list, which is a comma-delimited set of expressions
* The limit clause The optional limit clause specifies a limit on the number of output tuples that are produced by the select statement for a document.
* The select... into statement The select ... into statement is useful for defining a view and specifying that it is an output view in a single statement.









 The select list 

The select list in an AQL select or extract statement consists of a comma-delimited list of output expressions.



 Syntax 

Each select expression must be in one of the following forms:

select
<viewname>.<colname> as <alias> 
<viewname>.* 
<expr> as <alias> 
case
when <predfunction1()> then <expr1>
when <predfunction2()> then <expr2>...
when <predfunctionn()>
then <exprn>
From the navigation menu, select SQL. The Query workspace page opens.
2. Select the Presto engine from the Engine drop-down.
3. Select the catalog. In this scenario, consider the Apache Iceberg catalog, default schema, order_detail table, to run the query.
4. Click the overflow menu and select the required query.



* For a catalog and schema, you can run the Generate Path query.
* For a table, you can run the Generate path, Generate SELECT, Generate ALTER, and Generate DROP query.
* For a column, you can run the Generate path, Generate SELECT, and Generate DROP query.



Consider the following sample query to view the details from iceberg-beta:

Example:

!/bin/bash
SELECT * FROM "iceberg-beta"."default"."order_detail" LIMIT 10;
5. Click the Run on button to run the query.
6. Select Result set or Details tab to view the results. If required, you can save the query.
7. Click Saved queries to view the saved queries.
8. Click [Explain](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-explain_sql_query) to view the logical or distributed plan of execution for a specified SQL query.
You can now enter CLPPlus commands or SELECT statements and run scripts to work with the data in the database.





 Examples 

The following examples use a short script that retrieves rows from the sample table GOSALES.BRANCH. The script file is named cities.sql and is on the local Windows computer in the C:temp directory. The cities.sql file contains the following text:

SET ECHO ON
SELECT branch_code, city from GOSALES.BRANCH;



 Example 1 

To run the script interactively:



1. Start CLPPlus with your user ID and the alias that you created in the db2dsdriver.cfg file by running the following command:

clpplus <user_id>@<alias>
2. Enter your password.
3. At the SQL prompt, enter the following text:

start C:tempcities.sql







 Example 2 

Start CLPPlus with your user ID and the alias that you created in the db2dsdriver.cfg file and run the script in one step:

clpplus <user_id>/<password>@<alias> @C:tempcities.sql

Sample output from the cities.sql script follows:

BRANCH_CODE CITY
----------- --------------------------------------------------
6 Paris
7 Milano
9 Amsterdam
13 Hamburg
14 München
15 Kista
17 Calgary
18 Toronto
19 Boston
20 Seattle
21 Los Angeles
22 Miami
23 Lyon
24 Distrito Federal
25 Tokyo
26 Osaka City
28 Melbourne
29 Bilbao
30 Sao Paulo
31 Kuopio
32 Seoul
33 Singapore

BRANCH_CODE CITY
Thus, cataloging improves your query performance, especially for text-based data formats, such as CSV and JSON, where the schema inference requires a full scan of the data before the actual query execution.







 Select 

See the following examples for an outline of the general syntax of an SQL query statement that uses the query clause and the namedQuery clause.



 query 





 namedQuery 





 intoClause 

The query statement supports common table expressions. A common table expression permits defining a result table with a table name that can be specified as a table name in any FROM clause of the fullselect that follows.

Common table expressions are defined by using the reserved keyword WITH followed by one or more named queries. Each common table expression that is specified can also be referenced by name in the FROM clause of subsequent common table expressions.

Creating a common table expression avoids the overhead of creating and dropping an intermediate result object on Cloud Object Storage that is needed only for a certain query.

Moreover, a common table expression is beneficial when the same result table must be shared in a fullselect.





 Examples 

The common table expression examples use values clauses to define tables inline. For more information about the values clause, see [valuesClause](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencevaluesClause).

-- find the department with the highest total pay
WITH dtotal AS (
SELECT
col1 AS deptno,
SUM(col3+col4) AS totalpay
FROM VALUES -- deptno, empid, salary, bonus
(1, 1, 1000, 0), (1, 2, 2000, 500), (1, 3, 3000, 0),
(2, 4, 5000, 200), (2, 5, 6000, 0), (2, 6, 4000, 0),
(3, 7, 2000, 500), (3, 8, 2000, 500), (3, 9, 8000, 0)
GROUP BY col1
)
SELECT deptno
FROM dtotal
WHERE totalpay = (SELECT MAX(totalpay) FROM dtotal)
* Use the INTO clause of a [query](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencechapterSQLQueryStatement) to specify the output [URI](https://cloud.ibm.com/docs/sql-query?topic=sql-query-runningunique), that is, the location to which the result is to be written and the wanted result format.



2. The Target location field displays where the result is stored. An initial bucket in one of your Object Storage instances is automatically created for you when you open the UI. It is then chosen as your default location, if your query does not specify an INTO clause. To ensure the automatic setup of an initial bucket, do the following steps in advance:



* You must create an Object Storage instance.
* You must have at least 'Writer' access to the corresponding Object Storage bucket.



In the Details tab of the selected job, you can set any location that you specified in the INTO clause as your default location.
3. Click Run.

When the query completes, a preview of the query result is displayed in the query result tab of the UI. The preview function is only available for CSV and JSON result formats. You can run up to five queries simultaneously with a Standard plan instance of Data Engine.





 Sample queries 

What does a typical query look like? The following sample queries give you an idea to get you started:



 Example of a table exploration query 

The following query selects all columns of a table and limits the result to 50 rows. Use it to explore a particular table.

SELECT 
FROM cos://us-geo/sql/customers.csv STORED AS CSV
ORDER BY CustomerID
LIMIT 50





 Example of an exact target path specification 

The following query writes an SQL result into an exact result path. Normally, Data Engine always appends jobid=<jobid> to the provided target path to ensure a unique result location with each query execution. However, in the following sample query, this suffix is eliminated by adding JOBPREFIX NONE to the path in the INTO clause.
"bookmark": "g1AAAABweJzLYWBgYMpgSmHgKy5JLCrJTq2MT8lPzkzJBYorpBiYGJolWaSZmaWYpFqmGBiYmKclphhappqZmRglG5uD9HHA9BGlIwsAms8eJw",
"warning": "No matching index found, create an index to optimize query time."
}



Next, you find a document in the database by using multiple operators.





 Step 4: Running a query with operators 

In this example, the $gt (greater than) and $eq (equal) operators are used to search for documents that include a temperature that is greater than 20 degrees and a description that contains the value hot. The results include the descriptionField and temperatureField, shown in descending order by temperature.

You use a selector expression like the following example:

{
"selector": {
"temperatureField": {
"$gt": 50
},
"descriptionField": {
"$eq": "hot"
}
},
"fields": [
"descriptionField",
"temperatureField"
],
"sort": [{
"temperatureField": "desc"
}],
"use_index": "_design/query-demo-index"
}
Show more

To run the query, follow these steps:



1. Copy the following sample JSON to a file named query3.json.

{
"selector": {
"descriptionField": {
"$eq": "hot"
},
"temperatureField": {
"$gt": 50
}
},
"fields": [
"descriptionField",
"temperatureField"
],
"sort": [{
"temperatureField": "desc"
}],
"use_index": "_design/query-demo-index"
}
Show more
2. Run this query:

acurl "https://$ACCOUNT.cloudant.com/databasedemo/_find" -X POST -H "Content-Type: application/json" -d @query3.json
3. Review the query results:

{
"docs": [{
Go to Queries > Stored queries.
2. Select a query.
3. From the overflow menu, click Remove.
4. Confirm your choice by clicking Remove again.









 Query history 

To access the page, go to Queries > Query history or select Query history from the home page.

When you are on the Query history page, you can do the following:



* View your data in a table or card view.
* Export data to export your query history to a data file.
* Sort any column by placing the cursor on the column header.
* Find specific queries by using various filtering criteria.

For example, you can use it to find queries that are submitted by a particular user or group, or queries that run on a particular database.
* Search the query history but clicking Search. You can use a predefined search criteria, or create a new search option.
* Select the columns to display in the table.

Click the settings icon next to the Find query history field to edit columns.
* View metrics, access the explain graph, explain summary, explain verbose, explain distribution pages, and view the plan file and statistics status.





 Query history columns 



* Start time Specifies the time when the query started.
* End time Specifies the time when the query finished.
* Elapsed time Specifies the time that it took the query to run.
* Query text Specifies the SQL command of the query.
* Database Specifies the name of the database on which the query ran.
* Schema Specifies the schema that was used for the query.
* User name Species the name of the user that ran the query.
* Group Specifies the group of users from which the query originates.
* Result rows Specifies the number of result rows that were returned by the query.
* Prep time Specifies the preparation time that was needed for the query.
* Status Specifies the completion status of the query.
* Plan ID Specifies the ID of the system-generated plan for the query.
* Client IP Specifies the IP of the client that ran the SQL query.
* GRA time Specifies the time that the query spent at the GRA.
> SELECT 1 <=> '1'
: Result value

true
: Example of an SQL function usage fragment

> SELECT true <=> NULL
: Result value

false
: Example of an SQL function usage fragment

> SELECT NULL <=> NULL
: Result value

true





 = 

expr1 = expr2
: Returns true if expr1 equals expr2, or false otherwise.
: Arguments

expr1, expr2 - The two expressions must be same type or can be cast to a common type, and must be a type that can be used in equality comparison. Map type is not supported. For complex types such array and struct, the data types of fields must be orderable.
: Example of an SQL function usage fragment

> SELECT 2 = 2
: Result value

true
: Example of an SQL function usage fragment

> SELECT 1 = '1'
: Result value

true
: Example of an SQL function usage fragment

> SELECT true = NULL
: Result value

NULL
: Example of an SQL function usage fragment

> SELECT NULL = NULL
: Result value

NULL





 == 

expr1 == expr2
: Returns true if expr1 equals expr2, or false otherwise.
: Arguments

expr1, expr2 - The two expressions must be same type or can be cast to a common type, and must be a type that can be used in equality comparison. Map type is not supported. For complex types such array and struct, the data types of fields must be orderable.
: Example of an SQL function usage fragment

> SELECT 2 == 2
: Result value

true
: Example of an SQL function usage fragment

> SELECT 1 == '1'
: Result value

true
: Example of an SQL function usage fragment

> SELECT true == NULL
: Result value

NULL
: Example of an SQL function usage fragment

> SELECT NULL == NULL
: Result value

NULL





 > 

expr1 > expr2
SQL reference 



 Introduction 

With IBM Cloud® Data Engine, you can analyze and transform open data with SQL. It supports the various types of SELECT statements from the ANSI SQL standard.

The SELECT statement (or query statement) is used to read object data from IBM Cloud® Object Storage (COS), process the data, and store it back on Cloud Object Storage eventually.

You can use Data Engine as a data transformation service, as it always writes the results of a query to a specified location in either Object Storage or Db2 tables. Data Engine provides extended SQL syntax inside a special INTO clause to control how the result data is stored physically. This extended SQL syntax includes control over data location, format, layout, and partitioning.

A query statement can be submitted through Data Engine's web UI or programmatically, either by using the service's REST API, or by using the Python or Node.JS SDK. You can also use IBM Watson® Studio and the Python SDK to use Data Engine interactively with Jupyter Notebooks. In addition, you can submit SQL queries that use IBM Cloud® Functions.

In addition to the ad hoc usage of data in IBM Cloud® Object Storage, you can also register and manage your data in a catalog as tables, consisting of columns and partitions.

Several benefits to cataloging your data exist:



* It simplifies SQL SELECT statements because the SQL author does not need not know and specify exactly where and how the data is stored.
* The SQL execution can skip the inference of schema and partitioning because this information is available in the metastore. Thus, cataloging improves your query performance, especially for text-based data formats, such as CSV and JSON, where the schema inference requires a full scan of the data before the actual query execution.







 Select 

See the following examples for an outline of the general syntax of an SQL query statement that uses the query clause and the namedQuery clause.



 query 





 namedQuery 





 intoClause 

The query statement supports common table expressions. A common table expression permits defining a result table with a table name that can be specified as a table name in any FROM clause of the fullselect that follows.
tablesDF=spark.sql("SHOW TABLES")
tablesDF.show()
statesDF=spark.sql("SELECT * from COUNTRIESCAPITALS");
statesDF.show()

def main():
spark,sc = init_spark()
select_query_data_engine(spark,sc)

if __name__ == '__main__':
main()
Show more

Note that for the SELECT command to work, you must pass the IBM Cloud Object Storage identifiers as one of the standard Data Engine aliases, in this example, we have used ALIAS NAME. If you do not pass the expected ones, you might see the following error: Configuration parse exception: Access KEY is empty. Please provide valid access key.

select_query_data_engine_payload.json:

{
"application_details": {
"conf": {
"spark.hadoop.fs.cos.ALIAS NAME.endpoint": "CHANGEME",
"spark.hadoop.fs.cos.ALIAS NAME.access.key": "CHANGEME",
"spark.hadoop.fs.cos.ALIAS NAME.secret.key": "CHANGEME",
            "spark.hive.metastore.truststore.password" : "changeit",
            "spark.hive.execution.engine":"spark",
            "spark.hive.metastore.client.plain.password":"APIKEY-WITH-ACCESS-TO-DATA-ENGINE-INSTANCE",
            "spark.hive.metastore.uris":"THRIFT URL",
            "spark.hive.metastore.client.auth.mode":"PLAIN",
            "spark.hive.metastore.use.SSL":"true",
            "spark.hive.stats.autogather":"false",
            "spark.hive.metastore.client.plain.username":"<CHANGEME-CRN-DATA-ENGINE-INSTANCE>",
 for spark 3.3
[SQL Query Window](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/select-with-sql.jpg)

Figure 7. Access with SQL query window

The entry representing the job of the SELECT statement run previously is shown in Figure 8. There are two tabs, "Results" and "Details," at the top of the list that allow you to switch between seeing the results and more detailed information.

Zoom

![SQL Query Results](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/results-from-sql.jpg)

Figure 8. Access with SQL query jobs

The entry representing the details of running the SELECT statement run previously is shown in Figure 9.

Zoom

![SQL Query Details](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/details-from-sql.jpg)

Figure 9. Access with SQL query jobs





 Next Steps 

For more information on using Data Engine see the [Data Engine documentation](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overview) and [Analyzing Data with IBM Cloud SQL Query](https://www.ibm.com/cloud/blog/analyzing-data-with-ibm-cloud-sql-query).
* Prepare your data first. Transform input formats, such as .csv or .json to a compressed format, such as Parquet.
* Write results into [partitioned result sets](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencepartitionedClause). Start with a double digit partition value. An optimal single object size is about 128 MB. For more information, see [the following blog on big data layout](https://www.ibm.com/cloud/blog/big-data-layout).
* Use Parquet as the output format (STORED AS clause) instead of .csv or .json. Avoid working with Gzip objects. Depending on how a join clause is written, it can cause memory-intensive operations. Avoid to use functions in the join clause, such as trim. It is better to cleanse the data first with a preparation step and then to use the cleansed data in the join operation, without needing the functions.
* If you already follow the previous recommendations and still get the out of memory errors, try breaking down complex statements into multiple SQL statements.
* When you use time series functions, look at the following [FAQ](https://cloud.ibm.com/docs/sql-query?topic=sql-query-faq).







 How to work with large number of objects? 

If you have more than 150,000 objects in a single source location and you don't use a catalog table, it's possible that your query fails with an error, such as "Too many objects for a single query". If you don't use Hive-style partitioning or catalog tables, all objects in the source location must be listed as part of the query execution, even if only a single row from a single object is required for the query.

To successfully query the source location, use the following best practices.



* Create a table with the correct schema specified. If you don't specify the schema, schema inference causes the same error to be returned.
* If you use more than 10 UNION/JOIN constructs for different source URIs, try to lower the number of different sources.
gosales"."order_detail" AS header
ON details.order_number=header.order_number
LIMIT 10;
4. Click the Run on button to run the query.
5. Select Result set or Details tab to view the combined result. If required, you can save the query.
6. Click Saved queries to view the saved queries.
7. Click [Explain](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-explain_sql_query) to view the logical or distributed plan of execution for a specified SQL query.