vania-janet/multiturn-rag-retrieval-data huggingface dataset:
In this case, the names _C0, _C1, ... are used for the input columns. For more information, see [COS URI](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceCOSURI).

By default, if the format of the input data is JSON, each line must contain a separate, self-contained, and valid JSON object, also called newline-delimited JSON. However, if you specify the option MULTILINE, Data Engine can process JSON input data even if individual data records span multiple lines, such as when the data was formatted to make it easier to read. Specify this option only if you really need it because it limits input parallelization and can significantly reduce performance when you process large volumes of JSON data. If you need to frequently query large amounts of multiline JSON data, use Data Engine to transform the data into single -line JSON, or into a more performance optimized format, such as Parquet, before querying the transformed data.

If the file format is Parquet, with the optional MERGE SCHEMA clause you can handle Parquet schema evolution by specifying to scan all qualifying Parquet objects for their schema, and to merge the final schema across all objects. By default, for Parquet input only the first Parquet object that is found is used to infer the schema, which guarantees minimal overhead for compiling the SQL. Thus, use this option if your Parquet input data does not have a homogeneous schema.

If the object type identifier TEXT is used, the input is read line by line without inferring any schema. The whole line is returned in one column, called value. To query specific fields from the input data, you can use the options [regexp_extract](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sqlfunctionsregexp_extract) or [get_json_object](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sqlfunctionsget_json_object).





 timeSeriesProperties
spark.sql('select * from lakehouse.demodb.yellow_taxi_2022').count()

 load csv data into a dataframe
csvDF = spark.read.option("header",True).csv("s3a://source-bucket/zipcodes.csv")
csvDF.createOrReplaceTempView("tempCSVTable")
 load temporary table into an Iceberg table
spark.sql('create or replace table lakehouse.demodb.zipcodes using iceberg as select * from tempCSVTable')
 describe the table created
spark.sql('describe table lakehouse.demodb.zipcodes').show(25)
 query the table
spark.sql('select * from lakehouse.demodb.zipcodes').show()

 demonstration: Table maintenance
 load data for the month of Feburary to June into the table yellow_taxi_2022 created above
df_feb = spark.read.option("header",True).parquet("s3a://source-bucket/nyc-taxi/yellow_tripdata_2022-02.parquet")
df_march = spark.read.option("header",True).parquet("s3a://source-bucket/nyc-taxi/yellow_tripdata_2022-03.parquet")
df_april = spark.read.option("header",True).parquet("s3a://source-bucket/nyc-taxi/yellow_tripdata_2022-04.parquet")
df_may = spark.read.option("header",True).parquet("s3a://source-bucket/nyc-taxi/yellow_tripdata_2022-05.parquet")
df_june = spark.read.option("header",True).parquet("s3a://source-bucket/nyc-taxi/yellow_tripdata_2022-06.parquet")
The provisioning starts.
5. In the Resource List, locate the new instance under Databases and wait for it to be available (sometimes you may need to refresh the page). Click on the entry for your Db2 Warehouse on Cloud service.
6. Click on Open Console to launch the database console.







 Step 2: Create a table 

You need a table to hold the sample data. Create it using the console.



1. In the console for Db2 Warehouse on Cloud click on the upper left menu icon, then Run SQL in the navigation bar.
2. Click on the + symbol (Add a new script) next to the Untitled - 1 tab.
3. Click on From file and select the file cityschema.txt from the Github repository that was previously cloned to your local directory and open it.
4. Click on Run all to execute the statement. It should show a success message.







 Step 3: Load data 

Now that the table "cities" has been created, you are going to load data into it. This can be done in different ways, for example from your local machine or from cloud object storage (COS) or Amazon S3 interface. For this tutorial, you are going to upload data from your machine. During that process, you adapt the table structure and data format to fully match the file content.



1. In the console for Db2 Warehouse on Cloud click on the upper left menu icon, then Data in the navigation bar.
2. From the Load Data tab, click on My Computer.
3. Under File selection, click on Drag a file here or browse files to locate and pick the file "cities1000.txt" you downloaded in the first section of this guide.
4. Click Next to get to the schema overview. Choose the schema starting with BLUADMIN, then the table CITIES. Click on Next again.

Because the table is empty it does not make a difference to either append to or overwrite existing data.
5. Now customize how the data from the file "cities1000.txt" is interpreted during the load process.
* [group by<group by list>]

Groups the tuples that are produced from the same document by common values of a specified field. This clause is optional.
* [order by<order by list>]

Orders the output tuples that are produced by the select statement from each document. The order is based on the values of the order-by list, a comma-delimited list of expressions. This clause is optional.
* [limit <maximum number of output tuples for each document>]

Limits the number of output tuples for each document to the specified maximum. This clause is optional.







 Usage Notes 

The semantics of the select statement are as follows:



* Determine the input data (in tuples) by taking the Cartesian product of relations in the from list.
* For each input tuple that is generated, filter it by applying the predicates in the (optional) where clause.
* If the optional group by clause is present, group tuples that are produced from the same document by the values that are specified in the group-by list and compute the result of the aggregate functions within the select list.
* Consolidate any overlapping tuples, according to the policy defined in the (optional) consolidation clause. If the optional order by clause is present, order these tuples by the values of the order-by list.
* Compute all expressions within the select list on each tuple, and rename the columns as specified by the as clauses.
* If the optional limit clause is present, limit the number of output tuples to the specified number of tuples for each document.







 Examples 

An example of how to use the select statement is to extract phone numbers that match a pattern. Assume that the PhoneNumbers view that extracts phone numbers of the pattern XXX-XXX-XXXX for United States is already defined. This select statement evaluates the regular expression for the pattern 444-888-XXXX across the input text. The view has the output columns documentText and phoneNumber. In addition, the output is limited to the first occurrence of this phone number pattern that is identified per document.

create view PhoneNumbersPattern1 as
select D.documentText, D.phoneNumber
You can find examples that demonstrate how queries use indexes and what type of queries will have the best performance in the [Fabric CouchDB tutorial](https://hyperledger-fabric.readthedocs.io/en/release-2.2/couchdb_tutorial.htmluse-best-practices-for-queries-and-indexes).

Peers on the IBM Blockchain Platform have a set queryLimit, and will only return 10,000 entries from the state database. If your query reaches the queryLimit, you can use multiple queries to get the remaining results. If you need more results from a range query, start subsequent queries with the last key that is returned by the previous query. If you need more results from JSON queries, sort your query by using one of the variables in your data, then use the last value from the previous query in a 'greater than' filter for the next query.

Do not query the entire database for the purpose of aggregation or reporting. If you want to build a dashboard or collect large amounts of data as part of your application, you can query an off chain database that replicates the data from your blockchain network. This allows you to understand the data on the blockchain without degrading the performance of your network or disrupting transactions.

You can use block or chaincode events from your application to write transaction data to an off-chain database or analytics engine. For each block received, the block listener application would iterate through the block transactions and build a data store by using the key/value writes from each valid transaction's rwset. The [Peer channel-based event services](https://hyperledger-fabric.readthedocs.io/en/release-2.2/peer_event_services.html) provide replayable events to ensure the integrity of downstream data stores. For an example of how you can use an event listener to write data to an external database, see the [Off chain data sample](https://github.com/hyperledger/fabric-samples/tree/release-1.4/off_chain_data) in the Fabric Samples.





 Clock synchronization
> SELECT length('Spark SQL ')
: Result value

10
: Example of an SQL function usage fragment

> SELECT CHAR_LENGTH('Spark SQL ')
: Result value

10
: Example of an SQL function usage fragment

> SELECT CHARACTER_LENGTH('Spark SQL ')
: Result value

10





 levenshtein 

levenshtein(str1, str2)
: Returns the Levenshtein distance between the two indicated strings.
: Example of an SQL function usage fragment

> SELECT levenshtein('kitten', 'sitting')
: Result value

3





 like 

str like pattern
: Returns true if str matches pattern, null if any arguments are null, otherwise, false.
: Arguments

str - a string expression

pattern - A string expression. The pattern is a string that is matched literally, with exception to the following special symbols:

_ matches any one character in the input (similar to . in posix regular expressions)

% matches zero or more characters in the input (similar to .* in posix regular expressions)

The escape character is \. If an escape character precedes a special symbol or another escape character, the following character is matched literally. It is invalid to escape any other character.

String literals are unescaped in our SQL parser. For example, to match abc, the pattern is \abc.
: Example of an SQL function usage fragment

> SELECT '%SystemDrive%UsersJohn' like '%SystemDrive%\Users%'
: Result value

true

Note:

Use RLIKE to match with standard regular expressions.





 ln 

ln(expr)
: Returns the natural logarithm (base e) of expr.
: Example of an SQL function usage fragment

> SELECT ln(1)
: Result value

0.0





 locate 

locate(substr, str[, pos])
: Returns the position of the first occurrence of substr in str after position pos.
> SELECT inline_outer(array(struct(1, 'a'), struct(2, 'b')))
: Result value

1 a 2 b





 input_file_name 

input_file_name()
: Returns the name of the file that is being read, or empty string if not available.





 instr 

instr(str, substr)
: Returns the (1-based) index of the first occurrence of substr in str.
: Example of an SQL function usage fragment

> SELECT instr('SparkSQL', 'SQL')
: Result value

6





 int 

int(expr)
: Casts the value expr to the target data type int.





 isnan 

isnan(expr)
: Returns true if expr is NaN, or false otherwise.
: Example of an SQL function usage fragment

> SELECT isnan(cast('NaN' as double))
: Result value

true





 isnotnull 

isnotnull(expr)
: Returns true if expr is not null, or false otherwise.
: Example of an SQL function usage fragment

> SELECT isnotnull(1)
: Result value

true





 isnull 

isnull(expr)
: Returns true if expr is null, or false otherwise.
: Example of an SQL function usage fragment

> SELECT isnull(1)
: Result value

false





 json_tuple 

json_tuple(jsonStr, p1, p2, ..., pn)
: Returns a tuple like the function get_json_object, but it takes multiple names. All the input parameters and output column types are string.
: Example of an SQL function usage fragment

> SELECT json_tuple('{"a":1, "b":2}', 'a', 'b')
: Result value

1 2





 kurtosis 

kurtosis(expr)
: Returns the kurtosis value that is calculated from values of a group.





 lag
'statement': 'CREATE TABLE COUNTRIESCAPITALS (Country string,Capital string) USING PARQUET LOCATION cos://ALIAS NAME/mybucket/countriescapitals.parquet',
    }
    response = requests.post('https://api.dataengine.cloud.ibm.com/v3/sql_jobs', params=params, headers=headers_token, json=json_data)
    job_id = response.json()['job_id']
    time.sleep(10)
    response = requests.get(f'https://api.dataengine.cloud.ibm.com/v3/sql_jobs/{job_id}', params=params, headers=headers_token)
    if(response.json()['status']=='completed'):
        print(response.json())

Parameter values:
ALIAS NAME: Note that for the location URI (cos://ALIAS NAME/mybucket/countriescapitals.parquet) you need to pass one of the standard IBM Cloud Data Engine aliases. See [Data engine endpoints](/docs/sql-query?topic=sql-query-overviewendpoints)

The payload for the above application create_table_data_engine_payload.json also needs to provide the Data Engine credentials with the exact standard Data Engine alais, in this case: "ALIAS NAME"

Example:
Enter:

 {: codeblock}
{
"application_details": {
"conf": {
"spark.hadoop.fs.cos.ALIAS NAME.endpoint": "CHANGEME",
"spark.hadoop.fs.cos.ALIAS NAME.access.key": "CHANGEME",
"spark.hadoop.fs.cos.ALIAS NAME.secret.key": "CHANGEME"
        },
  "application": "cos://mybucket.ALIAS NAME/create_table_data_engine.py",
"allusers",
"getVerifiedEmails",
)

postViewOptions.SetDescending(true)
postViewOptions.StartKey = "alpha"
postViewOptions.EndKey = "beta"

viewResult, response, err := service.PostView(postViewOptions)
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(viewResult, "", " ")
fmt.Println(string(b))

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)

All Go examples require the service object to be initialized. For more information, see the API documentation's [Authentication section](https://cloud.ibm.com/apidocs/cloudant?code=goauthentication-with-external-configuration) for examples.

The end_key of beta is seen before the start_key of alpha, resulting in a query parse error.

The solution is to reverse not just the sort order, but also the start_key and end_key parameter values.

The following example shows correct filtering and reversing the order of output, by using the descending query argument, and reversing the start_key and end_key query arguments.

See the example that uses HTTP to apply correct filtering and sorting to a global query:

GET $SERVICE_URL/$DATABASE/_design/$DDOC/_view/$VIEW_NAME?descending=true&start_key="beta"&end_key="alpha" HTTP/1.1

See the example to apply correct filtering and sorting to a global query.

Client libraries use POST method instead of GET because they have the same behavior.



* Curl
* Java
* Node
* Python
* Go
Nulls are ordered lower than other objects.







 Examples 

Example 1: Order by multiple expressions

Assume that person is a field of type Span. The following order by clause specifies that the statement returns tuples within each document. They are ordered lexicographically by the text of the person field and then by the beginning of the person field.

order by GetText(P.person), GetBegin(P.person)







 The limit clause 

The optional limit clause specifies a limit on the number of output tuples that are produced by the select statement for a document.



 Syntax 

select <select list>
from <from list>
...
[limit <maximum number of output tuples for each document>];





 Description 



* <maximum number of output tuples for each document>

Specifies the maximum number of output tuples for each document. If the limit value is greater than or equal to the total number of tuples that could be returned, all of the tuples are returned.







 Examples 

Example 1: Limiting the number of returns

This example returns the first three person names in each document:

create view SampleView as
select 
from Person P
order by GetBegin(P.name)
limit 3;







 The select... into statement 

The select ... into statement is useful for defining a view and specifying that it is an output view in a single statement.



 Syntax 

select <select list>
into <output view name>
from <from list>
[where <where clause>]
[consolidate on <column> using '<policy>' with priority from <column> priority order]]]]
[group by <group by list>]
[order by <order by list>]
[limit <maximum number of output tuples for each document>];





 Description 



* <output view name>

Specifies the name of the output view that is defined by the statement. The select ... into statement is identical to the select statement, except for the additional into <output view name> clause.







 Examples 

Example 1: Defining a view
left_table.col1 AS l_col1,
left_table.col2 AS l_col2,
right_table.col1 AS r_col1,
right_table.col2 AS r_col2
FROM
VALUES (0, 10), (1, 11), (2, 12), (3,13), (4, 14), (5, 14) AS left_table
CROSS JOIN VALUES (0, 10), (2, 12), (4, 14), (6, 16) AS right_table

The result of the example query is shown in the following table.



Table 23. Query result for the example cross

 L_COL1 L_COL2 R_COL1 R_COL2 

 0 10 0 10 
 0 10 2 12 
 0 10 4 14 
 0 10 6 16 
 1 11 0 10 
 1 11 2 12 
 1 11 4 14 
 1 11 6 16 
 2 12 0 10 
 2 12 2 12 
 2 12 4 14 
 2 12 6 16 
 3 13 0 10 
 3 13 2 12 
 3 13 4 14 
 3 13 6 16 
 4 14 0 10 
 4 14 2 12 
 4 14 4 14 
 4 14 6 16 
 5 14 0 10 
 5 14 2 12 
 5 14 4 14 
 5 14 6 16 







 Anti join 

A LEFT ANTI or ANTI join returns only rows from the left table that do not have a matching row in the right table. Columns from the right table cannot be included in the column list of the select statement.

-- left anti join query
SELECT
left_table.col1 AS l_col1,
left_table.col2 AS l_col2
FROM
VALUES (0, 10), (1, 11), (2, 12), (3,13), (4, 14), (5, 14) AS left_table
ANTI JOIN
VALUES (0, 10), (2, 12), (4, 14), (6, 16) AS right_table
ON left_table.col1 = right_table.col1
If you are using Docker, replace the part after the | accordingly.
5. The script configuration above pushes about 10 lines/second. Stop the script after the desired number of messages have been streamed using control+C.
6. In your browser, return to the Data Engine UI and the Details tab. There, click on Query the result and then click Run to see some of the received messages under the Results tab of the batch job.
7. You can experiment with Event Streams by increasing or decreasing the number of lines value.









 Step 3: Investigating log data using Data Engine 

Depending on how long you ran the transfer, the number of files on Object Storage has certainly grown. You will now act as an investigator answering audit or compliance questions by combining Data Engine with your log file. The benefit of using Data Engine is that the log file is directly accessible - no additional transformations or database servers are necessary.



1. Back in the Details view edit the Data Engine.



* Click on the drop down next to Jobs and select Streaming.
* Open the Details and click on the Query the result.
* Notice the query editor, above, is populated with a query.
* Notice the FROM clause does not specify a specific parquet object in the bucket but references the job id, which means all of the objects in the job. Perfect!

cos://<REGION>/<BUCKET-NAME>/log-stream-landing/topic=webserver/jobid=<JOBID>
* Remove the INTO clause to display the output without storing the results.

INTO cos://<Region>/sql-<ID>/result/ STORED AS CSV
* Run the query.
* Observe the results when it is complete in the Result tab.
* Now lets do some investigation by modifying this basic query.



2. In the Data Engine UI, edit the SQL in the text area to look more like this, keep the FROM statement as is.

-- What are the top 10 web pages on NASA from July 1995?
-- Which mission might be significant?
SELECT REQUEST, COUNT(REQUEST)
* EXCEPT ALL
* MINUS ALL



The characteristics of a result set defined by a fullselect can be further defined by using the following clauses:



* ORDER BY: Define an overall ordering of the result set based on the criteria that are defined by the list of sortItem clauses. The default order direction is ascending if not explicitly specified by a sortItem clause. The order by clause cannot be used with cluster by, distribute by, or sort by clause. When you use partitioned output, the ORDER BY clause gets ignored. Use the sortClause instead.
* DISTRIBUTE BY: Distribute result set rows into new partitions based on the criteria that are defined by the list of expression clauses. Result set rows that have the same expression values are moved to the same partition. The distribute by clause cannot be used with order by or cluster by clause.
* SORT BY: You can define an overall ordering on a partition base as defined by the list of expression clauses. The default order direction is ascending if not explicitly specified by an expression clause. The sort by clause is used along with the distribute by clause.
* CLUSTER BY: Distribute result set rows into new partitions based on the criteria that are defined by the list of expression clauses. Moreover, each partition is sorted in ascending order based on the criteria that are defined by the set of expression clauses. Thus, this clause represents a shortcut for distribute by clause that is combined with sort by in ascending order. You cannot use the cluster by attribute with the order by, distribute by, sort by clause.
* LIMIT: Restrict the number of rows that are returned from the result set of the fullselect. The number of rows can be defined by an expression or by using the keyword ALL that causes all rows to be returned.



DISTRIBUTE BY, SORT BY, and CLUSTER BY have an effect only during your SQL query execution and do not influence the query result that is written back to Cloud Object Storage. Use these clauses only in execution of subqueries to optimize the outer query execution that works on the intermediate result sets produced by the sub queries.
Where



* SQL_URL is the SQL URL of the archive file in COS
* RESULTS_BUCKET is the SQL URL of the custom COS bucket that you plan to use to upload the query results



In the statement, the timestamp information is captured in the _ts entry of each row and is used to generate the year, day of year, and hour information. The result reflects the specified partitioning as Hive style partitioning. For example, you could see _hour=00 in the object name, which means that all log data belonging to that hour is included in that object.
2. Click Run.

You can see the query result in the Result area of the UI.

You can see the target COS URL in the Result location area of the UI. This URL points to the object storage bucket used by Data Engine to store the results.

With the lite plan, you can run only 1 query. You can run up to 5 queries simultaneously with a paid plan.
3. Copy the Result location URL. You need it to run other queries to analyze the data that is included in that archive file.



Complete the following steps to run the query to transform content from JSON into PARQUET:



1. In the SQL editor field of the Data Engine UI, enter the following SELECT statement:

SELECT * FROM cleancols(SQL_URL STORED AS JSON)
INTO RESULTS_BUCKET STORED AS PARQUET

Where



* SQL_URL is the SQL URL of the archive file in COS
* RESULTS_BUCKET is the SQL URL of the custom COS bucket that you plan to use to upload the query results
* Use [cleancols](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-reference) to avoid transformation problems into PARQUET format when the name of the columns include special characters or blanks.



For example, the following query is used to transform an archive file: