vania-janet/multiturn-rag-retrieval-data huggingface dataset:
ce8de101f7cf.cdc406pd09pasng7elgg.databases.appdomain.cloud:30929 --execute "select * from system.runtime.queries" --output-format CSV_HEADER --user ibmlhapikey output-format CSV > history.csv --password





 Importing query history 



1. To import the query history, create a schema in a catalog in which you have the write access.

create schema <non-system-catalog.schema-name> with (location=' s3a://<bucket-name>/<schema-name>')



* Example



./presto --server https://8dac613f-ba5b-4c3c-8c96- ce8de101f7cf.cdc406pd09pasng7elgg.databases.appdomain.cloud:30929 --execute "create schema iceberg_data.query_history with (location='s3a://8dac613f-ba5b-4c3c-8c96-ce8de101f7cf-customer/query_history')" --user ibmlhapikey --password
2. Create a table in same catalog.

This table must have same metadata as that of system.runtime.queries table. Use CREATE TABLE AS SELECT statement to create this table.

create table <non-system-table-name> as select * from system.runtime.queries where 1=0;



* Example



./presto --server https://8dac613f-ba5b-4c3c-8c96-ce8de101f7cf.cdc406pd09pasng7elgg.databases.appdomain.cloud:30929
--execute "create table iceberg_data.query_history.queries as select * from system.runtime.queries where 1=0"
--user ibmlhapikey --password
3.
The AQL Doc comment for a create view statement contains the following information:



* General description about the view.
* @field for every column name in the view.



Example

/
* Extracts all spans that match a phone number pattern from
* the input documents. It uses a regular expression to match
* phone number patterns.
* @field num phone number
* @field lc 3 tokens to the left of phone number/

create view PhoneNumber as
select P.num as num, LeftContextTok(P.num, 3) as lc
from
(
extract
regexes /+?([1-9]d{2})d{3}-d{4}/ and /+?[Xx].?d{4,5}/
on D.text as num
from Document D
) P;
Show more







 The output view statement 

The output view statement defines a view to be an output view. The runtime component outputs only the tuples of views that are marked as output views. The AQL compiler compiles only views that are marked as output or export, or reachable from views that are marked as output or export.



 Syntax 

output view <view-name> [as '<alias>'];





 Description 



* <view-name>

The name of the view to output, as known in the name space of the current module. The <view-name> is a simple identifier or a double-quoted identifier. The built-in view Document cannot be output.
* [as '<alias>']

Defines an <alias> name for the output view. When the optional alias is not specified, the view is output under the following names:



* In modular AQL, the view is output under the name <module-name>.<view-name> where <module-name> is the name of the module where the view is originally defined (which can be different from the module where the view is output).
* In AQL 1.4 or earlier, the view is output under the name <view-name> The output ...
Only one object contains the result set (jobid=<job_id>/<part-number>), and the other two are empty and don't contain any data.

It is important not to delete any of the files if you want to use the result set.

Each result is stored with an own job ID prefix that allows you to use the result directly in a query.

When you want to specify a result as input in your SQL query, specify the first (jobid=<job_id>) or the third one (jobid=<job_id>/<part-number>).

[Learn more about the result set created per query](https://cloud.ibm.com/docs/services/sql-query?topic=sql-query-overviewresult).

After you have the file converted to 'PARQUET` format, you can run queries to analyze its content.





 Step 3.5. Run a query to determine the number of events in the archive file 

To report on the total number of events that are included in the archive file, run the following query:

SELECT COUNT() AS NUMBER_EVENTS FROM <PARQUET_FILE> STORED AS PARQUET
INTO <RESULTS_BUCKET> STORED AS CSV

Where



* NUMBER_EVENTS is the name of the field that you want to use to report the numerical value
* PARQUET_FILE is the Result location URL that you get when you transform the archive file from JSON to PARQUET
* RESULTS_BUCKET is the SQL URL of the custom COS bucket that you plan to use to upload the query results



For example, to get the total number of events in a file, you can run the following query:

SELECT COUNT() AS logLines FROM cos://eu-de/results-at/jobid=f178778e-7707-46a9-982d-1e89261b63a5 STORED AS PARQUET
INTO cos://eu-de/results-marisa STORED AS CSV





 Step 3.6. Run a query to get a custom view of a subset of the event fields 

To see information about each event, run the following query:
<!DOCTYPE html>
<html>
<head>
<title>PHP Starter Application</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<link rel="stylesheet" href="style.css" />
</head>
<body>
<?php
if( getenv( "VCAP_SERVICES" ) )
{
 Get database details from the VCAP_SERVICES environment variable

 This can only work if you have used the Bluemix dashboard to
 create a connection from your dashDB service to your PHP App.

$details = json_decode( getenv( "VCAP_SERVICES" ), true );
$dsn = $details [ "dashDB" ][ "credentials" ];
$ssl_dsn = $details [ "dashDB" ][ "credentials" ];
 Build the connection string

$driver = "DRIVER={IBM DB2 ODBC DRIVER};";
$conn_string = $driver . $dsn; Non-SSL
$conn_string = $driver . $ssl_dsn; SSL
$conn = db2_connect( $conn_string, "", "" );
if( $conn )
{
echo "<p>Connection succeeded.</p>";
db2_close( $conn );
}
else
{
echo "<p>Connection failed.</p>";
}
}
else
{
echo "<p>No credentials.</p>";
}
?>
</body>
</html>
Show more
5. From your local working directory, push the updates to IBM Cloud by following the instructions in the "Getting Started" section of the PHP App in your IBM Cloud dashboard. Then, restart the App in IBM Cloud and view the App in a browser.
Issue: Define class error and Name or service not known 

When you restart the Presto engine, the following error is displayed: com.googlecode.aviator.exception.CompileExpressionErrorException. This error is displayed only once and for the first action you perform after the engine restart.

Workaround: If you see any of the define class error or name or service not known errors, wait for a few minutes for the server to completely restart. Then, retry the query for successful execution.





 Issue: Creating schema for target table 

You must create schema for the target table if the schema does not exist.





 Issue: Ingestion fails if CSV file contains bad record 

ibm-lh tool does not support skipping maximum bad records for CSV files if the mismatch field is greater than the table definition.





 Issue: Creating schema location with path 

Use one of the following location options when creating a schema:



* Location pointing to a bucket/subpath without a trailing /.
* Location pointing to a bucket/subpath with a trailing / – Recommended for better structuring.



Though you can use a location pointing to a bucket only with or without a trailing /, it might lead to failure. Therefore, it is recommended to use a subpath.





 Issue: Creating the database of type Memory gives an error 

Creating a database of the type Memory gives an error, pq: invalid input syntax for type integer: "".





 Issue: Presto does not support AS OF with iceberg tables 

Presto does not support AS OF <time stamp> in a SELECT query.

Workaround: Invoke CALL iceberg_data_rollback_to_snapshot to move to the desired timestamp.

If you use CALL iceberg_data_rollback_to_snapshot with a timestamp, you cannot call the stored procedure to move to a later timestamp. Use Spark SQL as an alternative.





 Issue: Only the creator has DROP access on the table in Apache Hive (API) 

Only the creator of a table can drop the table that is created in Apache Hive catalog. Other users cannot drop the table even if they have an explicit DROP access to the table. They will get the Access Denied message.
Explanations of common error and warning messages.



 Warning: Invalid training data found: The document was not returned in the top 100 search results for the given query, and will not be used for training 

This warning is caused by the document_ids in your training data not matching the document_ids in a search performed against the collection. Check your queries, and make sure that the document_id of the document you are rating is returned in the top 100 results for that query. If it is not, then you might want to check two things:



* If the document is not returned in the top 100, it might not be a good example of a high-quality result, and you might want to revisit why this document was chosen.
* If the document is not returned at all, then review why it is not returned, and see if there is any text in the document that matches portions of the query.



This warning indicates that you might have one or more bad queries. It is not an indication that training cannot happen.





 Error: Invalid training data found: Syntax error when parsing query 



* This means there is an issue with the actual query syntax. Validate that the queries return results and don’t raise a syntax error. This can only occur if you added a filter to your natural language query.







 Error: Invalid training data found: The query string provided exceeds the maximum length, please provide a shorter one 



* The maximum query string length is 2048. You’ll need to shorten the specific query. Including a filter in your query is one way to work around this.







 Error: This collection cannot be trained: your plan does not support training on this many top-level text fields. 



* This error only occurs with Lite plans. Top-level fields are fields that are not nested underneath another field. The training only occurs on top-level fields, and there are limits to how many fields can be used in the training process. The more top level fields in a collection, the more training data is required. Also, in cases where there are more than 10 top level fields, training is more likely to encounter errors.







 Error: Training data quality standards not met: You will need additional training queries with labeled examples.
LATERAL VIEW EXPLODE(master_child.col2) child_table AS child_id

The result of the example query is shown in the following table.



Table 17. Query result for example: deal with an n-m relation.

 MASTER_ID CHILD_ID 

 10 1 
 10 2 
 10 3 
 20 1 
 20 2 
 20 3 
 30 4 
 30 5 
 30 6 
 40 4 
 40 5 
 40 6 







 More topics - lateral view 

For more information about the clauses that are used in a lateral view clause, see the following topics:



* [expression](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexpression)
* [identifier](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceidentifier)
* [qualifiedName](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencequalifiedName)



Note:



* A simpleselect can contain multiple lateral view clauses that are evaluated in order of appearance.
* Within a lateral view clause, you can refer to columns defined in any (virtual) table on the left of the current lateral view clause.







 Related references - lateral view 

A lateral view clause is referenced by a [simpleselect](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencesimpleselect).





 Join types 

SELECT statements can retrieve and join column values from two or more tables into a single row. The retrieval is based on a specified condition, typically of matching column values.

The main characteristic of a join is, typically, matching column values in rows of each table that participates in the join. The result of a join associates rows from one table with rows from another table. Depending on the type of join operation, some rows might be formed that contain column values in one table that do not match column values in another table.
ON left_table.col1 = right_table.col1

The result of the example query is shown in the following table.



Table 21. Query result for example: full outer join query.

 L_COL1 L_COL2 R_COL1 R_COL2 

 1 11 null null 
 null null 6 16 
 3 13 null null 
 5 14 null null 
 4 14 4 14 
 2 12 2 12 
 0 10 0 10 







 Cross join 

A CROSS join creates a Cartesian product of the tables that are involved in the join operation, if a CROSS join that specifies a join condition behaves like an inner join.

-- cross join that specifies a join condition

SELECT
left_table.col1 AS l_col1,
left_table.col2 AS l_col2,
right_table.col1 AS r_col1,
right_table.col2 AS r_col2
FROM
VALUES (0, 10), (1, 11), (2, 12), (3,13), (4, 14), (5, 14) AS left_table
CROSS JOIN VALUES (0, 10), (2, 12), (4, 14), (6, 16) AS right_table
ON left_table.col1 = right_table.col1

The result of the example query is shown in the following table.



Table 22. Query result for example: cross join that specifies a join condition.

 L_COL1 L_COL2 R_COL1 R_COL2 

 0 10 0 10 
 2 12 2 12 
 4 14 4 14 



-- cross join that specifies no join condition
SELECT
left_table.col1 AS l_col1,
left_table.col2 AS l_col2,
right_table.col1 AS r_col1,
right_table.col2 AS r_col2
FROM
VALUES (0, 10), (1, 11), (2, 12), (3,13), (4, 14), (5, 14) AS left_table
You must use double quotation marks ("") when you are querying specific columns.



* To identify the passenger count, run:

select
sum("passenger_count")
from YELLOW_TAXI_JANUARY_2022_LOADED;

Output:

SUM
-------
3324167
(1 row)

You do not have to load whole tables into Netezza Performance Server. parquet is a columnar format so the Netezza Performance Server engine can load a subset of columns without having to transfer the entire table over the internet. This way, if you work with large tables, you can significantly reduce ingress traffic and achieve faster load times. The query engine always uses only the columns from a parquet table that are needed.
You can use the import statement to reference objects that are exported from other modules in the context of the current module.



 Syntax 

import view|dictionary|table|function <object-name>
from module <module-name> [as <alias>];





 Description 



* view|dictionary|table|function

Identifies the type of AQL object to be imported. The type of the object is mandatory.
* <object-name>

The <object-name> can be a simple identifier or a double-quoted identifier.
* <module-name>

The <module-name> must be a simple identifier.
* <alias>

This form of the import statement, also known as alias import, imports the specified AQL object under the <alias> name (not the original name) into the namespace of the current module. You can reference the imported element by using either an unqualified alias or an alias that is qualified with the current module name (the module where the AQL object was imported). You cannot use originalModule.elementName because the alias import statement imports the element under the alias name only and not under the qualified original name.







 Usage notes 



* An import statement without an alias specification imports the specified AQL object into the current module. It makes the AQL object accessible to AQL statements defined in the current module under its qualified name <original_module_name>.<object-name>.
* The import statement is only used to import AQL objects from modules other than the current module. An object that is declared in an AQL file of the module is visible to any other AQL file in that same module. An import statement puts objects in the context of the current module, and not in the context of the current file. Therefore, a view that is imported by 1.aql inside module A is made visible to 2.aql inside the same module without the need for any additional import statements.
* All import statements must follow immediately after the module declaration, and must precede all other types of statements. Only those AQL objects that are explicitly exported from any module can be imported in another module.
: Example of an SQL function usage fragment

> SELECT rtrim('LQSa', 'SSparkSQLS')
: Result value

SSpark





 second 

second(timestamp)
: Returns the second component of the string/timestamp.
: Example of an SQL function usage fragment

> SELECT second('2009-07-30 12:58:59')
: Result value

59





 sentences 

sentences(str[, lang, country])
: Splits str into an array of words.
: Example of an SQL function usage fragment

> SELECT sentences('Hi there! Good morning.')
: Result value

["Hi","there"],"Good","morning"]]





 sha 

sha(expr)
: Returns a sha1 hash value as a hex string of the expr.
: Example of an SQL function usage fragment

> SELECT sha('Spark')
: Result value

85f5955f4b27a9a4c2aab6ffe5d7189fc298b92c





 sha1 

sha1(expr)
: Returns a sha1 hash value as a hex string of the expr.
: Example of an SQL function usage fragment

> SELECT sha1('Spark')
: Result value

85f5955f4b27a9a4c2aab6ffe5d7189fc298b92c





 sha2 

sha2(expr, bitLength)
: Returns a checksum of SHA-2 family as a hex string of expr. SHA-224, SHA-256, SHA-384, and SHA-512 are supported. Bit length of 0 is equivalent to 256.
: Example of an SQL function usage fragment

> SELECT sha2('Spark', 256)
: Result value

529bc3b07127ecb7e53a4dcf1991d9152c24537d919178022b2c42657f79a26b
> SELECT parse_url('http://spark.apache.org/path?query=1', 'QUERY')
: Result value

query=1
: Example of an SQL function usage fragment

> SELECT parse_url('http://spark.apache.org/path?query=1', 'QUERY', 'query')
: Result value

1





 percent_rank 

percent_rank()
: Computes the percentage ranking of a value in a group of values.





 percentile 

percentile(col, percentage [, frequency])
: Returns the exact percentile value of numeric column col at the indicated percentage. The value of percentage must be between 0.0 and 1.0. The value of frequency must be a positive integral.

percentile(col, array(percentage1 [, percentage2]...) [, frequency])
: Returns the exact percentile value array of numeric column col at the indicated percentage. Each value of the percentage array must be between 0.0 and 1.0. The value of frequency must be a positive integral.





 percentile_approx 

percentile_approx(col, percentage [, accuracy])
: Returns the approximate percentile value of numeric column col at the indicated percentage. The value of percentage must be between 0.0 and 1.0. The accuracy parameter (default: 10000) is a positive numeric literal that controls approximation accuracy at the cost of memory. Higher value of accuracy yields better accuracy, 1.0/accuracy is the relative error of the approximation. When percentage is an array, each value of the percentage array must be between 0.0 and 1.0. In this case, returns the approximate percentile array of column col at the indicated percentage array.
: Example of an SQL function usage fragment

> SELECT percentile_approx(10.0, array(0.5, 0.4, 0.1), 100)
: Result value

[10.0,10.0,10.0]
: Example of an SQL function usage fragment

> SELECT percentile_approx(10.0, 0.5, 100)
* <expr>

Specifies an expression that is composed of scalar function calls, aggregate function calls, or a constant.
* <name>

Represents the name of the column that holds the result of the expression that is specified by <expr> as <alias>. If <alias> is not specified, the name of the column is by default the <name>. Can be a simple identifier or a double-quoted identifier.



* when<function1()> then <expr1> when <function2()> then <expr2> ... when <functionN()> then <exprn> [else ] <expr_default> as <name>



* <function1()>, <function2()>, <functionN()>

Specify scalar functions that return type Boolean.
* <expr1>, <expr2>, <exprn>, <expr_default>

Specify expressions that are composed of scalar function calls and must return the same type.



If the result of <function1()> is true, then the result of the case expression is the result of <expr1>, and none of the subsequent when clauses are evaluated. Otherwise, subsequent when clauses (if any) are evaluated in the same manner.

When none of the conditions of the when clauses are satisfied, the result of this case expression is the result of the default expression <expr_default>. This expression is specified in the optional else clause. If the [else] clause is absent, the result of this case expression is null.







 Usage notes 



* The following statement is not supported:

select * from Document;

The contents of the Document view might not be fully known in the current context or scope of the .aql file where this select statement is issued. The lack of information about the contents is because multiple require document with columns statements provided outside of the current .aql file might alter the eventual schema definition of this special Document view when it is used at the level of a module. The shorthand Document.* statement is not a valid AQL construct.
* You can explicitly select fields from the Document view.
import com.ibm.cloud.cloudant.v1.model.SearchResult;

import java.util.Arrays;

Cloudant service = Cloudant.newInstance();

PostSearchOptions searchOptions = new PostSearchOptions.Builder()
.db("examples")
.ddoc("cities-designdoc")
.index("cities")
.query("lat:\[0+TO+90\]")
.sort(Arrays.asList("<distance,lon,lat,-74.0059,40.7127,km>"))
.build();

SearchResult response =
service.postSearch(searchOptions).execute()
.getResult();

System.out.println(response);

import { CloudantV1 } from '@ibm-cloud/cloudant';

const service = CloudantV1.newInstance({});

service.postSearch({
db: 'examples',
ddoc: 'cities-designdoc',
index: 'cities',
query: 'lat:\[0+TO+90\]',
sort: ['<distance,lon,lat,-74.0059,40.7127,km>']
}).then(response => {
console.log(response.result);
});

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()

response = service.post_search(
db='examples',
ddoc='cities-designdoc',
index='cities',
query='lat:\[0+TO+90\]',
sort=['<distance,lon,lat,-74.0059,40.7127,km>']
).get_result()

print(response)

postSearchOptions := service.NewPostSearchOptions(
Assume that the PhoneNumbers view that extracts phone numbers of the pattern XXX-XXX-XXXX for United States is already defined. This select statement evaluates the regular expression for the pattern 444-888-XXXX across the input text. The view has the output columns documentText and phoneNumber. In addition, the output is limited to the first occurrence of this phone number pattern that is identified per document.

create view PhoneNumbersPattern1 as
select D.documentText, D.phoneNumber
from PhoneNumbers D
where MatchesRegex(/444-888-d{4}/,D.phoneNumber)
limit 1;

Another example of how you can use the select statement is to find approximate mappings of people and their corresponding phone numbers. Assume that the view Person is already defined, and that it has the columns person and the view PhoneNumbers. This select statement evaluates the where clause to find text spans that contain a person mention followed by a phone number within 1 to 3 words or tokens. The input to this statement is represented by a join of the Person and PhoneNumbers views in the from list.

create view PersonPhone as
select P1.documentText, P1.person, P2.phoneNumber, CombineSpans(P1.person,P2.phoneNumber) as personPhoneSpan
from Person P1, PhoneNumbers P2
where FollowsTok(P1.person,P2.phoneNumber,1,3);

The personPhoneSpan column will contain the matching spans that give the approximate person-phone mapping.

personPhoneSpan
John : 433-999-1000
Martha Mob 433-999-1001



* The select list The select list in an AQL select or extract statement consists of a comma-delimited list of output expressions.
* The from list The second part of a select or an extract statement in AQL is the from list. The from list is a comma-separated list that is the source of the tuples to be selected or extracted.