vania-janet/multiturn-rag-retrieval-data huggingface dataset:
SELECT statements can retrieve and join column values from two or more tables into a single row. The retrieval is based on a specified condition, typically of matching column values.

The main characteristic of a join is, typically, matching column values in rows of each table that participates in the join. The result of a join associates rows from one table with rows from another table. Depending on the type of join operation, some rows might be formed that contain column values in one table that do not match column values in another table.

A joined table specifies an intermediate result table that is the result of either an INNER join, an OUTER join, a CROSS join, or an ANTI join. The table is derived by applying one of the join operators to its operands.





 joinType 



 Inner join 

An INNER join combines each row of the left table with each row of the right table, keeping only the rows in which the join condition is true.

-- inner join query
SELECT
left_table.col1 AS l_col1,
left_table.col2 AS l_col2,
right_table.col1 AS r_col1,
right_table.col2 AS r_col2
FROM
VALUES (0, 10), (1, 11), (2, 12), (3,13), (4, 14), (5, 14) AS left_table
INNER JOIN
VALUES (0, 10), (2, 12), (4, 14), (6, 16) AS right_table
ON left_table.col1 = right_table.col1

The result of the example query is shown in the following table.



Table 18. Query result for example.

 L_COL1 L_COL2 R_COL1 R_COL2 

 0 10 0 10 
 2 12 2 12 
 4 14 4 14 







 Outer join 

An OUTER join includes the rows that are produced by the inner join, plus the missing rows, depending on the type of outer join.

A LEFT OUTER or LEFT join includes the rows from the left table that were missing from the inner join.

-- left outer join query
SELECT
left_table.col1 AS l_col1,
[group by <group by list>]
[order by <order by list>]
[limit <maximum number of output tuples for each document>];





 Description 



* <output view name>

Specifies the name of the output view that is defined by the statement. The select ... into statement is identical to the select statement, except for the additional into <output view name> clause.







 Examples 

Example 1: Defining a view

This example defines a view that is called PersonPhone and also specifies this view as an output view.

select P.name as name,
Ph.number as phoneNumber
into PersonPhone
from Person P, Phone Ph;

This example is equivalent to the following two statements:

create view PersonPhone as
select P.name as name,
Ph.number as phoneNumber
from Person P, Phone Ph;

output view PersonPhone;







 The detag statement 

The detag statement in AQL provides the function to detag or remove all the markup from HTML or XML documents before you run AQL extractors.

The detag statement can also retain the original locations of tags and any values that are stored in these tags. When a detag statement removes tags from a document, the runtime component remembers the mapping between offsets from the detagged text and the original markup source. The Remap function, a special built-in function that maps spans from the detagged text back to their equivalent spans from the original source.



 Syntax 

detag <input view name>.<text column>
as <output view name>
[detect content_type (always|never)]
[annotate
element '<element name>' as <auxiliary view name>
with attribute '<attribute name>' as <column name>]
and attribute '<attribute name>' as <column name>]
, element ...]];





 Description 



* <input view name>.<text column>



* <input view name>

Specifies the name of the input view on which to do the detag process.
./presto --server https://8dac613f-ba5b-4c3c-8c96-ce8de101f7cf.cdc406pd09pasng7elgg.databases.appdomain.cloud:30929
--execute "create table iceberg_data.query_history.queries as select * from system.runtime.queries where 1=0"
--user ibmlhapikey --password
3. To import the query history into the table that you have just created table, run the following query periodically.

INSERT INTO <non-system-table-name>
SELECT 
FROM system.runtime.queries
WHERE query_id NOT IN (SELECT query_id FROM <non-system-table-name>);



* Example



./presto --server https://8dac613f-ba5b-4c3c-8c96-ce8de101f7cf.cdc406pd09pasng7elgg.databases.appdomain.cloud:3092 –-execute "insert into iceberg_data.query_history.queries select * from system. runtime.queries where query_id not in (select query_id from iceberg_data.query_history.queries)"
--user ibmlhapikey --password
4. To retrieve query history from both tables, use following statement.

select * from <non-system-table-name> union select * from system.runtime.queries order by created;



* Example



./presto --server https://23b06b14-342b-4ed2-841d-7f02ca9ae788.cdc406pd09pasng7elgg.databases.appdomain.cloud:31530 --execute " select * from iceberg_data.query_history.queries union select * from system.runtime.queries order by created " --output-format=CSV_HEADER --user ibmlhapikey --password
* LIMIT: Restrict the number of rows that are returned from the result set of the fullselect. The number of rows can be defined by an expression or by using the keyword ALL that causes all rows to be returned.



DISTRIBUTE BY, SORT BY, and CLUSTER BY have an effect only during your SQL query execution and do not influence the query result that is written back to Cloud Object Storage. Use these clauses only in execution of subqueries to optimize the outer query execution that works on the intermediate result sets produced by the sub queries. To define the persistent target of the overall query that is written back to Cloud Object Storage, you need to use the dedicated [resultClause](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceresultClause) instead.





 Examples - values clause 

The set operator examples use values clauses to define result sets for the set operations. For more information about the values clause, see [valuesClause](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencevaluesClause).

-- set union eliminating duplicate rows
SELECT * FROM VALUES 1, 2 ,3 UNION VALUES 1, 2, 3

-- set union eliminating duplicate rows
SELECT * FROM VALUES 1, 2 ,3 UNION DISTINCT VALUES 1, 2, 3

The result of the example queries is shown in the following table.



Table 3. Query result for the example set

 COL1 

 1 
 3 
 2 



-- set union with duplicate rows
SELECT * FROM VALUES 1, 2 ,3 UNION ALL VALUES 1, 2, 3

The result of the example query is shown in the following table.



Table 4. Query result for the example set

 COL1 

 1 
 2 
 3 
 1 
 2 
 3 



-- intersecting set eliminating duplicate rows
SELECT * FROM VALUES 1, 2 ,3 INTERSECT VALUES 2, 2, 3, 3, 3

-- intersecting set eliminating duplicate rows
SELECT * FROM VALUES 1, 2 ,3 INTERSECT DISTINCT VALUES 2, 2, 3, 3, 3
* [namedWindows](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencenamedWindows)
* [relation](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencerelation)
* [resultColumn](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceresultColumn)







 Related references - simpleselect 

A simpleselect is referenced by the following clause:



* [fullselect](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencefullselect)







 Sort item clause 

Sort items are a component of a fullselect or a window specification.





 sortItem 

The semantics of the sort item components are as follows.



* expression: The expression represents a sort key. The value of the sort key is used to order the rows of the result.
* ASC: Uses the values of the sort key in ascending order. ASC is the default.
* DESC: Uses the values of the sort key in descending order.
* NULLS:



* FIRST: Specifies that NULL values appear first in the order.
* LAST: Specifies that NULL values appear last in the order.









 More topics - expression clause 

For more information about the clauses that are used in an expression clause, see the following topic:



* [expression](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexpression)







 Related references - sort item clause 

A sort item clause is referenced by the following clauses:



* [fullselect](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencefullselect)
* [windowSpec](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencewindowSpec)









 Relations
User_B connects to the catalog and runs grant select to User_C.

presto:default> grant select on useracat02.schema_test_01.tab_1 to "6ff74bf7-b71b-42f2-88d9-a98fdbaed304";
6. When the User_C connects to the catalog and runs select command on the table, the command fails with access denied message.

presto:default> select * from useracat02.schema_test_01.tab_1;
Query 20230612_073938_00132_hthnz failed: Access Denied: Cannot select from columns [name, id, salary, age] in table or view tab_1







 Issue: Creating schema without a location 

When you create a schema without a location, it is not listed in the schema list of any catalog. For example, if you create a schema without specifying the location of the bucket, the schema is created in HMS and not in the bucket. When you try to create a new schema with the same name, it fails and responds that the schema already exists.

Workaround: Specify the location of the bucket when creating a schema.





 Issue: Unique names for schema and bucket 

A schema and a bucket cannot be created with the same name. For example, if you create a schema named “sales” in one catalog, the same name cannot be used for another schema in another catalog. Similarly, if you register a bucket with the name “salesbucket”, another bucket with the same cannot be registered, even if the bucket is located in a different object store.

Workaround: Use unique names when creating schemas and buckets.





 Issue: Define class error and Name or service not known 

When you restart the Presto engine, the following error is displayed: com.googlecode.aviator.exception.CompileExpressionErrorException. This error is displayed only once and for the first action you perform after the engine restart.

Workaround: If you see any of the define class error or name or service not known errors, wait for a few minutes for the server to completely restart.
.getResult();

System.out.println(response);

from ibmcloudant.cloudant_v1 import CloudantV1, ViewQuery

service = CloudantV1.new_instance()

query1 = ViewQuery(
include_docs=True,
limit=5
)
query2 = ViewQuery(
descending=True,
skip=1
)

response = service.post_view_queries(
db='users',
ddoc='allusers',
queries=[query1, query2],
view='getVerifiedEmails'
).get_result()

print(response)

postViewQueriesOptions := service.NewPostViewQueriesOptions(
"users",
"allusers",
"getVerifiedEmails",
[]cloudantv1.ViewQuery{
{
IncludeDocs: core.BoolPtr(true),
Limit: core.Int64Ptr(5),
},
{
Descending: core.BoolPtr(true),
Skip: core.Int64Ptr(1),
},
},
)

viewQueriesResult, response, err := service.PostViewQueries(postViewQueriesOptions)
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(viewQueriesResult, "", " ")
fmt.Println(string(b))

import { CloudantV1 } from '@ibm-cloud/cloudant';

const service = CloudantV1.newInstance({});

const viewQueries: CloudantV1.ViewQuery[] = [
{
include_docs: true,
limit: 5
},
{
descending: true,
skip: 1
}
];
service.postViewQueries({
: Example of an SQL function usage fragment

> SELECT randn(null)
: Result value

1.1164209726833079





 rank 

rank()
: Computes the rank of a value in a group of values. The result is one plus the number of rows preceding or equal to the current row in the ordering of the partition. The values produce gaps in the sequence.





 regexp_extract 

regexp_extract(str, regexp[, idx])
: Extracts a group that matches regexp.
: Example of an SQL function usage fragment

> SELECT regexp_extract('100-200', '(\d+)-(\d+)', 1)
: Result value

100





 regexp_replace 

regexp_replace(str, regexp, rep)
: Replaces all substrings of str that match regexp with rep.
: Example of an SQL function usage fragment

> SELECT regexp_replace('100-200', '(\d+)', 'num')
: Result value

num-num





 repeat 

repeat(str, n)
: Returns the string that repeats the indicated string value n times.
: Example of an SQL function usage fragment

> SELECT repeat('123', 2)
: Result value

123123





 replace 

replace(str, search[, replace])
: Replaces all occurrences of search with replace.
: Arguments



* str - A string expression.
* search - A string expression. If search is not found in str, str is returned unchanged.
* replace - A string expression. If replace is not specified or is an empty string, nothing replaces the string that is removed from str.



: Example of an SQL function usage fragment

> SELECT replace('ABCabc', 'abc', 'DEF')
: Result value

ABCDEF





 reverse 

reverse(str)
: Returns the reversed indicated string.
: Example of an SQL function usage fragment
You can now access the original query field, formerly titled Enter a query or keyword, by clicking the Use the Discovery Query Language button.

The Build queries screen was redesigned, but all fields and options remain. : Following are the old and new names for the fields.

 Old field name New field or section name 

 Write and run a query Search for documents 
 Narrow your query results (Filter) Limit which documents you query 
 Group query results (Aggregation) Include analysis of your results 
 Fields to display Name did not change, but moved to the new Customize display options section. 
 Number of documents to return (Count) Number of documents to return [This field was moved to the Customize display options section.] 
 Include matching passages Include relevant passages [This field was moved to the Customize display options section.] 
 Number of query fields to skip at the beginning (Offset) Number of query results to skip at the beginning [This field was moved to the Customize display options section.] 
Show more





 5 June 2017 

Updated text display for Discovery News : Discovery News queries now display only the first 150 words of each article in the text and alchemyapi_text JSON fields. The blekko.snippet field displays only the first sentence of the snippet array.





 30 May 2017 

Update to 'passages' parameter : The passages parameter on the query API moved from beta to GA status.





 25 May 2017 

New query field highlighting for tooling : Added query field highlighting in this release. This feature adds yellow highlighting to field names in the JSON of the Results pane. All fields that are queried or filtered on are highlighted for each result even if the content of the field does not match the query. Any fields used in aggregations are also highlighted in the query results, but only the first aggregation operation is highlighted.





 10 May 2017 

New support for 'highlight' parameter : The query and notices methods now support the highlight parameter. The parameter is a boolean. When you run a query and specified highlight as true, the service returns output that includes a new highlight field in which words that match the query are wrapped in HTML * (emphasis) tags.
( <value> [, <value>]),
...
( <value> [, <value>]);





 Description 



* <table name>

Specifies the name of the table to create. The <table name> can be a simple identifier or a double-quoted identifier.
* <colname>

Specifies the name of the column to create.
* <type>

Specifies the AQL data type for the associated column. All columns must be of type Text , Integer, Float or Boolean.
* <value>

Specifies the tuples to be populated in the created table.







 Examples 

Example 1: Creating a table of company names

In this example, the create table statement adds more location metadata to company name annotations:

-- Create a dictionary of company names
create dictionary CompanyNames as
('IBM', 'BigCorp', 'Initech');

-- Find all matches of the company names dictionary.
create view Company as
extract
dictionary 'CompanyNames' on D.text as company
from Document D;

-- Create a table that maps company names to locations of
-- corporate headquarters.
create table NameToLocation
(name Text, location Text) as
values
('IBM', 'USA'),
('BigCorp', 'Apex'),
('Initech', 'Dallas'),
('Acme Fake Company Names', 'Somewhere');

-- Use the table to augment the Company view with location
-- information.
create view CompanyLoc as
select N2C.location as loc,
C.company as company
from Company C, NameToLocation N2C
where Equals(GetText(C.company), GetText(N2C.name));

output view CompanyLoc;
Show more





 Documenting the create table statement with AQL Doc 

The AQL Doc comment for a create table statement contains the following information:



* General description about the table.
(2017, 2 ,200),
(2017, 2 ,300),
(2018, 1 ,300),
(2018, 1 ,100),
(2018, 2 ,400) AS sales
GROUP BY col1, col2
GROUPING SETS ( (COL1), (COL1, COL2) )
ORDER BY year, quarter
Show more

The result of the example query is shown in the following table.



Table 10. Query result for the example rollup

 YEAR QUARTER AMOUNT 

 2017 null 650 
 2017 1 150 
 2017 2 500 
 2018 null 800 
 2018 1 400 
 2018 2 400 









 More topics - simpleselect 

For more information about the clauses that are used in a simpleselect, see the following topics:



* [booleanExpression](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencebooleanExpression)
* [expression](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexpression)
* [identifier](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceidentifier)
* [lateralView](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencelateralView)
* [namedWindows](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencenamedWindows)
* [relation](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencerelation)
* [resultColumn](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceresultColumn)







 Related references - simpleselect 

A simpleselect is referenced by the following clause:
[The new reference guide](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-reference) includes examples that can be copied and directly pasted into the web UI to be run.





 November 2018 

Support for hive-style partitioning
: Support for controlling the layout of SQL results. Including support for creating hive-style partitioning and paginated result data.

Support for Python SDK extensions
: Support for extensions in [Python SDK](https://pypi.org/project/ibmcloudsql) for result data partitioning, pagination, and exporting SQL job history to IBM Cloud Object Storage.





 October 2018 

Control your result's format
: Support for SELECT INTO to control the format the SQL result is written in.





 August 2018 

General availability
: IBM Cloud Data Engine is generally available. Its open beta phase ended.

New built-in functions
: Support for [new built-in SQL functions](https://issues.apache.org/jira/browse/SPARK-20746) released with Apache Spark 2.3.
: Set of SQL optimizer and ANSI SQL and Hive SQL compliance enhancements that are introduced with Apache Spark 2.3.





 June 2018 

ORC
: Support for ORC data (STORED AS ORC).

Geospatial functions
: Support for geospatial SQL functions for calculations, aggregations, and joins on location data.

ibmcloudsql Node.js client SDK
: Release of [ibmcloudsql Node.js client SDK](https://www.npmjs.com/package/ibmcloudsql).





 April 2018 

Introducing IBM Cloud Data Engine
: [IBM Cloud Data Engine release beta](https://www.ibm.com/cloud/blog/analyzing-data-with-ibm-cloud-sql-query?mhsrc=ibmsearch_a&mhq=ibm%20sql%20query).
Because we are using the wildcard operator, we also changed the term to lowercase.

{
"query":"test_results:p53"
}

With this syntax, occurrences of p53, tp53, P53, or TP53 are all returned.





 "" (Phrase query) 

Phrase queries only match occurrences of the whole phrase. The order of the words in the phrase must match.

For example, the following query returns only documents that contain a field named quotation with the text, There's no crying in baseball.

{
"query":"quotation:There's no crying in baseball"
}

A document with a quotation field that says Jimmy Dugan said there's no crying in baseball is also returned. However, documents that only mention baseball or crying without the entire phrase are not matched. Neither is a document with In baseball, there's no crying. Documents that contain the right text in the wrong field also are not matched. For example, a document with the text There's no crying in baseball in the text field is not returned.

Single quotation marks (') are not supported. You cannot use wildcards (*) in phrase queries.





 :: (Exact match) 

This operator specifies an exact match for the query term. Exact matches are case-sensitive.

For example, the following query searches for documents that contain entities of type Organization:

{
"query":"enriched_text.entities.type::Organization"
}

The entire content of the field that you specify must match the phrase you specify. For example, the following query finds documents in which only entity mentions of IBM Cloud are detected, not IBM Cloud Pak for Data or IBM cloud or Cloud.

{
"query":"enriched_text.entities.text::"IBM Cloud""
}





 :! (Does not include) 

This operator specifies that the results do not contain a match for the query term.

For example:

{
"query":"enriched_text.entities.text:!"cloud computing""
}





 ::! (Not an exact match)
2022-06-29 09:08:32.379442 PDT [18976]  DEBUG:  QUERY: CREATE EXTERNAL TABLE bulkETL_18976_0 ( c0 nvarchar(8),c1 nvarchar(10),c2 nvarchar(4),c3 nvarchar(6),c4 nvarchar(7),c5 nvarchar(9),c6 nvarchar(7),c7 nvarchar(9),c8 nvarchar(12),c9 nvarchar(12),c10 nvarchar(25),c11 nvarchar(13),c12 nvarchar(1) )  USING (  DATAOBJECT('/tmp/junk')  REMOTESOURCE 'jdbc'  DELIMITER ' '  ESCAPECHAR ''  CTRLCHARS 'YES'  CRINSTRING 'YES'  ENCODING 'INTERNAL'  MAXERRORS 1 QUOTEDVALUE 'YES' );
2022-06-29 09:08:32.525024 PDT [18976]  DEBUG:  QUERY: INSERT INTO "TEST_TABLE"("C1","C2","C3","C4","C5","C6","C7","C8","DATE_PROD","TIME_PROD","TIMESTMP","TIMETZ_PROD","C18") VALUES(bulkETL_18976_0.c0,bulkETL_18976_0.c1,bulkETL_18976_0.c2,bulkETL_18976_0.c3,bulkETL_18976_0.c4,bulkETL_18976_0.c5,bulkETL_18976_0.c6,bulkETL_18976_0.c7,bulkETL_18976_0.c8,bulkETL_18976_0.c9,bulkETL_18976_0.c10,bulkETL_18976_0.c11,bulkETL_18976_0.c12)
For more information about clauses that are used in a query, see the following topics:



* [COSURI](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceCOSURI)
* [CRN_URI](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceCRN_URI)
* [DB2_TABLE_URI](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceDB2_TABLE_URI)
* [fullselect](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencefullselect)
* [identifier](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceidentifier)
* [namedQuery](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencenamedQuery)
* [query](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencequery)







 Related references - query 

A query is referenced by the following clauses:



* [booleanExpression](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencebooleanExpression)
* [namedQuery](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencenamedQuery)
* [predicate](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencepredicate)
* [primaryExpression](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceprimaryExpression)







 Fullselect clause 





 fullselect 

A fullselect is the core component of a query.