vania-janet/multiturn-rag-retrieval-data huggingface dataset:
The contents of the Document view might not be fully known in the current context or scope of the .aql file where this select statement is issued. The lack of information about the contents is because multiple require document with columns statements provided outside of the current .aql file might alter the eventual schema definition of this special Document view when it is used at the level of a module. The shorthand Document.* statement is not a valid AQL construct.
* You can explicitly select fields from the Document view. The following example shows a valid explicit selection of fields from a Document view:

select D.label as label,
D.text as text
from Document D;







 Examples 

The following examples illustrate various forms of the select list.

Example 1: Explicit value assignment using a constant

This example shows the assignment of a constant value to a view attribute within the select list. The field that is called polarity indicates whether the polarity of PS.match is positive or negative (Notice the explicit assignment of a constant value to this attribute).

create view PositiveSentimentsWithPolarity as
select
'positive' as polarity,
PS.match as sentiment
from
PositiveSentiments PS;

create view NegativeSentimentsWithPolarity as
select
'negative' as polarity,
NS.match as sentiment
from
NegativeSentiments NS;

Example 2: Explicit value assignment using function call

The following example illustrates how the result of a function call is explicitly assigned to a view attribute within the select list.

create view Citizenship as
select
P.Name as name,
MatchesDict('USCities.dict', P.birthPlace) as isUSCitizen
from
Person P;

Example 3: Select list expression from a dictionary extraction

The following example illustrates how the select list expression can pick values of type Span from a result of dictionary extraction.

create view PersonNames as
select N.match as name
from
(extract
dictionary 'firstNames.dict'
on D.text
as match
from Document D
)N;

Example 4: Examples of case expressions
You must link the Db2 and Netezza databases to the Presto engine that is used to process the data.

To associate Db2 with the Presto engine, do the following steps:



1. From the Infrastructure manager, select Db2 database. Click the overflow menu icon at the end of the row and click Associate.
2. In the Associate with engine form, select the Presto engine that you want to use to process the data.
3. Click Associate and restart engine. The Db2 database is associated with the Presto engine.



Similarly, select the Netezza database and link it to the Presto engine.





 Step 6: Combine data 

You can also navigate to the Query workspace to create SQL queries to query your data.

To run the SQL query to join two tables, do the following steps:



1. From the navigation menu, select SQL. The Query workspace page opens.
2. Select the Presto engine from the Engine drop-down.
3. Click the overflow menu and select the required query.



* For a catalog and schema, you can run the Generate Path query.
* For a table, you can run the Generate path, Generate SELECT, Generate ALTER, and Generate DROP query.
* For a column, you can run the Generate path, Generate SELECT, and Generate DROP query.



Consider the following sample query to join the details from Db2 and Netezza:

Example:

!/bin/bash
SELECT * FROM "Db2"."default"."order_detail" AS details
LEFT JOIN "Netezza"."gosales"."order_detail" AS header
ON details.order_number=header.order_number
LIMIT 10;
4. Click the Run on button to run the query.
5. Select Result set or Details tab to view the combined result. If required, you can save the query.
6. Click Saved queries to view the saved queries.
7.
As you add field mappings, a preview of the search result is displayed with information from the corresponding fields of your data collection. This preview shows you what gets included in the search result response that is returned to users.

To get help with configuring the search, see [Troubleshooting](https://cloud.ibm.com/docs/services/assistant?topic=assistant-skill-search-addskill-search-add-troubleshoot).
3. Use the Message, No results found and Connectivity issue tabs to customize different messages to share with users based on the successfulness of the search.



Search result messages

 Tab Scenario Example message 

 Message Search results are returned I found this information that might be helpful: 
 No results found No search results are found I searched my knowledge base for information that might address your query, but did not find anything useful to share. 
 Connectivity issue I was unable to complete the search for some reason I might have information that could help address your query, but am unable to search my knowledge base at the moment. 



4. Choose whether to enable Emphasize the answer.

This option is available only if your Discovery instance uses the v2 Discovery API.

When you enable this feature, the sentence that is determined by Discovery to be the exact answer to the customer's question is highlighted in the block of text that is displayed to the customer as the search result.
5. In the Adjust result quantity section, specify the number of results to return.

The top three results are returned automatically. You can choose to show fewer or more (up to 10) results in the response.

By default, customers can choose to see more results. If you don't want to give customers this choice, clear the Include link for customers to view up to 10 results checkbox.
6. In the Set result selectivity section, decide whether to be more selective with the answers that are returned. By increasing result selectivity, Search returns fewer but more accurate results. In most cases, Search is accurate enough that the default setting (off) is sufficient.
7. Click Preview to open the Preview pane for testing. Enter a test message to see the results that are returned when your configuration choices are applied to the search.
(2, 4, 5000, 200), (2, 5, 6000, 0), (2, 6, 4000, 0),
(3, 7, 2000, 500), (3, 8, 2000, 500), (3, 9, 8000, 0)
GROUP BY col1
)
SELECT deptno
FROM dtotal
WHERE totalpay = (SELECT MAX(totalpay) FROM dtotal)

The result of the example queries is shown in the following table.



Table 1. Query result for the example find

 DEPTNO 

 2 



-- list products with a price above the average price
WITH products AS (
SELECT
col1 AS productid,
col2 AS price
FROM VALUES -- productid, price
(1, 10), (2, 20), (3, 30), (4, 40),
(5, 5), (6, 10), (7, 15), (8, 40),
(9, 100), (10, 200), (11, 300), (12, 400)
), avg_product_price AS (
SELECT
AVG(price) as avg_price
FROM products
)
SELECT
productid,
price,
(SELECT * FROM avg_product_price) AS avg_price,
price-(SELECT * FROM avg_product_price) AS price_dist
FROM products WHERE price > (SELECT * FROM avg_product_price)
Show more



Table 2. Query result for the example list

 PRODUCTID PRICE AVG_PRICE PRICE_DIST 

 9 100 97.5 2.5 
 10 200 97.5 102.5 
 11 300 97.5 202.5 
 12 400 97.5 302.5 







 cosResultClause 

You can use the Cloud Object Storage result clause to apply detailed control over the location, format, and layout of the SQL query result set being stored on Cloud Object Storage.

The default is JOBPREFIX JOBID, which means that jobid= is always appended to the target prefix.
For Choose an Operator select Trim Value. Using the index range specify a starting and ending character position so only the date is retained in the displayed output.
15. For Choose an Operator select Capture in Field to set the name of the field that will capture this information.
16. For Field Name specify a name that you will use to reference this type of information.
17. At the end of the page click Proceed to Validation.
18. For Set a query and verify parsed output with different sample lines enter a search that will return multiple records. For example, message.
19. If all log lines returned match the rule you specified, click Mark as Valid for each and then click Activate at the end of the page.

Until you validate all results, the Activate option will be disabled.



The rule will take affect in approximately 15 minutes.





 Step 6: Analyze the log line 

You might want to view a log line in context.



1. [Make sure you are logged in to the IBM Cloud and have accessed the IBM Log Analysis web UI](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-tutorial-use-logdnaLA-launch-ui).
2. Click the Views icon ![Views icon](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/views.png).
3. Select a log view. You might want to use the [custom view you created](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-tutorial-use-logdnaLA-create-custom-view).
4. Identify a log line you want to explore.
5. Open the log entry by clicking the down arrow. Information about line identifiers, tags, and labels is displayed.

!
By combining it with the SQL [Cloud Functions](https://hub.docker.com/r/ibmfunctions/sqlquery/), you can automatically run SQL-based post processing for new objects.

Query hints
: Data Engine has query hints for SQL queries that have potential for faster execution by using certain features of Data Engine. These hints are flagged with a light bulb icon in the job list and the specific hint is available inside the Details pane.





 September 2019 

Support for ETL to IBM® Db2® on Cloud
: You can specify [Db2 target tables](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewunique) in your SQL queries to process data from IBM Cloud Object Storage and save the Data Engine result into Db2 on Cloud.





 August 2019 

Support for DESCRIBE table transformation function
: Support for the [DESCRIBE](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencetableTransformer) table transformation function that enables easier exploration of the schema of data by returning the schema definition instead of data as the table content. Check out the new Starter Query sample in the UI.





 July 2019 

JSON preview
: You can directly preview query results in JSON format in the SQL console. Add INTO <COS URI> STORED AS JSON to your SQL statement to produce JSON output and preview it in the web console.

Support for Parquet schema evolution
: Support for Parquet schema evolution through the [MERGE SCHEMA](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexternalTableSpec) sub clause for STORED AS PARQUET input data. Check out the new samples in the UI.

New table transformation functions
: Support for [CLEANCOLS](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencetableTransformer) table transformation function that generically cleanses all input column names from characters that are not supported by the Parquet target format. Check out the new Samples.
Identify the file that you want to query.

Notice that the file name has the ID of your Log Analysis instance and the date, in UTC format.
6. For that file, copy the Object Data Engine URL.







 Step 3.3. Get information on the COS bucket that is used to store results from queries 

Complete the following steps:



1. In the COS instance UI, select Buckets.
2. Select the bucket name that you plan to use to store the results from queries.
3. For that bucket, select SQL URL.

A window opens that shows the URL.
4. Copy the URL.







 Step 3.4. Transform an archive file 

Complete the following steps to run the query to transform content from JSON into partitioned JSON objects:



1. In the SQL editor field of the Data Engine UI, enter the following SELECT statement:

SELECT , date_format(from_unixtime(_source._ts / 1000, 'yyyy-MM-dd HH:mm:ss'), 'yyyy') AS _year,
dayofyear(from_unixtime(_source._ts / 1000, 'yyyy-MM-dd HH:mm:ss')) AS _dayofyear,
date_format(from_unixtime(_source._ts / 1000, 'yyyy-MM-dd HH:mm:ss'), 'HH') AS _hour
FROM SQL_URL STORED AS JSON
INTO RESULTS_BUCKET STORED AS JSON PARTITIONED BY (_year, _dayofyear, _hour)

Where



* SQL_URL is the SQL URL of the archive file in COS
* RESULTS_BUCKET is the SQL URL of the custom COS bucket that you plan to use to upload the query results



In the statement, the timestamp information is captured in the _ts entry of each row and is used to generate the year, day of year, and hour information. The result reflects the specified partitioning as Hive style partitioning.
Querying historical data 

You can run the queries by using the command-line or the [query editor inside the web console](https://cloud.ibm.com/docs/netezza?topic=netezza-query-editor).

The following table definition is used for the example queries.

CREATE TABLE PRODUCT (PRODUCTID INTEGER, DESC VARCHAR (100), PRICE DECIMAL) DATA_VERSION_RETENTION_TIME 30;

The following rows are inserted at different times. The commit times of the inserts (the insert timestamps or _SYS_START values) are indicated in SQL comments.

INSERT INTO PRODUCT VALUES(1001, 'Jacket', 102.00); -- 2020-10-23 16:00:00
INSERT INTO PRODUCT VALUES(1002, 'Gloves', 20.50); -- 2020-10-23 16:05:00
INSERT INTO PRODUCT VALUES(1003, 'Hat', 18.99); -- 2020-10-23 16:10:00
INSERT INTO PRODUCT VALUES(1004, 'Shoes', 125.25); -- 2020-10-23 16:15:00



 Showing data with the insert and delete timestamps 

This SELECT command shows the table data with the associated insert and delete timestamp values at that instant when the query was issued. The _SYS_START and _SYS_END timestamps are available only in time travel queries, hence the use of AS OF NOW().

SELECT , _SYS_START, _SYS_END FROM <table_name> FOR SYSTEM_TIME AS OF NOW();

Example:

SELECT , _SYS_START, _SYS_END FROM PRODUCT FOR SYSTEM_TIME AS OF NOW();
PRODUCTID | DESCRIPTION | PRICE | _SYS_START | _SYS_END
----------+-------------+---------+---------------------+-----------
select N2C.location as loc,
C.company as company
from Company C, NameToLocation N2C
where Equals(GetText(C.company), GetText(N2C.name));

output view CompanyLoc;
Show more





 Documenting the create table statement with AQL Doc 

The AQL Doc comment for a create table statement contains the following information:



* General description about the table.
* @field for every column name in the schema of this table.



/ Create a table that maps company names to locations
/ of corporate headquarters.
* @field name name of the company
* @field location location of corporate headquarters
/

create table NameToLocation
(name Text, location Text) as
values
('IBM', 'USA'),
('Enron', 'UK'),
('Initech', 'Dallas'),
('Acme Fake Company Names', 'Somewhere');







 The create external table statement 

You can use the create external table statement to specify a table with established content when a compiled module is run across all input documents. You supply the table content during load time instead of in the source AQL code, and you do not have to recompile the module.

External tables are powerful constructs that allow the AQL developer to expose customization points in a compiled module.



 Syntax 

create external table <table-name>
(<colname> <type>
[, <colname> <type>]* )
allow_empty <true|false>;





 Description 



* <table-name>

Specifies name of the external table to create. The <table-name> can be a simple identifier or a double-quoted identifier.
* <colname>

Specifies the name of the column to create.
* <type>

Specifies the AQL data type for the associated column. All columns must be of type Text , Integer, Float or Boolean.
* [, <colname> <type>]
The following example uses the information from your connection string and the Java driver [jdbc](https://jdbc.postgresql.org/documentation/head/index.html) to connect to your database.

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import java.util.logging.;

public class PGConnect {

private final static Logger LOGGER = Logger.getLogger(Logger.GLOBAL_LOGGER_NAME);

private Connection connect() {

final String url = "jdbc:postgresql://host:port/ibmclouddb";

Properties props = new Properties();
props.setProperty("user","admin");
props.setProperty("password","mypassword123");
props.setProperty("ssl","true");
props.setProperty("sslmode","verify-full");
props.setProperty("sslrootcert", "/path/to/cert");

Connection conn = null;
while (conn == null) {
try {
conn = DriverManager.getConnection(url, props);
System.out.println("Connected to PG");
} catch (SQLException e) {
System.out.printf("%sn", e);
LOGGER.info("Not connected, retying ...");
}
}

return conn;

}

public static void main(String[] args) {

PGConnect icd = new PGConnect();

try {
Connection connection = icd.connect();
Statement stmt = connection.createStatement();
Query result for example: all department names that do not start with letter C.

 EMP_DEPT 

 D01 
 D02 







 RLIKE examples 

-- all rows that contain in col2 a value ending with 'bc'
SELECT 
FROM VALUES
(0, 'Abc'),
(1, 'xyz abc'),
(2, 'abcabcabc'),
(3, 'abc xyzxyz abc'),
(4, '123 456 789') AS data
WHERE data.col2 RLIKE 'bc$'

The result of the example query is shown in the following table.



Table 44. Query result for the example all

 COL1 COL2 

 0 Abc 
 1 xyz abc 
 2 abcabcabc 
 3 abc xyzxyz abc 



-- all rows that contain in col2 a sequence of 3 'abc' string occurrences
SELECT 
FROM VALUES
(0, 'Abc'),
(1, 'xyz abc'),
(2, 'abcabcabc'),
(3, 'abc xyzxyz abc'),
(4, '123 456 789') AS data
WHERE data.col2 RLIKE '(abc){3}'

The result of the example query is shown in the following table.



Table 45. Query result for the example all

 COL1 COL2 

 2 abcabcabc 



-- all rows that contain in col2 a sequence of integer values (three digits) separated by blank or tab
SELECT 
FROM VALUES
(0, 'Abc'),
(1, 'xyz abc'),
(2, 'abcabcabc'),
(3, 'abc xyzxyz abc'),
(4, '123t456 789') AS data
WHERE data.col2 RLIKE '\d{3}[ \t]\d{3}[ \t]\d{3}'

The result of the example query is shown in the following table.



Table 46.
Filtering by intent or entity requires slightly different syntax from filtering on other fields. For more information, see [Filtering by intent or entity](https://cloud.ibm.com/docs/assistant?topic=assistant-filter-referencefilter-reference-intent-entity).

Note: The filter query syntax uses some characters that are not allowed in HTTP queries. Make sure that all special characters, including spaces and quotation marks, are URL encoded when sent as part of an HTTP query. For example, the filter response_timestamp<2020-01-01 would be specified as response_timestamp%3C2020-01-01.





 Operators 

You can use the following operators in your filter query.



 Operator Description 

 : Fuzzy match query operator. Prefix the query term with : if you want to match any value that contains the query term, or a grammatical variant of the query term. Fuzzy matching is available for user input text and entity values. 
 :: Exact match query operator. Prefix the query term with :: if you want to match only values that exactly equal the query term. 
 :! Negative fuzzy match query operator. Prefix the query term with :! if you want to match only values that do not contain the query term or a grammatical variant of the query term. 
 ::! Negative exact match query operator. Prefix the query term with ::! if you want to match only values that do not exactly match the query term. 
 <=, >=, >, < Comparison operators. Prefix the query term with these operators to match based on arithmetic comparison. 
 `` Escape operator. Use in queries that include control characters that would otherwise be parsed as operators. For example, !hello would match the text !hello. 
 "" Literal phrase. Use to encloses a query term that contains spaces or other special characters. No special characters within the quotation marks are parsed by the API. 
 Approximate match. Append this operator followed by a 1 or 2 to the end of the query term to specify the allowed number of single-character differences between the query string and a match in the response object.
sqlClient = ibmcloudsql.SQLQuery(sqlquery_config['password'], sqlquery_config['crn'], sqlquery_config['target_cos_url'] + '/accidents')

data_source = sqlquery_config['target_cos_url'] + "/Traffic_Collision_Data_from_2010_to_Present.csv"

query = """
SELECT
Time Occurred AS time,
Area Name AS area,
Victim Age AS age,
Victim Sex AS sex,
Location AS location
FROM {}
WHERE
Time Occurred >= 1700 AND Time Occurred <= 2000 AND
Victim Age >= 20 AND Victim Age <= 35
""".format(data_source)

traffic_collisions = sqlClient.run_sql(query)
traffic_collisions.head()
Show more

The [function get_connection](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/project-lib-python.html?audience=wdp) retrieves the previously configured connection properties.







 Step 5: Visualize data using folium 

In this section, you will visualize the previous result set using a folium heat map to better identify patterns or hot spots for traffic incidents.



1. Create a common table expression to convert the location column to separate latitude and longitude columns. Run the following from another Notebook cell.

query = """
WITH location AS (
SELECT
id,
cast(split(coordinates, ',')[0] as float) as latitude,
cast(split(coordinates, ',')[1] as float) as longitude
FROM (SELECT
Dr Number as id,
regexp_replace(Location, '[()]', '') as coordinates
FROM {0}
)
)
SELECT
d.Dr Number as id,
d.Date Occurred as date,
d.Time Occurred AS time,
create view Organization as
(select * from OrganizationCandidates)
minus
(select * from InvalidOrganizations);

Example 4: Schema compatibility for minus

It is important to note that spans over different target text are not of the same type. Consider the following AQL example that explains this difference by using a String literal.

create view OneString as
select 'a string' as match
from Document D;

create view TheSameString as
select 'a string' as match
from Document D;

create view Subtraction as
(select R.match from OneString R)
minus
(select R.match from TheSameString R);

Instead of the expected output of an empty tuple list, the output is a set of records that have 'a string' as a field value.

Although the contents of the OneString and TheSameString views seem identical, the actual text values have different underlying AQL objects. The type of OneString.match is 'Span over OneString.match'. The type of TheSameString.match is 'Span over TheSameString.match'. Since the field types are different, they are not compatible for comparison purposes.

To obtain the wanted output of the empty tuple list, you must compare values of the same type. In the following example, the GetString() function converts span objects to string objects to pass compatible types to the minus operation.

create view Subtraction as
(select GetString(R.match) from OneString R)
minus
(select GetString(R.match) from TheSameString R);





 Documenting the create view statement with AQL Doc 

The AQL Doc comment for a create view statement contains the following information:



* General description about the view.
* @field for every column name in the view.



Example

/
* Extracts all spans that match a phone number pattern from
* the input documents. It uses a regular expression to match
* phone number patterns.
* @field num phone number
* @field lc 3 tokens to the left of phone number/