vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Start Tableau Desktop. Go to Connect > To a Server > More.
5. On the next page, you see a list of supported connectors. Select Other Databases (JDBC).
6. On the raised input form enter the following information:



* URL: jdbc:ibmcloudsql:{CRN of your Data Engine service instance}?targetcosurl={COS location for results}
* Dialect: SQL 92
* User: apikey
* Password: {your IBM Cloud API Key}



7. Click Sign in.



As Tableau does not support complex data types, such as struct, if a table contains one or more columns with complex data type, perform the following actions:



* Create a flattened view on the table.
* You can set the filterType option to view. This option effectively hides all tables and reveals views only to Tableau. To set the option with the JDBC URL, use the following URL:

jdbc:ibmcloudsql:{CRN of your Data Engine service instance}?targetcosurl={COS location for results}&filterType=view







 Using the driver with Cognos Analytics 

See the [Cognos Analytics Dynamic Query](https://www.ibm.com/docs/en/cognos-analytics/11.2.0?topic=administration-support-cloud-sql-query) documentation to learn how to create a data server connection to Data Engine through the Data Engine JDBC driver.





 ODBC connectivity that uses the JDBC driver with Progress Data Direct HDP 

The JDBC driver enables BI Tools to integrate data on Cloud Object Storage into business reports. However, some BI tools exist that provide only ODBC connectors but no JDBC connectors. If you use such business intelligence tools, you need another way to connect to your data.

An easy way to set up ODBC connectivity to Data Engine is to use the [Progress DataDirect Hybrid Data Pipeline](https://www.progress.com/cloud-and-hybrid-data-integration) product.

Zoom

![ODBC connectivity to Data Engine.]
CREATE TABLE IF NOT EXISTS hive.gosales.branch ("BRANCH_CODE" int, "ADDRESS1" varchar, "ADDRESS1_MB" varchar, "ADDRESS2" varchar, "ADDRESS2_MB" varchar, "CITY" varchar, "CITY_MB" varchar, "PROV_STATE" varchar, "PROV_STATE_MB" varchar, "POSTAL_ZONE" varchar, "COUNTRY_CODE" int, "ORGANIZATION_CODE" varchar, "WAREHOUSE_BRANCH_CODE" int) WITH ( format = 'CSV', external_location = 's3a://lhbeta/gosales/branch' );







 Step 3: Generate statistics with analyze table 

If you want to use the data without creating a new copy for a different table format or more table optimizations, you can generate statistics alone with analyze table.



1. To generate statistics with analyze table, run the following command:

analyze <TABLE_NAME>;

For example:

analyze hive.gosales.branch;







 Step 4: Convert data to analytics optimized formats (optional) 

You can use the data for creating different table format and more table optimizations. It is recommended to convert the data files to analytics optimized format in Parquet or ORC to improve query performance, reduce server and storage resource consumption. Table format like Iceberg can provide more performance improvements and features like snapshots, time travel, and transactional support for insert, update, and delete.

For example:



1. To create table for a data in CSV format to Parquet format, run Create table as command:

CREATE TABLE IF NOT EXISTS
<TABLE_NAME>
WITH ( format = 'PARQUET')
AS
SELECT 
FROM <TABLE_NAME>;

CREATE TABLE IF NOT EXISTS
hive.default.branch
WITH ( format = 'PARQUET')
AS
SELECT 
FROM hive.gosales.branch;
2. To change the table format to Iceberg, run Create table as command:

CREATE TABLE IF NOT EXISTS
<TABLE_NAME>
(1, 'xyz abc'),
(2, 'abcabcabc'),
(3, 'abc xyzxyz abc'),
(4, '123t456 789') AS data
WHERE data.col2 RLIKE '\d{3}[ \t]\d{3}[ \t]\d{3}'

The result of the example query is shown in the following table.



Table 46. Query result for example: all rows that contain in col2 a sequence of integer values (3 digits) separated by blank or tab.

 COL1 COL2 

 4 123 456 789 







 IS NULL examples 

--- all employees with missing salary information
SELECT
emp.col1 AS emp_id,
emp.col2 AS salary
FROM VALUES
(0, 1000),
(2, 2000),
(3, 3000),
(4, 4000),
(5, 5000),
(6, 6000),
(7, NULL),
(8, 8000),
(9,9000) AS emp
WHERE emp.col2 IS NULL

The result of the example query is shown in the following table.



Table 47. Query result for example: all employees with missing salary information.

 EMP_ID SALARY 

 7 null 











 CAST expression 

The syntax of a cast expression is described by the following syntax diagrams.

The cast specification returns the cast operand (the first operand) cast to the type specified by the data type.

If the specified data type is not supported, you receive an error.



 castExpression 

In case an expression cannot be cast to the data type specified in the cast expression, the expression result is null.





 More topics - cast expression 

For more information about the clauses that are used by a cast expression, see the following topics:



* [dataType](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencedataType)
Time travel query syntax and timestamps 



 Query syntax 

A SELECT query with one or more temporal clauses is a time travel query. Time travel queries might appear as sub-SELECTs in the INSERT, UPDATE, DELETE, MERGE, or CREATE TABLE AS SELECT (CTAS) statements.

Also, time travel queries might appear in a view definition (CREATE VIEW, with or without OR REPLACE) or a stored procedure definition (CREATE PROCEDURE, with or without OR REPLACE). In either case, timestamp expressions in the syntax (for example, CURRENT_TIMESTAMP - INTERVAL ‘1 day’) are not evaluated at view or procedure definition time, but at the time a user or application queries the view or calls the procedure.

Any base table reference (the table name, with or without database and schema name, and with or without an alias) in a SELECT or sub-SELECT might have an optional temporal clause, which consists of the keywords FOR SYSTEM_TIME followed by one of the following values:



* AS OF <TIMESTAMP EXPRESSION>
* BEFORE <TIMESTAMP EXPRESSION>
* BETWEEN <TIMESTAMP EXPRESSION 1> AND <TIMESTAMP EXPRESSION 2>
* FROM <TIMESTAMP EXPRESSION 1> TO <TIMESTAMP EXPRESSION 2>



Each TIMESTAMP EXPRESSION must be one of the following:



* A literal timestamp value. For example, ‘2022-10-31 20:00:00’.
* A query parameter or host variable whose value is a timestamp.
* A built-in function that returns or implicitly converts to a timestamp. For example, CURRENT_DATE, CURRENT_TIMESTAMP or (equivalently) NOW(), or CURRENT_TIMESTAMP(subsecond-digits) or (equivalently) NOW(subsecond-digits).
* An expression that evaluates to a single timestamp for all rows in the table. For example, CURRENT_TIMESTAMP - INTERVAL ‘1 day’. The expression cannot refer to table columns or to a non-deterministic function (for example, RANDOM()) or be a sub-SELECT.
The [Geospatial Toolkit](https://www.ibm.com/support/knowledgecenter/en/SSCJDQ/com.ibm.swg.im.dashdb.analytics.doc/doc/geo_intro.html) provides a set of [geospatial functions](https://www.ibm.com/support/knowledgecenter/en/SSCJDQ/com.ibm.swg.im.dashdb.analytics.doc/doc/geo_functions.html) that you can use to efficiently process and index spatial data. These functions are integrated into the Data Engine service and ready for immediate use. The Data Engine service also provides several sample queries that illustrate how to use these functions.







 Required user roles 

The following table shows which user roles are required to start a particular service action or API endpoint. Use this information to decide which access rights to grant your users when you create new user IDs.



Table 5. User roles

 Description Service action API endpoint Required user roles 

 Submit an SQL query. sql-query.api.submit POST/v2/sql_jobs/ Manager or Writer 
 Get information for all submitted jobs. sql-query.api.getalljobs GET/v2/sql_jobs/ Manager, Writer, or Reader 
 Get information for a specific submitted job. sql-query.api.getjobinfo GET/v2/sql_jobs/{job_id} Manager, Writer, or Reader 
 Submit a catalog or index management statement. sql-query.api.managecatalog POST/v2/sql_jobs/ Manager 







 Behavior of scanned data 

Data Engine reads as little data as possible based on your query. The amount of data that is scanned depends on the amount of data that Data Engine must read to run your query, and not on the actual size of your data. Several factors play a role when it comes to how much data needs to be accessed to run a query. First, data layout is important. Columnar formats, such as Parquet, lead to less data to be scanned, as Data Engine can selectively read ranges and single columns. Furthermore, the actual object layout determines how many objects need to be scanned.
* Automatic pause and resume is enabled in the web console.







 Components 



* Netezza Performance Server 11.2.2.5
* Web console 4.0.11







 Known issues 

If a common table expression or derived table query contains column names or column aliases, which begin with an underscore, Netezza Performance Server deletes these columns in the query result set.
If there are no columns to display, Netezza Performance Server returns the following error.

ERROR: No columns are selected due to column alias begin with underscore

Example:

create table t1 ( c1 int , c2 int);
CREATE TABLE
with tab1 as ( select c1 as _c1 , c2 as _c2 from t1 ) select tab1.* from tab1; ERROR: No columns are selected due to column alias begin with underscore
select tab1.* from ( select c1 as _c1 , c2 as _c2 from t1 ) as tab1; ERROR: No columns are selected due to column alias begin with underscore







 July 2022 

As of July 28, 2022, you can access data from data lakes and move data between applications with Kafka.



 New features 



* Use the technology preview of the Netezza Performance Server external tables to access and query parquet files that are stored outside of your database in data lakes (on AWS S3). For more information, see [Querying data from data lakes](https://cloud.ibm.com/docs/netezza?topic=netezza-overview_singularity).
* Use Netezza Performance Server as a data source or data sink. For more information, see [Using Netezza Performance Server as a data source](https://cloud.ibm.com/docs/netezza?topic=netezza-netezzakafkadatasourcekafka) and [Using Netezza Performance Server as a data sink](https://cloud.ibm.com/docs/netezza?topic=netezza-netezzakafkadatasinkkafka).







 Known issues 

datasource is a reserved keyword now.
* Ingesting documents into a collection that uses a custom Advanced Rules model]] built in Watson Knowledge Studio may fail if multiple extractors in the model internally use the same names for one or more output views. * If you delete a large number of documents, then immediately ingest a large number of documents, it may take longer for all the documents to become available. * The Classifier]] ! ! ! ] ] !!! ! ! ! ! enrichment doesn't work when FIPS (Federal Information Processing Standards) is enabled.<-- </ul> -->Also see the issues identified in all previous releases.<-- <section "id="section-2sept2020ki-fixed" "> --> 2.1.4 issues that were fixed in subsequent releases <-- <ul> --> * [Fixed in version 2.2 In the deployed Content Mining application, if you include the tilde () symbol in a search query to enable fuzzy matching or include an asterisk (*) symbol to represent a wildcard, the search customizations function properly, but the matching string is not highlighted in the query result. * [Fixed in version 2.2 A conversion error may occur when the Include in index field on the Manage fields tab in the tooling is changed. The document will not be indexed if this error occurs. To work around the issue:
<-- <ol> -->

1. oc edit sts core-discovery-converter
2. Edit between containers and - name: INGESTION_POD_NAME as follows:

containers:
- command:
- bash
- -c
- 
FILE=/opt/ibm/wex/zing/bin/converter.sh &&
sed -i "/choreo_2.11-9.1.1.jar/d" $FILE &&
sed -i "/disco-doc-conversion-commons_2.11-1.0.4.jar/d" $FILE &&
sed -i "/jackson-module-scala_2.11-2.10.4.jar/d" $FILE &&
-- all employees that work in a department that starts with letter C
SELECT
emp.col1 AS emp_id,
emp.col2 AS emp_dept
FROM VALUES
(0, 'D01'),
(2, 'C01'),
(3, 'C02'),
(4, 'D01'),
(5, 'D02'),
(6, 'C01'),
(7, 'D01'),
(8, 'C03'),
(9, 'D01') AS emp
WHERE emp.col2 LIKE 'C%'

The result of the example query is shown in the following table.



Table 42. Query result for the example all

 EMP_ID EMP_DEPT 

 2 C01 
 3 C02 
 6 C01 
 8 C03 



-- all department names that do not start with letter C
SELECT
DISTINCT emp.col2 AS emp_dept
FROM VALUES
(0, 'D01'),
(2, 'C01'),
(3, 'C02'),
(4, 'D01'),
(5, 'D02'),
(6, 'C01'),
(7, 'D01'),
(8, 'C03'),
(9,'D01') AS emp
WHERE emp.col2 NOT LIKE 'C%'

The result of the example query is shown in the following table.



Table 43. Query result for example: all department names that do not start with letter C.

 EMP_DEPT 

 D01 
 D02 







 RLIKE examples 

-- all rows that contain in col2 a value ending with 'bc'
SELECT 
FROM VALUES
(0, 'Abc'),
(1, 'xyz abc'),
(2, 'abcabcabc'),
(3, 'abc xyzxyz abc'),
[Pencil icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker//images/pencil.png) and enter a name for your parsing rule.

If you name a template before you click Build Parsing Template, the name is not changed.
12. Choose the Extract values by delimiter extractor.
13. For Delimiter specify a , (comma). A preview of the output delimited by a comma is presented for your review.
14. Choose one of the entries. For example, if your log contains a field called message you might select that line.
15. For Choose an Operator select Extract Values by Delimiter.
16. For Delimiter specify a delimiter that makes sense for your selected entry. For example, a reasonable delimiter might be a : (colon). A preview of the delimited output is presented for your review.
17. Select an entry containing a date value.
18. For Choose an Operator select Trim Value. Using the index range specify a starting and ending character position so only the date is retained in the displayed output.
19. For Choose an Operator select Capture in Field to set the name of the field that will capture this information.
20. For Field Name specify a name that you will use to reference this type of information.
21. At the end of the page click Proceed to Validation.
22. For Set a query and verify parsed output with different sample lines enter a search that will return multiple records. For example, message.
23. If all log lines returned match the rule you specified, click Mark as Valid for each and then click Activate at the end of the page.

Until you validate all results, the Activate option will be disabled.



The rule will take affect for views in approximately 15 minutes. Dashboards and screen updates can take a bit longer.
: Example of an SQL function usage fragment

> SELECT posexplode_outer(array(10,20))
0 10
1 20





 position 

position(substr, str[, pos])
: Returns the position of the first occurrence of substr in str after position pos. The indicated pos and return value are 1-based.
: Example of an SQL function usage fragment

> SELECT position('bar', 'foobarbar')
: Result value

4
: Example of an SQL function usage fragment

> SELECT position('bar', 'foobarbar', 5)
: Result value

7
: Example of an SQL function usage fragment

> SELECT POSITION('bar' IN 'foobarbar')
: Result value

4





 positive 

positive(expr)
: Returns the value of expr.





 pow 

pow(expr1, expr2)
: Raises expr1 to the power of expr2.
: Example of an SQL function usage fragment

> SELECT pow(2, 3)
: Result value

8.0





 power 

power(expr1, expr2)
: Raises expr1 to the power of expr2.
: Example of an SQL function usage fragment

> SELECT power(2, 3)
: Result value

8.0





 printf 

printf(strfmt, obj, ...)
: Returns a formatted string from printf-style format strings.
: Example of an SQL function usage fragment

> SELECT printf("Hello World %d %s", 100, "days")
: Result value

Hello World 100 days





 quarter 

quarter(date)
: Returns the quarter of the year for date, in the range 1 - 4.
: Example of an SQL function usage fragment

> SELECT quarter('2016-08-31')
: Result value

3





 radians 

radians(expr)
: Converts degrees to radians.
: Example of an SQL function usage fragment
In all other project types, the data in one of the tables is excluded from the index. For example, if you have two tables that have columns that are called DATA and the DATA column in one table is populated with dates and the column in the other table is populated with strings, the data in one of the tables is excluded from the index.
* Get the user credentials for a user who has permission to access the tables that you want to crawl.
* Before you can connect to a database, you must get the JDBC driver library for the database. When you set up the database data source, you are asked to specify the JDBC driver class path.
* Before you can connect to the Data Virtualization service by using JDBC, you must install IBM Data Server driver packages. For more information, see [Connecting applications to the Data Virtualization service](https://www.ibm.com/docs/SSQNUZ_4.5.x/svc-dv/connecting_apps_dv_service.html).
* If you want to connect to an instance of Data Virtualization that is hosted in a different cluster from your Discovery service, you must forward traffic that is routed for Data Virtualization from an external infrastructure node to the master nodes of your cluster. For more information, see [Updating HAProxy configuration file](https://www.ibm.com/docs/SSQNUZ_4.5.x/svc-dv/dv-netreqs.html).





1. Download the JAR files for the JDBC driver library from the database server or vendor's website.

The following files are associated with each database:



* Db2 and Data Virtualization: db2jcc4.jar
* Oracle: ojdbc8.jar
* SQL Server: mssql-jdbc-7.2.2.jre8.jar
* PostgreSQL: postgresql-42.2.6.jar



2. Compress the JAR files into a single compressed file.

If you have a JDBC driver that has only one JAR file, skip this step.
3. Make a note of where the driver is stored.
1004 | Shoes | 125.25 | 2020-10-23 16:15:00 | 2020-10-23 17:00:00
(4 rows)

In this example, the price for Shoes has been modified after or at the BEFORE timestamp. The system returns the previous valid row.

See also [the BEFORE subclause](https://cloud.ibm.com/docs/netezza?topic=netezza-runningqueries_tt).





 Querying data for all rows over a time period 



 With the FROM...TO subclause 

SELECT , _SYS_START, _SYS_END FROM <table_name> FOR SYSTEM_TIME FROM <TIMESTAMP EXPRESSION 1> TO “<TIMESTAMP EXPRESSION 2>” WHERE <condition>;

If you want to query historical data as far back as possible, you can use the RETENTION_START_TIMESTAMP keyword in your time travel queries. If you do this, you can avoid having to try to compute the right timestamp on your own. By extension, you reduce the risk of running into an error if the value turns out to be too old (older than the retention start timestamp).

Example:

SELECT , _SYS_START, _SYS_END FROM PRODUCT FOR SYSTEM_TIME FROM RETENTION_START_TIMESTAMP TO '2020-10-23 17:10:00' WHERE PRODUCTID = 1004;
PRODUCTID | DESCRIPTION | PRICE | _SYS_START | _SYS_END
-----------+-------------+--------+---------------------+---------------------
1004 | Shoes | 125.25 | 2020-10-23 16:15:00 | 2020-10-23 17:00:00
1004 Shoes 100.00 2020-10-23 17:00:00 
(2 rows)
Building queries with the Discovery Query Language 

In this tutorial, we will learn how to write a few different types of queries in the Discovery Query Language.

For more information about writing queries, see:



* [Query concepts](https://cloud.ibm.com/docs/discovery?topic=discovery-query-concepts)
* [Query reference](https://cloud.ibm.com/docs/discovery?topic=discovery-query-reference) (includes the list of parameters, operators, and aggregations available in the Discovery Query Language)



These example queries are built using the Discovery tooling. If you'd like to use the API instead, add the query parameters to your API call. For more information and examples, see the Queries section of the [API reference](https://cloud.ibm.com/apidocs/discoveryquery-your-collection).

You can also write natural language queries (such as "IBM Watson partnerships") using the Discovery tooling. This tutorial primarily focuses on how to write queries with Discovery Query Language because your requirements might necessitate a structured query, and filters and aggregations must be written in the Discovery Query Language.



 Before you begin 

Go to the Manage data screen and create a new collection named IBM Press Releases, and add these four documents to it: [test-doc1.html ![External link icon](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/discovery/test-doc1.html), [test-doc2.html !
(1 , 2),
(2, 2),
(null, 2),
(1, null),
(2, null),
(null, null)
) WHERE a IS DISTINCT FROM b

The result of the example query is shown in the following table.



Table 36. Query result for example: select all rows with distinct values in column A and B.

 A B 

 1 2 
 null 2 
 1 null 
 2 null 



-- all rows that have no distinct values in column A and B
SELECT * FROM (
SELECT
col1 AS a,
col2 AS b
FROM VALUES
(1, 2),
(2, 2),
(null, 2),
(1, null),
(2, null),
(null, null)
) WHERE a IS NOT DISTINCT FROM b

The result of the example query is shown in the following table.



Table 37. Query result for example: all rows that have no distinct values in column A and B.

 A B 

 2 2 
 null null 







 BETWEEN ... AND ... examples 

-- all employees with a salary between 4000 and 8000
SELECT
emp.col1 AS emp_id,
emp.col2 AS salary
FROM VALUES
(0, 1000),
(2, 2000),
(3, 3000),
(4, 4000),
(5, 5000),
(6, 6000),
(7, null),
(8, 8000),
(9,9000) AS emp
WHERE emp.col2 BETWEEN 4000 AND 8000

The result of the example query is shown in the following table.



Table 38. Query result for example: all employees with a salary in the range 4000 - 8000.

 EMP_ID SALARY 

 4 4000 
 5 5000 
 6 6000 
 8 8000 



-- all employees with a salary not between 4000 and 8000
SELECT
emp.col1 AS emp_id,
The IBM Cloudant search query syntax is based on the [Lucene syntax](https://lucene.apache.org/core/4_3_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.htmlOverview). Search queries take the form of name:value unless the name is omitted, in which case they use the default field, as demonstrated in the following examples:

See the following example search query expressions:

// Birds
class:bird

// Animals that begin with the letter "l"
l

// Carnivorous birds
class:bird AND diet:carnivore

// Herbivores that start with letter "l"
l* AND diet:herbivore

// Medium-sized herbivores
min_length:[1 TO 3] AND diet:herbivore

// Herbivores that are 2m long or less
diet:herbivore AND min_length:[-Infinity TO 2]

// Mammals that are at least 1.5m long
class:mammal AND min_length:[1.5 TO Infinity]

// Find "Meles meles"
latin_name:"Meles meles"

// Mammals who are herbivore or carnivore
diet:(herbivore OR omnivore) AND class:mammal

// Return all results
:
Show more

Queries over multiple fields can be logically combined, and groups and fields can be further grouped. The available logical operators are case-sensitive and are AND, +, OR, NOT, and -. Range queries can run over strings or numbers.

If you want a fuzzy search, you can run a query with to find terms like the search term. For instance, look finds the terms book and took.

If the higher bounds of a range query are both strings that contain only numeric digits, the bounds are treated as numbers not as strings.