vania-janet/multiturn-rag-retrieval-data huggingface dataset:
These functions can be used within the select list of a select statement but not within an extract statement.

The following example is the general form of an aggregate function call:

Aggregate_Function_Name(argument)

The argument can be:



* An expression that consists of a column of a view in the from clause, or a combination of scalar functions that involve columns of the views in the from clause.

In most cases, except as noted, null values for argument are ignored.
* The character in the special case of the Count() aggregate function.

In this case, all rows of the output are counted, including null values.





 Aggregate function Argument type Return type Return value 

 Avg(expression) Integer, Float Float The average of all input values or null if no rows are selected 
 Count(*) Integer The number of all input rows 
 Count(expression) Any Integer The number of all non-null input values 
 List(expression) Integer, Float, Text, Span List of scalar values of the same type as the input argument An unordered list of non-null input values: a bag, not a set, hence might contain duplicates. An empty list if only null values are selected 
 Max(expression) Integer, Float, Text, Span Same as the argument type The maximum element across all input values or null if no rows are selected 
 Min(expression) Integer, Float, Text, Span Same as the argument type The minimum element across all input values or null if no rows are selected 
 Sum(expression) Integer, Float Same as the argument type The sum of all input values or null if no rows are selected 



Limitations of the current version:

The current version of AQL supports the creation of scalar values through the aggregate function List.



 Examples 

The following example illustrates how aggregate functions can count the number of person name annotations, or compute sets of given names that are associated with each distinct surname that is identified in a document:

-- identify occurrences of given names in the document
create view FirstName as
extract dictionary 'firstNames.dict' on D.text as name
from Document D;

-- identify occurrences of surnames in the document
create view LastName as
extract dictionary 'lastNames.dict' on D.text as name
SELECT , _SYS_START, _SYS_END FROM PRODUCT FOR SYSTEM_TIME AS OF NOW();
PRODUCTID | DESCRIPTION | PRICE | _SYS_START | _SYS_END
----------+-------------+---------+---------------------+-----------
1001 Jacket 102.00 2020-10-23 16:00:00 
1002 Gloves 20.50 2020-10-23 16:05:00 
1003 Hat 18.99 2020-10-23 16:10:00 
1004 Shoes 125.25 2020-10-23 16:15:00 
(4 rows)





 Querying data for a specific time with AS OF 

SELECT , _SYS_START, _SYS_END FROM <table_name> FOR SYSTEM_TIME AS OF <TIMESTAMP EXPRESSION>;

Example:

SELECT , _SYS_START, _SYS_END FROM PRODUCT FOR SYSTEM_TIME AS OF '2020-10-23 16:30:00';
PRODUCTID | DESCRIPTION | PRICE | _SYS_START | _SYS_END
-----------+-------------+--------+---------------------+---------------------
1001 Jacket 102.00 2020-10-23 16:00:00 
1002 Gloves 20.50 2020-10-23 16:05:00 
1003 Hat 18.99 2020-10-23 16:10:00 
1004 | Shoes | 125.25 | 2020-10-23 16:15:00 | 2020-10-23 17:00:00
(4 rows)

In this example, the price for Shoes has been modified after the specified AS OF timestamp. The system returns the previous valid row.
The join types are specified in a [relation](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencerelation).





 Sampling table data 

Any table that is object stored on Cloud Object Storage, used in a from clause, can be associated with a table sample clause. The table sample clause defines how to retrieve a subset of rows from the underlying table (object stored on Cloud Object Storage). Thus, you can write queries for samples of the data, for example, for interactive data exploration and data mining.

The general syntax of a table sample clause is described by the following syntax diagram.





 sample 





 bucketSampleClause 

Three sampling types are supported:



* With TABLESAMPLE <number> PERCENT, you can sample a certain percentage of rows from the underlying table.
* With TABLESAMPLE <expression> ROWS, you can sample some rows from the underlying table.
* With TABLESAMPLE BUCKET x OUT OF y, you can bucketize the underlying data into y buckets and returns rows from bucket x. Buckets are numbered from 1 to y.







 Examples - sample 

The following examples demonstrate how to sample a subset of data from a Parquet object on Cloud Object Storage. The object that is referenced is accessible from the web UI as part of the provided sample queries.

-- retrieve 10 percent of employee data
SELECT * FROM cos://us-geo/sql/employees.parquet STORED AS PARQUET TABLESAMPLE (10 PERCENT)

-- retrieve 10 rows from the employee data object
SELECT * FROM cos://us-geo/sql/employees.parquet STORED AS PARQUET TABLESAMPLE (10 ROWS)

-- bucketize the employee data in 10 buckets and retrieve data from 2 buckets
SELECT * FROM cos://us-geo/sql/employees.parquet STORED AS PARQUET TABLESAMPLE (BUCKET 2 OUT OF 10)





 More topics - table sample clause 

For more information about the clauses that are used in a table sample clause, see the following topics:



* [expression](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexpression)
Specifies name of the external table to create. The <table-name> can be a simple identifier or a double-quoted identifier.
* <colname>

Specifies the name of the column to create.
* <type>

Specifies the AQL data type for the associated column. All columns must be of type Text , Integer, Float or Boolean.
* [, <colname> <type>]

Specifies additional columns and AQL objects to be used in the external table.
* allow_empty [true|false]

Specifies the value for the allow_empty clause.



* true

If the clause is allow_empty true, the module can be run successfully even if a URI to the external content is not supplied for this table. If the URI is not supplied, the runtime component treats it as an empty table.
* false

If the clause is allow_empty false, then you must supply a URI to the location of the file that contains the external content. The specified file must contain content. If the URI is not supplied, or if the file does not contain content, the runtime component throws an exception.









 Usage notes 



* The compiled representation of the module contains metadata about the external objects (views, dictionaries, and tables) that the module defines.
* When modules are being loaded, you must specify a list of URIs to external tables, as required by the modules that are being loaded.
* The supported format for the content of an external table is one CSV (.csv) file with header.



The following table shows the differences between the create external table statement and the create table statement:



 create external table create table 

 <br><br> * Defines a placeholder for a table whose content is supplied at initialization time.<br><br><br> <br><br> * Requires that the content of the table is available at compile time.<br> * Serialized in the compiled representation (.tam) of a module.<br><br><br> 







 Examples 

Example 1: Creating an external table that is populated at load time
Select Everything or a view.







 Step 2. Select the set of logs to display through a view by applying a search query 

To search for specific logs, you can apply a search query.



* You can do simple searches (single term string search), compound search (multiple search terms and operators), field searches if the log line can be parsed, and others.
* AND and OR operators are case-sensitive and must be capitalized.
* Use FieldName:==FieldValue to search for a specific field value.
* Use FieldName:Value to search for field values that start with that value.



You can only search logs for the number of days that is specified through the instance's service plan.

If you are searching for Windows logs, you must use double slashes (//) in the search query even though the console will show directories with single slashes.

Zoom

![Example search for Windows logs](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/windows-search.png)

Figure 1. Example search for Windows logs

Complete the following steps:



1. Enter a search query.
2. Press Enter.



When you run a query, notice that the name of the view changes to Unsaved View.



 Query for logs that are generated by a host 

To filter logs for a specific host on an IBM Log Analysis instance with platform logs, you need to enter the following query:

host:<HOSTNAME>

Where HOSTNAME is the name of a host providing platform logs. For example, to return IBM Cloud database logs specify the following query:

host:ibm-cloud-databases-prod





 Query by log criticallity 

Each log has a level field that defines the level of threat the log issue might have.

Valid values in ascending order of severity are shown in the following table.



Table 2. Levels

 Value Description 

 N/A Log entries with no level specified.
using System;
using IBM.Data.DB2;

namespace dotNetSSLTest
{
class Program
{
static void Main(string[] args)
{
DB2Command MyDB2Command = null;
// Use the dsn alias that you defined in db2dsdriver.cfg with the db2cli writecfg command in step 1.
String MyDb2ConnectionString = "database=alias;uid=userid;pwd=password;";
DB2Connection MyDb2Connection = new DB2Connection(MyDb2ConnectionString);
MyDb2Connection.Open();
MyDB2Command = MyDb2Connection.CreateCommand();
MyDB2Command.CommandText = "SELECT branch_code, city from GOSALES.BRANCH";
Console.WriteLine(MyDB2Command.CommandText);

DB2DataReader MyDb2DataReader = null;
MyDb2DataReader = MyDB2Command.ExecuteReader();
Console.WriteLine("BRANCHtCITY");
Console.WriteLine("============================");
while (MyDb2DataReader.Read())
{
for (int i = 0; i <= 1; i++)
{
try
{
if (MyDb2DataReader.IsDBNull(i))
{
Console.Write("NULL");
}
else
{
Console.Write(MyDb2DataReader.GetString(i));
}
}
catch (Exception e)
{
Console.Write(e.ToString());
}
Console.Write("t");

}
Console.WriteLine("");
}
Type: h for help with SQL commands
? for help on internal slash commands
g or terminate with semicolon to execute query
q to quit

SYSTEM.ADMIN(ADMIN)=>







 Loading data from local systems 



1. Prepare the local data file that you want to load:



1. Note the columns.
2. Note the location of the file.
3. Ensure that the data is saved in a delimited file, such as a comma-separated (CSV) file.



2. Create a table by using the [CREATE TABLE AS](https://www.ibm.com/docs/en/netezza?topic=npsscr-create-table-as-2) command.

The CREATE TABLE AS command creates a table on Netezza Performance Server and fills it with the data from your local data file.

As a part of this command, your local data file is turned into [a transient external table](https://www.ibm.com/docs/en/netezza?topic=et-transient-external-tables-2). In other words, your local data file is temporarily treated as a database table that you can query for loading to a Netezza Performance Server table. When you are finished, the transient external table is automatically deleted.

CREATE TABLE <table> AS SELECT * FROM EXTERNAL <file_path> (<col1>, <col2>, ...) USING (RemoteSource <source_type> DELIM <delimiter_type> SkipRows <number_of_rows>);

Where:



 Input Description 

 table Specifies a name for the table that you are creating. 
 file_path Specifies the location of the source data file that you are loading. This source data file is turned into a transient external table. 
 col1, col2, ... Specify column names that correspond to column names from the file that you are loading. 
 source_type Specifies that the source data file is remote. When you load data by using external tables, by default, the source data file path is assumed to be on the Netezza Performance Server host.
You can use the [Data Engine service REST API](https://cloud.ibm.com/apidocs/sql-query/sql-query-v3) to run queries and retrieve information about their status. This is especially helpful when you write code that automatically queries data.

Note: The Cloud Resource Name (CRN) is a mandatory part of a Data Engine REST endpoint call. The CRN Copy button copies your CRN to clipboard and you can paste it into your API call.





 Python applications and Notebooks 

For a Python application, you can also use the [ibmcloudsql package](https://pypi.org/project/ibmcloudsql/). Use IBM Watson Studio to run queries with Data Engine and visualize the query results with one of the various widget libraries available in [Watson Studio](https://cloud.ibm.com/catalog/services/data-science-experience).





 Cloud functions 

Data Engine is a serverless mechanism to submit SQL queries, making it a natural match for the serverless [IBM Cloud Functions](https://www.ibm.com/cloud/functions). You can use the generic [SQL Cloud function](https://hub.docker.com/r/ibmfunctions/sqlquery) to run Data Engine as an IBM Cloud function.





 Geospatial functions 

The [Geospatial Toolkit](https://www.ibm.com/support/knowledgecenter/en/SSCJDQ/com.ibm.swg.im.dashdb.analytics.doc/doc/geo_intro.html) provides a set of [geospatial functions](https://www.ibm.com/support/knowledgecenter/en/SSCJDQ/com.ibm.swg.im.dashdb.analytics.doc/doc/geo_functions.html) that you can use to efficiently process and index spatial data. These functions are integrated into the Data Engine service and ready for immediate use. The Data Engine service also provides several sample queries that illustrate how to use these functions.







 Required user roles
Use the DESCRIBE TABLE statement to verify the detected table schema:

DESCRIBE TABLE employees

If the DESCRIBE TABLE output shows partition information, you must run an ALTER TABLE ... RECOVER PARTITIONS statement to attach the partitions. For more information, see the section on [partitioned tables](https://cloud.ibm.com/docs/sql-query?topic=sql-query-getting_started_catalogpartitioned).

You can then query the table by name instead of specifying the Object Storage URI directly in the SQL statement:

SELECT * FROM employees LIMIT 10

If you want to use more specific data types than the data types inferred by automatic schema detection, you can also specify the table schema explicitly:

CREATE TABLE employees (
employeeID int,
lastName string,
firstName string,
title string,
titleOfCourtesy string,
birthDate timestamp,
hireDate timestamp,
address string,
city string,
region string,
postalCode string,
country string,
homePhone string,
extension int,
photo string,
notes string,
reportsTo string,
photoPath string
)
USING PARQUET
LOCATION cos://us-geo/sql/employees.parquet
Show more

If accessing the table in a SELECT statement does not work as expected, it is possibly caused by improper specification of the table schema in the CREATE TABLE statement. The column names and their data types in your CREATE TABLE statement must match the result of the following query:

SELECT * FROM describe (<data-location> stored as <storage-format>)

Column names are case-sensitive. Incorrect column name specification results in an empty column, that is, the column seems to contain no data. To solve such a problem, use the automatic schema detection, reorder the columns, or omit some columns.

The SHOW TABLES statement provides you with an overview of the existing tables in your instance. This statement allows an optional search filter to limit the number of results:

SHOW TABLES LIKE 'cus'

It is not possible to use a different namespace than default.
Objects of queryable data formats, as well as folders with multiple objects of a consistent queryable format (when shown in the "folders" view) are labeled as shown in Figure 4.

Zoom

![Object with SQL label](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/accessible-using-sql.jpg)

Figure 4. SQL label shows queryable objects

You can retrieve a SQL queryable URL for objects for a selected individual object (Object SQL URL) or for all objects currently displayed with an active prefix filter (Filtered SQL URL). You can use this URL inside the SQL statement as the table name.

Figure 5 shows how to access your data using Data Engine. When you click on the ellipses at the end of a row of an object that you can query, you will see a menu where you can "Access with SQL" by selecting that option.

Zoom

![Access with SQL](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/access-with-sql.jpg)

Figure 5. Access with SQL shows on queryable objects

The panel shown in Figure 6 shows how to access your data using Data Engine. The location of your object appears in the panel for reference outside of the console. The instances to which you have access appear in the dropdown list in the panel. After you specify the instance, click on "Open in SQL Query" to launch your instance already pre-populated with a sample query written in the appropriate SQL.

Zoom

!
* Struct types are not mapped and must be flattened first. See the FLATTEN[table transformation function](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencetableTransformer).
* Arrays, time series, and spatial data types are not mapped and must be converted with appropriate SQL functions.



The table name and optional schema are specified as part of the target URI. Important: If a table with the name that is indicated exists in the target database, that table is dropped before the query executes and all existing data is deleted.

Use the PARALLELISM x clause to specify that multiple parallel database connections are to be opened to write out the result. Depending on the size of your result and the network connectivity of your target database service, this clause can reduce the query processing time significantly.





 accessSecrets 

By default, either the credentials that are needed to access the target database are taken from the credentials object of a CRN_URI, or the IAM user who submits the statement is used to connect to the DB2_TABLE_URI. You can override this default by specifying either a combination of USER and PASSWORD or an APIKEY. However, the password or API key is not included in the SQL statement as plain text. Instead, you must store it as a custom key in a IBM® Key Protect for IBM Cloud® instance to which you have access. For a description how to store and manage the secrets in Key Protect, see [Setting up custom secrets in Key Protect](https://cloud.ibm.com/docs/sql-query?topic=sql-query-kpsetup).





 More topics - query 

For more information about clauses that are used in a query, see the following topics:



* [COSURI](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceCOSURI)
* [CRN_URI](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceCRN_URI)
Current activity 



  Queries 

To see a list of active queries, go to Current activity > Queries from the menu or homepage.

When you select Queries, you can do the following:



*  View queries
*  View statistics
*  View graphs
*  View graphs summaries
*  View verbose information
*  View distribution information
*  View plan file
*  View plan details



You can also export information about queries by clicking Export or refresh your view by clicking the refresh button next to Export.





  Sessions 

To see a list of active queries, go to Current activity > Sessions from the menu or homepage.

When you select Sessions, you can do the following:



*  View queries
*  Change priority
*  End transactions
*  End sessions



You can also export information about sessions by clicking Export or refresh your view by clicking the refresh button next to Export.
10 200 97.5 102.5 
 11 300 97.5 202.5 
 12 400 97.5 302.5 







 cosResultClause 

You can use the Cloud Object Storage result clause to apply detailed control over the location, format, and layout of the SQL query result set being stored on Cloud Object Storage.

The default is JOBPREFIX JOBID, which means that jobid= is always appended to the target prefix. You can optionally specify JOBID NONE, which skips the appending of jobid=. The results are then written exactly to the requested path. However, if that path contains existing objects (for example, from a previous query execution with the same target path), all existing objects get removed when the new result objects are written.

A query cannot run if JOBPREFIX NONE is specified and the target overlaps with at least one of the input URIs. The reason is that data cannot be overwritten by a query that is reading that same data. For example, SELECT * FROM cos://us-geo/mybucket/myprefix/mysubprefix INTO cos://us-geo/mybucket/myprefix JOBPREFIX NONE returns an error when you try to submit it.

In the COS result clause, you can explicitly specify the storage location and type of a query result on Cloud Object Storage. The storage location is specified by a COSURI.

Valid query result object formats are: AVRO, CSV, JSON, ORC, and PARQUET. Object type names are not case-sensitive.

Being able to explicitly specify the location and the type of the result object enables a user to take the following actions:



* Read raw data stored in one format, for example, CSV.
* Cleanse the data by using SQL constructs.
* Store the cleansed data in a format that offers benefits regarding query performance and storage consumption on Cloud Object Storage, for example, a column-based object format, such as Parquet.



Moreover, a user can explicitly define the way a query result is stored physically on Cloud Object Storage by using the result partitioned clause.
Table Name
: The table within a schema that you want to crawl.

Click the edit icon to specify more table crawl settings, including:

Primary key
: The primary key of the target database table. If the primary key is not configured in the target database table, you must specify the key in this field. The JDBC database crawler appends this primary key value to the URL of each crawled row to keep its uniqueness. When the primary key is a composite key, concatenate the key names by using a comma, for example key1,key2. If unspecified, the project defaults to the primary key fields of the table. If the primary key is configured in the target database table, this key is automatically detected.

Row filter
: Optional. Specify the SQL WHERE clause to designate which table rows to crawl. You must specify a Boolean expression that can be the condition of a WHERE clause in a SELECT statement. If there is an error in syntax or column names, the table is excluded from the crawl, and no documents are indexed.

Column with data to extract
: Name of the column with data that you want to crawl. If you don't specify the column, a column with text or with a single large object is chosen to be crawled.

MIME type of data
: Optional. The MIME type is detected if not specified.

The values that you specify in the table crawl settings dialog are not displayed with the schema and tables names, but the values are applied to the database connection.

The Column with data to extract and MIME type of data fields were added with the 4.6.5 release.
10. If you want the crawler to extract text from images in documents, expand More processing settings, and set Apply optical character recognition (OCR) to On.

When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).
11. Click Finish.



The collection is created quickly.