vania-janet/multiturn-rag-retrieval-data huggingface dataset:
New support for 'sort' parameter in the query API : The query API (GET /v1/environments/{environment_id}/collections/{collection_id}/query) now supports the sort parameter, which enables you to specify a comma-separated list of fields in the document to sort on. See the [Query your collection](https://cloud.ibm.com/apidocs/discoveryquery-your-collection) method in the API reference for information.

Improved handling by 'timeslice' parameter : The timeslice parameter for query aggregations now correctly handles dates in UNIX epoch format. See [Query reference](https://cloud.ibm.com/docs/discovery?topic=discovery-query-referenceaggregations) for information about aggregations and the timeslice parameter.

Update to JavaSDK : The Discovery Java SDK has been updated. See the [API Reference](https://cloud.ibm.com/apidocs/discovery?language=java) for details.

Fixed known issues with wildcard limitations in queries : Only one wildcard worked in any given query. For example, query-month:ctober worked, but query-month:ctobe generated a parsing error. This is resolved. : Wildcards did not work with queries that contained capital letters. For example, given the key/field pair {"borrower": "GOVERNMENT OF INDIA"}, query-borrower:ndia returned results but query-borrower:NDIA did not. This is resolved. : Wildcards are not necessary within phrases in queries. For example, given the key/field pair {"borrower": "GOVERNMENT OF TIMOR"}, query-borrower:"GOVERNMENT OF TIMOR" returns results, but query-borrower:"GOVERNMENT OF TIOR" does not. Using a wildcard is not applicable within phrases because all of the characters within the quotation marks (") of a phrase are escaped.





 24 March 2017 

New feature for My data insights : Added filtering to the "My data insights" screen in the Discovery tooling.





 15 March 2017 

Known issues : All fields that are ingested from HTML, PDF, and Word documents are typed as string.
Access with SQL shows on queryable objects

The panel shown in Figure 6 shows how to access your data using Data Engine. The location of your object appears in the panel for reference outside of the console. The instances to which you have access appear in the dropdown list in the panel. After you specify the instance, click on "Open in SQL Query" to launch your instance already pre-populated with a sample query written in the appropriate SQL.

Zoom

![Open with SQL](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/open-with-sql.jpg)

Figure 6. Access with SQL launch panel

Access is based on permissions, and you may wish to study more about [authentication and access](https://cloud.ibm.com/docs/sql-query?topic=sql-query-authenticationaccessauthentication).





 Getting Results 

Figure 7 shows a sample SQL query you can modify. By pressing the "Run" button, the list below the query will populate with a new entry that links to your results. The results will be stored in the location shown beneath the query.

Zoom

![SQL Query Window](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/select-with-sql.jpg)

Figure 7. Access with SQL query window

The entry representing the job of the SELECT statement run previously is shown in Figure 8. There are two tabs, "Results" and "Details," at the top of the list that allow you to switch between seeing the results and more detailed information.

Zoom

!
const info = JSON.parse(data);
console.log(info);
params.info = info;
resolve(params);
});

}).on("error", (err) => {
console.log(err);
reject({
statusCode: 500,
headers: {'Content-type': 'application/json'},
body: {message: 'Problem calling pagerduty', error: err}
});

});

req.write(JSON.stringify(pdparams));
req.end();

});

}
Show more



* [Click here](https://cloud.ibm.com/openwhisk/create/action) to create an action in IBM Cloud Functions.
* Give your action a name and select your choice of language. For the example above, Node.js 10 was selected.
* Click 'Create' and insert the example code, ensuring the appropriate fields are changed.
* Under 'Endpoints', tick the box next to 'Enable as Web Action'.
* Use the given url ending in .json as your endpoint url for your DR notification, ensuring the query parameter is inserted on the end.
Queries 



 Recent queries 

On the page, you can see the most recent 2000 queries.

You do not need to have the administrator privileges to view recent queries. You must have the List permission for database objects used in the queries to see the history records. The Recent Queries view is only showing SQL statements that involve the generation of a query plan, for example, INSERT, UPDATE, DELETE, or SELECT statements.





 Stored queries 



 Creating stored queries 



1. Go to Queries > Stored queries.
2. Type a name for your query.
3. Select the database in which you want to run the query.
4. Select the schema where you want to run the query.

If you do not pick a schema, the default database schema is selected.
5. Type the SQL query that you want to run.

If your query is a select statement, a Set Limit option appears to allow you to specify how many rows of data you want to retrieve. The default is No limit.
6. When you add the necessary information, you can do one of the following:



* Click Run to run the query.

The results of the query are displayed in the panel.
* Click the floppy disk icon that is next to the Query field name to save the query as a template.

The saved query is added to Queries > Recent Queries.
* Click Clear to clear the query.









 Finding stored queries 



1. Go to Queries > Recent queries.
2. Type the name of the query you are looking for in the search bar.







 Deleting stored queries 



1. Go to Queries > Stored queries.
2. Select a query.
3. From the overflow menu, click Remove.
4. Confirm your choice by clicking Remove again.









 Query history 

To access the page, go to Queries > Query history or select Query history from the home page.

When you are on the Query history page, you can do the following:



* View your data in a table or card view.
Complete the following steps:



1. In the COS instance UI, select Buckets.
2. Select the bucket name that you plan to use to store the results from queries.
3. For that bucket, select SQL URL.

A window opens that shows the URL.
4. Copy the URL.







 Step 3.4. Transform an archive file to PARQUET format 

When you query an archive file, the format of the data is JSON. You must transform the format to PARQUET to query successfully the data.

Parquet is an open source file format that stores nested data structures into a flat columnar format, and preserves the schema of the original data.

The Data Engine UI is an editor that lets you immediately start composing SQL queries. Since SQL Query uses Spark SQL, you can use Spark SQL functions and ANSI SQL to compose both simple and complex queries that involve large amounts of data.

Complete the following steps to run the query to transform content from JSON into PARQUET:



1. In the SQL editor field of the Data Engine UI, enter the following SELECT statement:

SELECT * FROM cleancols(SQL_URL STORED AS JSON)
INTO RESULTS_BUCKET STORED AS PARQUET

Where



* SQL_URL is the SQL URL of the archive file in COS
* RESULTS_BUCKET is the SQL URL of the custom COS bucket that you plan to use to upload the query results
* Use [cleancols](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-reference) to avoid transformation problems into PARQUET format when the name of the columns include special characters or blanks.



For example, the following query is used to transform an archive file:

SELECT * FROM cleancols(cos://ams03/at-eu-de/999999d8f1f.2019-06-03.62.json.gz STORED AS JSON)
INTO cos://eu-de/results-at STORED AS PARQUET
2. Click Run.
* [fullselect](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencefullselect)
* [namedWindows](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencenamedWindows)
* [simpleselect](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencesimpleselect)
* [sortItem](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencesortItem)
* [valuesClause](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencevaluesClause)







 Related references - fullselect 

A fullselect is referenced by the following clauses:



* [fullselect](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencefullselect)
* [query](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencequery)
* [relationPrimary](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencerelationPrimary)







 Simpleselect Clause 

A simpleselect is a component of a fullselect. Its syntax is defined by the following syntax diagram.





 simpleselect 

With a simpleselect, you can specify the following characteristics of a result set:



* The list of result columns from relations or lateral views that are part of the final result set. The result column list can be further redefined by using the following modifier keywords:



* DISTINCT: Eliminates all but one of each set of duplicate rows of the final result set.
* ALL: Retains all rows of the final result set, and does not eliminate redundant duplicates. It is the default.
Run a query to find all ERROR log rows within a time range 

For example, to find all ERROR log rows in the timeframe between 7 am and 8 am on July 17th, run the following command:

SELECT _source.data
FROM RESULTS_BUCKET STORED AS JSON
where _dayofyear=198 and _hour=07 and _source.level = "ERROR"
LIMIT 50
INTO QUERY_RESULTS_BUCKET STORED AS JSON

Where



* RESULTS_BUCKET is the SQL URL of the custom COS bucket that you use to upload the query results after transforming the data into partitioned JSON objects
* QUERY_RESULTS_BUCKET is the SQL URL of the custom COS bucket that you use to upload the query results









 EU-managed accounts 

To manage logs that are generated in your EU-supported account by using the IBM Log Analysis service, consider the following information:

Data Engine provides a serverless, no-ETL solution to easily query data stored in COS. [Learn more](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overview).

You can use this service to analyze data from logging archived files in COS.

Once you have the Data Engine service running on IBM Cloud, you can immediately start querying your data using the SQL Query user interface, programmatically by using either the REST API or the Python ibmcloudsql library, or write a serverless function by using Cloud Functions.

When you query logs, consider the following information:



* You must provision an instance of the Data Engine service in Frankfurt.
* You must restrict user access to work with that instance. Users need the platform viewer role to launch the UI, and the service writer role to run queries.
* When you open the UI, the Data Engine service automatically generates a unique COS bucket that will store all of the results as CSV files from your SQL queries. To make sure that you are using an EU-Supported bucket, create one. You can specify your custom bucket to store results in.
For example, map type is not orderable, so it is not supported. For complex types such array and struct, the data types of fields must be orderable.
: Example of an SQL function usage fragment

> SELECT 2 <= 2
: Result value

true
: Example of an SQL function usage fragment

> SELECT 1.0 <= '1'
: Result value

true
: Example of an SQL function usage fragment

> SELECT to_date('2009-07-30 04:17:52') <= to_date('2009-07-30 04:17:52')
: Result value

true
: Example of an SQL function usage fragment

> SELECT to_date('2009-07-30 04:17:52') <= to_date('2009-08-01 04:17:52')
: Result value

true
: Example of an SQL function usage fragment

> SELECT 1 <= NULL
: Result value

NULL





 <=> 

expr1 <=> expr2
: Returns same result as the EQUAL(=) operator for nonnull operands, but returns true if both are null, false if one of the them is null.
: Arguments

expr1, expr2 - The two expressions must be same type or can be cast to a common type, and must be a type that can be used in equality comparison. Map type is not supported. For complex types such array and struct, the data types of fields must be orderable.
: Example of an SQL function usage fragment

> SELECT 2 <=>; 2
: Result value

true
: Example of an SQL function usage fragment

> SELECT 1 <=> '1'
: Result value

true
: Example of an SQL function usage fragment

> SELECT true <=> NULL
: Result value

false
: Example of an SQL function usage fragment

> SELECT NULL <=> NULL
: Result value

true





 = 

expr1 = expr2
: Returns true if expr1 equals expr2, or false otherwise.
: Arguments
host:ibm-cloud-databases-prod AND label.database:postgresql







 Step 3. Create a custom view 

After you apply the search query to the Everything view or to an existing custom view, complete the following steps to save the outcome as a custom view:



1. In the web UI, click Unsaved View.
2. Select Save as new view. The Create new view page opens.
3. Enter a name for the view in the Name field.
4. Optionally, add a category. Enter a name and then click Add this as new view category.
5. Optionally, attach an alert. A new section is displayed for you to configure the alert.
6. Click Save View







 Step 4. Customize how log lines are displayed through a view 

There are different options to customize how you see data in a view:



* You can modify the properties of a view.
* You can rename a view, add or modify its description, and apply a specific line format.
* You can change the log format in the USER PREFERENCES section.
* You can apply a line template from the Tools section. Notice that this overrides any other line configuration. If you select Persist these settings, all views in the UI will show data per the line format that is specified in this section.
* You can apply color to terms or strings by setting Highlight Terms in the Tools section.





 Change the line format through the view properties page 

Complete the following steps to modify the format of an event line in a single view:



1. In your view, select Edit View Properties. The Edit View Properties page opens.
2. Enter a custom line format in the Custom %LINE Template section. The default is set to {{line}}.

For more information about the line template guidelines, see [Guidelines](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-viewsviews_line).
3. Click Save properties.







 Change the line format through the user preferences section
Click on View Table to browse the data. You may scroll down or click on column names to change the sort order.







 Step 4: Verify Loaded Data Using SQL 

The data has been loaded into the relational database. There were no errors, but you should run some quick tests anyway. Use the built-in SQL editor to type in and execute some SQL statements.



1. In the left navigation click on Run SQL to get back to the SQL editor. Click on the + symbol (Add new script) and Create new to create a new editor tab.

Instead of the built-in SQL editor you can use cloud-based and traditional SQL tools on your desktop or server machine with Db2 Warehouse on Cloud. The connection information can be found in the Administration menu in the left navigation.
2. In the editor type or copy in the following query:

select count() from cities;

Select the text of the query, then, in dropdown next to Run All, choose Run selected. In the results section the same number of rows as reported by the load process should be shown.
3. In the "SQL Editor" enter the following statement on a new line:

select countrycode, count(name) from cities
group by countrycode
order by 2 desc;

Mark the text of the above statement and click the Run selected button. Only this statement is executed, returning some by country statistics in the results section.
4. Finally, run the following statement similarly to retrieve details about San Francisco in California:

select * from cities
where name='San Francisco'
and countrycode='US';







 Step 5: Deploy the application code 

Change back to the terminal. Now you are going to deploy the application code, using a pre-built container image. You can modify the application code and build the container image on your own. See the [instructions in the GitHub repository](https://github.com/IBM-Cloud/cloud-sql-database) for details.



1. If you are not logged in, use ibmcloud login or ibmcloud login --sso to log in interactively.
> SELECT '%SystemDrive%UsersJohn' rlike '%SystemDrive%\Users.'

: Result value

true

Note:

Use LIKE to match with simple string pattern.





 rollup 





 round 

round(expr, d)
: Returns expr rounded to d decimal places by using HALF_UP rounding mode.
: Example of an SQL function usage fragment

> SELECT round(2.5, 0)
: Result value

3.0





 row_number 

row_number()
: Assigns a unique, sequential number to each row, starting with one, according to the ordering of rows within the window partition.





 rpad 

rpad(str, len, pad)
: Returns str, right-padded with pad to a length of len. If str is longer than len, the return value is shortened to len characters.
: Example of an SQL function usage fragment

> SELECT rpad('hi', 5, '??')
: Result value

hi???
: Example of an SQL function usage fragment

> SELECT rpad('hi', 1, '??')
: Result value

h





 rtrim 

rtrim(str)
: Removes the trailing space characters from str.

rtrim(trimStr, str)
: Removes the trailing string that contains the characters from the trim string from the str.
: Arguments



* str - A string expression.
* trimStr - The trim string characters to trim, the default value is a single space



: Example of an SQL function usage fragment

> SELECT rtrim(' SparkSQL ')
: Result value

SparkSQL
: Example of an SQL function usage fragment

> SELECT rtrim('LQSa', 'SSparkSQLS')
: Result value

SSpark





 second 

second(timestamp)
: Returns the second component of the string/timestamp.
: Example of an SQL function usage fragment

> SELECT second('2009-07-30 12:58:59')
: Result value

59





 sentences
jobid=<job_id>/<part-number>



Only one object contains the result set (jobid=<job_id>/<part-number>), and the other two are empty and don't contain any data.

It is important not to delete any of the files if you want to use the result set.

Each result is stored with an own job ID prefix that allows you to use the result directly in a query.

When you want to specify a result as input in your SQL query, specify the first (jobid=<job_id>) or the third one (jobid=<job_id>/<part-number>).

[Learn more about the result set created per query](https://cloud.ibm.com/docs/services/sql-query?topic=sql-query-overviewresult).





 Step 3.5. Run a query to find errors of the last 30 days 

For example, to find all ERROR log rows for the last 30 days, run the following command:

SELECT _source.data
FROM RESULTS_BUCKET STORED AS JSON
where dayofyear(_dayofyear > current_date - INTERVAL 30 day) and _source.level = "ERROR"
LIMIT 50
INTO QUERY_RESULTS_BUCKET STORED AS JSON

Where



* RESULTS_BUCKET is the SQL URL of the custom COS bucket that you use to upload the query results after transforming the data into partitioned JSON objects
* QUERY_RESULTS_BUCKET is the SQL URL of the custom COS bucket that you use to upload the query results







 Step 3.6. Run a query to find all ERROR log rows within a time range 

For example, to find all ERROR log rows in the timeframe between 7 am and 8 am on July 17th, run the following command:

SELECT _source.data
FROM RESULTS_BUCKET STORED AS JSON
where _dayofyear=198 and _hour=07 and _source.level = "ERROR"
LIMIT 50
INTO QUERY_RESULTS_BUCKET STORED AS JSON

Where
* [predicate](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencepredicate)
* [primaryExpression](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceprimaryExpression)







 Fullselect clause 





 fullselect 

A fullselect is the core component of a query. It is the only mandatory general component for a valid query statement. The other components outside of fullselect are optional. Its syntax is defined by the following syntax diagram.

The result set defined by a single fullselect can be combined with the result set of one or more other fullselects by using set operators.

The following set operators are supported and each set operator derives a result set from two other result sets R1 and R2:



* INTERSECT: The result consists of all rows in both R1 and R2.
* UNION: The result consists of all rows in R1 and all rows in R2.
* EXCEPT: The result consists of all rows that do not have a corresponding row in R2.
* MINUS: The minus operator is a synonym for the except operator and is supported for compatibility with other SQL implementations.



These set operators can be further refined by using the following modifiers:



* DISTINCT: This modifier ensures that the overall result set does not contain any duplicates. It is the default modifier that applies if no modifier is present.
* ALL: All rows of a fullselect's result set are combined by using a set operator. Thus, the overall result set can contain duplicates.



The following combinations of set operators and set modifiers are not supported:



* INTERSECT ALL
* EXCEPT ALL
* MINUS ALL



The characteristics of a result set defined by a fullselect can be further defined by using the following clauses:



* ORDER BY: Define an overall ordering of the result set based on the criteria that are defined by the list of sortItem clauses. The default order direction is ascending if not explicitly specified by a sortItem clause. The order by clause cannot be used with cluster by, distribute by, or sort by clause.
If you don't provide a sort parameter, relevance is used by default. The highest scoring matches are returned first.

If you provide a sort parameter, then matches are returned in that order, ignoring relevance.

If you want to use a sort parameter, and also include ordering by relevance in your search results, use the special fields -<score> or <score> within the sort parameter.





 POSTing search queries 

Instead of using the GET HTTP method, you can also use POST. The main advantage of POST queries is that they can have a request body, so you can specify the request as a JSON object. Each parameter in the previous table corresponds to a field in the JSON object in the request body.

See the following example that uses HTTP to POST a search request:

POST /db/_design/ddoc/_search/searchname HTTP/1.1
Content-Type: application/json
Host: $ACCOUNT.cloudant.com

See the following example that uses the command line to POST a search request:

curl "https://$ACCOUNT.cloudant.com/$DATABASE/_design/$DDOC/_search/$INDEX_NAME" -X POST -H "Content-Type: application/json" -d @search.json

See the following example JSON document that includes a search request:

{
"q": "index:my query",
"sort": "foo",
"limit": 3
}







 Query syntax 

The IBM Cloudant search query syntax is based on the [Lucene syntax](https://lucene.apache.org/core/4_3_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.htmlOverview). Search queries take the form of name:value unless the name is omitted, in which case they use the default field, as demonstrated in the following examples:

See the following example search query expressions:

// Birds
class:bird

// Animals that begin with the letter "l"
l

// Carnivorous birds
class:bird AND diet:carnivore