vania-janet/multiturn-rag-retrieval-data huggingface dataset:
See an example response from running the simple view query:

{
"total_rows": 2,
"offset": 0,
"rows": [
{
"id": "abc125",
"key": "amelie.smith@aol.com",
"value": {
"name": "Amelie Smith",
"email_verified": true,
"joined": "2020-04-24T10:42:59.000Z"
}
},
{
"id": "abc123",
"key": "bob.smith@aol.com",
"value": {
"name": "Bob Smith",
"email_verified": true,
"joined": "2019-01-24T10:42:59.000Z"
}
}
]
}
Show more





 Map function examples 

The definition of a view within a design document also creates an index based on the key information. The production and use of the index significantly increases the speed of access and searching or selecting documents from the view.

The following sections describe indexing with simple and complex keys, and reduce functions.

Your indexing functions work in a memory-constrained environment where the document forms part of the memory used in the environment. Your code's stack and document must fit within the memory. We limit documents to a maximum size of 64 MB.



 Indexing a field 

The following map function checks whether the object has a name field, and if so emits the value of this field. With this check, you can query against the value of the name field.

See an example of indexing a field:

function(doc) {
if (doc.name) {
emit("name", doc.name);
}
}





 An index for a one-to-many relationship 

If the object passed to emit has an _id field, a view query with include_docs set to true contains the document with the specific ID.

See an example of indexing a one-to-many relationship:
$conn = db2_connect( $conn_string, "", "" );
if( $conn )
{
echo "<p>Connection succeeded.</p>";
db2_close( $conn );
}
else
{
echo "<p>Connection failed.</p>";
}
}
else
{
echo "<p>No credentials.</p>";
}
?>
</body>
</html>
Show more
5. From your local working directory, push the updates to IBM Cloud by following the instructions in the "Getting Started" section of the PHP App in your IBM Cloud dashboard. Then, restart the App in IBM Cloud and view the App in a browser.
GET $SERVICE_URL/$DATABASE/_design/$DDOC/_view/$VIEW_NAME?start_key="alpha"&end_key="beta" HTTP/1.1

See the example of a global query that includes start_key and end_key query arguments.

Client libraries use POST method instead of GET because they have the same behavior.



* Curl
* Java
* Node
* Python
* Go



curl -X GET "$SERVICE_URL/users/_design/allusers/_view/getVerifiedEmails?start_key="alpha"&end_key="beta""

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.PostViewOptions;
import com.ibm.cloud.cloudant.v1.model.ViewResult;

Cloudant service = Cloudant.newInstance();

PostViewOptions viewOptions = new PostViewOptions.Builder()
.db("users")
.ddoc("allusers")
.view("getVerifiedEmails")
.startKey("alpha")
.endKey("beta")
.build();

ViewResult response =
service.postView(viewOptions).execute()
.getResult();

System.out.println(response);

const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

service.postView({
db: 'users',
ddoc: 'allusers',
view: 'getVerifiedEmails',
startKey: 'alpha',
endKey: 'beta'
}).then(response => {
console.log(response.result);
});

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()
Query operators 

You can use operators when you write queries to submit to Discovery by using the Query API.

The types of operators that are supported differ by query type:



* [Natural language queries](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsnlq-operator)
* [Discovery Query Language (DQL) queries](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsdql-operators)





 Natural Language Query (NLQ) operator 

The natural_language_query parameter accepts a string value.



 "" (Phrase query) 

Use quotation marks to emphasize a single word or phrase in the query that is most important to match. For example, the following request boosts documents that contain the term “nomination” in them.

{
"natural_language_query":"What is the process for "nomination" of bonds?"
}

Specifying a quoted phrase does not prevent documents without the phrase from being returned. It merely gives more weight to documents with the phrase than those without it. For example, the query results might also contain documents that mention “bonds” or “process” and do not contain the word “nomination”.

The following request boosts the phrase “change in monetary policy” and also matches “change” or “monetary” or “policy”.

{
"natural_language_query":""change in monetary policy""
}

Single quotation marks (') are not supported. You cannot use wildcards (*) in phrase queries.







 Discovery Query Language (DQL) operators 

Operators are the separators between different parts of a query.



 . (JSON delimiter) 

This delimiter separates the levels of hierarchy in the JSON schema

For example, the following query argument identifies the section of the enriched_text object that contains entities and the text recognized as an entity.

enriched_text.entities.text

The JSON representation of this section looks as follows:

Zoom

!
FROM cos://us-geo/sql/customers.csv STORED AS CSV
ORDER BY CustomerID
LIMIT 50





 Example of an exact target path specification 

The following query writes an SQL result into an exact result path. Normally, Data Engine always appends jobid=<jobid> to the provided target path to ensure a unique result location with each query execution. However, in the following sample query, this suffix is eliminated by adding JOBPREFIX NONE to the path in the INTO clause. Note: This action overwrites all objects that are currently stored in the provided result path.

SELECT 
FROM cos://us-geo/sql/employees.parquet STORED AS PARQUET
INTO cos://us-south/sql-7fb0b44d-2d76-4c5c-af1e-c746c84f9da1/result/employees.csv JOBPREFIX NONE





 Example of a self-join 

The following query uses a simple self-join to list the employees that are located in the same city as Steven.

SELECT e1.firstname employee, e2.firstname colleague, e1.city
FROM cos://us-geo/sql/employees.parquet STORED AS PARQUET e1,
cos://us-geo/sql/employees.parquet STORED AS PARQUET e2
WHERE e2.city = e1.city
AND e1.employeeid <> e2.employeeid
AND e1.firstname = 'Steven'
ORDER BY e1.city , e1.firstname







 Querying JSON logs 

To query JSON data where no schema can be inferred, for example, in logs, use the option to read the data AS TEXT. The input is then read without inferring any schema and each line is put into a single column named value.

To see how the data is structured, execute a basic select query.

SELECT 
FROM cos://us-geo/sql/LogDNA/_year=2019/_dayofyear=226/_hour=14 STORED AS TEXT
LIMIT 5
searchResult, _, err = service.PostSearch(searchOptions)
if err = nil {
panic(err)
}
}
The previous Go example requires the following import block: import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)
{
"total_rows": 65,
"rows":
{"_id":"10104153","_rev":"1-32aab6258c65c5fc5af044a153f4b994","name":"Silver Lake","latitude":34.08668,"longitude":-118.27023,"country":"US","population":32890,"timezone":"America/Los_Angeles"},
{"_id":"10104154","_rev":"1-125f589bf4e39d8e119b4b7b5b18caf6","name":"Echo Park","latitude":34.07808,"longitude":-118.26066,"country":"US","population":43832,"timezone":"America/Los_Angeles"},
{"_id":"4046704","_rev":"1-2e4b7820872f108c077dab73614067da","name":"Fort Hunt","latitude":38.73289,"longitude":-77.05803,"country":"US","population":16045,"timezone":"America/New_York"},
Improving your query results 

Learn about actions you can take to improve the quality of your query results.

You can use the tools that are built in to Discovery to make improvements.



 Results include more than exact matches 

Unlike some other search applications, adding quotation marks to a phrase that you submit does not return only exact matches. Queries that are submitted from the product user interface are natural language queries. When quoted text is submitted in a natural language query, the phrase is used to boost result scores. However, results are not limited to documents that contain the entire phrase.

If you want more control over how queries are handled, you must use the query API. For more information about the phrase operator of the query API, see [Query operators](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsphrase).





 A short query returns irrelevant results 

It might be that your query contains too many stop words and not enough distinct terms to trigger a meaningful search. When you submit a query, the query text is analyzed and optimized before it is submitted to the project. One of the changes that occurs is the removal of any stop words from the text. A stop word is a word that is considered to be not useful in distinguishing the semantic meaning of the content. Examples of stop words include terms such as and, the, and about. Discovery defines a list of stop words that it ignores automatically both when the data is indexed and when it is searched. When you submit a query that contains mostly or only stop words, such as About us, it is equivalent to submitting an empty query.

Although us is not included in the stop words list, it is lemmatized to we, which is listed as a stop word.

You can edit the stop words that are used by your collection. However, you can only augment the stop words list; you cannot remove stop words. And the stop words that you define are used only at query time. They do not affect the stop word list that is used by Discovery when data is added to a collection and the index is created.
The example shows how to retrieve the ID of the two most active employees in each department.

-- derive posts ranking by using a named window specification
SELECT 
FROM (
SELECT
posts.col1 AS emp_id,
posts.col2 AS dept_id,
posts.col3 AS posts,
DENSE_RANK() OVER post_ranking AS rank
FROM VALUES
(1, 1 ,100),
(2, 1 ,50),
(8, 1 ,250),
(3, 2 ,200),
(4, 2 ,300),
(9, 2 ,1000),
(5, 3 ,300),
(6, 3 ,100),
(7, 3 ,400) AS posts
WINDOW post_ranking AS (
PARTITION BY posts.col2
ORDER BY posts.col3 DESC
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)
)
WHERE rank <= 2
Show more

The result of the example query is shown in the following table.



Table 26. Query result for example: derive posts ranking by using a named window specification.

 EMP_ID DEPT_ID POSTS RANK 

 8 1 250 1 
 1 1 100 2 
 7 3 400 1 
 5 3 300 2 
 9 2 1000 1 
 4 2 300 2 



-- derive posts ranking by using an inline window specification
SELECT * FROM (
SELECT
posts.col1 AS emp_id,
posts.col2 AS dept_id,
posts.col3 AS posts,
DENSE_RANK() OVER (
PARTITION BY posts.col2
ORDER BY posts.col3 DESC
ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
) AS rank
FROM VALUES
(1, 1 ,100),
(2, 1 ,50),
(8, 1 ,250),
(3, 2 ,200),
(4, 2 ,300),
(9, 2 ,1000),
(5, 3 ,300),
(6, 3 ,100),
(7, 3 ,400) AS posts
The drilldown option is only available when you make global queries.

You can restrict results to documents with a dimension equal to the specified label. Restrict the results by adding drilldown=["dimension","label"] to a search query. You can include multiple drilldown parameters to restrict results along multiple dimensions.

Using a drilldown parameter is similar to using key:value in the q parameter, but the drilldown parameter returns values that the analyzer might skip.

For example, if the analyzer didn't index a stop word like "a", the drilldown parameter returns it when you specify drilldown=["key","a"].





 Ranges 

The ranges option is only available when you make global queries.

The range facet syntax reuses the standard Lucene syntax for ranges to return counts of results that fit into each specified category. Inclusive range queries are denoted by brackets ([, ]). Exclusive range queries are denoted by curly brackets ({, }).

The indexed values can't be mixed types. For example, if 100 strings are indexed, and one number, then the index can't be used for range operations. You can check the type by using the typeof operator, and convert it by using the parseInt, parseFloat, or .toString() functions.

See the following example of a request that uses faceted search for matching ranges:

?q=:&ranges={"price":{"cheap":"[0 TO 100]","expensive":"{100 TO Infinity}"}}

See the following example results after a ranges check on a faceted search:

{
"total_rows":100000,
"bookmark":"g...",
"rows":[...],
"ranges": {
"price": {
"expensive": 278682,
"cheap": 257023
}
}
}







 Geographical searches 

In addition to searching by the content of textual fields, you can also sort your results by their distance from a geographic coordinate.
To query specific fields from the input data, you can use the options [regexp_extract](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sqlfunctionsregexp_extract) or [get_json_object](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sqlfunctionsget_json_object).





 timeSeriesProperties 

The TIME_SERIES_FORMAT option triggers a read transformation mechanism that uses a set of timeSeriesProperties to dynamically generate one or more native time series columns (defined by the IN clause) from the specified value and key columns of the input data.

The parameters timetick and value are the only parameters that are required to be specified.

Following you see the descriptions of each parameter and how they affect the time series:



* timetick: The column that contains the timestamp or timetick. Ultimately, the resulting time series is sorted by this column. If two rows contain the same timetick, it is uncertain which timetick comes first in the time series.
* value: The column that contains the value.
* key: Optionally specify a key column that you can use to group each time series by. If a key is indicated, you can assume that n time series are created, where n is the set of all keys in the key column. If no key column is specified, a single time series is created from the indicated data set.
* starttime: Optionally specify a starttime string (any properly formatted [DateTime](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html)) for which to set the time series [TRS](https://cloud.ibm.com/docs/sql-query?topic=sql-query-TRS). If starttime is not indicated, and granularity is indicated, the starttime defaults to 1 January 1970 12 AM (midnight) GMT.
nullif(expr1, expr2)
: Returns null if expr1 equals to expr2, or expr1 otherwise.
: Example of an SQL function usage fragment

> SELECT nullif(2, 2)
: Result value

NULL





 nvl 

nvl(expr1, expr2)
: Returns expr2 if expr1 is null, or expr1 otherwise.
: Example of an SQL function usage fragment

> SELECT nvl(NULL, array('2'))
: Result value

["2"]





 nvl2 

nvl2(expr1, expr2, expr3)
: Returns expr2 if expr1 is not null, or expr3 otherwise.
: Example of an SQL function usage fragment

> SELECT nvl2(NULL, 2, 1)
: Result value

1





 octet_length 

octet_length(expr)
: Returns the byte length of string data or number of bytes of binary data.
: Example of an SQL function usage fragment

> SELECT octet_length('Spark SQL')
: Result value

9





 or 

expr1 or expr2
: Logical OR.





 parse_url 

parse_url(url, partToExtract[, key])
: Extracts a part from a URL.
: Example of an SQL function usage fragment

> SELECT parse_url('http://spark.apache.org/path?query=1', 'HOST')
: Result value

spark.apache.org
: Example of an SQL function usage fragment

> SELECT parse_url('http://spark.apache.org/path?query=1', 'QUERY')
: Result value

query=1
: Example of an SQL function usage fragment

> SELECT parse_url('http://spark.apache.org/path?query=1', 'QUERY', 'query')
: Result value

1





 percent_rank
Users need the platform viewer role to launch the UI, and the service writer role to run queries. For more information about roles, see [Required user roles](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewuser-roles).
* When you open the UI, the Data Engine service automatically generates a unique COS bucket that will store all of the results as CSV files from your SQL queries.



Use the Data Engine user interface (UI) to develop and test your queries, and the SQL Query REST API to automate them.

You can use the Data Engine to run SQL queries (that is, SELECT statements) to analyze, transform structured and semi-structured data, or clean up rectangular data. You cannot run actions such as CREATE, DELETE, INSERT, and UPDATE. The Data Engine service can process input data that is read from CSV, JSON, ORC, Parquet, or AVRO files.

The archive files from an Log Analysis instance contain data in JSON format.



* If your log data consists of rectangular data with heterogeneous data types, you should transform the format to partitioned JSON objects to query successfully the data.
* If your log data consists of rectangular data with homogeneous data types, you should transform the format to PARQUET to query successfully the data and obtain better performance. Parquet is an open source file format that stores nested data structures into a flat columnar format, and preserves the schema of the original data.
* Each query result can be written to a CSV, JSON, ORC, PARQUET, or AVRO file in a Object Storage instance of your choice.



For more information, see [Data Engine overview](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overview).



 Prerequisites 

To be able to use the Data Engine service to query archived event files, check the following prerequisites:



* You must have access to the COS instance where the logging data is available.

You must have access to a bucket that contains the Log Analysis archive files and a bucket to use to store results from your queries.
: Introduce the Mango query operator, $keyMapMatch that offers the ability to make queries on the keys of a map.

Improvements
: Internal bug fixes.

Database reporting
: Report the database that was used for authentication for a GET /_session request, provided it is configured.







 September 2020 



 1 September 2020 

The following changes were made in build 8162:

Improvements
: Internal bug fixes.

Drilldown parameters
: Drilldown parameters for text index searches can now be specified as a list of lists, which gives you the ability to avoid having to define it redundantly in a single query. Some languages don't have this facility.

couch_index server
: The couch_index server doesn't crash and log errors in the following cases: If a design document is deleted while that index is building, or when a design document is added immediately after database creation.

Invalid parameters
: IBM Cloudant now checks for and reports invalid parameters on database creation.







 July 2020 



 1 July 2020 

The following changes were made in build 8158:

Improvements
: Internal bug fixes.







 May 2020 



 15 May 2020 

The following changes were made in build 8153:

Improvements
: Internal bug fixes.







 April 2020 



 1 April 2020 

The following changes were made in build 8152:

Improvements
: Internal bug fixes.







 March 2020 



 15 March 2020 

The following changes were made in build 8142:

New! Endpoints
: New endpoints were added, so you can post multiple queries: POST /{db}/_all_docs/queries and POST /{db}/_design_docs/queries.

Multiple queries
: The ability to submit multiple queries against a view by using the POST to /{db}/_design/{ddoc}/_view/{view} with the ?queries option was replaced by the new queries endpoint. The same is true of the _all_docs and _design_docs endpoints. Specify a keys object when you POST to these endpoints.

disk_size and data_size fields
"poll.interval.ms": "60000",
"table.whitelist": "TEST_TABLE",
"batch.max.rows": "10000000",
"name": "test-source-jdbc"
},
"tasks": [],
"type": "source"
}
Show more
* For the sink connector, run the following command.

curl -s -X POST -H "Content-Type: Application/json" --data '{ "name": "test-sink", "config": { "connector.class": "io.aiven.connect.jdbc.JdbcSinkConnector", "tasks.max": "2", "connection.url":"jdbc:netezza://localhost:5480/db1;user=user1;password=secret","connection.user":"user1","connection.password":"secret","dialect.name":"GenericDatabaseDialect","topics": "TEST_TABLE", "insert.mode": "insert" } }' http://localhost:8083/connectors | jq
{
"name": "test-sink",
"config": {
"connector.class": "io.aiven.connect.jdbc.JdbcSinkConnector",
"tasks.max": "2",
"connection.url": "jdbc:netezza://localhost:5480/db1;user=user1;password=secret",
"connection.user": "user1",
"connection.password": "secret",
"dialect.name": "GenericDatabaseDialect",
"topics": "TEST_TABLE",
"insert.mode": "insert",
"name": "test-sink"
},
"tasks": [],
"type": "sink"
}
Show more



6.
[Check mark icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/images/checkmark-icon.svg) 
 Get all configured views and alerts GET 
 Delete a view and its associated alerts. DELETE 







 query (string) 

Specifies the search query that is applied to the view.

Check the query in the UI to validate that the data that is displayed through the view matches the data entry that you plan to monitor through that view.





 hosts (array of strings) 

Specifies the list of services from which you want to view data.

In the UI, the value that is set for Source in the Line identifiers section corresponds to the hosts value of the service that generates that auditing event.

For example, to enter multiple hosts, you must separate the hosts with a comma:

"hosts": ["is", "event-streams"]





 apps (array of strings) 

Specifies the service instance ID that generates the auditing event.

For example, to enter multiple apps, you must separate the apps with a comma:

"apps": ["apps1", "apps2"]

Notice that the apps value for global auditing events is the CRN of the account where the event is generated.





 channels (array of objects) 

Specifies the notification channels and trigger conditions that are associated with a view.



* You can configure 1 or more channels per view.
* You can configure any of the following channels through the Configuration API: email, webhook, PagerDuty.



"channels": [
{
"integration": "email",
"emails": "LIST OF EMAILS"], // array of strings (emails) that are comma-separated
"triggerlimit": 15,
"triggerinterval": "5m",
"immediate": true,
"terminal": true,
"operator": "presence",
"timezone": "TIMEZONE"
},
: Example of an SQL function usage fragment

> SELECT substr('Spark SQL', 5, 1)
: Result value

k





 substring 

substring(str, pos[, len])
: Returns the substring of str that starts at pos and is of length len, or the slice of byte array that starts at pos and is of length len.
: Example of an SQL function usage fragment

> SELECT substring('Spark SQL', 5)
: Result value

k SQL
: Example of an SQL function usage fragment

> SELECT substring('Spark SQL', -3)
: Result value

SQL
: Example of an SQL function usage fragment

> SELECT substring('Spark SQL', 5, 1)
: Result value

k





 substring_index 

substring_index(str, delim, count)
: Returns the substring from str before count occurrences of the delimiter delim. If count is positive, everything to the left of the final delimiter (counting from the left) is returned. If count is negative, everything to the right of the final delimiter (counting from the right) is returned. The function substring_index performs a case-sensitive match when you search for delim.
: Example of an SQL function usage fragment

> SELECT substring_index('www.apache.org', '.', 2)
: Result value

www.apache





 sum 

sum(expr)
: Returns the sum that is calculated from values of a group.





 tan 

tan(expr)
: Returns the tangent of expr.
: Example of an SQL function usage fragment

> SELECT tan(0)
: Result value

0.0





 tanh 

tanh(expr)
: Returns the hyperbolic tangent of expr.
: Example of an SQL function usage fragment

> SELECT tanh(0)
: Result value

0.0





 timestamp 

timestamp(expr)