vania-janet/multiturn-rag-retrieval-data huggingface dataset:
This index is useful for queries that involve both the publisher and the year, but if we introduce another field or make the query more complex (for example, by using the $or operator), then the index doesn't get used. We are back to a full database scan.

For a general-purpose search facility, we need IBM Cloudant Search, which is described in the next section.







 Step 3. Creating a search engine - IBM Cloudant Search 

IBM Cloudant Search is based on Apache Lucene and has its own query language that allows rich queries to be constructed. See the following example of a search:

publisher:Penguin AND (year:1972 OR year:1973) AND title:Crash

Unlike IBM Cloudant Query, you must specify the fields to index before you perform a query. IBM Cloudant Search indexes are defined by supplying IBM Cloudant with a JavaScript function that is called once for every document in the database - if the function calls index then data is added to the index.



1. From the IBM Cloudant Dashboard, select the books database.
2. Select the Design Documents tab.
3. Select New Search Index from the menu.
4. Enter a design document name.
5. Enter an index name.
6. Paste the following code into the search index function:

function (doc) {
index("author", doc.author);
index("publisher", doc.publisher);
index("title", doc.title);
index("year", doc.year);
}
7. Choose the "Standard Analyzer".

Zoom

![Select Standard Analyzer, and click Create Document and Build Index to create a search index.](https://cloud.ibm.com/docs-content/v1/content/522c6f62358b063f921283400a601c3f9bc66f08/Cloudant/images/indexingdashboard4.png)

Figure 3. New Search Index window
: Returns the hour component of the string/timestamp.
: Example of an SQL function usage fragment

> SELECT hour('2009-07-30 12:58:59')
: Result value

12





 hypot 

hypot(expr1, expr2)
: Returns sqrt(expr1**2 + expr2**2).
: Example of an SQL function usage fragment

> SELECT hypot(3, 4)
: Result value

5.0





 if 

if(expr1, expr2, expr3)
: If expr1 evaluates to true, then returns expr2; otherwise, returns expr3.
: Example of an SQL function usage fragment

> SELECT if(1 < 2, 'a', 'b')
: Result value

a





 ifnull 

ifnull(expr1, expr2)
: Returns expr2 if expr1 is null, or expr1 otherwise.
: Example of an SQL function usage fragment

> SELECT ifnull(NULL, array('2'))
: Result value

["2"]





 in 

expr1 in(expr2, expr3, ...)
: Returns true if expr equals to any valN.
: Arguments

expr1, expr2, expr3, ... - the arguments must be same type.
: Example of an SQL function usage fragment

> SELECT 1 in(1, 2, 3)
: Result value

true
: Example of an SQL function usage fragment

> SELECT 1 in(2, 3, 4)
: Result value

false
: Example of an SQL function usage fragment

> SELECT named_struct('a', 1, 'b', 2) in(named_struct('a', 1, 'b', 1), named_struct('a', 1, 'b', 3))
: Result value

false
: Example of an SQL function usage fragment
: Example of an SQL function usage fragment

> SELECT approx_percentile(10.0, array(0.5, 0.4, 0.1), 100)
: Result value

[10.0,10.0,10.0]
: Example of an SQL function usage fragment

> SELECT approx_percentile(10.0, 0.5, 100)
: Result value

10.0





 array 

array(expr, ...)
: Returns an array with the indicated elements.
: Example of an SQL function usage fragment

> SELECT array(1, 2, 3)
: Result value

[1,2,3]





 array_contains 

array_contains(array, value)
: Returns true if the array contains the value.
: Example of an SQL function usage fragment

> SELECT array_contains(array(1, 2, 3), 2)
: Result value

true





 ascii 

ascii(str)
: Returns the numeric value of the first character of str.
: Example of an SQL function usage fragment

> SELECT ascii('222')
: Result value

50
: Example of an SQL function usage fragment

> SELECT ascii(2)
: Result value

50





 asin 

asin(expr)
: Returns the inverse sine (also known as arcsine) the arc sin of expr if -1<=expr<=1 or NaN otherwise.
: Example of an SQL function usage fragment

> SELECT asin(0)
: Result value

0.0
: Example of an SQL function usage fragment

> SELECT asin(2)
: Result value

NaN





 assert_true 

assert_true(expr1 [, expr2])
: Returns null if column expr1 is true; throws an exception with the provided error message expr2 otherwise. Useful to check if a cast for a column is successful for every row.
: Example of an SQL function usage fragment
if (doc.class) {
index("class", doc.class, {"store": true});
}
}



 Index guard clauses 

The index function requires the name of the data field to index as the second parameter. However, if that data field doesn't exist for the document, an error occurs. The solution is to use an appropriate "guard clause" that checks whether the field exists. This clause contains the expected type of data before any attempt to create the corresponding index.

See the following example definition that doesn't have any validation on the type of the index data field:

if (doc.min_length) {
index("min_length", doc.min_length, {"store": true});
}

You might use the JavaScript typeof operator to implement the guard clause test. If the field exists and has the expected type, the correct type name is returned. The guard clause test succeeds, which means it's safe to use the index function. If the field does not exist, you wouldn't get back the expected type of field, that's why you wouldn't try to index the field.

JavaScript considers a result to be false if one of the following values is tested:



* 'undefined'
* Null
* The number +0
* The number -0
* NaN (not a number)
* "" (the empty string)



See the following example that uses a guard clause to check whether the required data field exists, and holds a number, before you try to index:

if (typeof doc.min_length === 'number') {
index("min_length", doc.min_length, {"store": true});
}

Use a generic guard clause test to ensure that the type of the candidate data field is defined.

See the following example of a "generic" guard clause:

if (typeof doc.min_length) !== 'undefined') {
// The field exists, and does have a type, so we can proceed to index using it.
...
}







 Analyzers
"selector":{
"_id":{
"$regex":"^_design/"
}
}
}

See the following example response (abbreviated) after you filter by using a selector:

{
"last_seq":"11-g1A...OaA",
"pending":0,
"results":[
{
"changes":
{
"rev":"10-304...4b2"
}
],
"id":"_design/ingredients",
"seq":"8-g1A...gEo"
},
{
"changes":
{
"rev":"123-6f7...817"
}
],
"deleted":true,
"id":"_design/cookbook",
"seq":"9-g1A...4BL"
},
{
"changes":
{
"rev":"6-5b8...8f3"
}
],
"deleted":true,
"id":"_design/meta",
"seq":"11-g1A...Hbg"
}
]
}
Show more





 The _view filter 

Using the _view filter, you can use an existing [map function](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-views-mapreducea-simple-view) as the filter.

The map function might emit output as the result of processing a specific document. When this situation occurs, the filter considers the document that is allowed and includes it in the list of documents that you changed.

See the following example application of the _view filter by using HTTP:

POST $SERVICE_URL/$DATABASE/_changes?filter=_view&view=$DDOC/$VIEW_NAME HTTP/1.1

See the following example applications of the _view filter:
For more information, see [Filtering by intent or entity](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-filter-referencefilter-reference-intent-entity).

Note: The filter query syntax uses some characters that are not allowed in HTTP queries. Make sure that all special characters, including spaces and quotation marks, are URL encoded when sent as part of an HTTP query. For example, the filter response_timestamp<2020-01-01 would be specified as response_timestamp%3C2020-01-01.





 Operators 

You can use the following operators in your filter query.



 Operator Description 

 : Fuzzy match query operator. Prefix the query term with : if you want to match any value that contains the query term, or a grammatical variant of the query term. Fuzzy matching is available for user input text and entity values. 
 :: Exact match query operator. Prefix the query term with :: if you want to match only values that exactly equal the query term. 
 :! Negative fuzzy match query operator. Prefix the query term with :! if you want to match only values that do not contain the query term or a grammatical variant of the query term. 
 ::! Negative exact match query operator. Prefix the query term with ::! if you want to match only values that do not exactly match the query term. 
 <=, >=, >, < Comparison operators. Prefix the query term with these operators to match based on arithmetic comparison. 
 `` Escape operator. Use in queries that include control characters that would otherwise be parsed as operators. For example, !hello would match the text !hello. 
 "" Literal phrase. Use to enclose a query term that contains spaces or other special characters. No special characters within the quotation marks are parsed by the API. 
 Approximate match. Append this operator followed by a 1 or 2 to the end of the query term to specify the allowed number of single-character differences between the query string and a match in the response object. For example, car1 would match car, cat, or cars, but it would not match cats.
For more information, see [Connecting with mysql](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-connecting-mysql).





 Connect with MySQL Workbench 

For more information, see [Connections in MySQL Workbench](https://dev.mysql.com/doc/workbench/en/wb-mysql-connections.html).





 Next Steps 

If you are using MySQL for the first time, see the [MySQL 5.7 Reference Manual](https://dev.mysql.com/doc/refman/5.7/en/).

You can connect, manage your databases, and manage data with MySQL's command-line interface (CLI) tool [mysql](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-connecting-mysql).

Looking for more tools on managing your deployment? Connect to your deployment with the [IBM Cloud CLI](https://cloud.ibm.com/docs/cli?topic=cli-install-ibmcloud-cli), the [Cloud Databases CLI plug-in](https://cloud.ibm.com/docs/databases-cli-plugin?topic=databases-cli-plugin-cdb-reference), or the [Cloud Databases API](https://cloud.ibm.com/apidocs/cloud-databases-api).

If you plan to use Databases for MySQL for your applications, check out [Connecting an external application](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-external-app) and [Connecting an IBM Cloud application](https://cloud.ibm.com/docs/databases-for-mysql?topic=databases-for-mysql-ibmcloud-app).
Using IBM Cloudant Search 

Search indexes provide a way to query a database by using [Lucene Query Parser Syntax](https://lucene.apache.org/core/4_3_0/queryparser/org/apache/lucene/queryparser/classic/package-summary.htmlOverview). A search index uses one or more fields from your documents.

You can use a search index to run queries, find documents based on the content they include, or work with groups, facets, or geographical searches.

To create a search index, you add a JavaScript function to a design document in the database. An index builds after it processes one search request or after the server detects a document update. The index function takes the following parameters:



1. Field name - The name of the field you want to use when you query the index. If you set this parameter to default, then this field is queried if no field is specified in the query syntax.
2. Data that you want to index, for example, doc.address.country.
3. (Optional) The third parameter includes the following fields: boost, facet, index, and store. These fields are described in more detail later.



By default, a search index response returns 25 rows. The number of rows that is returned can be changed by using the limit parameter. However, a result set from a search is limited to 200 rows. Each response includes a bookmark field. You can include the value of the bookmark field in later queries to look through the responses.

You can query the API by using one of the following methods: URI, IBM Cloudant Dashboard, curl, or a browser plug-in, such as Postman or RESTClient.

See the following example design document that defines a search index:

{
"_id": "_design/search_example",
"indexes": {
"animals": {
"index": "function(doc){ ... }"
}
}
}



 Search index partitioning type
* If the format of the input objects is CSV and a delimiter other than the default , (comma) is used, you must specify the delimiter by using the FIELDS TERMINATED BY option of the [STORED AS](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexternalTableSpec) clause. All single Unicode characters are allowed as delimiters.
* By default, it is assumed that CSV input objects have a header line that specifies the names of the input columns. If the objects don't have a header line, you must specify NOHEADER in the [STORED AS](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexternalTableSpec) clause.
* By default, it is assumed that JSON input objects consist of a single JSON record per line. If individual records span multiple lines, you must specify MULTILINE in the [STORED AS](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexternalTableSpec) clause.
* If required, you can use JOIN constructs to join data from several input URIs, even if those URIs point to different instances of Cloud Object Storage.
* Use the INTO clause of a [query](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencechapterSQLQueryStatement) to specify the output [URI](https://cloud.ibm.com/docs/sql-query?topic=sql-query-runningunique), that is, the location to which the result is to be written and the wanted result format.



2. The Target location field displays where the result is stored. An initial bucket in one of your Object Storage instances is automatically created for you when you open the UI. It is then chosen as your default location, if your query does not specify an INTO clause. To ensure the automatic setup of an initial bucket, do the following steps in advance:



* You must create an Object Storage instance.
SELECT * FROM cos://us-geo/sql/employees.parquet STORED AS PARQUET TABLESAMPLE (BUCKET 2 OUT OF 10)





 More topics - table sample clause 

For more information about the clauses that are used in a table sample clause, see the following topics:



* [expression](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceexpression)
* [identifier](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceidentifier)
* [qualifiedName](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencequalifiedName)
* [unsignedInteger](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceunsignedInteger)
* [unsignedNumber](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceunsignedNumber)







 Related references - table sample clause 

A table sample clause is referenced by the following clause:



* [relationPrimary](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencerelationPrimary)









 SQL functions 

The syntax for SQL function invocation is described by the following syntax diagram.



 functionOrAggregate 

Most function invocations look like function(argument1, ..., argumentN) but functions like TRIM(), POSITION(), FIRST(), LAST(), STRUCT(), EXTRACT(), and SUBSTRING() support a different invocation style.

Refer to section [SQL functions](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sqlfunctionssqlfunctions) for details about supported functions.





 More topics - function or aggregate clause 

For more information about the clauses that are used in a function or aggregate clause, see the following topics:
For example, the conversational flow might collect information about the type of device a customer wants to buy. When you know the device model, you can then send a model keyword in the query that is submitted to the search integration to get better results.

In the And then field of the step where you want the search to be triggered, choose Search for the answer.

To configure the search in Discovery, complete the following steps:



1. Click Edit settings.
2. Add values to one or both of the following fields:



* Custom search query. Add a word or phrase that you want to submit to Discovery as the query string for the search.

For example, you can specify a string such as, What cities do you fly to?.

For a more dynamic string, you can include a variable. For example, Do you have flights to ${destination}?

You are effectively defining the value that is used by the Discovery API as the natural_language_query parameter. For more information, see [Query parameters](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersnlq).

If you don't specify a text string, the action sends the most recently submitted user message as the search string.

If you want to use the original customer message that triggered the action as the query string instead, you need to plan ahead. You can follow these steps:



1. Create a session variable to store the initial user input. For example, named original message.
2. In Step 1, meaning the first step after the action trigger, set the value of the session variable. For more information about session variables, see [Defining session variables](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-search-addactions-variables-global).
3. Set the value of the variable by using an expression that looks like this: <? input.text ?>.



This expression captures the complete message that was submitted by the customer. As a result, your variable captures the customer message that triggered this action.



1.
SQL functions 

You can use any of the following functions in your query.



 ! 

expr
: Logical not.





 % 

expr1 % expr2
: Returns the remainder after expr1/expr2.
: Example of an SQL function usage fragment

> SELECT 2 % 1.8
: Result value

0.2
: Example of an SQL function usage fragment

> SELECT MOD(2, 1.8)
: Result value

0.2





 & 

expr1 & expr2
: Returns the result of bitwise AND of expr1 and expr2.
: Example of an SQL function usage fragment

> SELECT 3 & 5
: Result value

1





 * 

expr1 * expr2
: Returns expr1*expr2.
: Example of an SQL function usage fragment

> SELECT 2 * 3
: Result value

6





 + 

expr1 + expr2
: Returns expr1+expr2.
: Example of an SQL function usage fragment

> SELECT 1 + 2
: Result value

3





 - 

expr1 - expr2
: Returns expr1-expr2.
: Example of an SQL function usage fragment

> SELECT 2 - 1
: Result value

1





 / 

expr1 / expr2
: Returns expr1/expr2. It always performs floating point division.
: Example of an SQL function usage fragment

> SELECT 3 / 2
: Result value

1.5
: Example of an SQL function usage fragment

> SELECT 2L / 2L
: Result value

1.0





 < 

expr1 < expr2
: Returns true if expr1 is less than expr2.
: Arguments

expr1, expr2 - The two expressions must be same type or can be cast to a common type, and must be a type that can be ordered. For example, map type is not orderable, so it is not supported.
GET $SERVICE_URL/$DATABASE/_design/$DDOC/_view/$VIEW_NAME?descending=true&start_key="beta"&end_key="alpha" HTTP/1.1

See the example to apply correct filtering and sorting to a global query.

Client libraries use POST method instead of GET because they have the same behavior.



* Curl
* Java
* Node
* Python
* Go



curl -X GET "$SERVER_URL/users/_design/allusers/_view/getVerifiedEmails?descending=true&start_key="beta"&end_key="alpha""

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.PostViewOptions;
import com.ibm.cloud.cloudant.v1.model.ViewResult;

Cloudant service = Cloudant.newInstance();

PostViewOptions viewOptions = new PostViewOptions.Builder()
.db("users")
.ddoc("allusers")
.view("getVerifiedEmails")
.descending(true)
.startKey("beta")
.endKey("alpha")
.build();

ViewResult response =
service.postView(viewOptions).execute()
.getResult();

System.out.println(response);

const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

service.postView({
db: 'users',
ddoc: 'allusers',
view: 'getVerifiedEmails',
descending: true,
startKey: 'beta',
endKey: 'alpha'
}).then(response => {
console.log(response.result);
});

from ibmcloudant.cloudant_v1 import CloudantV1
: Example of an SQL function usage fragment

> SELECT degrees(3.141592653589793)
: Result value

180.0





 dense_rank 

dense_rank()
: Computes the rank of a value in a group of values. The result is one plus the previously assigned rank value. Unlike the function rank, dense_rank does not produce gaps in the ranking sequence.





 double 

double(expr)
: Casts the value expr to the target data type double.





 e 

e()
: Returns Euler's number, e.
: Example of an SQL function usage fragment

> SELECT e()
: Result value

2.718281828459045





 elt 

elt(n, input1, input2, ...)
: Returns the n-th input, for example, returns input2 when n is 2.
: Example of an SQL function usage fragment

> SELECT elt(1, 'scala', 'java')
: Result value

scala





 encode 

encode(str, charset)
: Encodes the first argument by using the second argument character set.
: Example of an SQL function usage fragment

> SELECT encode('abc', 'utf-8')
: Result value

abc





 exp 

exp(expr)
: Returns e to the power of expr.
: Example of an SQL function usage fragment

> SELECT exp(0)
: Result value

1.0





 explode 

explode(expr)
: Separates the elements of array expr into multiple rows, or the elements of map expr into multiple rows and columns.
: Example of an SQL function usage fragment

> SELECT explode(array(10, 20))
: Result value

10 20





 explode_outer 

explode_outer(expr)
: Separates the elements of array expr into multiple rows, or the elements of map expr into multiple rows and columns.
: Example of an SQL function usage fragment

> SELECT explode_outer(array(10, 20))
See the following example if statement to verify that the required fields exist in each document:

if (typeof doc.town == "string" && typeof doc.name == "string") {
index("town", doc.town, {facet: true});
index("name", doc.name, {facet: true});
}



 Counts 

The counts option is only available when you make global queries.

The counts facet syntax takes a list of fields, and returns the number of query results for each unique value of each named field.

The count operation works only if the indexed values are strings. The indexed values can't be mixed types. For example, if 100 strings are indexed, and one number, then the index can't be used for count operations. You can check the type by using the typeof operator, and convert it by using the parseInt, parseFloat, or .toString() functions.

See the following example query that uses the counts facet syntax:

?q=:&counts=["type"]

See the following example response after you use the counts facet syntax:

{
"total_rows":100000,
"bookmark":"g...",
"rows":[...],
"counts":{
"type":{
"sofa": 10,
"chair": 100,
"lamp": 97
}
}
}





 drilldown 

The drilldown option is only available when you make global queries.

You can restrict results to documents with a dimension equal to the specified label. Restrict the results by adding drilldown=["dimension","label"] to a search query. You can include multiple drilldown parameters to restrict results along multiple dimensions.

Using a drilldown parameter is similar to using key:value in the q parameter, but the drilldown parameter returns values that the analyzer might skip.