vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Multiple queries
: The ability to submit multiple queries against a view by using the POST to /{db}/_design/{ddoc}/_view/{view} with the ?queries option was replaced by the new queries endpoint. The same is true of the _all_docs and _design_docs endpoints. Specify a keys object when you POST to these endpoints.

disk_size and data_size fields
: The disk_size and data_size fields were retired from the database information object that is returned by GET /{db}.

/{db}/_changes feed
: The /{db}/_changes feed immediately returns headers now, even when no changes are available. This process prevents client's from becoming blocked.

Negative and noninteger heartbeat values
: Negative and noninteger heartbeat values now return a 400 Bad Request response status code.

Separate proxies
: Allow specifying separate proxies for both the source and target in a replication by using source_proxy and target_proxy keys.

POST view function
: The POST view function now supports identical parameters and behavior as specified in the /{db}/_design/{ddoc}/_view/{view}, /{db}/_all_docs, and /{db}/_design_docs endpoints. You can supply query string parameters as keys in a JSON object in the body of the POST request.

Replication errors
: Replication "info" errors are now JSON objects. Previously, they were strings.

Replication support
: A compatibility change was made to support replication with future databases that contain per-document access control fields.

Warning message
: Add a warning to the _find endpoint if multiple document scans were required to generate a result.

_find endpoint error
: Fix a bug in the _find endpoint whereby an error would be returned if a document matched by a text index was deleted while the query was being processed.







 January 2020 



 15 January 2020 

The following changes were made in build 8126:

Improvements
: Internal bug fixes.

Replication error reports
Improving result relevance with the API 

You can train Discovery to improve the relevance of query results for your particular organization or subject area. When you provide a Discovery instance with training data, the service uses machine-learning Watson techniques to find signals in your content and questions. The service then reorders query results to display the most relevant results at the top. As you add more training data, the service instance becomes more accurate and sophisticated in the ordering of results it returns.

Relevancy training is optional; if the results of your queries meet your needs, no further training is necessary. For information about use cases for relevancy training, see [Improve your natural language query results from Watson Discovery](https://developer.ibm.com/blogs/improving-your-natural-language-query-results-from-watson-discovery/).

For comprehensive information about the training APIs, see the [API reference](https://cloud.ibm.com/apidocs/discovery).

If you would prefer to use the Discovery tooling to train Discovery, see [Improving result relevance with the tooling](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-tooling).

Relevance training currently applies only to natural language queries in private collections. It is not intended for use with structured Discovery Query Language queries. For more about the Discovery Query Language, see [Query concepts](https://cloud.ibm.com/docs/discovery?topic=discovery-query-concepts).

Trained collections return a confidence score in the result of a natural language query. See [Confidence scores](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-toolingconfidence) for details.

Adding a custom stopwords list can improve the relevance of results for natural language queries. See [Defining stopwords](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsstopwords) for more information.
"productid": "1000042",
"taxonomy": [
"Home",
"Kitchen",
"Small Appliances"
],
"type": "product"
}
Show more



 Query parameters 

You can add some query parameters to the URL, for example /mydatabase/doc?attachments=true&conflicts=true.

All parameters are optional.



Table 1. Query parameters

 Name Type Description Default 

 attachments Boolean Includes attachments bodies in response. False 
 att_encoding_info Boolean Includes encoding information in attachment stubs if the particular attachment is compressed. False 
 atts_since Array of revision strings Includes attachments only since specified revisions. Doesn't include attachments for specified revisions. [] 
 conflicts Boolean Includes information about conflicts in documents. False 
 deleted_conflicts Boolean Includes information about deleted conflicted revisions. False 
 latest Boolean Forces retrieval of the most recent "leaf" revision, no matter what revision was requested. False 
 local_seq Boolean Includes last update sequence number for the document. False 
 meta Boolean Same as specifying the conflicts, deleted_conflicts, and open_revs query parameters. False 
 open_revs Array or all Retrieves documents of specified leaf revisions. Additionally, it accepts the value all to return all leaf revisions. [] 
 rev String Retrieves document of specified revision. <br><br> * <br><br><br> 
 revs Boolean Includes list of all known document revisions. False 
 revs_info Boolean Includes detailed information for all known document revisions. False 







 Read many 

To fetch more than one document at a time, [query the database](https://cloud.ibm.com/apidocs/cloudantpostalldocs) by using the include_docs option.
However, you can define and upload a custom list of stopwords that override the default list. Discovery applies the appropriate default or custom stopword list to your private collections, based on the language specified for that collection. For more information, see [Defining Stopwords](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsstopwords).





 What is a query expansion list? 

Query expansion terms are usually synonyms, antonyms, or typical misspellings for common terms. You can expand the scope of a query beyond exact matches - for example, you can expand a query for "ibm" to include "international business machines" and "big blue" - by uploading a list of query expansion terms. For more information, see [Query expansion](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsquery-expansion).





 What is the length of Watson Discovery News query results? 

IBM Watson™ Discovery News queries display approximately 50 words from each article in the text JSON field. For more information, see [Watson Discovery News](https://cloud.ibm.com/docs/services/discovery?topic=discovery-watson-discovery-news).





 How do you query Watson Discovery News using the API? 

For information about querying a collection via the API, see [API Reference](https://cloud.ibm.com/apidocs/discoveryquery-your-collection). The collection_id of the English language version of Watson Discovery News is news-en. The collection_id of the Korean collection is news-ko, the Spanish collection_id is news-es, the German collection_id is news-de, the French collection_id is news-fr, and the Japanese collection_id is news-ja. For more information, see [Querying Watson Discovery News](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsquerying-news).





 Can I upload JSON arrays? 

You can upload a JSON array, but each section must be uploaded individually. For example, the following JSON cannot be uploaded to the service:

[{
Improving result relevance with the tooling 

The relevance of natural language query results can be improved in IBM Watson™ Discovery with training. You can train your private collections using either the Discovery tooling, or the Discovery APIs. See [Improving the relevance of your query results with the API](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-api) if you would prefer to use the APIs.

Relevancy training is optional; if the results of your queries meet your needs, no further training is necessary. For information about use cases for relevancy training, see [Improve your natural language query results from Watson Discovery](https://developer.ibm.com/blogs/improving-your-natural-language-query-results-from-watson-discovery/).

To train Watson, you must provide the following:



* Example queries that are representative of the queries your users enter
* Ratings that indicate which results for each query are relevant and not relevant



After Watson has enough training input, the information that you provide about which results are good and bad for each query is used to learn about your collection. Watson does not just memorize, but it also learns from the specific information about individual queries and applies the patterns it detects to all new queries. It does so, using machine-learning Watson techniques that find signals in your content and questions. After training, Discovery then reorders the query results to display the most relevant results at the top. As you add more and more training data, Discovery becomes more accurate in the ordering of query results.

Relevance training currently applies only to natural language queries in private collections. It is not intended for use with structured Discovery Query Language queries. For more about the Discovery Query Language, see [Query concepts](https://cloud.ibm.com/docs/discovery?topic=discovery-query-concepts).

All private collections return a confidence score in the query results in most cases. For more information, see [Confidence scores](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-toolingconfidence).
index("myfield", "this is a string");
index("myfield", 123);

The function that is contained in the index field is a JavaScript function that is called for each document in the database. The function takes the document as a parameter, extracts some data from it, and then calls the function that is defined in the index field to index that data.

The index function takes three parameters, where the third parameter is optional.

The first parameter is the name of the field that you intend to use when querying the index, which is specified in the Lucene syntax portion of later queries. An example appears in the following query:

query=color:red

The Lucene field name color is the first parameter of the index function.

The query parameter can be abbreviated to q, so another way of writing the query is shown in the following example.

q=color:red

If the special value "default" is used when you define the name, you don't have to specify a field name at query time. The effect is that the query can be simplified:

query=red

The second parameter is the data to be indexed. Keep the following information in mind when you index your data:



* This data must be only a string, number, or boolean. Other types return an error from the index function call.
* If an error is returned when your function is running, for this reason or others, the document isn't added to that search index.



The third, optional, parameter is a JavaScript object with the following fields:



Table 1. Fields for the JavaScript object (optional parameter)

 Option Description Values Default 

 boost A number that specifies the relevance in search results. Content that is indexed with a boost value greater than 1 is more relevant than content that is indexed without a boost value. Content with a boost value less than one isn't so relevant. A positive floating point number 1 (No boosting) 
 facet Creates a faceted index. For more information, see [Faceting](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-cloudant-searchfaceting). true false
response.output.entities:value::soda

Similarly, you can filter on intents or entities sent as part of the request, as in this example:

request.input.intents:intent::hello





 Filtering by other fields 

To filter on another field in the log data, specify the location as a path identifying the levels of nested objects in the JSON response from the /logs API. Use dots (.) to specify successive levels of nesting in the JSON data. For example, the location request.input.text idenfities the user input text field as shown in the following JSON fragment:

"request": {
"input": {
"text": "Good morning"
}
}

Filtering is not available for all fields. You can filter on the following fields:



* assistant_id
* customer_id
* language
* request.context.global.system.user_id
* request.input.text
* request_timestamp
* response.context.global.system.user_id
* response.output.entities
* response.output.intents
* response_timestamp
* skill_id
* snapshot



Filtering on other fields is not currently supported.







 Examples 

The following examples illustrate various types of queries using this syntax.



 Description Query 

 The date of the response is in the month of July 2020. response_timestamp>=2020-07-01,response_timestamp<2020-08-01 
 The timestamp of the response is earlier than 2019-11-01T04:00:00.000Z. response_timestamp<2019-11-01T04:00:00.000Z 
 The message is labeled with the customer ID my_id. customer_id::my_id 
 The message was sent to a specific assistant. assistant_id::dcd5c5ad-f3a1-4345-89c5-708b0b5ff4f7 
 The user input text contains the word "order" or a grammatical variant (for example, orders or ordering. request.input.text:order
For more information about session variables, see [Defining session variables](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-search-addactions-variables-global).
3. Set the value of the variable by using an expression that looks like this: <? input.text ?>.



This expression captures the complete message that was submitted by the customer. As a result, your variable captures the customer message that triggered this action.



1. Add the session variable to the Custom query field (for example, ${original_message}).



* Custom results filter: Add a text string that defines information that must be present in any of the search results that are returned.

You are effectively defining the value that is used by the Discovery API as the filter parameter. For more information, see [Query parameters](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersfilter).

The syntax to use for the filter value is not intuitive. Here are a few examples of common use cases:



* To indicate that you want to return only documents with positive sentiment, for example, specify enriched_text.sentiment.document.label:positive.
* To filter results to include only documents that mention Boston, MA, specify enriched_text.entities.text:"Boston, MA".
* To filter results to include only documents that mention a city name that you saved in a context variable named $destination, you can specify enriched_text.entities.text:$destination.





If you add both a query and a filter value, the filter parameter is applied first to filter the data collection documents and cache the results. The query parameter then ranks the cached results.
3. If you want the search for an answer to be the last step in the action, select End the action after returning results.
4. Click Apply.







 Test the search integration 

After you configure search, you can send test queries to see the search results that get returned from Discovery by using the Preview page.

To test the full experience that customers have when they ask questions that are either answered by the action or trigger a search, use the Preview for your assistant.
If the field isn't present or has a value of something else than true, the user didn't complete the registration.

Using the emit function in a view function makes it easy to produce a list in response to running a query by using the view. The list consists of key and value pairs, where the key helps you identify the specific document and the value provides just the precise detail you want. The list also includes metadata such as the number of key:value pairs returned.

The document _id is automatically included in each of the key:value pair result records. The document _id is included to make it easier for the client to work with the results.

See an example of a simple view by using a map function:

function(user) {
if(user.email_verified === true) {
emit(user.email, {name: user.name, email_verified: user.email_verified, joined: user.joined});
}
}

See sample data for demonstrating the simple view example:

[
{
"_id":"abc123",
"name": "Bob Smith",
"email": "bob.smith@aol.com",
"email_verified": true,
"joined": "2019-01-24T10:42:59.000Z"
},
{
"_id":"abc125",
"name": "Amelie Smith",
"email": "amelie.smith@aol.com",
"email_verified": true,
"joined": "2020-04-24T10:42:59.000Z"
}
]
Show more

See an example response from running the simple view query:

{
"total_rows": 2,
"offset": 0,
"rows": [
{
"id": "abc125",
"key": "amelie.smith@aol.com",
"value": {
"name": "Amelie Smith",
"email_verified": true,
Curations API 

The Curations feature is beta functionality.

Use curations to specify the exact document to return in response to a specific natural language query. Curations can guarantee that frequent or important questions always return the most valuable document. The confidence_score for a curated query is always 1.00000.

This beta feature is only available from the API and is applied only to natural language queries, not queries that are specified by using the Discovery Query Language. Beta features are not available from the SDKs.

You can define up to 1,000 curations. For more information, see [Create curation](https://cloud.ibm.com/apidocs/discovery-datacreatecuration) in the API reference.

This example shows how a curation is added with the API. When querying with the same or similar natural_language_query the document with the document_id of document_id1234 is returned.

{
"natural_language_query": "curations in watson discovery",
"curated_results": [
{
"document_id": "document_id1234",
"collection_id": "collection_id1234"
}
]
}

The natural language query that is submitted by the customer must be an exact match for the query that is specified in the curation. Both queries, the one submitted by the user at run time and the one that is submitted by the curation API and then stored in the index, undergo query analysis. The query analyzer lemmatizes text, removes stop words, and adds query expansions.

You can optionally specify a hard-coded response to the query by including a snippet. A snippet is a response that you author and that is returned when the associated document is returned for the specified natural language query.

{
"curations": [
{
"curation_id": "c1175536f509405bc68a9f76235fa7bbb6f9af2f",
"natural_language_query": "What is a project",
"curated_results":
{
For each search result, the title, body, and url properties include content returned from the Discovery query. The search skill configuration determines which fields in the Discovery collection are mapped to these fields in the response. Your application can use these fields to display the results to the user (for example, you might use the body text to show an abstract or description of the matching document, and the url value to create a link the user can click to open the document).

In addition, the header property provides a message to display to the user about the results of the search. In the case of a successful search, header provides introductory text to be displayed before the search results (for example, I found the following information that might be helpful.). Different message text indicates that the search did not return any results, or that the connection to the Discovery service failed. You can customize these messages in the search skill configuration.

For more information about search skills, see [Creating a search skill](https://cloud.ibm.com/docs/assistant?topic=assistant-skill-search-add).





 User-defined 

A user-defined response type can contain up to 5000 KB of data to support a type of a response you have implemented in your client. For example, you might define a user-defined response type to display a special color-coded card, or to format data in a table or graphic.

The user_defined property of the response is an object that can contain any valid JSON data:

{
"output": {
"generic":[
{
"response_type": "user_defined",
"user_defined": {
"field_1": "String value",
"array_1":
1,
2
],
"object_1": {
"property_1": "Another string value"
}
}
}
]
},
"user_id": "faf4a112-f09f-4a95-a0be-43c496e6ac9a"
}
Show more

Your application can parse and display the data in any way you choose.
"clientCountryName": "US"
}
]
}





 Query for the next page using filter 

To get the next n results, specify a filter to exclude the last result from the previous query. Using the previous example, you can append the greater-than operator (_gt) to the clientCountryName field and the greater-or-equal operator to the datetime field. By making a specific order, you can get the most complete results.

firewallEventsAdaptive (limit: 2, orderBy: [datetime_ASC, clientCountryName_ASC], filter: {date_geq: "2018-11-12T00:00:00Z", clientCounterName_gt: "US"}) {
date
clientCountryName
}

The query response follows:

{
"firewallEventsAdaptive" : [
{
"datetime": "2018-11-12T00:00:00Z",
"clientCountryName": "UY"
},
{
"datetime": "2018-11-12T00:00:00Z",
"clientCountryName": "UZ"
}
]
}





 Query the previous page 

To get the previous n results, reverse the filters and sort order.

firewallEventsAdaptive (limit: 2, orderBy: [datetime_DESC, clientCountryName_DESC, filter: {date_leq: "2018-11-12T00:00:00Z", clientCountryName_lt: "UY"}]) {
datetime
clientCountryName
}

The query response follows:

{
"firewallEventsAdaptive" : [
{
"datetime": "2018-11-12T00:00:00Z",
"clientCountryName": "US"
},
{
"datetime": "2018-11-12T00:00:00Z",
* Search response type: If you add a search skill response type to a dialog node, then your assistant retrieves a passage from an external data source and returns it as the response to a particular question. This type of search occurs only when the individual dialog node is processed.

This approach is useful if you want to narrow down a user query before you trigger a search. For example, the dialog branch might collect information about the type of device the customer wants to buy. When you know the make and model, you can then send a model keyword in the query that is submitted to the search skill, and get better results.
* Search skill only: If only a search skill is linked to an assistant, and no dialog skill is linked to the assistant, then a search query is sent to the Discovery service when any user input is received from one of the assistant's integration channels.







 Test the search skill 

After you configure the search, you can send test queries to see the search results that get returned from Discovery by using the Try it out pane of the search skill.

You cannot test the full end-to-end user experience from the dialog skill's Try it out pane. The search skill is configured separately and attached to an assistant. The dialog skill has no way of knowing the details of the search, and therefore cannot show search results in its Try it out pane.

To test the full experience that customers will have when they ask questions that are either answered by the dialog or trigger a search, you must test it by using the API.



1. From the IBM Cloud Pak for Data web client, go to the details page for the provisioned instance.
2. Copy the URL from the "Access information" section of the page. You will specify this value as the {url}.
3. Copy the bearer token also. You will need to pass the token when you make an API call.
4. From the dialog builder in the user interface, add a search skill response type to a dialog node.
5. Make a note of the unique ID of the assistant to which you added the dialog that you edited in the previous step.