vania-janet/multiturn-rag-retrieval-data huggingface dataset:
See [Integrating with Watson Knowledge Studio](https://cloud.ibm.com/docs/discovery?topic=discovery-integrating-with-wks) for more information.







 keywords 

The keywords enrichment extracts instances of significant words within the text. To understand the difference between keywords, concepts, and entities, see [Understanding the difference between Entities, Concepts, and Keywords](https://cloud.ibm.com/docs/discovery?topic=discovery-configserviceudbeck).



* "sentiment" : boolean - optional - When true, sentiment analysis is performed on the extracted keyword in the context of the surrounding content.
* "emotion" : boolean - optional - When true, emotional tone analysis is performed on the extracted keyword in the context of the surrounding content.
* "limit" : INT - optional - The maximum number of keywords to extract from the ingested document. The default is 50.







 semantic_roles 

The semantic_roles enrichment identifies sentence components such as subject, action, and object within the ingested text.



* "entities" : boolean - optional When true, entities are extracted from the sentence components.
* "keywords" : boolean - optional When true, keywords are extracted from the sentence components.
* "limit" : INT - optional - The maximum number of semantic_roles objects to extract (sentences to parse) from the ingested document. The default is 50.







 sentiment 

The sentiment enrichment evaluates the overall sentiment level of entire document or specified target strings in the entire document.



* "document" : booleanoptional - When true the sentiment of the entire document is evaluated.
* "targets" : arrayoptional - A comma-separated array of target strings to evaluate sentiment of within the document.







 relations 

The relations enrichment extracts known relationships between identified entities within the document. Optionally, a Knowledge Studio custom model can be specified to extract custom relationships.



* "model" : string - optional - When specified, the custom model is used to extract relations instead of the public model. This option requires a Knowledge Studio custom model to be associated with your instance of Discovery.
A hypothesis is returned only if its confidence is greater than or equal to the specified threshold.

You can think of word alternatives as the timeline for a transcript that is chopped into smaller intervals, or bins. Each bin can have one or more hypotheses with different spellings and confidence. The word_alternatives_threshold parameter controls the density of the results that the service returns. Specifying a small threshold can potentially produce many hypotheses.



 Word alternatives results 

The service returns the results in a word_alternatives field that is an element of the results array. The word_alternatives field is an array of objects, each of which provides the following fields for an alternative word:



* start_time indicates the time in seconds at which the word for which hypotheses are returned starts in the input audio.
* end_time indicates the time in seconds at which the word for which hypotheses are returned ends in the input audio.
* alternatives is an array of hypothesis objects. Each object includes a confidence that indicates the service's confidence score for the hypothesis and a word that identifies the hypothesis. The confidence must be at least as great as the specified threshold to be included in the results. The service orders the alternatives by confidence.







 Word alternatives example 

The following example request specifies a rather low word_alternatives_threshold of 0.10. The simple input audio includes words with common homonyms and different possible acoustic interpretations.

IBM Cloud

curl -X POST -u "apikey:{apikey}" --header "Content-Type: audio/wav" --data-binary @{path}audio-file.wav "{url}/v1/recognize?word_alternatives_threshold=0.10"

IBM Cloud Pak for Data

curl -X POST --header "Authorization: Bearer {token}" --header "Content-Type: audio/wav" --data-binary @{path}audio-file.wav "{url}/v1/recognize?word_alternatives_threshold=0.10"

Because of the low threshold, the service returns multiple hypotheses and confidence scores for some of the words, along with the start and end times of all words.
For more information about defining entities, see [Defining information to look for in customer input](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-entities).

Pattern entities do not prevent spelling autocorrection
: Pattern entities that match all characters and words that are usually used to count input words do not prevent spelling autocorrection. For example, if a customer defines the ^..{0,19}$ pattern entity that matches the first 20 characters of an input, then the entity match does not affect spelling autocorrection. In this example, an input of cancl transaction is autocorrected to cancel transaction.

This change applies to the following languages: English and French. For more information, see [Correcting user input](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-runtime-spell-checkdialog-runtime-spell-check-rules).

Fuzzy matching updates
: Previously, an update was made so that interactions between the stemming and misspelling fuzzy matching features were not allowed. This change applied to the following languages: English, French, German, and Czech. This was updated so that this change applies only to the English language. For more information, see [How fuzzy matching works](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-entitiesentities-fuzzy-matching).

Improved irrelevance detection for Dutch
: Irrelevance detection for Dutch disregards any punctuation in an input sentence. For example, you can now expect the same confidence score for the following two inputs: ik ben een kleine krijger? and ik ben een kleine krijger. In this example, the question mark (?) doesn't affect the confidence score.

Improved enhanced intent detection
: The exact match in enhanced intent detection now better handles small differences between training examples and runtime utterances when the differences do not change the meaning of a sentence.
For example, for the word barrió, which has an accent and corresponds to the past tense of the verb barrer (to sweep), your assistant can also match the word barrio (neighborhood), but with a slightly lower confidence.

The system will provide the highest confidence scores in entities with exact matches. For example, barrio will not be detected if barrió is in the training set; and barrió will not be detected if barrio is in the training set.

You are expected to train the system with the proper characters and accents. For example, if you are expecting barrió as a response, then you should put barrió into the training set.

Although not an accent mark, the same applies to words using, for example, the Spanish letter ñ vs. the letter n, such as uña vs. una. In this case the letter ñ is not simply an n with an accent; it is a unique, Spanish-specific letter.

You can enable fuzzy matching if you think your customers will not use the appropriate accents, or misspell words (including, for example, putting a n instead of a ñ), or you can explicitly include them in the training examples.

Note: Accent normalization is enabled for Portuguese, Spanish, French, and Czech.

Workspace opt-out flag
: The Watson Assistant REST API now supports an opt-out flag for workspaces. This flag indicates that workspace training data such as intents and entities are not to be used by IBM for general service improvements. For more information, see the [API Reference](https://cloud.ibm.com/apidocs/assistant/assistant-v1?curl=data-collection)





 7 August 2017 

Next and last date interpretation
: The Watson Assistant service treats last and next dates as referring to the most immediate last or next day referenced, which may be in either the same or a previous week. See the [system entities](https://cloud.ibm.com/docs/assistant?topic=assistant-system-entitiessystem-entities-sys-date-time) topic for additional information.





 3 August 2017 

Fuzzy matching for additional languages (Beta)
For more information about the Fallback action, see [Editing the fallback action](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-handle-errorsfallback-action).





 Show warning 

The second and final step of the Trigger word detected action is the Show warning step. The Show warning step shows a customizable warning message to your customer if any trigger words are detected in the customer's input. Use this step to discourage customers from interacting with your assistant in unacceptable ways, such as using profanity.

For example, you might add darn, dang, and heck as trigger words for the Show warning step:

Zoom

![Adding trigger words to the Show warning step](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/show-warning-phrases.png)

Adding trigger words to the Show warning step

In this example, a customer enters darn, dang, or heck, the assistant responds with Please use appropriate language when interacting with the assistant. You can customize this message.

If the customer triggers the Show warning step again, the Fallback action is triggered. The default setting is if attempts exceed 2 total tries. You can customize this setting.

In the Fallback action, step 5 has:



* Profanity detected as the fallback reason.
* The default message: It seems this conversation would be best managed by a human agent. Let me connect you to one of our agents. You can customize this message.
* And then set to Connect to agent (action ends).



For more information about the Fallback action, see [Editing the fallback action](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-handle-errorsfallback-action).
![Add manual trigger](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/devsecops-pipeline-config-manual-trigger.png)

Figure 1. Manual trigger for Pruning Evidence
2. Select the prune evidence listener for the trigger

In the context of one pipeline, the evidence pruning tool appears as a new Tekton listener for the pipeline. Select prune-evidence-listener.

Zoom

![Prune evidence listener](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/devsecops-triggers-evidence-pruner.png)

Figure 2. One pipeline Tekton listener for pruning evidence
3. Configure the following environment properties for the trigger:



Table 1. Prune evidence trigger parameters

 Name Type Description Required or Optional 

 evidence-retention-days Text The duration days, for which evidence is retained. Default is 365. Required 
 evidence-pruner-batch-size Text The count of evidence files that undergo cleanup before the pull request is updated. Default is 1000. Optional 
 evidence-pruner-branch Text The branch within the evidence locker Git repository that is created to facilitate the cleanup and promotion of evidence within the default branch. Default is chore/prune-evidences. Optional 
 opt-in-v1-evidence-pruner Text When set to 1, all the v1 evidence files are removed. Default is 0. Optional 
 opt-in-v2-evidence-pruner Text When set to 1, all the v2 evidence files that are generated before evidence-retention-days days are removed. Default is 0. Optional 



Zoom

!
(https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/gs-ass-skip-user-input.png)







 Test order cancellations 

Test whether your assistant can recognize character patterns that match the pattern used for product order numbers in user input.



1. Click the ![Try it](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/ask_watson.png) icon to open the Try it out pane.
2. Enter, i want to cancel my order number TW12345.

Your assistant recognizes both the cancel_order intent and the @order_number entity. It responds with, If the pickup time is more than 48 hours from now, you can cancel your order. Just to confirm, you want to cancel order TW12345?
3. Enter, Yes.

Your assistant recognizes the yes intent and responds with, Ok. The order TW12345 is canceled. We hope we get the opportunity to bake a cake for you sometime soon.

![Shows the Try it out pane test of the cancel order number node when the user provides the order number in the initial input.](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/gs-ass-test-cancel-order-number-provided.png)

Now, try it when you don't know the order number.
4. Click Clear in the Try it out pane to start over. Enter, I want to cancel my order.

Your assistant recognizes the cancel_order intent, and responds with, If the pickup time is more than 48 hours from now, you can cancel your order.
var wsURI = '{ws_url}/v1/recognize'
+ '?access_token=' + access_token
+ '&model=es-ES_BroadbandModel';
var websocket = new WebSocket(wsURI);

websocket.onopen = function(evt) { onOpen(evt) };
websocket.onclose = function(evt) { onClose(evt) };
websocket.onmessage = function(evt) { onMessage(evt) };
websocket.onerror = function(evt) { onError(evt) };

The client can open multiple concurrent WebSocket connections to the service. The number of concurrent connections is limited only by the capacity of the service, which generally poses no problems for users.





 Initiate a recognition request 

To initiate a recognition request, the client sends a JSON text message to the service over the established connection. The client must send this message before it sends any audio for transcription. The message must include the action parameter but can usually omit the content-type parameter:

action (required string)
: Specifies the action to be performed:



* start begins a recognition request. It can also specify new parameters for subsequent requests. For more information, see [Send additional requests and modify request parameters](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-websocketsws-more).
* stop signals that all audio for a request has been sent. For more information, see [End a recognition request](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-websocketsws-stop).



content-type (optional string)
: Identifies the format (MIME type) of the audio data for the request. The parameter is required for the audio/alaw, audio/basic, audio/l16, and audio/mulaw formats.
[Boost](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/boost-modal.png)
5. You can keep entering words or phrases one at a time. When you're finished, click Apply.





 Testing recognition of boost words 

You can use Preview to test recognition of boost words. If a boost word or phrases matches, you can see Boost word detected in the Preview pane.

![Boost preview](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/boost-preview.png)
The map function might emit output as the result of processing a specific document. When this situation occurs, the filter considers the document that is allowed and includes it in the list of documents that you changed.

See the following example application of the _view filter by using HTTP:

POST $SERVICE_URL/$DATABASE/_changes?filter=_view&view=$DDOC/$VIEW_NAME HTTP/1.1

See the following example applications of the _view filter:



* Curl
* Java
* Node
* Python
* Go



curl -X POST "$SERVICE_URL/animaldb/_changes?filter=_view&view=views101/latin_name" -H "Content-Type: application/json" -d '{}'

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.ChangesResult;
import com.ibm.cloud.cloudant.v1.model.PostChangesOptions;

Cloudant service = Cloudant.newInstance();

PostChangesOptions changesOptions = new PostChangesOptions.Builder()
.db("animaldb")
.filter("_vew")
.view("views101/latin_name")
.build();

ChangesResult response =
service.postChanges(changesOptions).execute()
.getResult();

System.out.println(response);

const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

service.postChanges({
db: 'animaldb',
filter: '_view',
view: 'views101/latin_name'
}).then(response => {
console.log(response.result);
});

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()
(https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/gs-ass-skip-user-input.png)







 Test order cancellations 

Test whether your assistant can recognize character patterns that match the pattern used for product order numbers in user input.



1. Click the ![Try it](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/ask_watson.png) icon to open the "Try it out" pane.
2. Enter, i want to cancel my order number TW12345.

Your assistant recognizes both the cancel_order intent and the @order_number entity. It responds with, If the pickup time is more than 48 hours from now, you can cancel your order. Just to confirm, you want to cancel order TW12345?
3. Enter, Yes.

Your assistant recognizes the yes intent and responds with, OK. The order TW12345 is canceled. We hope we get the opportunity to bake a cake for you sometime soon.

![Shows the Try it out pane test of the cancel order number node when the user provides the order number in the initial input.](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/gs-ass-test-cancel-order-number-provided.png)

Now, try it when you don't know the order number.
4. Click Clear in the "Try it out" pane to start over. Enter, I want to cancel my order.

Your assistant recognizes the cancel_order intent, and responds with, If the pickup time is more than 48 hours from now, you can cancel your order.
The behavior of this predicate and the already defined NotNull predicate is different from all other predicates that return null on null input.





 MatchesDict 

The MatchesDict function takes a dictionary (as in a dictionary extraction), an optional flag specification, and a span as arguments:

MatchesDict('<dictionary>', ['<flags>', ]<span>)

The MatchesDict function returns TRUE if the span exactly matches one or more of the terms in the dictionary. The flags can be Exact or IgnoreCase.



* If Exact is used, a case-sensitive match is performed against each term in the dictionary.
* If IgnoreCase is used, the match that is performed against each term in the dictionary is not case-sensitive.
* If no flag is specified, the dictionary matches based on any flag that was specified when it was created. If no flag was specified during creation, it matches by using the IgnoreCase flag.



If any argument is null, the function returns null.

Dictionaries are always evaluated on token boundaries. For example, if a dictionary consists of the term fish, no match exists in the text Let's go fishing!.





 MatchesRegex 

The MatchesRegex function has a similar syntax to ContainsRegex. Unlike ContainsRegex function, the MatchesRegex function returns TRUE only if the entire text of the span, which is taken as a separate Java string, matches the regular expression. If any argument is null, the function returns null. The optional flags affect the matching behavior similar to flags used in Java regular expressions.

MatchesRegex(/<regular expression>/, ['<flags>', ]<span>)

The flags string is formed by combining a subset of these flags by using | as the separator:



* CANON_EQ
* CASE_INSENSITIVE
* DOTALL
* LITERAL
* MULTILINE
* UNICODE (meaningless without CASE_INSENSITIVE)
* UNIX_LINES



An example of a flags string is

'UNICODE | CASE_INSENSITIVE'

Consider this example where MatchesRegex is used to identify product names along with their version number mentions to the right.
For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).

After you enable OCR, if you want to remove annotated image text from the collection index, you can label the image so that you can exclude the associated text. You will learn about how to configure the index in the next procedure.

Zoom

![Shows an image in the page.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-image.png)

Figure 19. Shows an image in the page
14. When you reach the Bibliography, create a new label named bibliography.

Zoom

![Shows how to create the bibliography label.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-bibliography.png)

Figure 20. Creating a bibliography label

Apply the new label to each page.

Zoom

![Shows the bibliography label that is being applied to an entire page.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-apply-biblio.png)

Figure 21. A bibliography label is being applied
15. After you annotate and submit all the pages, click Apply changes and reprocess.

A notification is displayed to indicate that the collection was updated. You remain on the SDU tool page, but the Apply changes and reprocess button is disabled.
These words were also added to a custom model that is based on a previous-generation model.

{
"word": "NCAA",
"sounds_like": [
"N. C. A. A.",
"N. C. double A."
],
"display_as": "NCAA",
"count": 3,
"source": [
"corpus3",
"user"
]
}







 Deleting a custom word from a custom language model 

Use the DELETE /v1/customizations/{customization_id}/words/{word_name} method to delete a word from a custom language model.



* For custom models that are based previous-generation models, use the method to remove words that were added in error, for example, from a corpus with faulty data. You can remove any word that you added to the custom model's words resource via any means (for example, from corpora, from grammars, or directly.)
* For custom models that are based on next-generation models, you can delete only words that were added directly to the model. Words are not added to the model from corpora or grammars.



You cannot delete a word from the service's base vocabulary. If you delete a word that is also present in the service's base vocabulary, deleting the word from a custom model deletes only the custom pronunciation for the word. The word remains in the base vocabulary.

Removing a word from a custom model does not affect the model until you retrain it by using the POST /v1/customizations/{customization_id}/train method. If the model was previously trained on the word, the model continues to apply the word to speech recognition even after you delete the word from its words resource. You must retrain the model to reflect the deletion.



 Delete a custom word example 

The following example deletes the word IEEE from the custom model with the specified customization ID:

IBM Cloud