vania-janet/multiturn-rag-retrieval-data huggingface dataset:
To do so using Smart Document Understanding, open the Enrich Fields tab, choose the field(s) you'd like to enrich and select from the list of available Natural Language Understanding enrichments:



 Entity extraction 

Returns items such as persons, places, and organizations that are present in the input text. Entity extraction adds semantic knowledge to content to help understand the subject and context of the text that is being analyzed. The entity extraction techniques are based on sophisticated statistical algorithms and natural language processing technology, and are unique in the industry with their support for multilingual analysis and context-sensitive disambiguation. For a complete list of entity type systems, see [Entity type systems](https://cloud.ibm.com/docs/discovery?topic=discovery-entity-type-systems), or for a list of entity types and subtypes that are used in the Version 1 and Version 2 entity type systems, see [Entity types (Version 1)](https://cloud.ibm.com/docs/discovery?topic=discovery-entity-types-version-1) or [Entity types (Version 2)](https://cloud.ibm.com/docs/discovery?topic=discovery-entity-types-version-2). You can also create and add a [custom entity model](https://cloud.ibm.com/docs/discovery?topic=discovery-configservicecustom-entity-model) with IBM Watson™ Knowledge Studio.

Example portion of a document enriched with Entity Extraction:

{
"text": "The stockholders were pleased that Acme Corporation plans to build a new factory in Atlanta, Georgia.",
"enriched_text": {
"entities": [
{
"count": 1,
"sentiment": {
"score": 0
},
"text": "Acme Corporation",
"relevance": 0.98389,
"type": "Company"
},
{
"count": 1,
"sentiment": {
"score": 0
},
"text": "Atlanta",
"relevance": 0.532754,
* When a pattern entity starts or ends with a Unicode character, and includes word boundaries, for example bšb, the pattern match does not match the word boundary correctly. In this example, for input š zkouška, the match returns Group 0: 6-7 š (š zkouška), instead of the correct Group 0: 0-1 š (šzkouška).

The regular expression engine is loosely based on the Java regular expression engine. You will see an error if you try to upload an unsupported pattern, either by using the API or from within the Watson Assistant user interface.



For example, for entity ContactInfo, the patterns for phone, email, and website values can be defined as follows:



* Phone



* localPhone: (d{3})-(d{4}), e.g. 426-4968
* fullUSphone: (d{3})-(d{3})-(d{4}), e.g. 800-426-4968
* internationalPhone: ^((?+?[0-9])?)?[0-9_- ()]$, e.g., +44 1962 815000



* email: b[A-Za-z0-9._%+-]+@([A-Za-z0-9-]+.)+[A-Za-z]{2,}b, e.g. [name@ibm.com](mailto:name@ibm.com)
* website: (https?://)?([da-z.-]+).([a-z.]{2,6})([/w .-])/?$, e.g. [https://www.ibm.com](https://www.ibm.com)



3. Click Add value and repeat the process to add more entity values.



When you use pattern entities to find patterns in user input, you often need a way to store the part of the user input text that matches the pattern. To do so, you can use a context variable.
Although us is not included in the stop words list, it is lemmatized to we, which is listed as a stop word.

You can edit the stop words that are used by your collection. However, you can only augment the stop words list; you cannot remove stop words. And the stop words that you define are used only at query time. They do not affect the stop word list that is used by Discovery when data is added to a collection and the index is created.

For more information, see [Identifying words to ignore](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-stopwords).





 Results have too much text 

If the source document is large, consider splitting the document into smaller chunks.

To do so, you can create a Smart Document Understanding user-trained model. Find content in the document that can be used to consistently break your document into subsections. For example, maybe your document has chapters or subtitles. You can label the chapters with a custom label named chapter. After you teach the model to recognize the chapter content type, apply the model to your entire collection. For more information, see [Using Smart Document Understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fields).

You can then split the document by the chapter field to create many subdocuments segmented by chapter. For more information, see [Split documents to make query results more succinct](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-split-documents).





 Information from tables is not found 

The table understanding enrichment must be applied to your collection for information from tables to be searchable. The table understanding enrichment is applied to collections automatically in some situations. If it isn't and your collection has an HTML field in its index, you can apply the table understanding enrichment yourself.

For more information, see [Understanding tables](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-understanding_tables).





 Information from diagrams is not represented
JavaScript detections 

IBM Cloud® Internet Services (CIS) bot features include JavaScript detections. A small amount of JavaScript is injected into client devices using [Google’s Picasso fingerprinting technique](https://research.google/pubs/pub45581/). Picasso results are factored into bot scores and help CIS classify traffic as automated or human. BotScoreSrc: Not Computed and a score of 0 are relevant to Picasso JavaScript Fingerprinting requests. These requests are exempt from being blocked by any firewall rules.

This detection technique gathers general data about the machines reaching CIS. For example, if a particular user is accessing CIS via Google Chrome on a MacBook Pro, because there are millions of people using Google Chrome on a MacBook Pro, CIS cannot identify specific individuals. CIS also takes steps to anonymize and phase out data for added privacy.

JavaScript is only injected in response to requests for HTML pages or page views, excluding AJAX calls. API and mobile app traffic is unaffected. Additionally, code is not injected again until its 30-minute session life expires. The Picasso script is roughly 70 KB and execution time varies by device, anywhere from 90 ms to around 500 ms.

The snippets of JavaScript will contain a source pointing to the challenge platform with paths that start with /cdn-cgi/challenge-platform/....



 Enable JavaScript detections 

For no-cost Trial plans (Bot Fight Mode), JavaScript detections are automatically enabled and cannot be disabled.

For all other plans (Super Bot Fight Mode and Bot Management for Enterprise), JavaScript detections are optional. To adjust your settings, go to Security > Bots.





 Enforcing execution of JavaScript detections 

To ensure that visitors have previously run and passed JavaScript detections, you can create a firewall rule with the field cf.bot_management.js_detection.passed which performs a Managed Challenge action. Enter (cf.bot_management.js_detection.passed) into the Expression preview section as a singular rule.





 Considerations 



* If you have a Content Security Policy (CSP):



* Ensure that anything under /cdn-cgi/challenge-platform/ is allowed.
If the input is Yes., then the result is false because the user included a period and the expression expects only the exact text, Yes without any punctuation.





 String.equalsIgnoreCase(String) 

This method returns true if the specified string equals the input string, regardless of whether the case of the letters match.

Input: "yes"

This syntax:

{
"conditions": "input.text.equalsIgnoreCase('Yes')"
}

Results: The condition is true.

If the input is Yes., then the result is false because the user included a period and the expression expects only the text, Yes, in uppercase or lowercase letters without any punctuation.





 String.extract(String regexp, Integer groupIndex) 

This method returns a string from the input that matches the regular expression group pattern that you specify. It returns an empty string if no match is found.

This method is designed to extract matches for different regex pattern groups, not different matches for a single regex pattern. To find different matches, see the [getMatch](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-methodsdialog-methods-strings-getMatch) method.

In this example, the context variable is saving a string that matches the regex pattern group that you specify. In the expression, two regex patterns groups are defined, each one enclosed in parentheses. There is an inherent third group that is comprised of the two groups together. This is the first (groupIndex 0) regex group; it matches with a string that contains the full number group and text group together. The second regex group (groupIndex 1) matches with the first occurrence of a number group. The third group (groupIndex 2) matches with the first occurrence of a text group after a number group.

{
"context": {
"number_extract": "<? input.text.extract('([\d]+)(\b [A-Za-z]+)',n) ?>"
}
}
For example, you can look for mentions of an email address by looking for occurrences of the pattern {word}+@+{word}+.com. Or, you might have product order numbers that follow a very specific format, such as TWEX3433JKL. You can create a pattern to look for strings with that syntax in the user utterance.

To add an entity that recognizes a pattern:



1. Follow the standard procedure to create a dictionary-based entity, but select Patterns from the Type drop-down menu instead of Synonyms.

![Shows picking the Patterns type when creating an entity](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/define-entity.png)
2. Add a regular expression that defines the pattern you want to look for.



* For each entity value, there can be a maximum of up to 5 patterns.
* Each pattern (regular expression) is limited to 512 characters.



![Screen capture of defining a pattern entity](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/patternents1.png)

Follow these syntax rules:



* Entity patterns may not contain:



* Positive repetitions (for example x+)
* Backreferences (for example g1)
* Conditional branches (for example (?(cond)true))



* When a pattern entity starts or ends with a Unicode character, and includes word boundaries, for example bšb, the pattern match does not match the word boundary correctly. In this example, for input š zkouška, the match returns Group 0: 6-7 š (š zkouška), instead of the correct Group 0: 0-1 š (šzkouška).

The regular expression engine is loosely based on the Java regular expression engine.
If you want the crawler to extract text from images on the site, expand More processing settings, and set Apply optical character recognition (OCR) to On.

When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).
12. Click Finish.



The collection is created quickly. It takes more time for the data to be processed as it is added to the collection.

If you want to check the progress, go to the Activity page. From the navigation pane, click Manage collections, and then click to open the collection.
input.text.getMatch('(b [A-Za-z]+)',2) ?>

For the same user input, this expression returns and.





 String.isEmpty() 

This method returns true if the string is an empty string, but not null.

For this Dialog runtime context:

{
"context": {
"my_text_variable": ""
}
}

This syntax:

{
"conditions": "$my_text_variable.isEmpty()"
}

Results: The condition is true.





 String.length() 

This method returns the character length of the string.

For this input:

"Hello"

This syntax:

{
"context": {
"input_length": "<? input.text.length() ?>"
}
}

Results in this output:

{
"context": {
"input_length": 5
}
}





 String.matches(String regexp) 

This method returns true if the string matches the input regular expression.

For this input:

"Hello".

This syntax:

{
"conditions": "input.text.matches('^Hello$')"
}

Result: The condition is true because the input text matches the regular expression ^Hello$.





 String.startsWith(String) 

This method returns true if the string starts with the input substring.

For this input:

"What is your name?".

This syntax:

{
"conditions": "input.text.startsWith('What')"
}

Results: The condition is true.





 String.substring(Integer beginIndex, Integer endIndex) 

This method gets a substring with the character at beginIndex and the last character set to index before endIndex. The endIndex character is not included.

For this Dialog runtime context:

{
"context": {
curl -X POST -u "apikey:{apikey}" --header "Content-Type: application/json" --header "Accept: audio/wav" --output disabled-interjections.wav --data "{"text":"<say-as interpret-as='interjection' enabled='false'>Oh</say-as>, in addition, the <say-as interpret-as='interjection' enabled='false'>aha</say-as> wasp is endemic to Australia."}" "{url}/v1/synthesize?voice=en-US_AllisonExpesssive"

IBM Cloud Pak for Data

curl -X POST --header "Authorization: Bearer {token}" --header "Content-Type: application/json" --header "Accept: audio/wav" --output disabled-interjections.wav --data "{"text":"<say-as interpret-as='interjection' enabled='false'>Oh</say-as>, in addition, the <say-as interpret-as='interjection' enabled='false'>aha</say-as> wasp is endemic to Australia."}" "{url}/v1/synthesize?voice=en-US_AllisonExpesssive"





 Usage notes for interjections 

Keep the following in mind when using interjections:



* The <say-as> element with the interpret-as="interjection" attribute has no effect if the word to be modified is not one of the interjections aha or oh.
* Including punctuation with the text that is to be modified can affect whether the interjection is enabled or disabled. Do not include any punctuation characters with the word inside of the <say-as> tag.
* The word oh, when used colloquially to indicate the number zero, is never treated as an interjection. For example, the word is not emphasized as an interjection in the following sentence: The number is one oh three.
* The neutral style does not emphasize interjections to the extent of the default expressive voice or the other styles.
An error can occur, for example, if you add a custom word with an invalid sounds_like field, one that violates one of the rules for adding a pronunciation. You cannot train a custom model whose words resource includes a word with an error. You must correct or delete the word before you can train the model.



 List all custom words example 

The following example lists all of the words, regardless of type, from the custom model with the specified customization ID. The words are displayed in the default sort order, ascending alphabetical.

IBM Cloud

curl -X GET -u "apikey:{apikey}" "{url}/v1/customizations/{customization_id}/words"

IBM Cloud Pak for Data

curl -X GET --header "Authorization: Bearer {token}" "{url}/v1/customizations/{customization_id}/words"

The words resource for the model contains four words. The first word was added directly by the user, but its sounds_like field contains an error: The field cannot contain numbers. The other words were added by the user or by both the user and from corpora, which indicates that the words were added to a custom model that is based on a previous-generation model.

{
"words": [
{
"word": "75.00",
"sounds_like": "75 dollars"],
"display_as": "75.00",
"count": 1,
"source": "user"],
"error": {"75 dollars": "Numbers are not allowed in sounds_like. You can try for example 'seventy five dollars'."}]
},
{
"word": "HHonors",
"sounds_like":
"hilton honors",
"H. honors"
],
"display_as": "HHonors",
"count": 1,
"source":
"corpus1",
"user"
]
},
{
"word": "IEEE",
The characteristic words view is a word cloud that shows terms that are mentioned frequently in the documents you are analyzing.

You can click a word from the word cloud to add it to the existing query and filter the current document set to include only documents that also mention the specified word.

To find significant terms, complete the following steps:



1. From the search results page in guided mode, click Analyze cause or characteristics.

The characteristic words view is displayed.

Zoom

![Facet word cloud](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/cm-word-cloud.png)

Figure 6. Characteristic word cloud

The different font colors help to distinguish the words from one another; they have no statistical meaning.
2. Click a word in the cloud to limit the document set to include only documents that mention the word.







 Analyze relationships between two facets 

Use Pairs analysis to see how two facets are related to one another.

To compare two facets, complete the following steps:



1. From the Facet analysis pane, select Pairs.
2. Find the first facet that you want to compare in the list. Click either the X- or Y-axis icon that is associated with the facet to indicate where you want the facet values to be displayed in a two-dimensional graph.
3. Find the second facet, and then click the remaining axis icon. For example, if you selected the X-axis icon previously, select the Y-axis icon for the second facet.

Data from the two facets is displayed in a graph.

Zoom

![Facet pair comparison graph](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/cm-facet-pair.png)

Figure 7. Facet comparison graph
* text-to-speech.custom-voice-word-list.read -> text-to-speech.custom-model-word-list.read
* text-to-speech.custom-voice-word.read -> text-to-speech.custom-model-word.read



Update eventDeprecated action name -> New action name



* text-to-speech.custom-voice.update -> text-to-speech.custom-model.update



Delete eventsDeprecated action name -> New action name



* text-to-speech.custom-voice.delete -> text-to-speech.custom-model.delete
* text-to-speech.custom-voice-word.delete -> text-to-speech.custom-model-word.delete







 2 December 2020 

Multiple voice improvements
: The voices offered by the service have undergone significant change. The service supports new languages and voices, has improved the quality of many voices, and has deprecated many older voices. In addition, all of the service's voices are now customizable and generally available (GA) for production use.

New neural and enhanced neural voices
: To optimize the overall quality of voice synthesis, all available voices are now based on neural technology. The service offers two types of voices that are based on neural technology:



* Neural voices, which do not include the string V3 in their names, are now available for Arabic, Australian English, Chinese, Netherlands Dutch, and Korean. Neural voices support the use of the International Phonetic Alphabet (IPA) with the Speech Synthesis Markup Language (SSML) <phoneme> element.
* Enhanced neural voices, which include the string V3 in their names, are now available for Brazilian Portuguese, United Kingdom and United States English, French, German, Italian, Japanese, and Spanish (all dialects). Enhanced neural voices support the use of both IPA and IBM Symbolic Phonetic Representation (SPR) with the SSML <phoneme> element. Enhanced neural voices also achieve a slightly higher degree of natural-sounding speech. And further customization capabilities will be exposed for enhanced neural voices in the coming months.



The service no longer offers standard voices for any language.
For more information about using capture groups in your dialog, see [Storing and recognizing entity pattern groups in input](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-tipsdialog-tips-get-pattern-groups).







 How fuzzy matching works 

Fuzzy matching is available for languages noted in the [Supported languages](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-language-support) topic.

Fuzzy matching has these components:



* Stemming - The feature recognizes the stem form of entity values that have several grammatical forms. For example, the stem of bananas would be banana, while the stem of running would be run.
* Misspelling - The feature is able to map user input to the appropriate corresponding entity despite the presence of misspellings or slight syntactical differences. For example, if you define giraffe as a synonym for an animal entity, and the user input contains the terms giraffes or girafe, the fuzzy match is able to map the term to the animal entity correctly.
* Partial match - With partial matching, the feature automatically suggests substring-based synonyms present in the user-defined entities, and assigns a lower confidence score as compared to the exact entity match.



For English, fuzzy matching prevents the capturing of some common, valid English words as fuzzy matches for a given entity. This feature uses standard English dictionary words. You can also define an English entity value/synonym, and fuzzy matching will match only your defined entity value/synonym. For example, fuzzy matching may match the term unsure with insurance; but if you have unsure defined as a value/synonym for an entity like @option, then unsure will always be matched to @option, and not to insurance.

Your fuzzy matching setting has no impact on synonym recommendations. Even if fuzzy matching is enabled, synonyms are suggested for the exact value you specify only, not the value and slight variations of the value.
Regex change
: The regular expressions that are supported by SpEL functions like find, matches, extract, replaceFirst, replaceAll and split have changed. A group of regular expression constructs are no longer allowed, including look-ahead, look-behind, possessive repetition and backreference constructs. This change was necessary to avoid a security exposure in the original regular expression library.





 12 June 2017 

Updates
: This release includes the following updates:



* The maximum number of workspaces that you can create with the Lite plan (formerly named the Free plan) changed from 3 to 5.
* You can now assign any name to a dialog node; it does not need to be unique. And you can subsequently change the node name without impacting how the node is referenced internally. The name you specify is treated as an alias and the system uses its own internal identifier to reference the node.
* You can no longer change the language of a workspace after you create it by editing the workspace details. If you need to change the language, you can export the workspace as a JSON file, update the language property, and then import the JSON file as a new workspace.







 6 June 2017 

Learn
: A new Learn about IBM Watson® Assistant page is available that provides getting started information and links to service documentation and other useful resources. To open the page, click the icon in the page header.

Bulk export and delete
: You can now simultaneously export a number of intents or entities to a CSV file, so you can then import and reuse them for another Watson Assistant application. You can also simultaneously select a number of entities or intents for deletion in bulk.

Updates to Korean
: Korean tokenizers have been updated to address informal language support. IBM continues to work on improvements to entity recognition and classification.

Emoji support
: Emojis added to intent examples, or as entity values, will now be correctly classified/extracted.

Only emojis that are included in your training data will be correctly and consistently identified; emoji support may not correctly classify similar emojis with different color tones or other variations.

Entity stemming (Beta - English only)