vania-janet/multiturn-rag-retrieval-data huggingface dataset:
You can set a threshold below which such events are ignored.



By default, speech activity detection is configured to provide optimal performance for the general case for each model. For specific cases, the default settings might not be optimal and can lead either to slow transcription or to word insertions and deletions. You are encouraged to experiment with different settings to determine which values work best for your audio.





 Speech detector sensitivity 

Use the speech_detector_sensitivity parameter to adjust the sensitivity of speech activity detection. Use the parameter to suppress word insertions from music, coughing, and other non-speech events. The service biases the audio it passes for speech recognition by evaluating chunks of the input audio against prior models of speech and non-speech activity.

Specify a float value between 0.0 and 1.0. The default value is 0.5, which provides a reasonable compromise for the level of sensitivity. A value of 0.0 suppresses all audio (no speech is transcribed). A value of 1.0 suppresses no audio (speech detection sensitivity is disabled). The values increase on a monotonic curve of sensitivity versus speech. Specifying one or two decimal places of precision (for example, 0.55) is typically more than sufficient.

This parameter can affect both the quality and the latency of speech recognition:



* Lower values can decrease latency because less audio is potentially passed for speech recognition. However, a low setting might discard chunks of audio that contain actual speech, losing viable content from the transcript.
* Higher values can increase latency because more audio is potentially passed for speech recognition. However, a high setting might pass chunks of audio that contain non-speech events, adding spurious content to the transcript.





 Speech detector sensitivity example 

The following example request specifies a value of 0.6 for the speech_detector_sensitivity parameter with the synchronous HTTP interface. The service recognizes slightly more potential non-speech events than it would by default.

IBM Cloud

curl -X POST -u "apikey:{apikey}" --header "Content-Type: audio/flac" --data-binary @{path}audio-file1.flac "{url}/v1/recognize?speech_detector_sensitivity=0.6"
Removing a word from a custom model does not affect the model until you retrain it by using the POST /v1/customizations/{customization_id}/train method. If the model was previously trained on the word, the model continues to apply the word to speech recognition even after you delete the word from its words resource. You must retrain the model to reflect the deletion.



 Delete a custom word example 

The following example deletes the word IEEE from the custom model with the specified customization ID:

IBM Cloud

curl -X DELETE -u "apikey:{apikey}" "{url}/v1/customizations/{customization_id}/words/IEEE"

IBM Cloud Pak for Data

curl -X DELETE --header "Authorization: Bearer {token}" "{url}/v1/customizations/{customization_id}/words/IEEE"
[Bidi options](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/bidi-options.png)
3. Click the X to close the page. Your changes are saved automatically.







 Working with accented characters 

In a conversational setting, users might or might not use accents while interacting with the Watson Assistant service. As such, both accented and non-accented versions of words might be treated the same for intent detection and entity recognition.

However for some languages, like Spanish, some accents can alter the meaning of the entity. Thus, for entity detection, although the original entity might implicitly have an accent, your assistant can also match the non-accented version of the same entity, but with a slightly lower confidence score.

For example, for the word "barrió", which has an accent and corresponds to the past tense of the verb "barrer" (to sweep), your assistant can also match the word "barrio" (neighborhood), but with a slightly lower confidence.

The system will provide the highest confidence scores in entities with exact matches. For example, barrio will not be detected if barrió is in the training set; and barrió will not be detected if barrio is in the training set.

You are expected to train the system with the proper characters and accents. For example, if you are expecting barrió as a response, then you should put barrió into the training set.

Although not an accent mark, the same applies to words using, for example, the Spanish letter ñ vs. the letter n, such as "uña" vs. "una". In this case the letter ñ is not simply an n with an accent; it is a unique, Spanish-specific letter.

You can enable fuzzy matching if you think your customers will not use the appropriate accents, or misspell words (including, for example, putting a n instead of a ñ), or you can explicitly include them in the training examples.
* To define synonyms that are recognized and tagged when a document is ingested and that can be retrieved by search, create a dictionary and add synonyms for the dictionary term entry. A dictionary defines special terms that you want to tag in your documents, such as product names or industry-specific terminology. You can use the dictionary terms later to create facets and to filter documents. For more information, see [Dictionary](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-dictionary).
* To define synonyms that are applied to the query text that is submitted by users to expand the meaning of the query, add synonyms by using the Synonyms tool on the Improve relevance section of the Improve and customize page. For more information, see [Expanding the meaning of queries](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-search-settingsquery-expansion).



* Can I use Discovery to detect sentiment?

You can use Discovery to detect both phrase and document sentiment. Document sentiment is a built-in Natural Language Processing enrichment that is available for all project types. Document sentiment evaluates the overall sentiment that is expressed in a document to determine whether it is positive, neutral, or negative. Phrase sentiment does the same. However, phrase sentiment can detect and assess multiple opinions in a single document and, in English and Japanese documents, can find specific phrases. For more information about document sentiment, see [Sentiment](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-nlunlu-sentiment). For more information about phrase sentiment, see [Detecting phrases that express sentiment](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-phrase-sentiment). You cannot detect the sentiment of entities or keywords in v2.
* What is a nested field?

When you ingest a file or crawl an external data source, the data that you add to Discovery is processed and added to the collection as a document. Fields from the original file are converted to document fields and are added to the collection's index.
Recognized entities are: <? entities.toJson() ?>



 When placement of entities in the input matters 

When you use the shorthand expression, @city.contains('Boston'), in a condition, the dialog node returns true only ifBoston is the first entity detected in the user input. Only use this syntax if the placement of entities in the input matters to you and you want to check the first mention only.

Use the full SpEL expression if you want the condition to return true any time the term is mentioned in the user input, regardless of the order in which the entities are mentioned. The condition, entities['city']?.contains('Boston') returns true when at least one 'Boston' city entity is found in all the @city entities, regardless of placement.

For example, a user submits "I want to go from Toronto to Boston." The @city:Toronto and @city:Boston entities are detected and are represented in the array that is returned as follows:



* entities.city[0].value = 'Toronto'
* entities.city[1].value = 'Boston'



The order of entities in the array that is returned matches the order in which they are mentioned in the user input.





 Entity properties 

Each entity has a set of properties associated with it. You can access information about an entity through its properties.



 Property Definition Usage tips 

 confidence A decimal percentage that represents your assistant's confidence in the recognized entity. The confidence of an entity is either 0 or 1, unless you have activated fuzzy matching of entities. When fuzzy matching is enabled, the default confidence level threshold is 0.3. Whether or not fuzzy matching is enabled, system entities always have a confidence level of 1.0. You can use this property in a condition to have it return false if the confidence level is not higher than a percent you specify. 
 location A zero-based character offsets that indicates where the detected entity values begin and end in the input text. Use .literal to extract the span of text between start and end index values that are stored in the location property. 
 value The entity value identified in the input.
input.text.extract('[\d]+',0) ?>"
}
}

Important: To process \d as the regular expression, you have to escape both the backslashes by adding another \: \\d

Result:

{
"context": {
"number_extract": "123456"
}
}





 String.find(String regexp) 

This method returns true if any segment of the string matches the input regular expression. You can call this method against a JSONArray or JSONObject element, and it will convert the array or object to a string before making the comparison.

For this input:

"Hello 123456".

This syntax:

{
"conditions": "input.text.find('^[^d][d]{6}[^d]$')"
}

Result: The condition is true because the numeric portion of the input text matches the regular expression ^[^d][d]{6}[^d]$.





 String.isEmpty() 

This method returns true if the string is an empty string, but not null.

For this Dialog runtime context:

{
"context": {
"my_text_variable": ""
}
}

This syntax:

{
"conditions": "$my_text_variable.isEmpty()"
}

Results: The condition is true.





 String.length() 

This method returns the character length of the string.

For this input:

"Hello"

This syntax:

{
"context": {
"input_length": "<? input.text.length() ?>"
}
}

Results in this output:

{
"context": {
"input_length": 5
}
}





 String.matches(String regexp) 

This method returns true if the string matches the input regular expression.

For this input:

"Hello".

This syntax:

{
The interjection attribute is an SSML extension specific to the Text to Speech service. It is supported for use only with the expressive neural voices.

With the expressive neural voices, the service automatically emphasizes the following interjections: aha, hmm, huh, oh, uh, uh-huh, and um. You can use the interjection value to enable or disable the service's emphasis of the interjections aha and oh. Include the additional enabled attribute with a value of true or false to enable or disable the interjection.

The following example disables emphasis of both the aha and oh interjections in the text:

<say-as interpret-as='interjection' enabled='false'>Oh</say-as>, in addition, the <say-as interpret-as='interjection' enabled='false'>aha</say-as> wasp is endemic to Australia.

For more information, see [Emphasizing interjections](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-synthesis-expressivesyntheses-expressive-interjections).





 letters 

The letters value spells out the characters in the word within the element. (The value also pronounces individually any numeric characters that are included in the enclosed string.) The following example spells the letters of the word hello.

<say-as interpret-as="letters">Hello</say-as>





 number 

The number value offers an alternative to the cardinal and ordinal values. You can use the optional format attribute to indicate how a series of numbers is to be interpreted. The first example omits the format attribute to pronounce the number as a cardinal value. The second example explicitly specifies that the number is to be pronounced as a cardinal value. The third example specifies that the number is to be pronounced as an ordinal value.

<say-as interpret-as="number">123456</say-as>
<say-as interpret-as="number" format="cardinal">123456</say-as>
Can I use Discovery to detect sentiment? 

You can use Discovery to detect both phrase and document sentiment. Document sentiment is a built-in Natural Language Processing enrichment that is available for all project types. Document sentiment evaluates the overall sentiment that is expressed in a document to determine whether it is positive, neutral, or negative. Phrase sentiment does the same. However, phrase sentiment can detect and assess multiple opinions in a single document and, in English and Japanese documents, can find specific phrases. For more information about document sentiment, see [Sentiment](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-nlunlu-sentiment). For more information about phrase sentiment, see [Detecting phrases that express sentiment](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-phrase-sentiment). You cannot detect the sentiment of entities or keywords in v2.





 What is a nested field? 

When you ingest a file or crawl an external data source, the data that you add to Discovery is processed and added to the collection as a document. Fields from the original file are converted to document fields and are added to the collection's index. Some content is added to root-level index fields and some information is stored in nested fields. Where data gets stored differs by file type. Most of the fields from structured data sources are stored as root-level fields. For files with unstructured data, much of the body of the file is stored in the text field in the index. Other information, such as the file name, is stored in nested fields with names like extracted_metadata.filename. You can determine whether a field is a nested field by its name. If the field name includes a period, it is a nested field. For more information about how different file types are handled, see [How your data source is processed](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-index-overview).





 Which type of query should I use in my custom app?
If it finds a term that it can match to an existing dictionary entity value or synonym, it adds the term to the list of words that belong to the skill, and does not correct it.

For example, if a user enters a sentence like I wnt to buy a boook, fuzzy matching recognizes that the term boook means the same thing as your entity value book, and adds it to the protected words list. Your assistant corrects the input to be, I want to buy a boook. Notice that it corrects wnt but does not correct the spelling of boook. If you see this type of result when you are testing your dialog, you might think your assistant is misbehaving. However, your assistant is not. Thanks to fuzzy matching, it correctly identifies boook as a @reading_material entity mention. And thanks to autocorrection revising the term to want, your assistant is able to map the input to your buy_something intent. Each feature does its part to help your assistant understand the meaning of the user input.
IBM Cloud

curl -X POST -u "apikey:{apikey}" --header "Content-Type: audio/wav" --data-binary @{path}audio-file.wav "{url}/v1/recognize?&redaction=true&max_alternatives=3&keywords=birth%2Cbirthday&keywords_threshold=0.5"

IBM Cloud Pak for Data

curl -X POST --header "Authorization: Bearer {token}" --header "Content-Type: audio/wav" --data-binary @{path}audio-file.wav "{url}/v1/recognize?&redaction=true&max_alternatives=3&keywords=birth%2Cbirthday&keywords_threshold=0.5"







 Profanity filtering 

The profanity filtering feature is generally available for US English and Japanese only.

The profanity_filter parameter indicates whether the service is to censor profanity from its results. By default, the service obscures all profanity by replacing it with a series of asterisks in the transcript. Setting the parameter to false displays words in the output exactly as transcribed.

The service censors profanity from all final transcripts and from any alternative transcripts. It also censors profanity from results that are associated with word alternatives, word confidence, and word timestamps. The sole exception is keyword spotting, for which the service returns all words as specified by the user, regardless of whether profanity_filter is true.



 Profanity filtering example 

The following example shows the results for a brief audio file that is transcribed with the default true value for the profanity_filter parameter. The request also sets the word_alternatives_threshold parameter to a relatively high value of 0.99 and the word_confidence and timestamps parameters to true.

IBM Cloud
/society/crime/sexual offence /society/crime/sexual offense 
 /society/crime/sexual offence/paedophilia /society/crime/sexual offense/pedophilia 
 /society/crime/sexual offence/prostitution /society/crime/sexual offense/prostitution 
 /society/crime/sexual offence/rape /society/crime/sexual offense/rape 
 /style and fashion/men 's fashion /style and fashion/men's fashion 
 /technology and computing/consumer electronics/ebook reader /technology and computing/consumer electronics/e-book reader 









 8 July 2020 

Arabic keyword support
: Support for Arabic keywords is now available, for all public and premium service instances. For details, see [Language support](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-language-support).





 10 April 2020 

Custom Sentiment (Beta)
: Custom Sentiment (Beta) is now available. The Beta supports English and is available only for service instances in the Dallas location. For details, see [Creating custom sentiment models](https://cloud.ibm.com/docs/services/natural-language-understanding?topic=natural-language-understanding-customizingcustom-sentiment).





 March 2020 



 20 March 2020 

Error response format change for advanced rules model endpoints (Beta)
: Changed the error response format for advanced rules model endpoints (Beta) to be consistent with other endpoints.



* Renamed status to code.
* Renamed message to error.



* Previous format:



{
"status": 404,
"message": "not found"
}



* New format:



{
"code": 404,
"error": "not found"
}







 11 March 2020 

Enhanced sentiment analysis for English
: Enhanced English sentiment analysis to better identify and understand idioms in text.







 December 2019 



 12 December 2019 

Full support for IBM Cloud IAM
: Natural Language Understanding now supports the full implementation of IBM Cloud Identity and Access Management (IAM). API keys for Watson services are no longer limited to a single service instance.
Expression that converts the current observation’s value to a string.

"TS_EXP_MATCH” (StringExpression, StringExpression) Output: Boolean Expression
Expression that does a string match on the indicated expression.





 Anchor creation functions 

Each of these functions creates an anchor artifact, which is used by other functions to determine where to segment a time series.

Anchor functions are deprecated, use expression creation functions instead. Expression creation functions are similar to anchor functions but they are more feature rich.

TS_ANCHOR_EQ

[2]Output: AnchorType
Creates an anchor that sets an anchor point when an observation value equals the value of the input parameter.

TS_ANCHOR_CONTAINS(String) Output: AnchorType
Creates an anchor that sets an anchor point when an input string is contained in the value of an observation value.

TS_ANCHOR_CONTAINED_IN(String) Output: AnchorType
Creates an anchor that sets an anchor point when an observation value is contained in the input string.

TS_ANCHOR_STARTS_WITH(String) Output: AnchorType
Creates an anchor that sets an anchor point when an observation value begins with the input string.

TS_ANCHOR_ENDS_WITH(String) Output: AnchorType
Creates an anchor that sets an anchor point when an observation value ends with the input string.

TS_ANCHOR_GT(Double) Output: AnchorType
Creates an anchor that sets an anchor point when an observation value is greater than (>) the input value.

TS_ANCHOR_GTE(Double) Output: AnchorType
Creates an anchor that sets an anchor point when an observation value is greater than or equal to (>=) the input value.

TS_ANCHOR_LT(Double) Output: AnchorType
Creates an anchor that sets an anchor point when an observation value is less than (<) the input value.

TS_ANCHOR_LTE(Double) Output: AnchorType
Creates an anchor that sets an anchor point when an observation value is less than or equal to (<=) the input value.

TS_ANCHOR_DIVISIBLE_BY(Double) Output: AnchorType
Configuring detect-secrets scan 

Detect-secrets is a client-side security tool that detects secrets within a codebase to remediate and prevent secret leaks.

For more information, see [IBM/detect-secrets](https://github.com/IBM/detect-secrets).

Detect-secrets scans the entire current codebase and outputs a snapshot of currently identified secrets. The secrets list is then used to audit the identified secrets as true positives or false positives, the DevSecOps toolchains then create Git issues for the true positives.



 Using the baseline file 

A baseline file is the snapshot of a detect secret scan. The file lists potential secrets on every run and marks false positives, which are not flagged as issues in subsequent runs.

The baseline file must be updated periodically. Update and audit the baseline file by committing it and pushing it back into the repo. Including these false positives in the baseline ensures that no issues are created for these existing files during future scans.





 Classifying secrets 

When a detect-secrets scan is complete, the detected secrets have yet to be classified as real positives or false positives.

Three conditions classify a potentially detected secret:



* Unaudited - A secret is categorized as unaudited if it is a potential secret that must be reviewed in the audit process.
* Audited as real - A secret is categorized as audited as real if it is marked as a true positive during an audit and the baseline file still contains the secret.
* Live - A secret is categorized as live if the secret is confirmed as a live secret by the plug-ins that are used by detect-secrets.







 Managing the baseline 

The following commands manage the secrets baseline:



* The detect-secrets scan command generates a baseline file or updates the existing file to contain a snapshot of all detected potential secrets of the run.
* The detect-secrets audit command labels secrets so that you can narrow down your checklist of secrets to migrate.



Audits identify the detected secrets as either true or false positives, you can then update the baseline file so the false positives can be excluded from the future scan runs.





 Updating the baseline 

Take the following steps to update your baseline:



1.