vania-janet/multiturn-rag-retrieval-data huggingface dataset:
[and case (exact | insensitive )]
[and lemma_match];







 Description 



* <dictionary name>

Specifies a name for the new internal or external dictionary. Can be a simple identifier or double-quoted identifier.
* '<file name>'

Specifies the name of the file that contains dictionary entries. Dictionary files are carriage-return-delimited text files with one dictionary entry per line. Entries in a dictionary file can consist of multiple tokens.
* <table name>

Specifies the name of the table from which to add dictionary entries. Dictionaries cannot be created from a table that is imported from another module.
* <column name>

Specifies the name of the column in the table from which to add dictionary entries.
* required [true|false]

Specifies whether external content for the external dictionary is required to run the module.



* true

If the clause is required true, you must supply a URI to the location of the file that contains the external content. The specified file must contain content. If the URI is not supplied, or if the file does not contain content, the runtime component throws an exception.
* false

If the clause is required false, the module can be run successfully even if a URI to the external content is not supplied for this dictionary. If a URI is not supplied, the runtime component treats it as an empty dictionary.



The use of create external dictionary <dictionary-name> allow_empty is now deprecated and results in a compiler warning.
* '<language code(s)>'

Specifies a comma-delimited list of two-letter language codes, such as en (English) or zh (Chinese) for the languages, or for the document languages for external dictionaries, on which to evaluate the dictionary. The dictionary produces no results on documents whose language code is not contained in this string.

If the language parameter is omitted, the dictionary language defaults to one of the following language sets:



* The language sets that are specified through the set default language statement, if it is declared, in the containing module.
A Spark runtime version comes with default runtime language version. IBM extend support for new language versions and remove the existing language version to keep the runtime free from any security vulnerabilities. The system also provides settling time to transition your workloads when ever there is a new language versions. You can test your workload with a language version by passing an environment variable that points to the language version of the application.

Example:

"application_details": {
"application": "/opt/ibm/spark/examples/src/main/python/wordcount.py",
"arguments": [
"/opt/ibm/spark/examples/src/main/resources/people.txt"
],
"env": {
"RUNTIME_PYTHON_ENV": "python310"
}
}
}





 Learn more 

When managing your Spark applications, follow the recommended [Best practices](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-best-practices-serverless).
context['integrations']['browser_info'] ?>.





 Shorthand syntax for entities 

The following table shows examples of the shorthand syntax that you can use when referring to entities.



 Shorthand syntax Full syntax in SpEL 

 @year entities['year']?.value 
 @year == 2016 entities['year']?.value == 2016 
 @year != 2016 entities['year']?.value != 2016 
 @city == 'Boston' entities['city']?.value == 'Boston' 
 @city:Boston entities['city']?.contains('Boston') 
 @city:(New York) entities['city']?.contains('New York') 



In SpEL, the question mark (?) prevents a null pointer exception from being triggered when an entity object is null.

If the entity value that you want to check for contains a ) character, you cannot use the : operator for comparison. For example, if you want to check whether the city entity is Dublin (Ohio), you must use @city == 'Dublin (Ohio)' instead of @city:(Dublin (Ohio)).





 Shorthand syntax for intents 

The following table shows examples of the shorthand syntax that you can use when referring to intents.



Intents shorthand syntax

 Shorthand syntax Full syntax in SpEL 

 `#help` `intent == 'help'` 
 `! #help` `intent != 'help'` 
 `NOT #help` `intent != 'help'` 
 `#help` or `#i_am_lost` `(intent == 'help' || intent == 'I_am_lost')` 









 Built-in global variables 

You can use the expression language to extract property information for the following global variables:



 Global variable Definition 

 context JSON object part of the processed conversation message. 
 entities[ ] List of entities that supports default access to 1st element. 
 input JSON object part of the processed conversation message. 
 intents[ ] List of intents that supports default access to first element. 
 output JSON object part of the processed conversation message. 







 Accessing entities
Language Phonetic symbols for obsolete neural voices Phonetic symbols for enhanced neural and expressive neural voices 

 Australian English [English (Australian) symbols (obsolete)](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-auSymbols) [English (Australian) symbols](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-auSymbols-new) 
 Dutch Netherlands [Dutch (Netherlands) symbols (obsolete)](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-nlSymbols) [Dutch (Netherlands) symbols](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-nlSymbols-new) 
 Korean [Korean symbols (obsolete)](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-koSymbols) [Korean symbols](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-koSymbols-new) 



You need to be aware of the following information when validating your custom words:



* Enhanced neural and expressive neural voices support both standard International Phonetic Alphabet (IPA) and IBM Symbolic Phonetic Representation (SPR) phonetic symbols. Obsolete neural voices supported only IPA symbols.
* Enhanced neural and expressive neural voices use IPA symbols that are different from those that are available with obsolete neural voices. Some overlap exists, but you need to verify and test the translations of all words to ensure that they are defined to your satisfaction.
* If you update custom words in a custom model, adding a new translation for a word that already exists overwrites the word's existing translation.



Because custom models are based on languages, not on specific voices, you do not need to create new custom models. But you might find it easier to work with new models to avoid confusion between existing and new custom words that the models contain.

For more information about about working with custom models and entries for custom words, see the following topics:
"words": [
"Hello", 0.0629826778195474, 0.2590192737303819]],
"world", 0.2598829173456253, 0.5322130804452672]]
]
}

Also, the level of precision for word timings and marks is now reduced to three decimal places. For more information about the new responses, see [Generating word timings](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-timing).

Note: Results for enhanced neural and neural voices were different previously. These inconsistencies could cause errors for the Watson SDKs. The results for all voices are now consistent.





 26 January 2022 

Defect fix: Improve SSML documentation
: Defect fix: The SSML documentation was updated to correct the following errors:



* The examples of the <break> element are now correct. The element is unary, as now shown in the examples. The previous examples included open and close tags with embedded text. The embedded text was not spoken by the service. For more information, see [The <break> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsbreak_element).
* The service supports SSML version 1.1. All references and examples now use the correct version. The documentation previously referred to version 1.0.







 3 December 2021 

New Czech neural voice: cs-CZ_AlenaVoice
: A new language, Czech, with a new female voice, cs-CZ_AlenaVoice, is now available. The voice is a neural voice.

New Belgian Dutch neural voice: nl-BE_BramVoice
: A new male Belgian Dutch (Flemish) voice, nl-BE_BramVoice, is now available. The voice is a neural voice.

Defect fix: Improve SSML and speech synthesis
: Defect fix: The following defects for the Speech Synthesis Markup Language (SSML) and speech synthesis were fixed with this release:
curl -X GET --user "apikey:{apikey}" --output "curriculum-fr.html" "{url}/v3/documents/bae02796-0d28-435c-9115-888359fdde62/translated_document?version=2018-05-01"





 Step 4: Translate a previously submitted document 

The following example request translates the original English file curriculum.html, which has document ID bae02796-0d28-435c-9115-888359fdde62, to Portuguese.

curl -X POST --user "apikey:{apikey}" --form "document_id=bae02796-0d28-435c-9115-888359fdde62" --form "source=en" --form "target=pt" "{url}/v3/documents?version=2018-05-01"

{
"document_id": "a0ge2746-ad38-7d5c-7025-4cd3g9f451ab"
}

The response contains a new document ID. Repeat step two with the new document ID to check the status of the translation. When the status becomes available, use the new document ID to download the translated file as shown in step three.

When translating a previously submitted document, the target language must be different from the target language of the original request when the document was initially submitted.





 Step 5: Delete documents 

The service automatically deletes original documents and any associated translated documents after they have not been used for a certain period of time. For more information, see [Information security](https://cloud.ibm.com/docs/language-translator?topic=language-translator-information-security).

To delete documents manually, use the Delete document method. In this tutorial, the English file curriculum.html was involved with two translations, so two requests are required to delete all copies of the original document.
Previously, a comma in a translation could occasionally cause the translation to fail to generate valid audio when used for speech syntheses. This problem was identified in US English custom models.

Defect fix: French synthesis of dates is now consistent
: Defect fix: French synthesis no longer includes the article "le" before dates of the form "the ordinal of month." Previously, the article was included only for the first day of the month for French (for example, "the first of September," "le premier septembre").

Known limitation with using the Ogg audio format with the Safari browser
: By default, the service returns audio in the Ogg audio format with the Opus codec (audio/ogg;codecs=opus). However, the Ogg audio format is not supported with the Safari browser. If you are using the the Text to Speech service with the Safari browser, you must specify a different format in which you want the service to return the audio.



* For more information about the available formats, see [Supported audio formats](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-audio-formatsformats-supported).
* For more information about specifying a format, see [Specifying an audio format](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-audio-formatsformats-specify).







 31 August 2022 

New beta rate_percentage query parameter for controlling the global speaking rate
: The service offers a new rate_percentage query parameter to modify the speaking rate for a speech synthesis request. The speaking rate is the speed at which the service speaks the text that it synthesizes into speech. A higher rate causes the text to be spoken more quickly; a lower rate causes the text to be spoken more slowly. The parameter changes the per-voice default rate for an entire request. For more information, see [Modifying the speaking rate](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-synthesis-paramsparams-rate-percentage).

New beta pitch_percentage query parameter for controlling the global speaking pitch
For more information about custom language models, see [Creating a custom language model](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-languageCreate).

The following example shows the hesitation "uhm" in a US English transcript:

{
"result_index": 0,
"results": [
{
"alternatives":
{
"confidence": 0.99,
"transcript": ". . . that uhm that's a . . ."
}
],
"final": true
}
]
}







 Capitalization 

For most languages, the service does not use capitalization in response transcripts. If capitalization is important for your application, you must capitalize the first word of each sentence and any other terms for which capitalization is appropriate.

The service applies automatic capitalization only to the following languages and models. The service always applies this capitalization, regardless of whether you use smart formatting.



* For US English previous-generation models, the service capitalizes many proper nouns. For example, the service returns the following transcript for the phrase barack obama graduated from columbia university:

Barack Obama graduated from Columbia University

The service does not capitalize proper nouns for US English with next-generation models.
* For German next-generation models, the service capitalizes many nouns. For example, the service returns the following transcript for the phrase er braucht erst einen neuen eintrag ins vokabular punkt:

er braucht erst einen neuen Eintrag ins Vokabular Punkt

The service does not capitalize nouns for German with previous-generation models.







 Punctuation 

The service does not insert punctuation in response transcripts by default. You must add any punctuation that you need to the service's results.

For some languages, you can use smart formatting to direct the service to substitute punctuation symbols, such as commas, periods, question marks, and exclamation points, for certain keyword strings.
For billing purposes, whitespace characters are not counted. For more information, see [Data limits](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-service-featuresfeatures-data-limits).





 How does customization work? 

The customization interface of the Text to Speech service creates a dictionary of words and their translations for a specific language. This dictionary is referred to as a custom model. For more information, see [Understanding customization](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-customIntro).





 How do I create a custom model? 

Review the guidelines for working with the customization interface before you begin. Then, see the steps and examples for creating, querying, updating, and deleting custom models in [Creating and managing custom models](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-customModels). Also review [Creating and managing custom entries](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-customWords) for examples and guidance about adding relevant training data.





 Can I create a custom voice? 

IBM Cloud

As a premium customer, you can work with IBM to train a new custom voice for your specific use case and target market. Creating a custom voice is different from customizing one of the service's existing voices. A custom voice is a unique new voice that is based on audio training data that the customer provides. IBM can train a custom voice with as little as one hour of training data.

To request a custom voice or for more information, complete and submit this [IBM Request Form](https://form.asana.com/?k=CBuXK1uwlAf5ek6FwB6mcg&d=8612789739828).





 How do I use the Tune by Example feature? 

Tune by Example lets you control exactly how specified text is spoken by the service. You provide text and spoken audio to add a custom prompt to a custom model.
However, if you override the default pronunciation rules for the string St. to translate it as saint, the service can no longer pronounce the word based on context. Applying a custom model that includes such a translation causes the service to pronounce the previous input sentence as

Saint Anthony lives on Henry saint

Consider such cases when you develop word/translation pairs.





 Trailing periods 

The service applies a word from a custom model only to those strings in the input text that match the word exactly. A trailing . (period) in a word entry changes how the word is synthesized:



* A word that does not have a trailing period can contain practically any character. Characters include letters, digits, punctuation (other than a trailing period), non-letter symbols (such as %, &, and @), quotation marks, parentheses, brackets, and so on. Its translation can include any legal input to the service, including white space and a phonetic representation in SSML format.
* A word that has a trailing period can contain only letters, periods, and internal apostrophes (not as the first or last character). The word's translation can contain only normal words with ordinary spelling that are separated by white space or hyphens. It cannot contain a phonetic representation.



An example of a word with a trailing period is "div.". Assume that a custom model includes the entry {word='div.', translation='division'}. The service does not apply the translation to the string "div" because it does not include a trailing period and therefore does not match the entry.





 Phonetic translation for foreign words 

One use of phonetic translation is to add pronunciations for words that are foreign to the base language of the custom model. For example, you might add a pronunciation for a French word to a custom model that is based on English. In this case, you must use the phonetic symbols for the language of the custom model, English.

The same phonetic symbol can produce different sounds for different languages. Also, not all phonetic symbols are supported for all languages. Make sure to use the phonetic symbols for the base language of the custom model when defining a translation.





 Working with IBM SPR entries
Using grammars with custom language models 

The IBM Watson® Speech to Text service supports the use of grammars with custom language models. You can add grammars to a custom language model and use them for speech recognition. Grammars restrict the set of phrases that the service can recognize from audio.

A grammar uses a formal language specification to define a set of production rules for transcribing strings. The rules specify how to form valid strings from the language's alphabet. When you apply a grammar to speech recognition, the service can return only one or more of the phrases that are generated by the grammar.

For example, when you need to recognize specific words or phrases, such as yes or no, individual letters or numbers, or a list of names, using grammars can be more effective than examining alternative words and transcripts. Moreover, by limiting the search space for valid strings, the service can deliver results faster and more accurately.

When you use a custom language model and a grammar for speech recognition, the service can return a valid phrase from the grammar or an empty result. If the result is not empty, the service includes a confidence score with the final transcript, as it does for all recognition requests. For grammars, the score indicates the likelihood that the response matched the grammar. False-positives are always possible, especially for simple grammars, so you must always consider the confidence of the service's results when evaluating its response.

For more information about the languages and models that support grammars and their level of support (generally available or beta), see [Language support for customization](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-support).



 Supported grammar formats 

The Speech to Text service supports grammars that are defined in the following standard formats:



* Augmented Backus-Naur Form (ABNF), which uses a plain-text representation that is similar to traditional BNF grammar. The media type for this format is application/srgs.
* XML Form, which uses XML elements to represent the grammar. The media type for this format is application/srgs+xml.



Both grammar formats have the expressive power of a Context-Free Grammar (CFG).
curl -X PUT --header "Authorization: Bearer {token}" --header "Content-Type: application/json" --data "{"translation":"<phoneme alphabet=\"ipa\" ph=\"ɲɯːjoːkɯ\"></phoneme>", "part_of_speech":"Mesi"}" "{url}/v1/customizations/{customization_id}/words/%EF%BC%AE%EF%BC%B9"
* Phonetic IBM SPR:

IBM Cloud

curl -X PUT -u "apikey:{apikey}" --header "Content-Type: application/json" --data "{"translation":"<phoneme alphabet=\"ibm\" ph=\"nyu:yo:ku\"></phoneme>", "part_of_speech":"Mesi"}" "{url}/v1/customizations/{customization_id}/words/%EF%BC%AE%EF%BC%B9"

IBM Cloud Pak for Data

curl -X PUT --header "Authorization: Bearer {token}" --header "Content-Type: application/json" --data "{"translation":"<phoneme alphabet=\"ibm\" ph=\"nyu:yo:ku\"></phoneme>", "part_of_speech":"Mesi"}" "{url}/v1/customizations/{customization_id}/words/%EF%BC%AE%EF%BC%B9"







 Querying a single word from a custom model 

To query the translation of a single word from a custom model, use the GET /v1/customizations/{customization_id}/words/{word} method. You specify the word in the URL of the method. The translation is returned as it is defined in the custom model (sounds-like or phonetic).

The following example queries a custom model for the translation of the word IEEE:

IBM Cloud
Verify that the sounds-like pronunciations of the words are accurate and correct. Also look for typographical and other errors in the words themselves. For more information about validating and correcting the words in a custom model, see [Validating a words resource for previous-generation models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWordsvalidateModel).





 Train the custom language model 

The final step before you can use a grammar with a custom language model is to train the model. You use the same process that you use for training a custom model on new corpora or new custom words. Training the model compiles the grammar for use during speech recognition. The service processes the grammar from its original text-based format to a binary runtime format for speech recognition. You cannot use the grammar until you train the model.

You use the POST /v1/customizations/{customization_id}/train method to train a custom model. You pass the method the customization ID of the model that you want to train, as in the following example.

IBM Cloud

curl -X POST -u "apikey:{apikey}" "{url}/v1/customizations/{customization_id}/train"

IBM Cloud Pak for Data

curl -X POST --header "Authorization: Bearer {token}" "{url}/v1/customizations/{customization_id}/train"

The training method is asynchronous. Training typically takes only seconds or a few minutes to complete. But it can take longer depending on the size and complexity of the grammar and the current load on the service. For information about checking the status of a training operation, see [Monitoring the training request](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-grammarAddmonitorTraining-grammar).



 Monitoring the training request 

The service returns a 200 response code if the training process is successfully initiated.