vania-janet/multiturn-rag-retrieval-data huggingface dataset:
German (de) Advanced rules models, Built-in entities, Classifier (Document and Text), Custom entities, Dictionary, Document sentiment, Keywords, Machine Learning, Optical character recognition v1 (Installed), Optical character recognition v2 (Cloud-managed), Parts of speech, Regular expressions, Smart Document Understanding, Stemmer, Table Understanding 
 Hebrew (he) Classifier (Document and Text), Custom entities, Dictionary, Optical character recognition v2 (Cloud-managed), Parts of speech, Regular expressions, Smart Document Understanding, Table Understanding 
 Hindi (hi) Classifier (Document and Text), Custom entities, Dictionary, Parts of speech, Regular expressions, Stemmer 
 Italian (it) Advanced rules models, Built-in entities, Classifier (Document and Text), Custom entities, Dictionary, Document sentiment, Keywords, Machine Learning, Optical character recognition v1, Parts of speech, Regular expressions, Smart Document Understanding, Stemmer, Table Understanding 
 Japanese (ja) Advanced rules models, Built-in entities, Classifier (Document and Text), Custom entities, Dictionary, Document sentiment, Keywords, Machine Learning, Optical character recognition v1, Parts of speech, Phrase sentiment, Regular expressions, Smart Document Understanding, Table Understanding 
 Korean (ko) Advanced rules models, Built-in entities, Classifier (Document and Text), Custom entities, Dictionary, Document sentiment, Keywords, Machine Learning, Optical character recognition v1, Parts of speech, Regular expressions, Smart Document Understanding, Table Understanding 
 Norwegian (Bokmål) (nb) Classifier (Document and Text), Custom entities, Dictionary, Optical character recognition v1, Parts of speech, Regular expressions, Smart Document Understanding, Stemmer, Table Understanding 
 Norwegian (Nynorsk) (nn) Classifier (Document and Text), Custom entities, Dictionary, Optical character recognition v1, Parts of speech, Regular expressions, Smart Document Understanding, Stemmer, Table Understanding 
 Polish (pl) Classifier (Document and Text), Custom entities, Dictionary, Optical character recognition v1, Parts of speech, Regular expressions, Smart Document Understanding, Table Understanding
Using a grammar for speech recognition 

Once you create and train your custom language model with your grammar, you can use the grammar in speech recognition requests:



* Use the language_customization_id query parameter to specify the customization ID (GUID) of the custom language model for which the grammar is defined. A custom model can be used only with the base model for which it is created. If your custom model is based on a model other than the default, you must also specify that base model with the model query parameter. For more information, see [Using the default model](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-usemodels-use-default). You must issue the request with credentials for the instance of the service that owns the model.
* Use the grammar_name parameter to specify the name of the grammar. You can specify only a single grammar with a request.



When you use a grammar, the service recognizes only words from the specified grammar. The service does not use custom words that were added from corpora, that were added or modified individually, or that are recognized by other grammars.

For more information about the languages and models that support grammars and their level of support (generally available or beta), see [Language support for customization](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-support).



 Examples of using a grammar with a custom language model 

The following examples show the use of a grammar with a custom language model for each speech recognition interface:



* For the [WebSocket interface](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-websockets), you first specify the customization ID with the language_customization_id parameter of the /v1/recognize method. You use this method to establish a WebSocket connection with the service.

var access_token = {access_token};
var wsURI = '{ws_url}/v1/recognize'
+ '?access_token=' + access_token
For more information, see [Creating a custom language model](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-languageCreatecreateModel-language) and [Listing custom language models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-manageLanguageModelslistModels-language).

New names for UK English models
: The names of the language models en-UK_BroadbandModel and en-UK_NarrowbandModel have been deprecated. The models are now available with the names en-GB_BroadbandModel and en-GB_NarrowbandModel.

The deprecated en-UK_{model} names continue to function, but the GET /v1/models method no longer returns the names in the list of available models. You can still query the names directly with the GET /v1/models/{model_id} method.





 1 July 2017 

Language model custom now generally available for US English and Japanese
: The language model customization interface of the service is now generally available (GA) for both of its supported languages, US English and Japanese. IBM does not charge for creating, hosting, or managing custom language models. As described in the next bullet, IBM now charges an extra $0.03 (USD) per minute of audio for recognition requests that use custom models.

Updates to pricing plans for the service
: IBM updated the pricing for the service by



* Eliminating the add-on price for using narrowband models
* Providing Graduated Tiered Pricing for high-volume customers
* Charging an additional $0.03 (USD) per minute of audio for recognition requests that use US English or Japanese custom language models



For more information about the pricing updates, see



* The Speech to Text service in the [IBM Cloud Catalog](https://cloud.ibm.com/catalog/services/speech-to-text)
* The [Pricing FAQs](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-faq-pricing)



Empty body no longer required for HTTP POST requests
"extract": "taxonomy,entity,relation,doc-emotion,doc-sentiment,concept,keyword",
"sentiment": true,
"quotations": true,
"language": "english"
}
}
]


Fixed known issue : Improved performance and stability of the service.





 8 March 2017 

Improved back end performance : Optimized the back end, including the addition of new timeouts, to improve overall performance.

Fixed defect : Fixed a bug that caused the environment status of free (0-sized) environments to report a status of pending regardless of the real status.

English language support : The only national language currently supported by Discovery is U.S. English (en_US). [Update](https://cloud.ibm.com/docs/discovery?topic=discovery-language-support)





 3 March 2017 

New My data insights screen : Added the My data insights screen to the Discovery tooling.





 26 February 2017 

Improved performance and error codes : The performance of the Discovery News environment is faster and stable. : Improved error codes messaging and delivery.

Change to number of results : The Discovery News service returns only 50 results at a time. As a workaround, use the offset parameter in your query to page through results. : You can submit a new configuration with an individual document by using the following command:


curl -X POST -u apikey:{apikey_value} -F "file=@wikipedia-sample.html" -F "configuration=$(cat config.json)" "https://gateway.watsonplatform.net/v1/environments/{environment_id}/collections/{collection_id}/documents?version=2016-12-01"


: The PDF and Word converters create HTML as a middle step. The service can apply additional transforms and normalizations on the intermediary HTML before the final transformation to normalized JSON.





 16 February 2017 

New features for JSON fields and environments : You can now use CSS selectors to select JSON fields that you can then apply enrichments to.
* For more information about using SSML and its <phoneme> element, see [Understanding SSML](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-ssml).
* For more information about specifying SPR and IPA symbols and translations, see [Understanding phonetic symbols](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-symbols).









 Mixed sounds-like and phonetic translation 

You can mix the sounds-like and phonetic methods in the same translation. This feature can reduce the work that is involved in composing a translation.

For instance, assume that you used the sounds-like method to get part of a word pronounced satisfactorily. But you now need to fine-tune the remaining elements of the word. You can use the phonetic method to specify the difficult aspects of the word. The following example applies mixed translation to the word trinitroglycerin:

try<phoneme alphabet="ipa" ph="nˈaɪtɹəglɪsəɹɨn"></phoneme>
Service features 

The IBM Watson® Speech to Text service offers many advanced features to help you get the most from your audio transcription. The service offers multiple speech recognition interfaces, and these interfaces support many features that you can use to manage how you pass your audio to the service and the results that the service returns. You can also customize the service to enhance its vocabulary and to accommodate the acoustic characteristics of your audio. And as with all Watson services, SDKs are available to simplify application development in many programming languages.



 Using languages and models 

The service supports speech recognition for the many languages listed in [Language support](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-aboutabout-languages). The service provides different models for the languages that it supports. Most language models are generally available (GA) for production use; a few are beta and subject to change.



* For most languages, the service offers previous-generation Broadband and Narrowband models. Most previous-generation models are GA. For more information, see [Previous-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models).
* For a growing set of languages, the service offers next-generation Multimedia and Telephony models that improve upon the speech recognition capabilities of the previous-generation models. All next-generation models are GA. Next-generation models return results with greater throughput and higher accuracy than previous-generation models. For more information, see [Next-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-ng).



For most languages, you can transcribe audio at one of two sampling rates:



* Use Broadband or Multimedia models for audio that is sampled at a minimum sampling rate of 16 kHz.
* Use Narrowband or Telephony models for audio that is sampled at a minimum sampling rate of 8 kHz.



Effective 31 July 2023, all previous-generation models will be removed from the service and the documentation. Most previous-generation models were deprecated on 15 March 2022.
The keywords_threshold and word_alternatives_threshold parameters no longer accept a null value
: The keywords_threshold and word_alternatives_threshold parameters of the recognition methods no longer accept a null value. To omit keywords and word alternatives from the response, omit the parameters. A specified value must be a float.





 22 September 2016 

New beta language model customization interface
: The service now offers a new beta language model customization interface for US English. You can use the interface to tailor the service's base vocabulary and language models via the creation of custom language models that include domain-specific terminology. You can add custom words individually or have the service extract them from corpora. To use your custom models with the speech recognition methods that are offered by any of the service's interfaces, pass the customization_id query parameter. For more information, see



* [Creating a custom language model](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-languageCreate)
* [Using a custom language model for speech recognition](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-languageUse)
* [API & SDK reference](https://cloud.ibm.com/apidocs/speech-to-text)



New support for audio/mulaw audio format
: The list of supported audio formats now includes audio/mulaw, which provides single-channel audio encoded using the u-law (or mu-law) data algorithm. When you use this format, you must also specify the sampling rate at which the audio is captured. For more information, see [audio/mulaw format](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-audio-formatsaudio-formats-mulaw).

New supported_features identified when listing models
: The GET /v1/models and GET /v1/models/{model_id} methods now return a supported_features field as part of their output for each language model. The additional information describes whether the model supports customization.
curl -X POST --user "apikey:{apikey}" --form forced_glossary=@glossary.tmx "{url}/v3/models?version=2018-05-01&base_model_id=en-es&name=custom-english-to-spanish"

The API response contains details about your custom model, including its model ID. Use the model ID to check the status of your model and to translate sentences once the status of the model becomes available.

{
"model_id": "96221b69-8e46-42e4-a3c1-808e17c787ca",
"source": "en",
"target": "es",
"base_model_id": "en-es",
"domain": "general",
"customizable": false,
"default_model": false,
"owner": "4937aab7-0f3a-4a75-bee1-0b7a12b6b8",
"status": "uploaded",
"name": "custom-english-to-spanish",
"train_log": null
}







 Step 3: Check the status of your model 

Model training might take anywhere from a couple of minutes (for forced glossaries) to several hours (for large parallel corpora) depending on how much training data is involved. To see if your model is available, use the [Get model details](https://cloud.ibm.com/apidocs/language-translator/language-translatorgetmodel) method and specify the model ID that you received in the service response in Step 2. Also, you can check the status of all of your models with the [List models](https://cloud.ibm.com/apidocs/language-translator/language-translatorlistmodels) method.

The status response attribute describes the state of the model in the training process:



* uploading
* uploaded
* dispatching
* queued
* training
* trained
* publishing
* available
They include several extra benefits, such as finer-grained access control by using IBM Cloud Identity and Access Management (IAM), the ability to connect service instances to apps and service across different regions, and an easy way to view usage per group. For more information about migration, see [Migrating Cloud Foundry service instances and apps to a resource group](https://cloud.ibm.com/docs/account?topic=account-migrate).

Defect fix: Specifying large cardinal numbers with the <say-as> element no longer causes errors for English voices
: Defect fix: You can now use the <say-as> element to pronounce large numbers as cardinal numbers. Previously, enclosing a large number in the <say-as> element with the attribute interpret-as="cardinal" could cause speech synthesis to fail for English voices. For example, <say-as interpret-as="cardinal">3,200</say-as> could cause the service to generate an error. For more information, see [cardinal](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementssay-as-cardinal) in the topic SSML elements.

Defect fix: Homonyms and other words are now pronounced correctly by English voices
: Defect fix: The service now pronounces homonyms and other words correctly based on their context in English text that is to be synthesized. Previously, words such as advocate and wifi could be pronounced incorrectly by English voices.





 30 November 2022 

Defect fix: Add rules for custom model naming documentation
: Defect fix: The documentation now provides detailed rules for naming custom models. For more information, see



* [Creating a custom model](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-customModelscuModelsCreate)
* [API & SDK reference](https://cloud.ibm.com/apidocs/text-to-speech)







 7 October 2022 

The service now enforces stricter SSML validation
: The service now enforces stricter validation of input text that includes Speech Synthesis Markup Language (SSML) elements.
For more information, see [Working with custom words for previous-generation models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWordsworkingWords).
* Delete extraneous words that are introduced in error (for example, by typographical or other mistakes in a corpus) by using the DELETE /v1/customizations/{customization_id}/words/{word_name} method. For more information, see [Deleting a word from a custom language model](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-manageWordsdeleteWord).



* If the word was extracted from a corpus, you can instead update the corpus text file to correct the error and then reload the file by using the allow_overwrite parameter of the POST /v1/customizations/{customization_id}/corpora/{corpus_name} method. For more information, see [Add a corpus to the custom language model](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-languageCreateaddCorpus).
* If the word was extracted from a grammar, you can update the grammar file to correct the error and then reload the file by using the allow_overwrite parameter of the POST /v1/customizations/{customization_id}/grammars/{grammar_name} method. For more information, see [Add a grammar to the custom language model](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-grammarAddaddGrammar).
* Brazilian Portuguese narrowband model (pt-BR_NarrowbandModel)
* Mandarin Chinese broadband model (zh-CN_BroadbandModel)
* Mandarin Chinese narrowband model (zh-CN_NarrowbandModel)



For more information, see [Previous-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models).

New support for audio/ogg;codecs=opus audio format
: The HTTP POST requests /v1/sessions/{session_id}/recognize and /v1/recognize, as well as the WebSocket /v1/recognize request, support transcription of a new media type: audio/ogg;codecs=opus for Ogg format files that use the Opus codec. In addition, the audio/wav format for the methods now supports any encoding. The restriction about the use of linear PCM encoding is removed. For more information, see [audio/ogg format](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-audio-formatsaudio-formats-ogg).

New sequence_id parameter for long polling of sessions
: The service now supports overcoming timeouts when you transcribe long audio files with the HTTP interface. When you use sessions, you can employ a long polling pattern by specifying sequence IDs with the GET /v1/sessions/{session_id}/observe_result and POST /v1/sessions/{session_id}/recognize methods for long-running recognition tasks. By using the new sequence_id parameter of these methods, you can request results before, during, or after you submit a recognition request.

New capitalization feature for US English transcription
: For the US English language models, en_US_BroadbandModel and en_US_NarrowbandModel, the service now correctly capitalizes many proper nouns. For example, the service would new return text that reads "Barack Obama graduated from Columbia University" instead of "barack obama graduated from columbia university." This change might be of interest to you if your application is sensitive in any way to the case of proper nouns.

New HTTP error code
* It gives the OOV word IBM홈페이지 a sounds-like of 아이 비 엠 홈페이지.













 Working with custom words for previous-generation models 

You can use the POST /v1/customizations/{customization_id}/words and PUT /v1/customizations/{customization_id}/words/{word_name} methods to add new words to a custom model's words resource. You can also use the methods to modify or augment a word in a words resource.

You might, for instance, need to use the methods to correct a typographical error or other mistake that is made when a word is added from a corpus. You might also need to add sounds-like definitions for an existing word. If you modify an existing word, the new data that you provide overwrites the word's existing definition in the words resource. The rules for adding a word also apply to modifying an existing word.

You are likely to add most custom words from corpora. Make sure that you know the character encoding of your corpus text files. The service preserves the encoding that it finds in the text files. You must use that same encoding when working with custom words in the custom model. For more information, see [Character encoding for custom words](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-manageWordscharEncoding).



 Using the sounds_like field 

The sounds_like field specifies how a word is pronounced by speakers. By default, the service automatically attempts to complete the field with the word's spelling. But the service cannot generate a pronunciation for all words. After adding or modifying words, you must validate the words resource to ensure that each word's definition is complete and valid. For more information, see [Validating a words resource for previous-generation models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWordsvalidateModel).
* Japanese narrowband model (ja-JP_NarrowbandModel)
* US English broadband model (en-US_BroadbandModel)



By default, the service automatically uses the updated models for all speech recognition requests. If you have custom language or custom acoustic models that are based on the models, you must upgrade your existing custom models to take advantage of the updates by using the following methods:



* POST /v1/customizations/{customization_id}/upgrade_model
* POST /v1/acoustic_customizations/{customization_id}/upgrade_model



For more information, see [Upgrading custom models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgrade).

Language model customization now available for Dutch and Italian
: Language model customization is now supported for Dutch and Italian with the new versions of the following models:



* Dutch broadband model (nl-NL_BroadbandModel)
* Dutch narrowband model (nl-NL_NarrowbandModel)
* Italian broadband model (it-IT_BroadbandModel)
* Italian narrowband model (it-IT_NarrowbandModel)



For more information, see



* [Parsing of Dutch, English, French, German, Italian, Portuguese, and Spanish](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWordscorpusLanguages)
* [Guidelines for Dutch, French, German, Italian, Portuguese, and Spanish](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWordswordLanguages-esES-frFR)



Because the Dutch and Italian models are beta, their support for language model customization is also beta.

Japanese narrowband model now includes some multigram word units
: The Japanese narrowband model (ja-JP_NarrowbandModel) now includes some multigram word units for digits and decimal fractions. The service returns these multigram units regardless of whether you enable smart formatting. The smart formatting feature understands and returns the multigram units that the model generates.