vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* en-US_Telephony
* ja-JP_Multimedia
* ja-JP_Telephony



Visible improvements to the models: The new technology improves the default behavior of the new English and Japanese models. Among other changes, the new technology optimizes the default behavior for the following parameters:



* The default customization_weight for custom models that are based on the new versions of these models changes from 0.2 to 0.1.
* The default character_insertion_bias for custom models that are based on the new versions of these models remains 0.0, but the models have changed in a manner that makes use of the parameter for speech recognition less necessary.



Upgrading to the new models: To take advantage of the improved technology, you must upgrade any custom language models that are based on the new models. To upgrade to the new version of one of these base models, do the following:



1. Change your custom model by adding or modifying a custom word, corpus, or grammar that the model contains. Any change that you make moves the model to the ready state.
2. Use the POST /v1/customizations/{customization_id}/train method to retrain the model. Retraining upgrades the custom model to the new technology and moves the model to the available state.

Known issue: At this time, you cannot use the POST /v1/customizations/{customization_id}/upgrade_model method to upgrade a custom model to one of the new base models. This issue will be addressed in a future release.



Using the new models: Following the upgrade to the new base model, you are advised to evaluate the performance of the upgraded custom model by paying special attention to the customization_weight and character_insertion_bias parameters for speech recognition. When you retrain your custom model:



* The custom model uses the new default customization_weight of 0.1 for your custom model. A non-default customization_weight that you had associated with your custom model is removed.
* The custom model might no longer require use of the character_insertion_bias parameter for optimal speech recognition.



Improvements to language model customization render these parameters less important for high-quality speech recognition:
See [Adjudication](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-build-groundtruthwks_haperform). 
 Train the model Create the machine learning model. See [Creating a machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-train-mlwks_madocsets). 
 Evaluate the model Evaluate the accuracy of the model. See [Evaluating annotations added by the model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-train-mlwks_matest). Depending on model accuracy, this step might result in the need to repeat earlier steps again and again until optimal accuracy is achieved. See [Analyzing machine learning model performance](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-evaluate-ml) for ideas about what to update based on common performance issues. 
 Publish the model Export the model. See [Using the machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-publish-ml).
The new models offer improved speech recognition. By default, the service automatically uses the updated models for all recognition requests. If you have custom language or custom acoustic models that are based on these models, you must upgrade your existing custom models to take advantage of the updates by using the following methods:



* POST /v1/customizations/{customization_id}/upgrade_model
* POST /v1/acoustic_customizations/{customization_id}/upgrade_model



For more information, see [Upgrading custom models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgrade).

Keyword spotting and word alternatives features now generally available
: The keyword spotting and word alternatives features are now generally available (GA) rather than beta functionality for all languages. For more information, see



* [Keyword spotting](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-spottingkeyword-spotting)
* [Word alternatives](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-spottingword-alternatives)



Defect fix: Improve documentation for customization interface
: Defect fix: The following known issues that were associated with the customization interface have been resolved and are fixed in production. The following information is preserved for users who may have encountered the problems in the past.



* If you add data to a custom language or custom acoustic model, you must retrain the model before using it for speech recognition. The problem shows up in the following scenario:



1. The user creates a new custom model (language or acoustic) and trains the model.
2. The user adds additional resources (words, corpora, or audio) to the custom model but does not retrain the model.
3. The user cannot use the custom model for speech recognition. The service returns an error of the following form when used with a speech recognition request:

{
"code_description": "Bad Request",
"code": 400,
"error": "Requested custom language model is not available.
For more information, see [Upgrading a custom language model](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgradecustom-upgrade-language).
2. Ensure that the custom acoustic model is in either the ready or the available state. You can check a model's status by using the GET /v1/acoustic_customizations/{customization_id} method. If the model's state is ready, upgrade the model before training it on its latest data.
3. Upgrade the custom acoustic model by using the POST /v1/acoustic_customizations/{customization_id}/upgrade_model method:

IBM Cloud

curl -X POST -u "apikey:{apikey}" "{url}/v1/acoustic_customizations/{customization_id}/upgrade_model"

IBM Cloud Pak for Data

curl -X POST --header "Authorization: Bearer {token}" "{url}/v1/acoustic_customizations/{customization_id}/upgrade_model"

The upgrade method is asynchronous. It can take on the order of minutes or hours to complete, depending on the amount of audio data that the model contains and the current load on the service. As with training, upgrading generally takes as long as the cumulative amount of audio data that the model contains.
4. If the custom acoustic model was trained with a custom language model, upgrade the custom acoustic model again, this time with the previously upgraded custom language model. Use the custom_language_model_id query parameter to specify the customization ID of the custom language model.

IBM Cloud

curl -X POST -u "apikey:{apikey}" "{url}/v1/acoustic_customizations/{customization_id}/upgrade_model?custom_language_model_id={custom_language_model_id}"

IBM Cloud Pak for Data
See [Adjudication](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-build-groundtruthwks_haperform). 
 Train the model Create the machine learning model. See [Creating a machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-train-mlwks_madocsets). 
 Evaluate the model Evaluate the accuracy of the model. See [Evaluating annotations added by the model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-train-mlwks_matest). Depending on model accuracy, this step might result in the need to repeat earlier steps again and again until optimal accuracy is achieved. See [Analyzing machine learning model performance](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-evaluate-ml) for ideas about what to update based on common performance issues. 
 Publish the model Export the model. See [Using the machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-publish-ml).
Choose the version of the model that you want to deploy, or click Export current model.

If there is only one working version of the model, save the current model for deployment by clicking Save for Deployment. This versions the model, which enables you to deploy one version, while you continue to improve the current version. Saving the version might take a few minutes. The option to deploy does not appear until after the version is created.
4. Click Export, and click Export again to confirm.

The model is saved as a PEAR file, and you are prompted to download the file. A PEAR (Processing Engine ARchive) file is the UIMA standard packaging format for UIMA components. The model is saved in PEAR format so it can be distributed and reused within UIMA applications.







 What to do next 

From the Discovery service, follow the steps to create a Machine Learning enrichment, which include uploading the PEAR file. For more details, see [Machine Learning models](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domainmachinelearning) in the Discovery v2 documentation.







 Leveraging a rule-based model in IBM Watson Explorer 

Export the PEAR file that is produced when the rule-based model is created so it can be used in IBM Watson Explorer.



 Procedure 

To leverage a rule-based model in IBM Watson Explorer, complete the following steps.



1. Log in as a Knowledge Studio administrator or project manager, and select your workspace.
2. Select the Rule-based Model > Versions > Rule-based Model tab.
3. Click Export current model.

The model is saved as a PEAR file, and you are prompted to download the file. A PEAR (Processing Engine ARchive) file is the UIMA standard packaging format for UIMA components. The model is saved in PEAR format so it can be distributed and reused within UIMA applications.
4. Download the file to your local system.
5. From the IBM Watson Explorer application, import the PEAR file.
For more information about all available next-generation models, see [Next-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-ng).

If you have custom language models that are based on the updated model, you must upgrade your existing custom models to take advantage of the updates by using the POST /v1/customizations/{customization_id}/upgrade_model method. For more information, see [Upgrading custom models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgrade).

Update to Japanese ja-JP_Multimedia model to support low latency
: The next-generation Japanese model ja-JP_Multimedia now supports low latency. You can use the low_latency parameter with speech recognition requests that use the model. You do not need to upgrade custom models that are based on the updated Japanese base model. For more information about the next-generation models and low latency, see [Next-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-ng) and [Low latency](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-interimlow-latency).





 11 February 2022 (Version 4.0.5) 

Defect fix: Improve custom model upgrade and base model version documentation
: Defect fix: The documentation that describes the upgrade of custom models and the version strings that are used for different versions of base models has been updated. The documentation now states that upgrade for language model customization also applies to next-generation models. Also, the version strings that represent different versions of base models have been updated. And the base_model_version parameter can also be used with upgraded next-generation models.

For more information about custom model upgrade, when upgrade is necessary, and how to use older versions of custom models, see



* [Upgrading custom models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgrade)
For more information about deploying multiple instances of the same model version see [Deploying the same model version to multiple services](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-publish-mlwks_secdep)

Added models method
: Added the models method to the Natural Language Understanding service allowing users to list deployed Knowledge Studio models.







 September 2018 



 New features and changes 

Support for additional document types
: Introduced support for HTML, DOC, DOCX, and PDF files. For more information about the supported document types, size limits, and other information, see [Creating a workspace](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-create-projectcp-mlm). For more information about adding documents for annotation, see [Adding documents to a workspace](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotationwks_projadd).







 August 2018 



 New features and changes 

New automated migration option
: Introduced a new option for automated migration of Standard plan instances from the deprecated IBM Marketplace platform to the IBM Cloud platform. If you have a Standard instance on the deprecated platform, you'll have the option to migrate. For more information, see [Migrating to IBM Cloud](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-migrate).







 July 2018 



 New features and changes 

Deployed Models page updated
: The Deployed Models page was updated to include models from Knowledge Studio instances that are managed by [IAM resource groups](https://cloud.ibm.com/docs/iam?topic=iam-userroles), in addition to models that are managed by [Cloud Foundry organizations](https://cloud.ibm.com/docs/iam?topic=iam-cfaccess).
For more information, see [Smart formatting](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-formattingsmart-formatting).





 1 March 2018 

Updates to French and Spanish broadband models for improved speech recognition
: The French and Spanish broadband models, fr-FR_BroadbandModel and es-ES_BroadbandModel, have been updated for improved speech recognition. By default, the service automatically uses the updated models for all recognition requests. If you have custom language or custom acoustic models that are based on either of these models, you must upgrade your custom models to take advantage of the updates by using the following methods:



* POST /v1/customizations/{customization_id}/upgrade_model
* POST /v1/acoustic_customizations/{customization_id}/upgrade_model



For more information, see [Upgrading custom models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgrade). The section presents rules for upgrading custom models, the effects of upgrading, and approaches for using upgraded models.





 1 February 2018 

New Korean models
: The service now offers language models for Korean: ko-KR_BroadbandModel for audio that is sampled at a minimum of 16 kHz, and ko-KR_NarrowbandModel for audio that is sampled at a minimum of 8 kHz. For more information, see [Previous-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models).

For language model customization, the Korean models are generally available (GA) for production use; for acoustic model customization, they are beta functionality. For more information, see [Language support for customization](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-support).
For an introduction to customization, including acoustic model customization, see [Understanding customization](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-customization).

New version parameter for speech recognition
: The various methods for making recognition requests now include a new version parameter that you can use to initiate requests that use either the older or upgraded versions of base and custom models. Although it is intended primarily for use with custom models that have been upgraded, the version parameter can also be used without custom models. For more information, see [Making speech recognition requests with upgraded custom models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgrade-usecustom-upgrade-use-recognition).

Updates to US English models for improved speech recognition
: The US English models, en-US_BroadbandModel and en-US_NarrowbandModel, have been updated for improved speech recognition. By default, the service automatically uses the updated models for all recognition requests. If you have custom language or custom acoustic models that are based on the US English models, you must upgrade your custom models to take advantage of the updates by using the following methods:



* POST /v1/customizations/{customization_id}/upgrade_model
* POST /v1/acoustic_customizations/{customization_id}/upgrade_model



For more information about the procedure, see [Upgrading custom models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgrade). The section presents rules for upgrading custom models, the effects of upgrading, and approaches for using upgraded models. Currently, the methods apply only to the new US English base models. But the same information will apply to upgrades of other base models as they become available.

Language model customization now available for UK English
: The service now supports language model customization for the UK English models, en-GB_BroadbandModel and en-GB_NarrowbandModel. Although the service handles UK and US English corpora and custom words in a generally similar fashion, some important differences exist:
{{{site.data.keyword.attribute-definition-list}}



 Using the machine learning model 

Leverage a machine learning model that you trained with Knowledge Studio for IBM Cloud Pak for Data by making it available to other Watson applications.

You can deploy or export a machine learning model. A dictionary can only be used to pre-annotate documents within Knowledge Studio.

You can also pre-annotate new documents with the machine learning model. See [Pre-annotating documents with the machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-preannotationwks_preannotsire) for details.



 Exporting a machine learning model 

To export a machine learning model as a .zip file, complete the following steps:



1. Log in as a Knowledge Studio administrator or project manager, and select your workspace.
2. Select Machine Learning Model > Versions.
3. Choose the version of the model that you want to export, or select Export current model.

If there is only one working version of the model, create a snapshot of the current model. This versions the model, which enables you to deploy one version, while you continue to improve the current version. The option to deploy does not appear until you create at least one version.
4. Click Export, and then click Export again to confirm.







 Deploying a machine learning model to IBM Watson Discovery for IBM Cloud Pak for Data 

When you are satisfied with the performance of the model, you can export a version to IBM Watson™ Discovery for IBM Cloud Pak for Data. This feature enables your applications to use the deployed machine learning model to enrich the insights that you get from your data to include the recognition of entities and relations that are relevant to your domain.



 Before you begin 

You must have administrative access to a [Discovery for IBM Cloud Pak for Data](https://cloud.ibm.com/docs/discovery-data) deployment.





 Procedure 



1.
Historically, the behavior of these existing ML model did not change unless the skill was updated, at which point the skill was retrained and a new model generated to replace the older one. This meant that many older models never benefited from improvements in our ML algorithms.

Watson Assistant uses continuous retraining. The Watson Assistant service continually monitors all ML models, and automatically retrains those models that have not been retrained in the previous 6 months. Watson Assistant retrains using the selected algorithm version. If the version you had selected is no longer supported, Watson Assistant retrains using the version labeled as Previous. This means that your assistant will automatically have the supported technologies applied. Assistants that have been modified during the previous 6 months will not be affected.

In most cases, this retraining will be seamless from an end-user point of view. The same inputs will result in the same actions, intents, and entities being detected. In some cases, the retraining might cause changes in accuracy.





 Instructions for Watson Assistant for IBM Cloud Pak for Data 

IBM Cloud Pak for Data

When upgrading your instance of Watson Assistant for IBM Cloud Pak for Data, as long as your existing models have been trained using an algorithm version that is still supported, your models will not be retrained during or after the upgrade.

New algorithm versions will be included in major releases (for example, 4.0.0 or 5.0.0) or minor releases (for example, 4.5.0 or 4.6.0). A monthly release might include a new algorithm version if there is more than 6 months between major or minor releases. Each algorithm version supports 2 major/minor releases or a maximum time of 12 months, whichever is first. For more information, see the [IBM Cloud Pak for Data Software Support Lifecycle Addendum](https://www.ibm.com/support/pages/ibm-cloud-pak-data-software-support-lifecycle-addendum)

Each new release includes full support for the version listed as Latest in the most recent prior release. This version is then labeled as Previous after you upgrade. In addition, each new release will support running models that were trained on the version prior to that so that upgrading won't impact your runtime.
For more information about custom model upgrade, when upgrade is necessary, and how to use older versions of custom models, see



* [Upgrading custom models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgrade)
* [Using upgraded custom models for speech recognition](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgrade-use)



Defect fix: Update capitalization documentation
: Defect fix: The documentation that describes the service's automatic capitalization of transcripts has been updated. The service capitalizes appropriate nouns only for the following languages and models:



* All previous-generation US English models
* The next-generation German model



For more information, see [Capitalization](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-basic-responseresponse-capitalization).





 2 February 2022 

New beta en-WW_Medical_Telephony model is now available
: A new beta next-generation en-WW_Medical_Telephony is now available. The new model understands terms from the medical and pharmacological domains. Use the model in situations where you need to transcribe common medical terminology such as medicine names, product brands, medical procedures, illnesses, types of doctor, or COVID-19-related terminology. Common use cases include conversations between a patient and a medical provider (for example, a doctor, nurse, or pharmacist).

The new model is available for all supported English dialects: Australian, Indian, UK, and US. The new model supports language model customization and grammars as beta functionality. It supports most of the same parameters as the en-US_Telephony model, including smart_formatting for US English audio. It does not support the following parameters: low_latency, profanity_filter, redaction, and speaker_labels.