vania-janet/multiturn-rag-retrieval-data huggingface dataset:
For more information about next-generation models and low latency, see



* [Next-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-ng)
* [Customization support for next-generation models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-supportcustom-language-support-ng)
* [Low latency](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-interimlow-latency)



Improved language model customization for next-generation English and Japanese models
: The service now provides improved language model customization for next-generation English and Japanese models:



* en-AU_Multimedia
* en-AU_Telephony
* en-IN_Telephony
* en-GB_Multimedia
* en-GB_Telephony
* en-US_Multimedia
* en-US_Telephony
* ja-JP_Multimedia
* ja-JP_Telephony



Visible improvements to the models: The new technology improves the default behavior of the new English and Japanese models. Among other changes, the new technology optimizes the default behavior for the following parameters:



* The default customization_weight for custom models that are based on the new versions of these models changes from 0.2 to 0.1.
* The default character_insertion_bias for custom models that are based on the new versions of these models remains 0.0, but the models have changed in a manner that makes use of the parameter for speech recognition less necessary.



Upgrading to the new models: To take advantage of the improved technology, you must upgrade any custom language models that are based on the new models. To upgrade to the new version of one of these base models, do the following:



1. Change your custom model by adding or modifying a custom word, corpus, or grammar that the model contains. Any change that you make moves the model to the ready state.
2.
For example, you can update commands such as pip/pip3 for discovering Python packages that require a later pip version.

The following example shows how to use the cra-custom-script to update the pip version:

!/usr/bin/env bash

if [ "${PIPELINE_DEBUG:-0}" == 1 ]]; then
trap env EXIT
env | sort
set -x
fi

python3 -m pip install --upgrade pip

If your Dockerfile uses an image from a private Docker registry, you can use the cra-custom-script-path parameter to authenticate to a private Docker registry before you run Code Risk Analyzer and to allow Code Risk Analyzer to pull this image for scanning.

The following example shows how to use the cra-custom-script to authenticate to the ibmcloud container registry:

!/usr/bin/env bash

if [ "${PIPELINE_DEBUG:-0}" == 1 ]]; then
trap env EXIT
env | sort
set -x
fi

ibmcloud cr login





 Debugging the Code Risk Analyzer in DevSecOps 

To help with debugging, you can run the Code Risk Analyzer locally as a command-line interface (CLI) on your own local machine. For information about running the ibmcloud cra bom-generate command to generate a BOM file, see [Bill of Materials (BOM)](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-cra-cli-pluginbom-generate-command). After you generate the BOM file, use the ibmcloud cra cve command to list any vulnerabilities. For more information about running the ibmcloud cra cve command, see [Vulnerability scan](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-cra-cli-pluginvulnerability-command).

Make sure that the run-cra task does not contain any errors.
[update-collection](https://cloud.ibm.com/apidocs/discoveryupdate-collection) Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, environment_id, collection_id, name, description, configuration_id Update a collection. 
 [create-expansions](https://www.ibm.com/watsoncreate-expansions) Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, environment_id, collection_id, expansions Create or update the expansion list. 
 [delete-expansions](https://cloud.ibm.com/apidocs/discoverydelete-expansions) Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, environment_id, collection_id Delete the expansion list. 
 [list-expansions](https://cloud.ibm.com/apidocs/discoverylist-expansions) Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, environment_id, collection_id Get the expansion list. 
 [add-document](https://cloud.ibm.com/apidocs/discoveryadd-document) Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, environment_id, collection_id, file, metadata, file_content_type Add a document.
update-counterexample Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, workspace_id, text, new_text Update a counterexample. 
 create-entity Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, workspace_id, entity, description, metadata, values, fuzzy_match Create an entity. 
 delete-entity Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, workspace_id, entity Delete an entity. 
 get-entity Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, workspace_id, entity, export, include_audit Get information about an entity. 
 list-entities Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, workspace_id, export, page_limit, include_count, sort, cursor, include_audit List entities. 
 update-entity Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, workspace_id, entity, new_entity, new_description, new_metadata, new_fuzzy_match, new_values Update an entity. 
 create-value Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, workspace_id, entity, value, metadata, synonyms, patterns, value_type Add an entity value.
* Upgrade model train classic operator:

./cpd-cli upgrade --repo ./repo.yaml --assembly modeltrain-classic --arch Cluster_architecture --namespace <Project> --transfer-image-to <Registry_location> --cluster-pull-prefix <Registry_from_cluster> --ask-pull-registry-credentials --ask-push-registry-credentials
* Upgrade Elasticsearch operator:

./cpd-cli upgrade --repo ./repo.yaml --assembly ibm-cloudpakopen-elasticsearch-operator --namespace <Project> --transfer-image-to <Registry_location> --cluster-pull-prefix <Registry_from_cluster> --ask-pull-registry-credentials --ask-push-registry-credentials

where <Project> is the namespace where your Discovery for Cloud Pak for Data 2.2.0 instance is installed, where <Registry_location> is the location of the images that you pushed to the registry server, and where <Registry_from_cluster> is the location from which pods on the cluster can pull images.



* When you install on IBM Cloud Pak for Data 3.5, you might encounter the following issue:



* Error: If you try to provision the Discovery service on a cluster where Planning Analytics is running, some of the Discovery pods don't start and installation fails. The logs for the pod show messages such as, java.lang.NumberFormatException: For input string.
* Cause: An environment variable named COUCHDB_PORT is added to the Kubernetes cluster by the couchdb service that is installed with Planning Analytics. Discovery does not use couchdb, and therefore does not specify a value for this environment variable. However, some pods attempt to parse the variable, which results in the error.
* Solution: [Install patch cpd-watson-discovery-2.2.1-patch-1](https://www.ibm.com/docs/en/cloud-paks/cp-data/3.5.0?topic=iwd-installing-watson-discoverysvc-install__patches-section), which fixes this issue.





Also, see the issues in all previous releases.
Upgrading does not support upgrading across multiple releases and must be performed, release by release. Some updates will require execution of Terraform upgrade commands to modify the config files also changes to the Terraform state file. These steps cannot be performed within Schematics. Workspace Terraform templates must be upgraded using a local copy of Terraform. Follow the steps below for upgrading the 0.x releases.



Terraform versions list

 Version Recommendation 

 v0.12 Review the [v0.13 upgrade guide](https://developer.hashicorp.com/terraform/language/v1.1.x/upgrade-guides/0-13) and follow the instructions [Upgrading a Terraform v0.12 Workspace to v0.13](https://cloud.ibm.com/docs/schematics?topic=schematics-migrating-terraform-versionmigrate-steps12). Schematics has deprecated Terraform v0.12. 
 v0.13 For the Terraform v0.14 upgrade, you must run terraform apply with Terraform v0.14 to complete its state format upgrades. If you get any errors, see the [v0.14 upgrade guide](https://developer.hashicorp.com/terraform/language/v1.1.x/upgrade-guides/0-14). Follow the instructions upgrade-13-to10 
 v0.14 You can upgrade directly to the [Terraform v1.0](https://developer.hashicorp.com/terraform/language/v1.1.x/upgrade-guides/1-1) version. Review the [v0.15 upgrade guide](https://developer.hashicorp.com/terraform/language/v1.1.x/upgrade-guides/0-15). 
 v0.15 You can upgrade directly to the [Terraform v1.0](https://developer.hashicorp.com/terraform/language/v1.1.x/upgrade-guides/1-0) version.
You can upgrade to a new version when [restoring a backup](https://cloud.ibm.com/docs/messages-for-rabbitmq?topic=cloud-databases-dashboard-backupsrestoring-a-backup) from the Backups tab of your Deployment Overview. Clicking Restore on a backup brings up a dialog box where you can change some options for the new deployment. One of them is the database version, which is auto-populated with the versions available for you to upgrade to. Select a version and click Restore to start the provision and restore process.





 Upgrading through the CLI 

When you upgrade and restore from backup through the IBM Cloud CLI, use the provisioning command from the resource controller.

ibmcloud resource service-instance-create <service-name> <service-id> <service-plan-id> <region>

The parameters service-name, service-id, service-plan-id, and region are all required. You also supply the -p with the version and backup ID parameters in a JSON object. If you need to use a key protect key, resize the deployment, or allocate dedicated cores, you can do so with the optional parameters key_protect_key, members_disk_allocation_mb, members_memory_allocation_mb, or members_cpu_allocation_count, along with their respective values to the body of the request.

ibmcloud resource service-instance-create example-upgrade messages-for-rabbitmq standard us-south -p '{"backup_id": "crn:v1:bluemix:public:messages-for-rabbitmq:us-south:a/54e8ffe85dcedf470db5b5ee6ac4a8d8:1b8f53db-fc2d-4e24-8470-f82b15c71717:backup:06392e97-df90-46d8-98e8-cb67e9e0a8e6", "version":3.8}'





 Upgrading through the API
If you use Python, review the following steps and examples to migrate to Secrets Manager API v2.



1. Update your environment to use the latest version (2.0.0) by updating the requirements.txt or running pip install --upgrade "ibm-secrets-manager-sdk".
2. Update your import statements to use v2.
3. Remove the secret type, and the metadata and resources wrapper from your resource for each operation that you're using.



Responses don't contain the resources object. They contain only the secret JSON, or a secrets field, in case of a list.



 Comparing v1 and v2 Python examples 

Compare the following examples to see the changes between the v1 Python code snippet and how the code must look like in v2.

v1 example

response = secretsManager.create_secret(
'arbitrary'
{'collection_type: 'application/vnd.ibm.secrets-manager.secret+json', 'collection_total': 1},
[{'name': 'example-arbirary-secret', 'description': 'Extended description for this secret.', 'payload': 'secret-data'}]
)

v2 example

secret_prototype_model = {
'description': 'Extended description for this secret.',
'expiration_date': '2023-10-05T11:49:42Z',
'name': 'example-arbitrary-secret',
'secret_group_id': 'default'
'secret_type':'arbitrary'
'payload': 'secret-data'
}
response = secretsManager.create_secret(
secret_prototype=secret_prototype_model,
)







 Migrating Go applications 

If you use Go, review the following steps and examples to migrate to Secrets Manager API v2.



1. Update your go.mod file to use the latest version (2.0.0).
2. Update your import statements to use v2.
3.
istio-system istiod-5b9b5bfbb7-khcht 1/1 Running 0 60m



7. [Update your istioctl client and sidecars](https://cloud.ibm.com/docs/containers?topic=containers-istioupdate_client_sidecar).







 Updating the istioctl client and sidecars 

Whenever the Istio managed add-on is updated, update your istioctl client and the Istio sidecars for your app.

For example, the patch version of your add-on might be updated automatically by IBM Cloud Kubernetes Service, or you might [update the minor version of your add-on](https://cloud.ibm.com/docs/containers?topic=containers-istioistio_minor). In either case, update your istioctl client and your app's existing Istio sidecars to match the Istio version of the add-on.



1. Get the version of your istioctl client and the Istio add-on control plane components.

istioctl version --short=false

Example output

client version: version.BuildInfo{Version:"1.11.2"}
pilot version: version.BuildInfo{Version:1.18.0}
pilot version: version.BuildInfo{Version:1.18.0}
data plane version: version.ProxyInfo{ID:"istio-egressgateway-77bf75c5c-vp97p.istio-system", IstioVersion:1.18.0}
data plane version: version.ProxyInfo{ID:"istio-egressgateway-77bf75c5c-qkhgm.istio-system", IstioVersion:1.18.0}
data plane version: version.ProxyInfo{ID:"istio-ingressgateway-6dcb67b64d-dffhq.istio-system", IstioVersion:1.18.0}
This is currently an [experimental](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks-faqsexperimental) feature of the service.



 Before you begin 

You must have administrative access to a Watson Discovery service instance, and know the IBM Cloud space and instance names that are associated with it.





 Procedure 

To deploy a rule-based model to Watson Discovery, complete the following steps:



1. Log in as a Knowledge Studio administrator or project manager, and select your workspace.
2. Select the Rule-based Model > Versions > Rule-based Model tab.
3. Choose the version of the model that you want to deploy.

If there is only one working version of the model, save the current model for deployment by clicking Save for Deployment. This versions the model, which enables you to deploy one version, while you continue to improve the current version. Saving the version might take a few minutes. The option to deploy does not appear until after the version is created.

Note: Each version can be deployed to only one service instance. If you want to deploy the same model to more than one instance, create a version for each instance.
4. Click Deploy, choose to deploy it to Discovery, and then click Next.
5. Provide the IBM Cloud space and instance. If necessary, select a different region.
6. Click Deploy.
7. The deployment process might take a few minutes. To check the status of the deployment, click Status on the Versions tab next to the version that you deployed.

If the model is still being deployed, the status indicates "publishing". After deployment completes, the status changes to "available" if the deployment was successful, or "error" if problems occurred.

Once available, make a note of the model ID (model_id).







 What to do next 

To use the model, you must export the model, and then import it into Discovery.



1. Select the Rule-based Model > Versions > Rule-based Model tab.
2. Click Export current model.
If you are running the deployable architecture as-is, no updates are necessary. Your configuration might require you to update this file. If you need to specify a particular Terraform or provider version, complete the following steps:



1. Open the versions.tf file.
2. Update required_version to the Terrafrom version that users need to use.
3. Update version to the provider version that users need to use.









 Testing your customized deployable architecture 

Before you onboard your customized deployable architecture to a private catalog, test your customization and ensure that the architecture runs as intended. To test your architecture with the Terraform command line, complete the following steps:



1. After you customize the architecture, initialize the Terraform CLI. For more information, see [Initializing Working Directories](https://www.terraform.io/cli/init).

terraform init
2. Provision the resources. For more information, see [Provisioning Infrastructure with Terraform](https://www.terraform.io/cli/run).



1. Run terraform plan to generate a Terraform execution plan to preview the proposed actions.

terraform plan
2. Run terraform apply to create the resources that are defined in the plan.

terraform apply









 Creating a release 

If your tests were successful, you can prepare to onboard your architecture to a private catalog by making sure that your files are in the repository and creating a release. For more information, see [Creating a release](https://cloud.ibm.com/docs/sell?topic=sell-source-repo-setupcreate-release).





 Leveraging automation to share to a private catalog 

The customization bundle has an automation directory that includes a couple of options for starter pipelines. These pipelines can be used to automate the lifecycle of your customized products that you share by using a private catalog on IBM Cloud. This includes managing the onboarding, validation, and sharing of the deployable architecture version to a private catalog. README files are included for each pipeline option that provide details on how to leverage it.
Go to the backup directory that you specified in the ${BACKUP_DIR} parameter in the previous procedure.
4. Run the following command to download the pgmig tool from the [GitHub Watson Developer Cloud Community](https://github.com/watson-developer-cloud/community/tree/master/watson-assistant/data) repository.

In the first command, update <WA_VERSION> to the version that you want to restore. For example, update <WA_VERSION> to 4.6.0 if you want to restore Watson Assistant 4.6.0.

wget https://github.com/watson-developer-cloud/community/raw/master/watson-assistant/data/<WA_VERSION>/pgmig
chmod 755 pgmig
5. Create the following two configuration files and store them in the same backup directory:



* resourceController.yaml: The Resource Controller file keeps a list of all provisioned Watson Assistant instances. See [Creating the resourceController.yaml file](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-backupbackup-resource-controller-yaml).
* postgres.yaml: The Postgres file lists details for the target Postgres pods. See [Creating the postgres.yaml file](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-backupbackup-postgres-yaml).



6. Get the secret:

oc get secret ${INSTANCE}-postgres-ca -o jsonpath='{.data.ca.crt}' | base64 -d | tee ${BACKUP_DIR}/ca.crt | openssl x509 -noout -text



* Replace ${INSTANCE} with the name of the Watson Assistant instance that you want to back up.
* Replace ${BACKUP_DIR} with the directory where the postgres.yaml and resourceController.yaml files are located.



7.
For more information, see [Training failures for custom acoustic models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-acousticfailedTraining-acoustic).
* training indicates that the model is being trained. The progress field changes from 0 to 100 when training is complete.
* available indicates that the model is trained and ready to use. The progress field is 100.
* upgrading indicates that the model is being upgraded. The progress field is 0.
* failed indicates that training of the model failed. The progress field is 0. For more information, see [Training failures for custom acoustic models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-acousticfailedTraining-acoustic).



Use a loop to check the status of the training once a minute until the model becomes available. For more information about other fields that are returned by the method, see [Listing custom acoustic models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-manageAcousticModelslistModels-acoustic).





 Training failures for custom acoustic models 

Training fails to start if the service is handling another request for the custom acoustic model. A conflicting request could be another training request or a request to add audio resources to the model. The service returns a status code of 409.

Training also fails to start for the following reasons:



* The custom model contains less than 10 minutes of audio data.
* The custom model contains more than 200 hours of audio data.
* One or more of the custom model's audio resources is invalid.
* You passed a custom language model with the custom_language_model_id query parameter that is not in the available state. A custom language model must be fully trained and available to be used to train a custom acoustic model.
* You passed an incompatible custom language model with the custom_language_model_id query parameter. Both custom models must be based on the same version of the same base model.



The service returns a status code of 400 and sets the custom model's status to failed.
For more information, see [Upgrading to Terraform v0.13 documentation](https://developer.hashicorp.com/terraform/language/v1.1.x/upgrade-guides/0-13). The upgrade command generates a versions.tf file with a terraform configuration block.
5. Edit the versions.tf file to set the source parameter to source = "IBM-Cloud/ibm" as shown in the code block.

versions.tf file

terraform {
required_providers {
ibm = {
 TF-UPGRADE-TODO

 No source detected for this provider. You must add a source address
 in the following format:

source = "IBM-Cloud/ibm"

 For more information, see the provider source documentation:

}
}
required_version = ">= 0.13"
}
Show more
6. Download the Terraform state file from the existing Schematics Workspace, by using [Schematics state pull](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referencestate-pull) command.

When, the workspace is created with the tfstate Schematics considers it as a secure file. Also, you cannot pull the created tfstate file through UI. You need to use the command line to [pull](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referencestate-pull) the state file and [create workspace](https://cloud.ibm.com/docs/schematics?topic=schematics-schematics-cli-referenceschematics-workspace-new).

Copy the downloaded state file as terraform.tfstate into the Terraform execution folder.
7. Run the state replace provider command in on the command-line to update the IBM Cloud provider version in the state file.

terraform state replace-provider registry.terraform.io/-/ibm registry.terraform.io/ibm-cloud/ibm.
8.