vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Generating word timings 

You can use the WebSocket interface of the IBM Watson® Text to Speech service to obtain timing information for user-specified locations such as word boundaries or for all words of the input text:



* Include the SSML <mark> element in input text to identify the time at which the marker occurs in the audio.
* Specify the timings parameter of a JSON text message to obtain timing information for all strings of the input text.



Timing information is useful for synchronizing the audio and the input text. For example, you can coordinate an avatar's or robot's gestures with the content of the synthesized speech.

The <mark> element and the timings parameter are available only with the WebSocket interface, not with the HTTP interface. Also, the timings parameter is not supported for Japanese input text.



 How the service returns word timings 

To return mark or word timing information, the service multiplexes independent binary and text streams to construct its response:



* For each <mark> element, the service returns a JSON text message. Each message indicates the exact time from the beginning of the synthesized audio at which the mark occurs.
* For word timings for all strings, the service returns one or more JSON text messages. Each message contains an array of words and their start and end times from the beginning of the synthesized audio.



The binary and text streams that the service sends are independent. So the service has little control over the number of audio chunks that it delivers and when the user receives the text and audio messages. For example, if the audio is synthesized more quickly than it is compressed, all text messages might arrive before any of the audio arrives.

In practical terms, the service can send an arbitrary number of audio chunks, including multiple chunks of audio before and after each text message. It is also possible for a single binary chunk to contain audio data that both precede and follow the timing information for a mark or word.

However, the text message that contains the timing information always arrives before the binary chunk that contains the corresponding audio. Moreover, the audio messages always arrive in order so that you can construct complete and accurate audio of the synthesized text from the binary results.





 Specifying an SSML mark
The following example shows the token %HESITATION for a US English transcript:

{
"result_index": 0,
"results": [
{
"alternatives":
{
"confidence": 0.99,
"transcript": ". . . that %HESITATION that's a . . ."
}
],
"final": true
}
]
}

Hesitation markers can appear in both interim and final results. Enabling smart formatting prevents hesitation markers from appearing in final results for US English. For more information, see [Smart formatting](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-formattingsmart-formatting).

Hesitation markers can also appear in other fields of a transcript. For example, if you request [Word timestamps](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-metadataword-timestamps) for the individual words of a transcript, the service reports the start and end time of each hesitation marker.

{
"result_index": 0,
"results": [
{
"alternatives":
{
"timestamps":
. . .

"that",
7.31,
7.69
],

"%HESITATION",
7.69,
7.98
],

"that's",
7.98,
8.41
],

"a",
8.41,
8.48
],
. . .
],
"confidence": 0.99,
"transcript": ". . . that %HESITATION that's a . . ."
}
],
"final": true
}
]
}
Show more
"confidence": 0.55,
"final": false
},
{
"from": 3.57,
"to": 3.95,
"speaker": 1,
"confidence": 0.63,
"final": true
}
]
}
Show more

The transcript field shows the final transcript of the audio, which lists the words as they were spoken by all participants. By comparing and synchronizing the speaker labels with the timestamps, you can reassemble the conversation as it occurred. The confidence field for each speaker label indicates the service's confidence in its identification of the speaker.



Table 1. Speaker labels example

 Timestamps <br>(0.00 - 2.14) Speaker labels <br>(0.00 - 2.14) Timestamps <br>(2.15 - 3.95) Speaker labels <br>(2.15 - 3.95) 

 0.28, <br>0.79 <br>"hello", "from": 0.28, <br>"to": 0.79, <br>"speaker": 2, <br>"confidence": 0.52, <br>"final": false 2.15, <br>2.44 <br>"how's", "from": 2.15, <br>"to": 2.44, <br>"speaker": 2, <br>"confidence": 0.51, <br>"final": false 
 1.07, <br>1.53 <br>"yeah", "from": 1.07, <br>"to": 1.53, <br>"speaker": 1, <br>"confidence": 0.63, <br>"final": false 2.45, <br>3.03 <br>"Billy", "from": 2.45, <br>"to": 3.03, <br>"speaker": 2, <br>"confidence": 0.55, <br>"final": false
For more information, see [Uninstalling Watson Text to Speech](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=speech-uninstalling-watson-text).



New guidance for scaling up your installation
: The service now provides updated guidance about scaling up your installation. The information includes specifying the number of pods and the maximum number of concurrent sessions for enhanced neural or neural voices. For more information, see [Administering Watson Text to Speech](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=speech-administering-watson-text).

Command-line updates to import and export utilities
: The commands that are used with the import and export utilities for the Speech services include new options and arguments. The import and export utilities are also the foundation for backing up and restoring the services and for migrating from IBM Cloud Pak for Data version 3.5 to version 4.0.3. For more information about using the utilities, see



* [Administering Watson Text to Speech](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=speech-administering-watson-text)
* [Upgrading Watson Text to Speech](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.0?topic=speech-upgrading-watson-text)



New property for managing concurrency and speech synthesis
: The new global.ttsVoiceMarginalCPU property manages the tradeoff between concurrency and speech synthesis speed. The default value of 400 offers a reasonable balance for most customers and maintains real-time synthesis. For information about modifying this value to suit your needs, contact IBM Support.

New support for neural voices
: All neural voices that are currently available for Text to Speech for IBM Cloud are now also available for installation on Text to Speech for IBM Cloud Pak for Data. The following languages and voices are now available:



* Arabic:ar-MS_OmarVoice
IBM Cloud Pak for Data

curl -X POST --header "Authorization: Bearer {token}" --data-binary @healthcare.txt "{url}/v1/customizations/{customization_id}/corpora/healthcare"

The method also accepts an optional allow_overwrite query parameter that overwrites an existing corpus for a custom model. Use the parameter if you need to update a corpus file after you add it to a model.

The method is asynchronous. It can take on the order of minutes to complete. The duration of the operation depends on the total number of words in the corpus and the current load on the service. For custom models that are based on previous-generation models, the duration also depends on the number of new words that the service finds in the corpus. For more information about checking the status of a corpus, see [Monitoring the add corpus request](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-languageCreatemonitorCorpus).

You can add any number of corpora to a custom model by calling the method once for each corpus text file. The addition of one corpus must be fully complete before you can add another. A corpus has the status being_processed when you first add it to a model. Its status becomes analyzed when the service finishes processing it.

For custom models that are based on previous-generation models, after the addition of a corpus is complete, examine the new custom words that were extracted from it to check for typographical and other errors. For more information, see [Validating a words resource for previous-generation models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWordsvalidateModel).



 Monitoring the add corpus request 

The service returns a 201 response code if the corpus is valid. It then asynchronously processes the contents of the corpus. You cannot submit requests to add data to the custom model or to train the model until the service's analysis of the corpus for the current request completes.
stop_timestamp string. Time in the ISO format yyyy-MM-ddTHH:mm:ss.SSSZ Time when the transaction ended. 
 response_milliseconds number Latency in milliseconds between when silence is detected in the caller's speech and a final result from Speech to Text is received. 
 echo_detected Boolean Indicates whether an echo was detected. The value can be true or false. 
 confidence number The confidence score of the final utterance. 



The response object contains the following keys:



Keys for the response object

 Key Type Description 

 type string The response type: <br>text_to_speech - a command to play an utterance to the caller <br>sms - a command to send a SMS to the caller <br>url - a command to play an audio file to the caller <br>transfer - a command to transfer a call <br>text_to_speech_config - a command to change Text to Speech settings <br>speech_to_text_config - a command to change the Speech to Text settings <br>pause_speech_to_text - a command to stop speech recognition <br>unpause_speech_to_text - a command to start speech recognition <br>pause_dtmf - a command to stop DTMF recognition <br>unpause_dtmf - a command to start speech recognition <br>enable_speech_barge_in - a command to enable speech barge-in so that callers can interrupt playback by speaking <br>disable_speech_barge_in - a command to disable speech barge-in so that playback isn't interrupted when callers speak over top of the played back audio <br>enable_dtmf_barge_in - a command to enables DTMF barge-in so that callers can interrupt playback from the phone integration by pressing a key. <br>disable_dtmf_barge_in - a command to disable DTMF barge-in so that playback from the phone integration isn't interrupted when callers press keys <br>dtmf - a command to send DTMFs to the caller <br>hangup - a command to disconnect a call
But many factors contribute to the amount of data that you need for an effective custom language model. Although it is not possible to provide the exact number of words that you need to add for any custom model or application, even adding a few words to a custom model can improve speech recognition. For more information about limits on the number of words that you can add and for other factors that affect the amount of data that you need, see [How much data do I need?](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWordswordsResourceAmount).
* How does custom model upgrading work?

When a new version of a previous-generation base model is released to improve the quality of speech recognition, you must upgrade any custom language and custom acoustic models that are based on that model to take advantage of the updates. When you upgrade a custom model, you do not need to upgrade its resources individually. The service upgrades the resources automatically. Custom model upgrading applies only to previous-generation models.



* For more information about upgrading a custom model, see [Upgrading custom models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgrade).
* For more information about using an upgraded custom model for a speech recognition request, see [Using upgraded custom models for speech recognition](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgrade-use).



* Can the Speech to Text service transcribe numbers as digits instead of strings?

For US English, Japanese, and Spanish audio, you can use smart formatting to convert certain strings, such as digits and numbers, to more conventional representations. Smart formatting is beta functionality. For more information, see [Smart formatting](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-formattingsmart-formatting).



Text to Speech



* How do I access my service credentials?

How you access your service credentials depends on whether you are using Text to Speech with IBM Cloud® or IBM Cloud Pak® for Data.
Asynchronous HTTP Query parameter of POST /v1/recognitions method 







 timestamps 

An optional boolean that indicates whether the service produces timestamps for the words of the transcript. By default (false), timestamps are not returned. For more information, see [Word timestamps](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-metadataword-timestamps).



Table 27. The timestamps parameter

 Availability and usage Description 

 Previous-generation models Generally available for all languages. 
 Next-generation models Generally available for all languages. 
 WebSocket Parameter of JSON start message 
 Synchronous HTTP Query parameter of POST /v1/recognize method 
 Asynchronous HTTP Query parameter of POST /v1/recognitions method 







 Transfer-Encoding 

An optional value of chunked that causes the audio to be streamed to the service. By default, audio is sent all at once as a one-shot delivery. For more information, see [Audio transmission](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-inputtransmission).



Table 28. The Transfer-Encoding parameter

 Availability and usage Description 

 Previous-generation models Generally available for all languages. 
 Next-generation models Generally available for all languages. 
 WebSocket Not applicable; always streamed 
 Synchronous HTTP Request header of POST /v1/recognize method 
 Asynchronous HTTP Request header of POST /v1/recognitions method 







 word_alternatives_threshold 

An optional double between 0.0 and 1.0 that specifies the threshold at which the service reports acoustically similar alternatives for words of the input audio. By default, word alternatives are not returned. For more information, see [Word alternatives](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-spottingword-alternatives).



Table 29. The word_alternatives_threshold parameter

 Availability and usage Description 

 Previous-generation models Generally available for all languages.
Several new APIs have been introduced to the IBM Cloud Object Storage SDKs to provide support for applications working with retention policies. Select a language (HTTP, Java, JavaScript, or Python) at the beginning of this page to view examples that use the appropriate Object Storage SDK.

All code examples assume the existence of a client object that is called cos that can call the different methods. For details on creating clients, see the specific SDK guides.

All date values used to set retention periods are Greenwich mean time. A Content-MD5 header is required to ensure data integrity, and is automatically sent when using an SDK.



 Add a retention policy on an existing bucket 

This implementation of the PUT operation uses the protection query parameter to set the retention parameters for an existing bucket. This operation allows you to set or change the minimum, default, and maximum retention period. This operation also allows you to change the protection state of the bucket.

Objects written to a protected bucket cannot be deleted until the protection period has expired and all legal holds on the object are removed. The bucket's default retention value is given to an object unless an object-specific value is provided when the object is created. Objects in protected buckets that are no longer under retention (retention period has expired and the object does not have any legal holds), when overwritten, will again come under retention. The new retention period can be provided as part of the object overwrite request or the default retention time of the bucket will be given to the object.

The minimum and maximum supported values for the retention period settings MinimumRetention, DefaultRetention, and MaximumRetention are a minimum of 0 days and a maximum of 365243 days (1000 years).

A Content-MD5 header is required. This operation does not make use of extra query parameters.

For more information about endpoints, see [Endpoints and storage locations](https://cloud.ibm.com/docs/services/cloud-object-storage?topic=cloud-object-storage-endpointsendpoints)

Syntax

PUT https://{endpoint}/{bucket-name}?protection= path style
PUT https://{bucket-name}.{endpoint}?protection= virtual host style

Example request
* For an overview of the next-generation models and their technology, see [Next-Generation Watson Speech to Text](https://medium.com/ibm-watson-speech-services/next-generation-watson-speech-to-text-650fd66d95d0).
* For more information about the technology that underlies the next-generation models, see [Advancing RNN Transducer Technology for Speech Recognition](https://arxiv.org/abs/2103.09935).
* For information about migrating from previous-generation to next-generation models, see [Migrating to next-generation models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-migrate).





 Next-generation model types 

The service makes available two types of next-generation models:



* [Telephony models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-ngmodels-ng-telephony) are intended specifically for audio that is communicated over a telephone. Like previous-generation narrowband models, telephony models are intended for audio that has a minimum sampling rate of 8 kHz.
* [Multimedia models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-ngmodels-ng-multimedia) are intended for audio that is extracted from sources with a higher sampling rate, such as video. Use a multimedia model for any audio other than telephonic audio. Like previous-generation broadband models, multimedia models are intended for audio that has a minimum sampling rate of 16 kHz.



Choose the model type that most closely matches the source and sampling rate of your audio. The service automatically adjusts the sampling rate of your audio to match the model that you specify. To achieve the best recognition accuracy, also consider the frequency content of your audio.
Previously, in some cases the day of the week was omitted and the date was presented incorrectly. Note that smart formatting is beta functionality.

Defect fix: Update documentation for speech hesitation words for next-generation models
: Defect fix: Documentation for speech hesitation words for next-generation models has been updated. More details are provided about US English and Japanese hesitation words. Next-generation models include the actual hesitation words in transcription results, unlike previous-generation models, which include only hesitation markers. For more information, see [Speech hesitations and hesitation markers](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-basic-responseresponse-hesitation).

Security vulnerabilities addressed
: The following security vulnerabilities have been fixed:



* [Security Bulletin: IBM Watson Speech Services Cartridge for IBM Cloud Pak for Data is vulnerable to a denial of service in Python (CVE-2020-10735)](https://www.ibm.com/support/pages/node/6988073)
* [Security Bulletin: IBM Watson Speech Services Cartridge for IBM Cloud Pak for Data is vulnerable to phishing attacks in Python (CVE-2021-28861)](https://www.ibm.com/support/pages/node/6988075)
* [Security Bulletin: IBM Watson Speech Services Cartridge for IBM Cloud Pak for Data is vulnerable to a denial of service in Pypa Setuptools (CVE-2022-40897)](https://www.ibm.com/support/pages/node/6988077)
* [Security Bulletin: IBM Watson Speech Services Cartridge for IBM Cloud Pak for Data is vulnerable to a sensitive information exposure in systemd (CVE-2022-4415)](https://www.ibm.com/support/pages/node/6988079)
* [Security Bulletin: IBM Watson Speech Services Cartridge for IBM Cloud Pak for Data is vulnerable to a denial of service in Python (CVE-2022-45061)](https://www.ibm.com/support/pages/node/6988081)
For more information, see [Activity log](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-activity-log).





 16 April 2023 

Allow changing topics in free text and regex responses
: By default, customers can't change topics when the assistant is asking for a free text response or when an utterance matches the pattern in a regex response. Now free text and regex customer response types have a setting to allow a user to digress and change topics. For more information, see [Enabling changing the topic for free text and regex customer responses](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-change-topicchange-topic-free-text-regex).

Autolearning beta for dialog removed
: As of this release, the autolearning beta has been removed from the Analytics section in dialog. You can use actions to test a new and improved autolearning beta. For more information, see [Using autolearning to improve assistant responses](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-autolearn).





 7 April 2023 

Never return choice when customer changes topics
: If a customer changes a topic during a conversation, there might be some situations when you might not want them to return to the previous action. If you need to do this, a new Never return choice is available in Action settings. For more information, see [Disabling returning to the original topic](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-change-topicchange-topic-never-return).





 22 March 2023 

Autolearning
: Autolearning is a beta feature that is available for evaluation and testing purposes in English-language assistants only. Use autolearning to enable your assistant to learn from interactions with your customers and improve responses. For more information, see [Using autolearning to improve assistant responses](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-autolearn).
For more information about all available next-generation models, see [Next-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-ng).

Updated Korean telephony and multimedia next-generation models
: The existing Korean next-generation models have been updated:



* The ko-KR_Telephony model has been updated for improved low-latency support for speech recognition.
* The ko-KR_Multimedia model has been updated for improved speech recognition. The model now also supports low latency.



Both models are generally available, and both support language model customization and grammars. You do not need to upgrade custom language models that are based on these models. For more information about all available next-generation models, see [Next-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-ng).

Defect fix: Confidence scores are now reported for all transcription results
: Defect fix: Confidence scores are now reported for all transcription results. Previously, when the service returned multiple transcripts for a single speech recognition request, confidence scores might not be returned for all transcripts.





 11 April 2022 

New Brazilian Portuguese pt-BR_Multimedia next-generation model
: The service now offers a next-generation multimedia model for Brazilian Portuguese: pt-BR_Multimedia. The new model supports low latency and is generally available. It also supports language model customization and grammars. For more information about next-generation models and low latency, see



* [Next-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-ng)
* [Customization support for next-generation models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-supportcustom-language-support-ng)