Lingua: Addressing Scenarios for Live Interpretation and Automatic Dubbing (2022)

Abstract: Lingua is an application developed for the Church of Jesus Christ of Latter-day Saints that performs both real-time interpretation of live speeches and automatic video dubbing (AVD). Like other AVD systems, it can perform synchronized automatic dubbing, given video files and optionally, corresponding text files using a traditional ASR–MT–TTS pipeline. Lingua’s unique contribution is that it can also operate in real-time with a slight delay of a few seconds to interpret live speeches. If no source-language script is provided, the translations are exactly as recognized by ASR and translated by MT. If a script is provided, Lingua matches the recognized ASR segments with script segments and passes the latter to MT for translation and subsequent TTS. If a human translation is also provided, it is passed directly to TTS. Lingua switches between these modes dynamically, enabling translation of off-script comments and different levels of quality for multiple languages. (see extended abstract)

## introduction lingua is an automatic speech-to-speech (sts) interpreter and video dubber that is being developed and piloted at the church of jesus christ of latter-day saints. this application is suitable for interpreting live speeches and video recordings on-the-fly from english to 38 other languages. it can also assist in creating exactly synchronized audio tracks for the professional dubbing of video recordings. a common pitfall of sts systems built on a cascaded architecture is the propagation of errors from one component of the pipeline to the later components @xcite . for instance, if the asr module registers "ice cream" as "i scream", the mt module is typically incapable of rectifying the mistake, and it will indiscriminately translate the incorrect transcription. lingua affords the possibility of course corrections by allowing the user to upload a script of the speech in advance. it then uses a dynamic programming algorithm to align the asr transcription with the official script on the phonemic level and override the asr when a likely match is found. not only does this feature improve the accuracy of downstream outputs, it can also reduce the décalage or lag between the original speaker and the live interpretation @xcite . rather than waiting for the speaker to finish their sentence, lingua can detect which sentence is being uttered early on and get a head start on producing the appropriate outputs. lingua is intended to facilitate the distribution of multimedia content into languages for which it would otherwise be cost-inefficient to provide manual interpretation and dubbing. it is of special interest for improving the accessibility of live events and undubbed video recordings for speakers of underserved languages when interpreters are not readily available. it can also accelerate the process of dubbing previously recorded content, including not only speeches but also short films and other multimedia presentations. ## related work there has been significant research over the past few years in the field of automatic speech-tospeech translation and more specifically in its application to automatic video dubbing (avd). for example, improvements detailed by amazon researchers include the use of human-like, customizable neural voices, integration with neural mt, adjustments to the duration and prosody of translated utterances, and the handling of background noise and reverberation @xcite . several companies are now joining the fray with their own research and product development. apptek recently announced plans to release an avd tool that includes speaker diarization, limited prosody transfer, and basic utterance length control, with planned improvements in transfer of emotion, utterance length adjustment, and simulated lip movement @xcite . other companies are already providing self-serve dubbing services on the internet. for example, both maestra video dubber @xcite and aloud @xcite provide capabilities for users to upload video files and corresponding text files. if the latter are not uploaded, the systems can transcribe the videos to produce text. users can then edit the transcriptions and their corresponding, mt-produced translations, selecting from available synthetic voices, and making needed modifications to ensure acceptable video dubbing quality. ## speech-to-speech pipeline lingua is similar to these systems in that it can also perform automatic dubbing given video files and, optionally, corresponding text files in either srt or a proprietary xml format. it currently uses microsoft's cognitive services apis in a traditional asr -mt -tts pipeline. the asr component has been customized with 50+ hours of audio from church speeches and their aligned human transcriptions, resulting in a reduction in word error rate from 7.2% to 3.3%. the mt systems have also been customized using hundreds of thousands to millions of sentence pairs from the church's extensive translation memories, resulting in an increase of bleu scores over generic mt baselines between 6 and 22 points, with an average increase of 13 points. the tts component has not been customized, but the switch to neural voices for all languages in early 2021 was a significant, albeit subjectively evaluated, improvement. ## future work the dynamic speed adjustment for the tts sometimes results in dubbing that is unnaturally fast. additional hyperparameter tuning may help to distribute the dubbing more evenly. however, this strategy is unlikely to resolve the problem entirely, as translations are generally longer than the source @xcite . manual dubbings often intentionally abbreviate the translations to improve the alignment with the original. methods similar to those described in @xcite and @xcite may be implemented to optimize the mt to generate outputs that are roughly equivalent in length to the input. to prepare lingua for deployment in real-world settings, we will need to run additional user studies. these experiments will be essential to assess the subjective acceptability of the outputs across all of the target languages. ## conclusion in this paper, we described lingua, an application that is capable of interpreting live speeches and creating synchronized dubbings for videos. it mitigates the shortcomings of cascaded sts systems by following an optional uploaded script, although it can revert to the default cascade if the speaker goes off script. we assessed the performance of the fuzzy matching classifier, and we found that it achieves f1 @xmath0.95 on a difficult test set. we discussed the primary hyperparameters and demonstrated how they affect the performance of the fuzzy matching algorithm. finally, we proposed future avenues of research to improve the tts component and prepare lingua for deployment. we anticipate that lingua will increase accessibility to church multimedia content for underserved linguistic communities. it will decrease the time, cost, and expertise required to perform live interpretation and produce professional dubbings.