From Speech-to-Speech Translation to Automatic Dubbing (2020)

Abstract: We present enhancements to a speech-to-speech translation pipeline in order to perform automatic dubbing. Our architecture features neural machine translation generating output of preferred length, prosodic alignment of the translation with the original speech segments, neural text-to-speech with fine tuning of the duration of each utterance, and, finally, audio rendering to enriches text-to-speech output with background noise and reverberation extracted from the original audio. We report and discuss results of a first subjective evaluation of automatic dubbing of excerpts of TED Talks from English into Italian, which measures the perceived naturalness of automatic dubbing and the relative importance of each proposed enhancement.

## introduction automatic dubbing can be regarded as an extension of the speech-to-speech translation (stst) task @xcite , which is generally seen as the combination of three sub-tasks: (i) transcribing speech to text in a source language (asr), (ii) translating text from a source to a target language (mt) and (iii) generating speech from text in a target language (tts). independently from the implementation approach @xcite @xcite @xcite , the main goal of stst is producing an output that reflects the linguistic content of the original sentence. on the other hand, automatic dubbing aims to replace all speech contained in a video document with speech in a different language, so that the result sounds and looks as natural as the original. hence, in addition to conveying the same content of the original utterance, dubbing should also match the * * contribution while the author was with amazon. while stst has been addressed for long time and by several research labs @xcite @xcite @xcite , relatively less and more sparse efforts have been devoted to automatic dubbing @xcite @xcite , although the potential demand of such technology could be huge. in fact, multimedia content created and put online has been growing at exponential rate, in the last decade, while availability and cost of human skills for subtitling and dubbing still remains a barrier for its diffusion worldwide. foot_0 professional dubbing @xcite of a video file is a very labor intensive process that involves many steps: (i) extracting speech segments from the audio track and annotating these with speaker information; (ii) transcribing the speech segments, (iii) translating the transcript in the target language, (iv) adapting the translation for timing, (v) casting the voice talents, (vi) performing the dubbing sessions, (vii) fine-aligning the dubbed speech segments, (viii) mixing the new voice tracks within the original soundtrack. automatic dubbing has been addressed both in monolingual cross-lingual settings. in @xcite , synchronization of two speech signals with the same content was tackled with time-alignment via dynamic time warping. in @xcite automatic monolingual dubbing for tv users with special needs was generated from subtitles. however, due to the poor correlation between length and timing of the subtitles, tts output fre-quently broke the timing boundaries. to avoid unnatural time compression of tts's voice when fitting its duration to the duration of the original speech, @xcite proposed phonedependent time compression and text simplification to shorten the subtitles, while @xcite leveraged scene-change detection to relax the subtitle time boundaries. regarding crosslingual dubbing, lip movements synchronization was tackled in @xcite by directly modifying the actor's mouth motion via shuffling of the actor's video frames. while the method does not use any prior linguistic or phonetic knowledge, it has been only demonstrated on very simple and controlled conditions. finally, mostly related to our contribution is @xcite , which discusses speech synchronization at the phrase level (prosodic alignment) for englishto-spanish automatic dubbing. in this paper we present research work to enhance a stst pipeline in order to comply with the timing and rendering requirements posed by cross-lingual automatic dubbing of ted talk videos. similarly to @xcite , we also shorten the tts script by directly modifying the mt engine rather than via text simplification. as in @xcite , we synchronize phrases across languages, but follow a fluencybased rather than content-based criterion and replace generation and rescoring of hypotheses in @xcite with a more efficient dynamic programming solution. moreover, we extend @xcite by enhancing neural mt and neural tts to improve speech synchronization, and by performing audio rendering on the dubbed speech to make it sound more real inside the video. in the following sections, we introduce the overall architecture (section 2) and the proposed enhancements (sections 3-6). then, we present results (section 7) of experiments evaluating the naturalness of automatic dubbing of ted talk clips from english into italian. to our knowledge, this is the first work on automatic dubbing that integrates enhanced deep learning models for mt, tts and audio rendering, and evaluates them on real-world videos. ## automatic dubbing with some approximation, we consider here automatic dubbing of the audio track of a video as the task of stst, i.e. asr + mt + tts, with the additional requirement that the output must be temporally, prosodically and acoustically close to the original audio. we investigate an architecture (see figure 1 ) that enhances the stst pipeline with (i) enhanced mt able to generate translations of variable lengths, (ii) a prosodic alignment module that temporally aligns the mt output with the speech segments in the original audio, (iii) enhanced tts to accurately control the duration of each produce utterance, and, finally, (iv) audio rendering that adds to the tts output background noise and reverberation extracted from the original audio. in the following, we describe each component in detail, with the exception of asr, for which we use @xcite an of-the-shelf online service foot_2 . ## machine translation our approach to control the length of mt output is inspired by target forcing in multilingual neural mt @xcite . we partition the training sentence pairs into three groups (short, normal, long) according to the target/source string-length ratio. in practice, we select two thresholds t 1 and t 2 , and partition training data according to the length-ratio intervals [0, t 1 ), [t 1 , t 2 ) and [t 2 , ∞]. at training time a length token is prepended to each source sentence according to its group, in order to let the neural mt model discriminate between the groups. at inference time, the length token is instead prepended to bias the model to generate a translation of the desired length type. we trained a transformer model @xcite with output length control on web crawled and proprietary data amounting to 150 million english-italian sentence pairs (with no overlap with the test data). the model has encoder and decoder with 6 layers, layer size of 1024, hidden size of 4096 on feed forward layers, and 16 heads in the multi-head attention. for the reported experiments, we trained the models with thresholds t 1 = 0.95 and t 2 = 1.05 and generated at inference time translations of the shortest type, resulting, on our test set, in an average length ratio of 0.97. a reason for the length exceeding the threshold could be that for part of test data the model did not learn ways to keep the output short. a detailed account of the approach, the followed training procedure and experimental results on the same task of this paper, but using slightly different thresholds, can be found in @xcite . the paper also shows that human evaluation conducted on the short translations resulted in a minor loss in quality with respect to the model without output length control. finally, as baseline mt system for our evaluation experiments we used an online service foot_3 4 prosodic alignment prosodic alignment @xcite is the problem of segmenting the target sentence to optimally match the distribution of words and pauses 4 . let @xmath0, e 2 , . . . , e n be a source sentence of n words which is segmented according to k breakpoints 1 ≤ i 1 < i 2 < . . . i @xmath1, shortly denoted with i. given a target sentence @xmath2, f 2 , . . . , f m of m words, the goal is to find within it k corresponding breakpoints 1 ≤ j 1 < j 2 < . . . j @xmath3: by assuming a markovian dependency on j, i.e.: and omitting from the notation the constant terms i, e, f , we can derive the following recurrent quantity: the first term computes the relative match in duration between the corresponding t-th segments foot_5 , while the second term measure the linguistic plausibility of a placing a break after the j t in f . for this, we simply compute the following ratio of normalized language model probabilities of text windows centered on the break point, by assuming or not the presence of a pause (br) in the middle: the rational of our model is that we want to favor split points were also tts was trained to produce pauses. tts was in fact trained on read speech that generally introduces pauses in correspondence of punctuation marks such as period, comma, semicolon, colon, etc. notice that our interest, at the moment, is to produce fluent tts speech, not to closely match the speaking style of the original speaker. in our implementation, we use a larger text window (last and first two words), we replace words with parts-of speech, and estimate the language model with kenlm @xcite on the training portion of the must-c corpus tagged with parts-of-speech using an online service 6 . ## experimental evaluation we evaluated our automatic dubbing architecture (figure 1 ), by running perceptual evaluations in which users are asked to grade the naturalness of video clips dubbed with three configurations (see we adopt the mushra (multi stimulus test with hidden reference and anchor) methodology (mushra, 2014), originally designed to evaluate audio codecs and later also tts. we asked listeners to evaluate the naturalness of each versions of a video clip on a 0-100 scale. figure 2 shows the user interface. in absence of a human dubbed version of each clip, we decided to use, for calibration purposes, the clip in the original language as hidden reference. the clip versions to evaluate are not labeled and randomly ordered. the observer has to play each version at least once before moving forward and can leave a comment about the worse version. in order to limit randomness introduced by asr and tts across the clips and by mt across vergiven its very poor quality, as also reported in @xcite . other intermediate configurations were not explored to limit the workload of the subjects participating in the experiment. ## conclusions we have perceptually evaluated the naturalness of automatic speech dubbing after enhancing a baseline speech-to-speech translation system with the possibility to control the verbosity of the translation output, to segment and synchronize the target words with the speech-pause structure of the source utterances, and to enrich tts speech with ambient noise and reverberation extracted from the original audio. we tested our system with both italian and non-italian listeners in order to evaluate both language independent and language dependent naturalness of dubbed videos. results show that while we succeeded at achieving synchronization at the phrasal level, our prosodic alignment step negatively impacts on the fluency and prosody of the generated language. the impact of these disfluencies on native listeners seems to partially mask the effect of the audio rendering with background noise and reverberation, which instead results in a major increase of naturalness for non-italian listeners. future work will be devoted to better adapt machine translation to the style used in dubbing and to improve the quality of prosodic alignment, by generating more accurate sentence segmentation and by introducing more flexible synchronization.