The Two Shades of Dubbing in Neural Machine Translation (2020)

Abstract: Dubbing has two shades; synchronisation constraints are applied only when the actor’s mouth is visible on screen, while the translation is unconstrained for off-screen dubbing. Consequently, different synchronisation requirements, and therefore translation strategies, are applied depending on the type of dubbing. In this work, we manually annotate an existing dubbing corpus (Heroes) for this dichotomy. We show that, even though we did not observe distinctive features between on- and off-screen dubbing at the textual level, on-screen dubbing is more difficult for MT (-4 BLEU points). Moreover, synchronisation constraints dramatically decrease translation quality for off-screen dubbing. We conclude that, distinguishing between on-screen and off-screen dubbing is necessary for determining successful strategies for dubbing-customised Machine Translation.

## introduction dubbing is a form of audiovisual translation (avt) which consists in replacing the original audio track of a film dialogue with another track containing the dialogue translated in a different language. it is the preferred type of avt in countries with large film and streaming markets, e.g. germany, italy, spain and parts of latin america @xcite . dubbing, as a form of translation, is among the few where machine translation has found no steady ground yet. the first reason is the particularity of dubbing as a genre. translation for dubbing is a text written to be spoken, therefore it should bear a close approximation to orality and reflect oral unlabored dialog @xcite . however, dubbing is not spontaneous, impromptu speech but rather a carefully-prepared imitation of spoken language. this prefabricated orality @xcite is what confers dubbing particular linguistic characteristics. the second challenge lies in the constraints affecting the translation. the most distinctive characteristic of dubbing is the need for synchronisation when an actor's mouth appears on screen. isochrony requires that the duration of the source and target utterance is equal, in order for the translated dialogue to exactly fit the time during which the actor speaks. at a second level, lip-sync consists in adapting the translation to match the articulatory mouth movements of the actor, mainly matching open vowels and bilabials. 1 for example, the sentence "i get strong off other people's fear" is translated as "el miedo de los demás me da fuerzas", instead of a more literal translation "me fortalezco con el miedo de otras personas", in order to match the duration of the source utterance and the overlap of f between fear-fuerzas. from the above, it becomes evident that there is a clear dichotomy in dubbing strategies when an actor's mouth is visible on screen or not (off screen). while on-screen dubbing is bound by synchronisation constraints, off-screen dubbing should simply be representative of orality. very few works have attempted to automatise dubbing through neural machine translation (nmt). @xcite attempt to integrate the constraint of isochrony by selecting the translation which has a similar number of syllables to the source. @xcite use the nmt attention mechanism to segment the translation into prosodic phrases in order to improve a text-to-speech system for dubbing. @xcite adapt an nmt system to generate translations of the same length as the source, although in terms of characters, which does not necessarily reflect duration of utterance. however, none of these works has taken into consideration the on/off-screen dichotomy. in this work, we address, to our knowledge for the first time, the dichotomy between on-screen and off-screen dubbing in nmt. as a first contribution, we manually annotate the heroes corpus @xcite by explicitly marking whether the actor's mouth appears on screen or not. this annotation foot_0 is the first of its kind and will provide an invaluable resource for the study of dubbing and a benchmark for systems' evaluation. second, we show that synchronisation is hardly discernible at the textual level, since distinctive text features to tell apart on/off-screen dubbing are particularly hard to find. despite this, we demonstrate that an nmt system tuned to the dubbing domain achieves significantly worse results for on-screen compared to off-screen dubbing, and that applying isochrony constraints to off-screen dubbing is detrimental for nmt quality. our findings suggest that using a single nmt system for all "shades" of dubbing is not optimal, but viable solutions for dubbing-customised nmt should take into account the on/off dichotomy. ## data, annotation and analysis data: the heroes corpus @xcite is the only freely available dubbing corpus and is based on the drama television series heroes. it contains 7,000 single-speaker english utterance segments and their translations into spanish, as well as time-alignments that allow for re-aligning the text with the video. before starting the annotation, we verified that synchronisation is actually present in heroes by watching the dubbed videos. we found that dubbing is convincing and that synchronisation is indeed present, with a stronger emphasis on preserving isochrony. annotation: a first annotator watched all videos and annotated the source side of the corpus on the word-level, specifying whether speech is on-screen (mouth visible on screen) or off. consequently, each utterance can be fully on-screen, fully off-screen, or mixed (if the actor's mouth is visible only for some words). to validate the reliability of the annotation, a second annotator worked on 700 randomly selected utterances (10 % of the corpus). this yielded an inter-annotator agreement (cohen's kappa) of κ = .73 on the word level and .73 on the utterance level. 3 we conclude that annotation as on/off/mixed is possible with a substantial agreement across annotators. during annotation, we fixed several transcription errors and identified non-english source utterances, resulting in a total of 6,977 segments. ## neural machine translation for on-and off-screen dubbing we now work on the hypothesis that, if there is no difference between on/off-screen dubbing, as suggested from the ratios and classification results in section 2, an mt system should achieve comparable performance across the different classes. if this is not the case, there should be different dubbing strategies mirrored in the translation, possibly due to synchronisation constraints. to validate this hypothesis, we test the performance of an nmt model on on/off-screen dubbing. data: we create a test set by randomly selecting 400 sentences from each class, paying attention to avoid data leakage in the training set. from the remaining sentences, 10% from each class is sampled to form the development set. since the number of sentences among the classes is unbalanced, we over-sample the two smaller classes (off and mixed) until they reach the size of the on class, resulting in a training corpus of 9,500 sentences. in order to obtain reliable results, we perform three rounds of cross-validation and report the mean and standard deviation of bleu scores. ## analysis the bleu scores and the improvements from fine-tuning are higher for the off class. our first hypothesis was that this difference is due to different contents/styles in the on/off test sets; for example, off-screen contains more narratives that are easier to translate, whereas on-screen more informal language. however, the majority of segments come from dialogues between characters, and narratives account only for around 10 segments in the whole corpus. since the styles of the source segments are comparable, the lower bleu scores for the on and mixed class suggest that (partially and fully) on-screen dubbing is highly constrained translation, which depends not only on semantics, but on factors to which the nmt model does not have access, such as phonetics and visemics. therefore, this difference should be investigated in the interplay between the source and target in terms of translation solutions. to account for the difference in bleu, we compute perplexity of the human dubbing and nmt outputs on a 5-gram language model @xcite trained on a corpus of general spanish language @xcite ( ## conclusions and future work we have explored for the first time the characteristics of on/off-screen dubbing from a computational perspective. to this end, we annotated the heroes corpus, distinguishing between the two shades of dubbing. our annotation is an important starting point towards fulfilling the multimodal requirements necessary for fully functional dubbing engines which will incorporate detection of visuals for identifying the shade and deciding whether to apply synchronisation @xcite . despite the lack of distinctive features at the textual level in our dataset, nmt seems to suffer from a (still) elusive difference between on/off-screen dubbing, as witnessed by lower bleu scores for on and mixed. we have further shown that isochrony constraints significantly hurt nmt performance for off-screen dubbing. understanding the language of dubbing is still an open problem. we hope that our findings will set a first cornerstone towards the successful integration of nmt in dubbing. in the future, we will investigate isochrony and especially lip-sync from a phonetic perspective.