Dubbing in Practice: A Large Scale Study of Human Localization With Insights for Automatic Dubbing (2023)

Abstract: We investigate how humans perform the task of dubbing video content from one language into another, leveraging a novel corpus of 319.57 hours of video from 54 professionally produced titles. This is the first such large-scale study we are aware of. The results challenge a number of assumptions commonly made in both qualitative literature on human dubbing and machine-learning literature on automatic dubbing, arguing for the importance of vocal naturalness and translation quality over commonly emphasized isometric (character length) and lip-sync constraints, and for a more qualified view of the importance of isochronic (timing) constraints. We also find substantial influence of the source-side audio on human dubs through channels other than the words of the translation, pointing to the need for research on ways to preserve speech characteristics, as well as transfer of semantic properties such as emphasis and emotion, in automatic dubbing systems.

## introduction considerable attention has been paid to the dubbing of video content from one language into another, both in the literature of several disciplines and in the daily practice of the entertainment industry. one influential line of work, in the fields of film studies and audiovisual translation, studies human dubbing from a qualitative perspective @xcite @xcite , as a profession and semiotic activity. this literature has developed a rich body of theory on the nature of the human dubbing task, and the ways humans approach it, but has little engagement with large-scale data. more recently, machine-learning practitioners have taken up the task of building multimodal systems for automatic dubbing (e.g., @xcite @xcite , but lack deep empirical or theoretical bases for how to organize their work. what is missing from both literatures, and can help bridge the gap between them, is a large-scale study of human dubbing in practice: a data-driven examination of the way humans actually perform this task. such an analysis can have benefits for both the qualitative study of human dubbing, by providing empirical evidence of how dubbing teams approach their work, and informing future machine-learning work on automatic dubbing systems. it is exactly this analysis we undertake in this work. human dubbing involves a sequence of human contributors each with control over a different aspect of the process @xcite @xcite . the first step is an approximately literal translation of the original script, done by a dialogue translator. next, a dialogue adaptor will modify this translation into a plausible script meeting the requirements for dubbing such as isochrony, lip sync, kinesic synchrony, and so on. finally, the translated and modified script will go to a production team. voice actors, with input from a dubbing director or supervisor, have been noted to often have freedom to improvise or make small changes to the dialogue as it is being recorded @xcite @xcite . we, however, aim to understand human dubbing by studying not its process, but its product: a large set of actual dubbed dialogues from real tv shows, obtained from amazon studios. as compared to qualitative work or interviews with dubbers, this approach has the particular virtue of capturing tacit knowledge brought to bear in the human dubbing process but difficult to write down or explain. we organize our investigation around one of the most fundamental insights from the qualitative literature, that of human dubbing (and subtitling, which we do not consider here) as ''constrained translation'' @xcite . a dub, after all, is not just a translation of the original content-indeed, it is not a purely textual product at all. as a translation, it should preserve the meaning of the original; as spoken language, it should sound natural; as an accompaniment to a video track, it should fit with the timing of actors' mouth movements, body language, and the flow of the story @xcite . simultaneously satisfying all of these constraints is very difficult, and in general may not be possible. we are accordingly interested in how human dubbers balance the competing interests of semantic fidelity, natural speech, timing constraints, and convincing lip sync. each can be traded off against the others, with varying effects on the audience's experience of the resulting product. we operationalize this broad question as several more specific ones about the human dubbing process: 1 isochrony do dubbers respect timing constraints imposed by the video and original audio? isometry do the original and dub texts have approximately the same number of characters? speech tempo how much do voice actors vary their speaking rates, possibly compromising speech naturalness, to meet timing constraints? lip sync how closely do the voice actors' words match visible mouth movements of the original actors? translation quality how much will dubbers reduce translation accuracy (i.e., adequacy and fluency) to meet other constraints? source influence do source speech traits influence the target in ways not mediated by the wordsofthe dub, indicatingsemantictransfer? after exploring each of these questions, we provide insights on several research directions to address weaknesses we uncover in current automatic dubbing approaches. ## corpus description & preprocessing we begin with a dataset consisting of every tv show produced by amazon studios which was available on prime video at the end of 2021 for which we were able to locate a hand-curated transcript (for english shows) or dubbing script (for dubbed shows). these scripts are produced as part of the human dubbing process (see § 1 for more details). this dataset contains 674 episodes of 54 shows, constituting 319.57 hours of content from 9,215 distinct speakers. a detailed summary of this dataset is provided in we perform extensive quality filtering prior to analysis. data amounts for the entire corpus as well as each genre/language subset, at each stage of processing/filtering described below, are provided in ## insights for automatic dubbing our analysis of the human dubbing process points to several directions that should (and perhaps should not) be pursued in automatic dubbing, which we summarize below. translation quality and speech naturalness appear to be paramount. the input to the dubbing process mostly consist of dialogue with challenging issues for automatic translation systems, like ambiguous speaker gender, ambiguous addressee gender and number, and formality between characters. speaker gender and number issues are especially critical since the audience can often both hear and see the speakers and addressees. we note a stark lack of literature on automatic translation of dialogues, compared to common domains in literature like news. likewise, naturalness for tts systems is challenging enough under normal circumstances, but tv shows often include yelling, crying, whispering, and so forth, making the problem even harder. while research does exist in this space, we suspect there is much room for improvement. we find strong evidence for several levels of non-textual transfer of source audio properties into human dubs: speaker characteristics, dialogue line-level effects, and emotion/emphasis transfer when considering semantic alignments at the word level. this points to a glaring issue with pipeline approaches employed by the vast majority of automatic dubbing literature: without a mechanism to encode emotion/emphasis, individual vocal profiles and other traits of the source speech, we expect them to be nearly impossible to replicate in synthetic target speech. the high rates of isochrony that we observe in human dubs support the need for continued research on isochronic mt, especially given the observed unwillingness of human dubbers to vary their speaking rate, which shows that automatic dubbing systems should not simply vary speaking rates to achieve isochronic constraints. however, our findings do not support the use of isometric mt. our work refutes the claim that isometry is a good proxy for isochrony, and it also casts doubt on the claim that isochrony is more necessary with tts than with human voice actors because tts is less able to vary speaking rates (i.e., we find that human dubbers are not varying speaking rate to meet isochronic constraints, and thus automatic dubbing systems should likely not either). the authors suspect that directly optimizing isochrony (as opposed to isometry) is likely a better approach for automatic dubbing. finally, the low rates of lip sync that we observe (and the very small if still statistically significant difference between on-and off-screen rates) in human dubs suggest that research on automatic lip sync can be marginally useful, at best, for automatic dubbing. efforts like @xcite , however, which edit mouth movements in the video, may be an exception. ## future work this work focused on two language pairs: english-german and english-spanish. in future work, we hope to analyze more distant language pairs (e.g., english-chinese or english-arabic), as well as non-english source material. our analysis has shown that isometry is a poor proxy for isochrony in human dubs, yet several prior works have claimed that isometric mt benefits automatic dubbing. in future work, we hope to perform analysis to understand this discrepancy. the scope of this work necessitated automatic metrics. however, in future work, we hope to verify some of these findings (e.g., translation quality on-vs off-screen) using human annotators. finally, the aggregate analysis in this work is necessary to provide high-level insights for automatic dubbing. however, it likely also hides interesting variations across different individual translators, adaptors, dubbers, dubbing studios, and so on. we hope to better explore these dimensions in future work. ## conclusion we present the first large-scale quantitative study of how humans perform the task of dubbing video content from one language into another. our results challenge a number of popular assumptions in both qualitative and machine learning literature: human dubbers display less respect for isochrony and especially lip sync than is suggested by qualitative literature, while being surprisingly unwilling to vary speaking rates or sacrifice translation quality to hit other constraints. our analysis provides insights on research directions to address weaknesses in current automatic dubbing approaches.