Automatic Video Dubbing at A pp T ek (2022)

Abstract: Video dubbing is the activity of revoicing a video while offering a viewing experience equivalent to the original video. The revoicing usually comes with a changed script, mostly in a different language, and the revoicing should reproduce the original emotions, coherent with the body language, and lip synchronized. In this project, we aim to build an AD system in three phases: (1) voice-over; (2) emotional voice-over; (3) full dubbing, while enhancing the system with human-in-the-loop capabilities for a higher quality.

## introduction video dubbing is the activity of revoicing a video while offering a viewing experience equivalent to the original video. the revoicing usually comes with a new script, and it should reproduce the original emotions, coherent with the body language, and be lip synchronized. @xcite and @xcite introduced two automatic dubbing systems as a cascade of automatic speech recognition (asr), machine translation (mt) and text-to-speech (tts), enhanced with a prosodic alignment (pa) component to transfer prosody through the pipeline. in this project, we aim to build an ad system in two phases: (1) voiceover; (2) full dubbing, and enhance it with humanin-the-loop capabilities for a higher quality. the product will be released in the form of rest apis and a web interface in q3 of the current year. the pricing will follow a pay-per-use scheme, with possible variations according to requested quality control or if the script to dub is provided by the user for higher quality dubbing. our current system is designed as an enhanced pipeline of asr, mt and tts. our asr system includes speaker diarization (the task of detecting "who speaks when") so that consecutive segments from the same speaker can be translated as coherent units, and each speaker is assigned a unique voice. our mt system is a transformerbased encoder-decoder, augmented with metadata features for style adaptation @xcite and output length control @xcite . the translations are performed from and to subtitle files to preserve the timestamps and use them as boundaries for the synthesized voices. additionally, we use speaker-adaptive tts to reproduce the voice features of the original actor for the given segment in the new language. finally, the background sound, obtained via source separation, is merged with the synthesized voices for the final audio and video rendering. this system can already translate video contents and dub the output videos in a voice-over style. ## voice-over voice-over is a simpler solution than dubbing, where the original voice's volume is lowered down, and the new voice is rendered with a natural volume over it, usually with a delay of some frames. our system is already capable of performing voice-over for some language pairs foot_0 but some aspects can be improved: diarization: speaker diarization can be improved in the cases when the audio quality is low, or one speaker speaks for less than one second. prosody alignment: we plan to add prosody alignment for transferring the pauses from the source to the target speech, but also the emphasis applied to sentences and single words. mt output length: although in voice-over we have time constraints less strict than in dubbing, some translations do not fit the allocated space, and it is important to have a fine-grained control over the mt output length. ## emotional voice-over the main limitation of the current system is the synthesized voice speaking with a "flat" tone, which does not match the emotions expressed in the original video. our research effort for achieving emotional speech is aimed to release the feature in 2023 and will affect the whole pipeline: emotion detection: emotions need to be detected from the source audio and matched with the recognized text, in order to annotate the latter with emotions tags. emotion-aware mt: expand apptek's mt systems to support emotions as part of their metadata. additional research effort will focus on letting the mt system annotate the output text with emotions at a word level, to be used from our tts system. emotion-aware tts: develop tts systems that can generate emotional speech for different emotions. such a task can be challenging given the low data availability, particularly for languages other than english. ## full dubbing a fully-fledged ad system improves the voiceover approach by fully synchronizing audio and video time. lip-syncing is a strict requirement that can be achieved using orthogonal technologies: ## language support our initial release will include english-to-arabic and english-to-spanish. in the following two years we plan to expand it to english to many european languages, including french, german, italian, polish and ukrainian, plus russian and chinese. the reverse directions will also be rolled out soon after. ## human in the loop an ad system can make errors in multiple points of its pipeline, and the earlier the errors occur, the more harmful they can be for the final result. for this reason, we plan to let users adding manual transcripts or the final scripts to obtain a higherquality video at the cost of more manual work, using our internal tool for easy editing parallel data. ## conclusion apptek dubbing is an ambitious pioneering project that combines mt with other technologies to provide a high-quality and localized translated video, with the goal of making dubbing accessible beyond the movie industry. intermediate product releases will support simpler re-voicing modes and a human-in-the-loop approach to allow the users to trade-off costs with quality.