Pause-Aware Automatic Dubbing using LLM and Voice Cloning (2024)

Abstract: Automatic dubbing aims to translate the speech of a video into another language, ensuring the new speech naturally fits the original video. This paper details Huawei Translation Services Center’s (HW-TSC) submission for IWSLT 2024’s automatic dubbing task, under an unconstrained setting. Our system’s machine translation (MT) component utilizes a Transformer-based MT model and an LLM-based post-editor to produce translations of varying lengths. The text-to-speech (TTS) component employs a VITS-based TTS model and a voice cloning module to emulate the original speaker’s vocal timbre. For enhanced dubbing synchrony, we introduce a parsing-informed pause selector. Finally, we rerank multiple results based on lip-sync error distance (LSE-D) and character error rate (CER). Our system achieves LSE-D of 10.75 and 12.19 on subset1 and subset2 of DE-EN test sets respectively, superior to last year’s best system.

## introduction the task of automatic dubbing is to translate spoken language in a video into another language such that the translated speech can be seamlessly blended with the original video. a unique aspect of dubbing is isochrony, which refers to the property that the speech translation is time-aligned with the original speaker's visual cues. the spoken words should match the speaker's lip movements, ensuring the audio is heard when the lips move and is silent when they don't. to address this challenge, a unified model that simultaneously processes translations and speech timing is optimal, allowing for adjustments in translation to fit timing constraints. @xcite accomplish this by simply binning target phoneme durations and interleaving them with target phonemes during training and inference. @xcite enhance this approach by predicting the durations of phonemes as target factors. however, these methods fail to utilize pre-trained machine translation (mt) models and large language models (llm) that are trained on massive text corpora. moreover, constructing large-scale datasets with phoneme duration labels is challenging, thus limiting the translation quality. therefore, a disentangled approach that considers mt and dubbing synchrony separately can achieve better results. our system @xcite from last year first generated a set of translation candidates and later reranked them based on speech overlaps, achieving better mean opinion scores (mos) than the baseline systems. therefore, this year we extend last year's system by using more advanced pre-trained models and a more sophisticated pause-aware dubbing pipeline. specifically, our method comprises the following key components: in this paper, we provide detailed analyses of the components mentioned above. our system 1 60 achieves an lse-d of 10.75 on subset1 and 12.19 on subset2 of the de-en test sets, respectively, outperforming last year's best system. additionally, we take into account the vocal timbre of the speech, which can enhance the perceptual quality. ## experimental setups we fine-tuned the nllb-1.3b foot_3 model for 20 epochs on the covost2 (changhan wang, 2020) de-en subset, using a learning rate of 3×10 - foot_4 and a batch size of 512. for the llm-based post-editor, when employing the "complex" indicator, we sampled three answers and selected the one with the highest comet score @xcite compared to the original translation. for the "short" indicator, we sampled only once. when adjusting the speech speed, we set the lower bound to 0.75× and the upper bound to 2.5×. we adopted several evaluation metrics: the bleu score and the comet score to evaluate mt quality, and the lip-sync error distance (lse-d) (chung and zisserman, 2017) 5 and asr character error rate (cer) to measure dubbing performance. we used the wav2vec2base model foot_5 , fine-tuned on librispeech, as the asr model, which utilizes a character-level vocabulary. we opted not to use a more advanced asr model since the less robust model is more sensitive to speech quality. during rerank, we considered four translations: the original translation, the translation using llm-based post editor without indicating the output length, the "complex" translation, and the "simple" translation. additionally, we attempted to enhance the speech by applying denoising and audio super-resolution techniques foot_6 , which remove noise and upscale the audio from 16khz to 44.1khz. ## discussion compared to last year's system, which utilized a length-aware mt system that employed a length tag to indicate the desired output length, this year's approach aims to enhance translation quality by fine-tuning a pre-trained mt model rather than training one from scratch. although we attempted to incorporate length tags in the fine-tuning process, we found that they failed to produce translations with varying lengths due to the limited number of epochs and fine-tuning data. consequently, we used an llm which has robust rewriting capabilities. we submitted a single entry for the english-chinese subtask, which presents significantly greater challenges than the german-english subtask due to factors such as long-form video, speaker changes, and background music. to address these challenges, we enhanced our automatic dubbing system with an open-source diarization model @xcite , a source separation tool @xcite , and a tts api 8 . however, given the complexity of the task and the lack of labeled test set, we have not provided a detailed analysis. in movie dub-bing, it is crucial that the emotion of the dubbed speech matches that of the original speech, therefore, expressive tts models are preferred. we evaluated the seamless expressive model @xcite , however, we observed that the speech quality was inconsistent, and for non-english languages, the speech did not sound native. ## conclusion in this paper, we propose a novel pause-aware automatic dubbing system that ensures translated speech signals are not only accurate but also maintain the timbre of the original speech. the key components involve a novel pause selector, informed by parsing, to align dubbing with the video's pace, a vc model to convert the tone color, and an llm to provide translation candidates. for future work, we plan to carry out more systematic experiments on long-form, movie-like videos and provide more expressive dubbed videos.