Length-Aware NMT and Adaptive Duration for Automatic Dubbing (2023)

Abstract: This paper presents the submission of Huawei Translation Services Center for the IWSLT 2023 dubbing task in the unconstrained setting. The proposed solution consists of a Transformer-based machine translation model and a phoneme duration predictor. The Transformer is deep and multiple target-to-source length-ratio class labels are used to control target lengths. The variation predictor in FastSpeech2 is utilized to predict phoneme durations. To optimize the isochrony in dubbing, re-ranking and scaling are performed. The source audio duration is used as a reference to re-rank the translations of different length-ratio labels, and the one with minimum time deviation is preferred. Additionally, the phoneme duration outputs are scaled within a defined threshold to narrow the duration gap with the source audio.

## introduction automatic dubbing (ad) @xcite @xcite technology uses artificial intelligence (ai) to automatically generate dubbed audio for video content. dubbing is the process of replacing the audio with a translation of the original audio in a different language. ai dubbing technology automates this process by using machine learning algorithms to translate the original audio and synthesize a new voice that sounds natural and resembles a human voice. the synthesized voice is then synchronized with the lip movements of the characters in the video to produce dubbed audio. this technology has the potential to significantly reduce the time and cost of creating dubbed audio and make it easier to reach a global audience by translating video content into multiple languages. recent advances in the field of automatic dubbing have contributed to the development of more efficient and cost-effective methods for producing localized content. researchers have utilized var-ious techniques and technologies, including machine translation (mt) @xcite , speech synthesis @xcite , and speech recognition @xcite , to improve the accuracy and quality of automatic dubbing systems. isometric machine translation @xcite is a technique used in automatic dubbing where translations should match a given length to allow for synchronicity between source and target speech. for neural mt, generating translations of length close to the source length, while preserving quality is a challenging task. controlling mt output length comes at a cost to translation quality, which is usually mitigated with a two-step approach of generating n-best hypotheses and then re-ranking based on length and quality. another area of research focuses on the synchronization of the dubbed audio with the original source audio. this is essential for ensuring that the dubbed audio matches the timing and intonation of the original speech. researchers have developed various methods for achieving accurate synchro-nization, including the use of phoneme duration predictors and machine learning algorithms to detect and align speech segments @xcite @xcite . one of the latest developments in automatic dubbing research is the use of deep neural networks for speech synthesis @xcite . these networks enable the creation of more naturalistic and expressive speech, improving the overall quality of the dubbed audio. in conclusion, recent research in automatic dubbing has shown significant progress and promise for the future of localized content production. by combining advanced machine learning techniques with speech synthesis, speech recognition, and sentiment analysis, researchers are developing more accurate, efficient, and cost-effective automatic dubbing systems. the iwslt 2023 @xcite dubbing task focuses on isochrony in dubbing, which refers to the property that the speech translation is time aligned with the original speaker's video. the task assumes that the front automatic speech recognition (asr) output text and subsequent text-to-speech (tts) models already exist, and the goal is to predict the phonemes and their durations. our proposed solution involves using a transformerbased @xcite machine translation model and a phoneme duration predictor. a deep transformer @xcite model is utilized to handle multiple target-to-source lengthratio class labels, which are used to control target lengths. the phoneme duration predictor is based on the variation predictor used in fastspeech2 @xcite . to optimize isochrony in dubbing, the solution utilizes re-ranking and scaling techniques. the translations generated by different length-ratio labels are re-ranked based on their time deviation from the source audio duration, with the minimum deviation one preferred. the phoneme duration outputs are also scaled within a predefined threshold to narrow the duration gap with the source audio. these techniques help to ensure that the translated speech is synchronized with the original speaker's video. ## data the data provided in the constrained setting is derived from covost2 @xcite de-en data, consisting of german source text, english target text, speech durations, and english phonemes and durations @xcite . we additionally apply wmt2014 de-en data for training the mt model. the amount of data for both sets is shown in ## system the system consists of four parts: pause alignment, machine translation, phoneme duration variation predictor, and re-ranking and scaling. figure 1 shows the system pipeline. the following describes the four parts in detail. ## experiments we used sentencepiece @xcite to process nmt bilingual text and obtain subword vocabularies, resulting in a german vocabulary of 29k and an english vocabulary of 25k. we trained a transformer nmt model using fairseq @xcite , with an encoder of 25 layers, a decoder of 6 layers, 8 attention heads, embeddings of 512, and ffn embeddings of 2048. the model was optimized using adam @xcite with an initial learning rate of 5e-4, and warmup steps of 4000. dropout was set to 0.1. the model was trained on 8 gpus, with a batch size of 2048 tokens and an update frequency of 4. during the inference phase, an open-source vad tool was used to process the source speech and obtain speech segments and durations for subsequent selection of nmt translated text lengths and adjusting the duration of synthetic dubbings. the nmt translated text was then converted to phoneme sequences using an open-source grapheme-tophoneme tool, and the initial phoneme durations were predicted using a pre-trained variation predictor module in fastspeech2. as the main evaluation method for this task is manual evaluation, and our method allows for adjustment of phoneme duration prediction, we mainly experiment and compare bleu @xcite under different strategies of machine translation. to measure the synchronicity between source and dubbed speech, we use speech overlap (so) @xcite metric. it should be noted that the metrics presented don't take into account speech naturalness, which is extremely important to people viewing dubs. @xcite showed that human dubbers produces natural speech even at the cost of isochrony. the experimental results on the two test sets of the task are shown in too long translations will result in lower quality of machine translation, while short translations will result in insufficient duration for generating dubbing. after re-ranking, the translations can achieve more moderate results in translation quality and duration. moreover, by setting appropriate scaling thresholds, scaling operation can further improve the isochrony without affecting bleu. we also compared the results without pause alignment, as shown in the last row of ## conclusion this paper describes the submission of huawei translation services center for the iwslt 2023 dubbing task under the unconstrained setting. our solution consists of four parts: pause alignment, machine translation, phoneme duration variation predictor, re-ranking and scaling. pause alignment is used to align source audio and source text to improve synchronization between synthetic dubbing and source video. the machine translation model is trained using the deep transformer structure. to control the output translation length, multiple target-to-source length-ratio tags are used to adjust the length. pre-trained variation predictor in fast-speech2 is used to predict phoneme durations. in order to optimize the isochrony in dubbing, the results of different lengths of the machine translation output are re-ranked and scaled. using the source audio duration as a reference, the translations with different length ratios are re-ranked, and the output with the smallest time deviation is preferred. in addition, the phoneme duration output is scaled within a defined threshold, further narrowing the duration gap from the source audio. we compare the experimental results of different length-ratio strategies, and our method can achieve a balanced result in bleu and speech overlap.