S tyle D ubber: Towards Multi-Scale Style Learning for Movie Dubbing (2024)

Abstract: Given a script, the challenge in Movie Dubbing (Visual Voice Cloning, V2C) is to generate speech that aligns well with the video in both time and emotion, based on the tone of a reference audio track. Existing state-of-the-art V2C models break the phonemes in the script according to the divisions between video frames, which solves the temporal alignment problem but leads to incomplete phoneme pronunciation and poor identity stability. To address this problem, we propose StyleDubber, which switches dubbing learning from the frame level to phoneme level. It contains three main components: (1) A multimodal style adaptor operating at the phoneme level to learn pronunciation style from the reference audio, and generate intermediate representations informed by the facial emotion presented in the video; (2) An utterance-level style learning module, which guides both the mel-spectrogram decoding and the refining processes from the intermediate embeddings to improve the overall style expression; And (3) a phoneme-guided lip aligner to maintain lip sync. Extensive experiments on two of the primary benchmarks, V2C and Grid, demonstrate the favorable performance of the proposed method as compared to the current state-of-the-art. The code will be made available at https://github.com/GalaxyCong/StyleDubber.

## introduction movie dubbing @xcite , also known as visual voice cloning (v2c), aims to convert a script into speech with the voice characteristics specified by the reference audio, while maintaining lip-sync with a video clip, and reflecting the character's emotions depicted therein (see figure 1 (a)). v2c is more challenging than conventional text-to-speech (tts) @xcite 2021), and has obvious applications in the film industry and audio aigc, including broadening the audience for existing video. existing methods broadly fall into two groups. one group of methods focus primarily on achieving audio-visual sync. for example, a duration aligner is introduced in @xcite to explicitly control the speed and pause-duration of speaker content by mapping textual phonemes to video frames. then, an upsampling process is used to expand the video frame sequence to the length of mel-spectrogram frame sequence by multiplying by a fixed coefficient. however, the frame level alignment makes it hard to learn complete phoneme pronunciations, and often leads to seemingly mumbled pronunciations. the other family of methods focuses on maintaining identity consistency between the generated speech and the reference audio. to enable the model to handle a multi-speaker environment, a speaker encoder is used to extract identity embeddings through averaging and nor-1 6767 malizing per speaker embeddings @xcite . in contrast, @xcite and @xcite try to learn desired speaker voices based on facial appearances. although humans' faces can reflect some vocal attributes (e.g., age and identity) to some extent, they rarely encode speech styles, such as pronunciation habits or accents. according to @xcite , human speech can be perceived as a compound of multi-acoustic factors: (1) unique characteristics, such as timbre, which can be reflected on utterance level (see left panel of figure 1 (b) ); (2) pronunciation habits, such as the rhythm and regional accent, which are usually reflected at the phoneme level (see pink rectangles in figure 1 (b) ). we also note that one's voice can be affected by emotions. for example, the voice can be significantly different when one gets angry. based on these observations, we propose to learn phoneme level representations from the speaker's pronunciation habits reflected in the reference audio, and take both facial expressions and overall timbre characteristics at the utterance level of the reference audio into consideration when generating speech. in light of the above, we propose styledubber, which learns a desired style at the phoneme and utterance levels instead of the conventional video frame level. specifically, a multimodal phoneme adaptor (mpa) is proposed to capture the pronunciation styles at the phoneme level. by leveraging the cross-attention relevance between textual phonemes of the script and the reference audio as well as visual emotions, mpa learns the reference style and then generates intermediate speech representations with consideration of the required emotion. our model also introduce an utterance-level style learning (usl) module to strengthen personal characteristics during both the mel-spectrogram decoding and refining processes from the above intermediate representations. for the temporal alignment between the resulting speech and the video, we propose a phoneme-guided lip aligner (pla) to synchronize lip-motion and phoneme embeddings. at last, hifigan @xcite is used as a vocoder to convert the predicted melspectrogram to the time-domain waves of dubbing. the main contributions are summarized as: ## related work text to speech is a longstanding problem, but recent models represent a dramatic improvement @xcite @xcite @xcite . fastspeech2 @xcite , for example, alleviates the one-to-many text-to-speech mapping problem by explicitly modeling variation information. @xcite , in contrast, improves generalization through episodic meta-learning and generative adversarial networks. recently @xcite proposed a non-autoregressive flowmatching model for mono or cross-lingual zeroshot text-to-speech synthesis. despite the impressive speech they generate, these methods cannot be applied to the v2c task as they lack the required emotion modelling and lip sync. visual voice cloning is proposed to address the problem of film dubbing @xcite and has attracted a lot of attention in cross-modality alignment field @xcite @xcite @xcite @xcite . then, @xcite proposed a hierarchical prosody dubbing model by associating with lip, face, and scene and focus on frame-level prosody learning @xcite . to handle multi-speaker scenes, hassid et al. (2022) matches identities by normalizing each speaker to the unit norm, and lu et al. (2022) adopts a lookup table to match the d-vector. recently, face-tts (lee et al., 2023) used biometric information extracted directly from the face image as style to improve identity modelling using a score-based diffusion model. unlike the above methods, styledubber address the challenge of insufficient identity information by introducing adaptive utterance-level embedding and detailed pronunciation variations based on the reference audio and video. 2 6768 reference audio (input) text: "you can if it's true love." phoneme sequence:{y,uw1,ae1,...,l,ah1,v} text / subtitle (input) phoneme encoder a) multimodal phoneme-level adaptor c) utterance-level style learning mel-post net usln usln usln conv-1d multi-head attention n mel-linear conv-1d speaker encoder self attentions emotion encoder acoustic reference transformer emotion reference transformer n dub-style adapting with two-level silent movie (input) monotonic attention lip-text duration predictor (softplus) output speech affine transform lr mel-decoder human pronunciation modeling aims to learn individual pronunciation variations, which is crucial to generate comprehensible, natural, and acceptable speech @xcite . compared with fixed speaker representations, phoneme-dependent methods @xcite can better control speech and describe more pronunciation features, as phonemes are the basic sound units in a language @xcite . recently, @xcite analysed the correlation between local pronunciation content and speaker embeddings at the quasi-phoneme level by reference attention. here, in contrast, we propose a multimodal style adaptor to capture the fine-grained pronunciation variation, which not only imitates the reference style acoustically, but also conveys emotional expression by reference transformer. ## experiments we evaluate our method on two primary v2c datasets, v2c-animation and grid. below, we first provide implementation details. then, we briefly introduce the datasets and evaluation metrics, followed by quantitative and qualitative results. ablation studies are also conducted to thoroughly evaluate our model. ## conclusion in this work, we propose styledubber for movie dubbing, which imitates the speaker's voice at both phoneme and utterance levels while aligning with a reference video. styledubber uses a multimodal phoneme-level adaptor to improve pronunciation that captures speech style while considering the visual emotion. moreover, a phoneme-guided lip aligner is devised to synchronize vision and speech without destroying the phoneme unit. the proposed model sets new state-of-the-art on the v2c and grid benchmarks under three settings.