Uni-Dubbing: Zero-Shot Speech Synthesis from Visual Articulation (2024)

Abstract: In the field of speech synthesis, there is a growing emphasis on employing multimodal speech to enhance robustness. A key challenge in this area is the scarcity of datasets that pair audio with corresponding video. We employ a methodology that incorporates modality alignment during the pre-training phase on multimodal datasets, uniquely facilitating zero-shot generalization through the process of freezing the video modality feature extraction component and the encoder module within the pretrained weights, thereby enabling effective cross-modal and cross-lingual transfer. We have named this method ‘Uni-Dubbing’. Our method finely tunes with both multimodal and single-modality audio data. In multimodal scenarios, it achieves a reduced word error rate (WER) of 31.73%, surpassing the previous best of 33.9%. It also excels in metrics like tone quality and synchronization. With single-modality audio, it achieves a WER of 36.08%, demonstrating adaptability to limited data. Its domain generalization capabilities are proven across various language tasks in video translation and audio generation. Trained on 433 hours of audio data, it surpasses techniques using 200 hours of audiovisual data. The code and demo are available at https://diracer.github.io/unidubbing.

## introduction with the widespread use of short videos and online meetings in daily life and the workplace @xcite , the barrier of cross-linguistic communication has become an urgent problem, and thus multimodal technologies have attracted much attention @xcite . recently, many researchers have conducted corresponding studies in this area, such as lip reading task @xcite @xcite that transfers video domain to text domain, lip task(prajwal * *corresponding author @xcite @xcite ) that transfers video domain to audio domain, and lip translation @xcite that converts to the target language directly based on lips. in the case of the field of visual tasks, the biggest challenge for researchers is the extreme scarcity of training data. in addition, the relationship between lips and speech is not always a simple one-to-one mapping; for example, the same word may have very different lip shapes for people with different accents @xcite . therefore, maintaining accurate intonation poses a significant challenge, and this has led to the emergence of many important research findings. for these reasons, we adopt the strategy of using discrete units as intermediate targets, i.e., transforming audio and video data into discrete units for alignment, which can effectively circumvent the disadvantage of insufficient paired audio and video data. on top of this, we employ the rvq @xcite module thus enabling the method to achieve better timbre preservation, i.e. high fidelity, after full-shot training. furthermore, in order to cope with the lack of data for contemporary visual tasks, we also use mhubert @xcite and k-means of re-combining with discrete units, which enables our model to achieve better semantic consistency and reach zero-shot capability. as mentioned earlier, the barriers to cross-language communication are equally significant challenges and a lot of good work has emerged, but unfortunately none of the current methods have been able to achieve zero-shot cross-language video translation yet. we further explored learning cross-language and cross-modal lip2wav mappings from the audio domain, i.e., zero-shot transspeech, based on the zero-shot lip2wav model, have verified that the method is capable of crosslanguage migration. in summary, our goals in the current crosslanguage video-to-speech translation are twofold: based on these two goals, in this paper, the innovation of this study lies in proposing a framework that requires only cross-linguistic audio speech training, without the need for visual speech training inputs, to achieve direct synthesis of visual speech to cross-linguistic audio speech. this framework can predict the corresponding audio speech output by analyzing an individual's lip movements, and this prediction is not limited to the language system of the input visual speech. our method utilizes an advanced zero-shot learning strategy @xcite that aligns audio and visual phonemes with audio data alone during the training process, thus enabling the prediction of audio outputs in a target language that has not been seen before in seemingly impossible cross-modal scenarios. the main contributions of this paper are: ## related work in our paper, for the cross-language lip2wav synthesis task we mainly divide it into two steps: first implementing high-fidelity video-to-speech synthesis, followed by zero-shot cross-language videoto-speech translation. a great deal of excellent research work has preceded our study. ## conclusion this paper introduces uni-dubbing, an innovative approach trained on multimodal audio-video datasets, which achieved the best wer, estoi, and synchronization metrics on the lrs3 dataset. additionally, by utilizing implicit visual embeddings and acoustic tokens, we successfully preserved partial speaker information on the crossspeaker lrs3 dataset. we then implemented a zero-shot strategy, transitioning from audio to video modalities in cross-modal lip2wav tasks, and cross-lingual lip2wav translation tasks. this method significantly reduces the dependency on multimodal datasets and demonstrates potential for application in a wider range of tasks. to further validate the practicality of this method, our research utilized only the audio portion of existing multimodal datasets. in future work, we plan to explore the use of larger single-modality audio datasets, aiming to further expand the applicability and enhance the effectiveness of this method. through such research, we hope to deepen our understanding and utilization of single-modality audio data in multimodal tasks, thereby paving new paths for development in this field.