T rans F ace: Unit-Based Audio-Visual Speech Synthesizer for Talking Head Translation (2024)

Abstract: Direct speech-to-speech translation achieves high-quality results through the introduction of discrete units obtained from self-supervised learning. However, talking head translation, converting audio-visual speech (i.e., talking head video) from one language into another, still confronts several challenges compared to audio speech: (1) Existing methods invariably rely on cascading, synthesizing via both audio and text, resulting in delays and cascading errors. (2) Talking head translation has a limited set of reference frames. If the generated translation exceeds the length of the original speech, the video sequence needs to be supplemented by repeating frames, leading to jarring video transitions. In this work, we propose a model for talking head translation, TransFace , which can directly translate audio-visual speech into audio-visual speech in other languages. It consists of a speech-to-unit translation model to convert audio speech into discrete units and a unit-based audio-visual speech synthesizer, Unit2Lip, to re-synthesize synchronized audio-visual speech from discrete units in parallel. Furthermore, we introduce a Bounded Duration Predictor, ensuring isometric talking head translation and preventing duplicate reference frames. Experiments demonstrate that Unit2Lip significantly improves synchronization and boosts inference speed by a factor of 4.35 on LRS2. Additionally, TransFace achieves impressive BLEU scores of 61.93 and 47.55 for Es-En and Fr-En on LRS3-T and 100% isochronous translations. The samples are available at https://transface-demo.github.io .

## introduction with the rapid advancement of online communication technology, the demand for talking head translation technology @xcite ., * equal contribution. † corresponding author. to address these challenges, we propose a direct talking head translation framework, transface, consisting of a speech-to-unit translation model (s2ut) to convert audio speech into discrete units and a unit-based audio-visual speech synthesizer (unit2lip) to re-synthesize synchronized audio-visual speech from discrete units in parallel. unit2lip achieves acceleration with parallel synthesis of audio and visual speech (unit-to-audio & visual) instead of the traditional serial synthesis (unit-to-audio-to-visual). meanwhile, since the s2ut model can learn cross-linguistic mapping from parallel audio data, we directly apply this cross-linguistic mapping to the talking head translation to realize zero-shot talking head translation without parallel visual corpus. finally, to address the challenge of the fixed number of reference frames, we propose a bounded duration predictor module, which can uniformly coordinate the number of frames sustained by each discrete unit according to the target length, and thus control the total length, realizing isometric talking head translation and improving the translation result. the main contributions are as follows: ## transface the illustration of direct talking head translation system (transface) for audio-visual speech has been presented in figure 1 . (1) as shown in section 3.1, we employ a self-supervised learning (ssl) model hubert @xcite that has been pre-trained on the audio speech corpus to derive discrete units of target audio speech. these units are then employed to train the speech-to-unit translation model (s2ut). (2) furthermore, the ssl model is also adopted to obtain discrete units of audio speech from the audio-visual speech, which are used to train the unit-based synthesizer. in section 3.2, we introduce a speech-to-unit translation (s2ut) model to translate the source audio speech into the target units. (3) subsequently, a unit-based audio-visual speech synthesizer, which is trained separately on target language audio-visual dataset, is applied to convert the translated units into target audio-visual speech, as detailed in the section 4. ## unit-based audio-visual speech synthesizer in this section, we present a unit-based audio-visual speech synthesizer, unit2lip, that effectively converts discrete units into synchronized audio and visual speech. although the unit-based vocoder @xcite , which transforms units into audio speech, has been extensively studied and applied, the synthesizer for audio-visual speech encounters several additional challenges: (1) ensuring the synchronization of audio and visual speech, and (2) preserving the output video length identical to the source video length. ## conclusion talking head translation aims to translate speech from one language to audio-visual speech (i.e., talking head video) in other languages, which is widely used in online conferencing, online healthcare and online education. however, all current methods rely on cascading multiple models, severely hampering inference speed and rendering them unsuitable for online streaming applications. in this paper, we introduce transface, a direct talking head translation model comprising a speech-to-discreteunit translator and a unit-based audio-visual speech synthesizer, unit2lip. this framework facilitates parallel synthesis of audio-visual speech, significantly enhancing the speed of generating translated talking heads. additionally, we propose a bounded duration predictor to ensure consistent translation without compromising quality, addressing the challenge of abrupt video variations.