AV - T ran S peech: Audio-Visual Robust Speech-to-Speech Translation (2023)

Abstract: Direct speech-to-speech translation (S2ST) aims to convert speech from one language into another, and has demonstrated significant progress to date. Despite the recent success, current S2ST models still suffer from distinct degradation in noisy environments and fail to translate visual speech (i.e., the movement of lips and teeth). In this work, we present AV-TranSpeech, the first audio-visual speech-to-speech (AV-S2ST) translation model without relying on intermediate text. AV-TranSpeech complements the audio stream with visual information to promote system robustness and opens up a host of practical applications: dictation or dubbing archival films. To mitigate the data scarcity with limited parallel AV-S2ST data, we 1) explore self-supervised pre-training with unlabeled audio-visual data to learn contextual representation, and 2) introduce cross-modal distillation with S2ST models trained on the audio-only corpus to further reduce the requirements of visual data. Experimental results on two language pairs demonstrate that AV-TranSpeech outperforms audio-only models under all settings regardless of the type of noise. With low-resource audio-visual data (10h, 30h), cross-modal distillation yields an improvement of 7.6 BLEU on average compared with baselines. Audio samples are available at https://AV-TranSpeech.github.io/ .

## introduction speech-to-speech translation (s2st) models @xcite @xcite relying on speech data have achieved high performance and significantly broken down communication barriers between people not sharing a common language, which attracts broad interest in the machine learning community @xcite . among them, direct systems @xcite leverage recent progress on self-supervised discrete units learned from unlabeled speech for building textless s2st, further supporting translation between unwritten languages @xcite . as speech production @xcite @xcite is accompanied by the movement of lips and teeth, it can be visually interpreted to understand speech. in recent years, significant research @xcite has introduced joint modeling of spoken language and vision: @xcite investigate to learn lip-based audio-visual speaker embeddings, where the speaker's mouth area is used alongside speech as inputs. @xcite focus on audio-visual speech enhancement and separation which better integrates visual information. despite their success, it is unclear how lip can contribute to audio-based s2st, and how to incorporate visual modality as auxiliary information in s2st. a visual translator may open up a host of practical applications: improving speech translation in noisy environments, enabling dictation, or dubbing archival films. despite the benefits of audio-visual approaches, training direct speech translation models without relying on intermediate text typically requires a large amount of training data, while there are very few resources providing parallel audio-visual speech due to the heavy workload. to mitigate the data scarcity, researchers have leveraged multitask learning @xcite , data augmentation @xcite , and weekly-supervised data with synthesized speech @xcite in audio s2st. in this work, we propose av-transpeech, introducing the first av-s2st system without using text. as illustrated in figure 1 , our textless av-transpeech inherits speech-to-unit translation (s2ut) framework @xcite , which consists of an audio-visual speechto-unit translation (av-s2ut) model followed by a unit-based vocoder that converts discrete units to speech. av-transpeech complements the audio stream with the auxiliary visual information, which is invariant to speaking environments and promotes system robustness. to tackle the challenges of data shortage, we 1) build upon the recently introduced audio-visual hubert (av-hubert) which learns contextual representations through self-supervised masked prediction, and show that large-scale pre-training benefits av-s2st training; 2) introduce cross-modal distillation and leverage s2st models trained on the audio-only corpus, which further reduces the requirements of visual data and boosts the performance of visual systems in low-resource scenarios. experimental results on two language pairs demonstrate the robustness of av-transpeech in noisy scenarios, outperforming audio-only s2st under all settings regardless of the snr and the type of noise. with low-resource audio-visual data (10h, 30h), cross-modal distillation yields an improvement of 7.6 bleu on average compared with baselines. the main contributions of this work include: ## av-transpeech in this section, we first overview the encoderdecoder architecture for av-transpeech, following which we introduce the cross-modal distillation procedure for few-shot transfer learning with lowresource data. the overall architecture has been illustrated in figure 1 , and we put more details on the encoder and decoder block in appendix a. ## conclusion in this work, we proposed av-transpeech, the first audio-visual speech-to-speech translation (av-s2st) model without relying on intermediate text. av-transpeech complemented the audio stream with the visual information to promote robustness in noisy environments, opening up a host of practical applications: silent dictation or dubbing archival films. to mitigate the data scarcity for training av-s2st models, we 1) built upon the av-hubert with a self-supervised learning framework fir contextual representations, showing that large-scale pre-training benefited the av-s2st training; 2) leveraged cross-modal distillation with s2st models trained on the audio-only corpus, which further reduced the visual data requirements and boosted performance in low-resource scenarios. experimental results on two language pairs demonstrated that av-transpeech achieved significant robustness in noisy environments, outperforming audio-only s2st models under all settings regardless of the type of noise. with low-resource audio-visual data (10h, 30h), cross-modal distillation yielded an improvement of 7.6 bleu on average compared with baselines. we envisage that our work will serve as a basis for future audiovisual speech-to-speech translation studies, unlocking the ability for high-quality translation given a user-defined modality input.