XLAVS - R : Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception (2024)

Abstract: Speech recognition and translation systems perform poorly on noisy inputs, which are frequent in realistic environments. Augmenting these systems with visual signals has the potential to improve robustness to noise. However, audio-visual (AV) data is only available in limited amounts and for fewer languages than audio-only resources.To address this gap, we present XLAVS-R, a cross-lingual audio-visual speech representation model for noise-robust speech recognition and translation in over 100 languages. It is designed to maximize the benefits of limited multilingual AV pre-training data, by building on top of audio-only multilingual pre-training and simplifying existing pre-training schemes. Extensive evaluation on the MuAViC benchmark shows the strength of XLAVS-R on downstream audio-visual speech recognition and translation tasks, where it outperforms the previous state of the art by up to 18.5% WER and 4.7 BLEU given noisy AV inputs, and enables strong zero-shot audio-visual ability with audio-only fine-tuning.

## introduction speech recognition and speech-to-text translation, two core speech perception tasks, have witnessed rapid developments in the past two years. still, the performance of state-of-the-art (sota) models such as whisper @xcite and @xcite degrades sharply in noisy environments @xcite . audio-only approaches to make such systems more robust @xcite remain challenged by frequent noise types such as intense babble noise and overlapped speech @xcite . complementing audio inputs with visual signals such as video frames of speaker's lips offers an alternative approach @xcite @xcite ., * work done at meta ai hours languages a av a av audio-only pre-training data avformer (seo et al., 2023) 60k 0 1 ⋆ 0 fava @xcite 2.8k 0 >100 0 audio-only & audio-visual pre-training data u-hubert @xcite 0.5k 1.8k 1 ⋆ 1 ⋆ vatlm @xcite 4.3k 1.8k 1 ⋆ 1 ⋆ xlavs-r (this work) 436k 1.2k 128 9 xlavs-r (+ extended av data) 436k 8.3k 128 100+ to bridge this gap, we present xlavs-r, a cross-lingual audio-visual speech representation learning model for noise-robust speech perception across over 100 languages. its training strategy is designed to maximize the efficiency of multilingual av pre-training by injecting audio-visual (av) signals ( §3.2) after pre-training on audioonly (a-only) data that is easier to scale in size and language coverage ( §3.1). we improve av-hubert @xcite with simplified training protocol and updated model architecture, and scale model size up to 2b parameters for performance in the multilingual setting. we conduct extensive evaluations on the muavic benchmark @xcite . xlavs-r yields sota performance on speech recognition in 9 languages and speech-to-text translation in 6 x-to-english pairs ( §5.1, §5.2), notably improving their robustness to noisy inputs. furthermore, xlavs-r enables zero-shot audio-visual ability in downstream tasks with audio-only fine-tuning (ft): fine-tuning our xlavs-r 2b model without supervised av downstream data achieves the best audio-visual performances in noisy settings even compared to the corresponding audio-visual supervised fine-tuned model ( §5.4). this confirms the benefits of our pre-training strategy and relaxes the data requirements for downstream tasks, making it possible to tackle avsr and avstt tasks without labeled-transcribed or translated-av data. ## related work self-supervised audio-only speech representation. self-supervised learning (ssl) for speech aims to establish a general speech representation with unlabeled speech data applicable to various downstream applications, including speech recognition and spoken language understanding tasks @xcite . many of today's successful ssl approaches @xcite @xcite are based on masked span prediction. specifically, wav2vec 2.0 @xcite applies a contrastive predictive coding-like (van den oord et al., 2018) loss on masked speech utterances. hubert @xcite utilizes masked speech features to predict hidden units derived from layerwise features. instead of using discrete units, data2vec @xcite directly regresses dense features from an exponential moving average (ema) teacher. ssl approaches notably reduce the need for labeled speech data. in asr benchmarks @xcite , they yield performance on par with or better than fully supervised models with far fewer transcriptions. this has motivated sustained efforts toward multilingual ssl models such as xlsr-53 @xcite , xls-r @xcite , and mms @xcite , which particularly excel in low-resource language scenarios. self-supervised audio-visual speech representation. audio-visual ssl approaches draw heavy inspiration from their audio-only counterparts. av-hubert @xcite extends hubert @xcite to the audio-visual setting by taking the masked audio-visual stream as input and predicting the hidden units initialized with mfcc clusters, iteratively refining them with layerwise features. this framework has proven effective for multiple downstream tasks, including lip reading @xcite , audio-visual speech recognition and translation @xcite . in av2av @xcite and our work of xlavs-r, it has been extended to a multilingual setting. raven @xcite leverages modality-specific ema teachers to generate targets for masked prediction, thus avoiding an iterative refinement process. similarly, av-data2vec @xcite is based on data2vec @xcite and regresses multimodal features with an audio-visual ema teacher. av2vec @xcite further combines av-data2vec with the masked prediction objective in av-hubert. some prior works also investigated the use of unpaired unimodal data, notably audio, to improve multimodal representation. u-hubert @xcite augments av-hubert with audio-only speech during pretraining, resulting in improved audio-visual speech recognition performance. vatlm @xcite further added text-only data and trained the model on an arbitrary modality stream. our work builds on the insights of av/u-hubert with adaptations for continual ssl training with visual modality injection to a-only model and for the multilingual setting. ## xlavs-r our approach (figure 1 ) requires first training an audio-only ssl model, which is used to generate unit targets for audio-visual ssl pre-training. we inject visual modality into the resulting model to continue training with audio-visual ssl pretraining. through this process, xlavs-r introduces three key improvements over baseline approaches. first, we leverage the abundant availability of audio-only speech data to enhance overall performances and achieve greater robustness on domain shift ( §3.1), and inject visual modality to build an audio-visual pre-training model on top of audio-only pre-training model ( §3.2). second, we use a learnable audio feature extractor to better capture the various phonetic information of multilingual audios ( §3.3). third, we improve the training efficiency of the audio-visual pre-training stage by single-round training with unit targets from audioonly contextualized representation ( §3.4). ## experiments data. for av pre-training, we leverage a total 1.2k hours of data in 9 languages: english (en), arabic (ar), german (de), greek (el), spanish (es), french (fr), italian (it), portuguese (pt) and russian (ru). we also experiment with additional data for domain and language coverage. on top of the above 1.2k hour data, we add 7.1k hours of inhouse av data in 100+ languages and train xlavs-r with a total of 8.3k hours in 100+ languages, as summarized in the last row of for a-only pre-training, we follow babu et al. (2022) for training on 436k-hour data in 128 languages. for all the pre-trained models, we perform multilingual fine-tuning on muavic labeled data for audio-visual speech recognition (avsr) in all 9 languages and audio-visual speech-to-text translation (avs2tt) in 6 language pairs into english-el, es, fr, it, pt, ru. we use muavic for in-domain evaluation on both audio-only and audio-visual test modes. we use fleurs @xcite for audio-only out-of-domain (ood) evaluation. we report an average of all 9 languages for avsr and 6 xto-english language pairs for avs2tt in both indomain and out-of-domain evaluation. we remove extremely short utterances (less than 0.2 seconds) and long utterances (more than 20 seconds) for better training stability. modeling. we build xls-r ( §3.1) and xlavs-r ( §3) models in two model sizes: 300m and 2b. the number of encoder layers/embedding dimension/feed forward dimension are 24/1024/4096 and 48/1920/7680 respectively. for audio-visual training targets of xlavs-r, we extract audio-only speech representation from the 36th layer of xls- ## conclusion we introduced xlavs-r, a cross-lingual audiovisual speech representation for noise-robust speech perception in over 100 languages. since audio-visual data is harder to come by than audioonly data, we injected the visual modality by continuing training an audio-only pre-trained model. we also design a simpler yet more effective training scheme and improved architecture compared to previous state-of-the-art models. extensive evaluation on the muavic benchmark shows xlavs-r achieves sota performance on both audio-visual speech recognition and translation tasks, yielding particularly large improvements in noisy settings. we also show that xlavs-r effectively leverages unlabeled audio-only multilingual speech data and audio-visual data in self-supervised learning resulting in enhanced zero-shot audio-visual ability for downstream tasks, and extended av pre-training data augments robustness against domain shift.