Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition (2022)

Abstract: Training Transformer-based models demands a large amount of data, while obtaining aligned and labelled data in multimodality is rather cost-demanding, especially for audio-visual speech recognition (AVSR). Thus it makes a lot of sense to make use of unlabelled unimodal data. On the other side, although the effectiveness of large-scale self-supervised learning is well established in both audio and visual modalities, how to integrate those pre-trained models into a multimodal scenario remains underexplored. In this work, we successfully leverage unimodal self-supervised learning to promote the multimodal AVSR. In particular, audio and visual front-ends are trained on large-scale unimodal datasets, then we integrate components of both front-ends into a larger multimodal framework which learns to recognize parallel audio-visual data into characters through a combination of CTC and seq2seq decoding. We show that both components inherited from unimodal self-supervised learning cooperate well, resulting in that the multimodal framework yields competitive results through fine-tuning. Our model is experimentally validated on both word-level and sentence-level tasks. Especially, even without an external language model, our proposed model raises the state-of-the-art performances on the widely accepted Lip Reading Sentences 2 (LRS2) dataset by a large margin, with a relative improvement of 30%.

## introduction audio-visual speech recognition (avsr) is a speech recognition task that leverages both an audio input of human voice and an aligned visual input of lip motions. it has been one of the successful application fields that involve multiple modalities in recent years. due to the limited amount of labeled, multimodal aligned data and the difficulty of recognition from the visual inputs (i.e., lip reading), it is a challenging task to tackle. existing avsr models tend to use extra data to increase the performance of the system, in a form of inserting an extra supervised learning stage in the training process. for example, many existing methods rely on an extra sequence level classification to bootstrap its learning on visual features. @xcite ; @xcite train their visual front-end on lrw (chung and zisserman, 2016) before learning on the avsr task. @xcite chunks the mv-lrs data @xcite into pieces of words and pre-train the model through classification. @xcite are also used in @xcite for the same purpose. learning an effective visual front-end could still be notoriously hard, even with these extra supervised learning tasks. sometimes curriculum learning is required to adapt the learned visual front-end into avsr task @xcite . end-to-end learning of large-scale avsr data hasn't been successful until recently @xcite . although self-supervised learning could enable leveraging unlabelled or even unaligned data, it hasn't been adequately explored on this task. @xcite is among the few attempts in this facet, in which it predicts lip motions from audio inputs. their proposed learning schemes yield strong emotion recognition results but are relatively weak in speech recognition. moreover, since in avsr it is the lip shape and motions between frames rather than the objects in a single image that matters for recognizing speech contents, if pre-trained visual models tailored for tasks targeting at single frame images could work for avsr remains unknown. in another scenario, selfsupervised learning in unimodality has been well established as a paradigm to learn general repre-sentations from unlabelled examples, such as in natural language processing @xcite , speech recognition @xcite , and computer vision @xcite @xcite . in this work, we rely on a simple but effective approach, which is to utilize unlabelled unimodal data by using pre-trained models that are trained in single-modality through self-supervised learning. specifically, we use @xcite pretrained on the large librilight @xcite dataset as our audio front-end. for visual front-end, we found that it is not as straight-forward for it to leverage pre-trained models, as we have to substitute the first convolutional layer in moco v2 @xcite ) by a 3-d convolutional layer and finetune it through lrw. in total, our approach doesn't require a curriculum learning stage, and the overall training time has been decreased. experimental results show that our new frontends significantly outperform previous ones by a big margin in both audio-only and visual-only settings, and a new state-of-the-art has been achieved in the final avsr setting. to our best knowledge, this is the first work that successfully applies unimodal pre-trained models in the multimodal setting of avsr. ## architecture the overall architecture of our model is shown in fig. 1 . the audio-visual model is comprised of four components, the front-ends and back-ends for both modalities, the fusion module, and the decoders. ## experiments in this section, we will first introduce the datasets and various settings we used in each component of our model. then we will present results of audioonly, visual-only and audio-visual settings. we also present a breakdown of the relative contribution of every component through ablation study.