A udio VSR : Enhancing Video Speech Recognition with Audio Data (2024)

Abstract: Visual Speech Recognition (VSR) aims to predict spoken content by analyzing lip movements in videos. Recently reported state-of-the-art results in VSR often rely on increasingly large amounts of video data, while the publicly available transcribed video datasets are insufficient compared to the audio data. To further enhance the VSR model using the audio data, we employed a generative model for data inflation, integrating the synthetic data with the authentic visual data. Essentially, the generative model incorporates another insight, which enhances the capabilities of the recognition model. For the cross-language issue, previous work has shown poor performance with non-Indo-European languages. We trained a multi-language-family modal fusion model, AudioVSR. Leveraging the concept of modal transfer, we achieved significant results in downstream VSR tasks under conditions of data scarcity. To the best of our knowledge, AudioVSR represents the first work on cross-language-family audio-lip alignment, achieving a new SOTA in the cross-language scenario.

## introduction visual speech recognition (vsr), also known as lip reading, aims to recognize the content of speech based on lip movements, without relying on the audio stream. previous work primarily utilizes video information to build vsr models. however, for any language, the amount of recorded audio information far exceeds that of video information. this raises the question: can these audio resources be leveraged to enhance the capabilities of vsr models? there are two methods to leverage audio data to build a strong vsr model: combining synthetic visual data with authentic visual data and leveraging the alignment of the audio and video. for the method of combining synthetic visual data with the authentic visual data, previous research @xcite has successfully used generated data to enhance image classification and contrastive learning capabilities. however, the effectiveness of this method varies from problem to problem and lacks discussion regarding the lip-reading task. does the chicken-andegg problem apply to the task of vsr? syncvsr @xcite applies synthetic data to vsr tasks but does not provide an in-depth discussion of mixing ratios and cross-language scenarios. our work settles these problems and validates the data inflation method in different scenarios. the effectiveness and robustness of lip-reading can be further enhanced with data inflation. this is essentially because the addition of new data increases the coupling of the dataset, which leads to betterlearned knowledge. based on this, we enrich the dataset using talking head generation (tfg) models. we compare multiple tfg models and find that using the tfg model for data inflation needs to be combined with the original data. furthermore, since using generated data for data inflation does not always work @xcite , we give a relatively stable mixing ratio. our method achieves new sota results on zeroshot and full-shot vsr tasks. for the method of leveraging the alignment of the audio and video, previous work @xcite has conducted extensive research on indo-european languages, proposing a cross-language model within the language family. however, there is a weakness when crossing language families, as there are significant linguistic differences between the language families. based on this, we propose a language-independent pre-training model for the two major language families, and use this model to accomplish the downstream vsr task. specifically, for any language, its lips and sounds always correspond to each other, so we use multilingual audio and video data to pre-train the audio-lip alignment model. considering that the audio data for any language is much larger than its video data @xcite @xcite , we transfer the knowledge of audiolip alignment gained in the pre-training phase to the downstream task, using audio to augment the capabilities of the vsr model. our contributions are highlighted as follows. ## method to achieve an effective vsr model in both zeroshot and full-shot scenarios, we rethink the essence of deep learning in sec. 3.1. based on these insights, we employ a talking head generation (tfg) model for data inflation in sec. 3.2. for the crosslanguage challenge, aiming for language independence, we focus on universal applicability across language families in sec. 3.3. consequently, we utilize self-supervised learning and introduce an audio-lip-alignment model in sec. 3.4. ## limitation for zero-shot and full-shot scenarios, there is a possibility that the model may not perform effectively. for cross-language scenarios, the results of audiovsr still need improvement in the sino-tibetan setting compared to the indo-european setting. furthermore, our model only includes the two major language families and is not universally applicable to all language families. in the future, we will explore the effectiveness of our method in noisy scenarios. ## conclusion we enhanced the performance of vsr across multiple scenarios. in zero-shot and full-shot situations, we employed an audio-to-lip model for data inflation, leveraging the knowledge learned from the generative model to enhance the lip-reading model. for cross-linguistic scenarios, we considered the impact of different language families on lip-reading tasks and trained an audio-lip-alignment model using self-supervised learning. we provided solutions for challenges such as insufficient video data and cross-linguistic scenarios. our method achieved state-of-the-art (sota) results in zero-shot, fullshot, and cross-language tasks.