Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model (2023)

Abstract: We present a novel approach to multilingual audio-visual speech recognition tasks by introducing a single model on a multilingual dataset. Motivated by a human cognitive system where humans can intuitively distinguish different languages without any conscious effort or guidance, we propose a model that can capture which language is given as an input speech by distinguishing the inherent similarities and differences between languages. To do so, we design a prompt fine-tuning technique into the largely pre-trained audio-visual representation model so that the network can recognize the language class as well as the speech with the corresponding language. Our work contributes to developing robust and efficient multilingual audio-visual speech recognition systems, reducing the need for language-specific models.

## introduction with the great advancements of deep learning, automatic audio-visual speech recognition (avsr) technology has achieved remarkable progress @xcite @xcite @xcite . it utilizes multimodal inputs, including both audio and visual cues, providing several advantages to the deep learning-based speech recognition branch. one of the benefits is that it can accurately recognize speech in noisy environments, such as crowded restaurants or corrupted video conferencing situations. this capability is critical for advancing the field of automatic speech recognition technology in the future. nevertheless, outstanding performances in audio-visual speech recognition have been mostly shown in monolingual datasets, particularly in english. few recent studies have started focusing on multilingual speech recognition tasks, but they are still in their infancy. one work @xcite ., † corresponding author. while the recent multilingual speech recognition studies have shown remarkable advances, they have only focused on pre-training the multilingual speech recognition model or audio-visual speech representation model, followed by fine-tuning with the specific language @xcite . this is due to the imbalance of dataset language distribution and each language's distinctive characteristic. however, producing a language-specific speech recognition model can be time-consuming and inefficient. most importantly, it does not correspond to a real-life situation, where humans intuitively recognize the language when others are speaking. inspired by the human understanding perspective, in this paper, we design a single model multilingual audio-visual speech recognition framework that the model can not only determine which language is taken into the input speech but also recognize the speech correctly. to do so, we newly introduce an audio-visual guiding linguistic representation extractor. with the largely trained audio-visual speech representation model @xcite , we fine-tune the model by utilizing prompts so that the model can extract comprehensive linguistic information from the audio and video inputs. we set only a small amount of downstream task-specific parameters as the learnable parameters for extracting linguistic representation into the input space so that comprehensive linguistic information can be produced. furthermore, we consider the imbalanced distribution issue that the multilingual datasets contain, with some languages having significantly fewer samples than others. inspired by @xcite , we suggest a weighted objective function in order to balance each language distribution. by training our model on a diverse set of languages, we aim to capture their inherent similarities and differences, allowing our model to recognize and transcribe speech in multiple languages with greater accuracy and efficiency. we validate the effectiveness of our proposed model using muavic @xcite , a multilingual audio-visual corpus for robust speech recognition and robust speech-to-text translation providing 1200 hours of audio-visual speech in 9 languages. therefore, our work contributes to developing efficient and robust multilingual audio-visual speech recognition systems. we also believe that our proposed approach has several potential benefits: reducing the need for language-specific models, improving the performance of speech recognition in low-resource languages, and enabling more effective multilingual communication. ## methodology given the input multilingual video sequence, x @xmath0, . . . , x l } ∈ r l×h×w ×c where l, h, w , and c are the frame length, height, width, and channel sizes, respectively, and the paired input audio sequence, x @xmath1, where s represents the length of audio, we design a model that properly recognizes the given input video and audio with the correct language. we aim to utilize both visual and audio information so that the proposed architecture can successfully recognize not only the language but also the content of the input video. to this end, we initially propose a language prompt adopted from the largely pre-trained audio-visual speech representation model @xcite , which we call it linguistic representation extractor. further, we design a multilingual transformer decoder given the inputs of language class, linguistic representation, and the combined features from the audio and visual transformer encoders. we will explain the detailed aforementioned techniques in the following subsections. ## experimental result in the experimental result in clean environment. we evaluate audio-visual speech recognition in a clean environment. as shown in the first section of noisy environment. in the second section of ## conclusion we introduce the multilingual audio-visual speech recognition model by training a single model on a diverse set of languages. the proposed model finetunes a largely trained audio-visual representation model with prompts to provide meaningful language information. it has presented a promising starting point for future research endeavors.