Where Visual Speech Meets Language: VSP - LLM Framework for Efficient and Context-Aware Visual Speech Processing (2024)

Abstract: In visual speech processing, context modeling capability is one of the most important requirements due to the ambiguous nature of lip movements. For example, homophenes, words that share identical lip movements but produce different sounds, can be distinguished by considering the context. In this paper, we propose a novel framework, namely Visual Speech Processing incorporated with LLMs (VSP-LLM), to maximize the context modeling ability by bringing the overwhelming power of LLMs. Specifically, VSP-LLM is designed to perform multi-tasks of visual speech recognition and translation, where the given instructions control the type of task. The input video is mapped to the input latent space of an LLM by employing a self-supervised visual speech model. Focused on the fact that there is redundant information in input frames, we propose a novel deduplication method that reduces the embedded visual features by employing visual speech units. Through the proposed deduplication and low rank adaptation, VSP-LLM can be trained in a computationally efficient manner. In the translation dataset, the MuAViC benchmark, we demonstrate that VSP-LLM trained on just 30 hours of labeled data can more effectively translate compared to the recent model trained with 433 hours of data.

## introduction along with audio, visual speech (e.g., lip movements) plays a critical role in human communication. with the increasing acknowledgment of the importance of visual speech, a diverse range of visual-based speech processing technologies @xcite @xcite @xcite is emerging. for instance, visual speech recognition (vsr) @xcite @xcite allows for the identification of spoken words through the observation of lip movements alone, without the need for audio access. most recently, the exploration has begun into visual speech translation (vst) @xcite , which directly generates translated text in the target language from the input lip movements of the source language. one key challenge in visual speech processing is to distinguish homophenes @xcite . homophenes refer to the words having different sounds but showing the same lip movements. therefore, a crucial aspect of developing visual speech processing systems is in the modeling of context so that the same lip movements can be mapped into correct different pronunciations (that is distinguishing homophenes). recently, large language models (llms) @xcite @xcite are attracting significant attention across various fields @xcite @xcite , thanks to their versatility and strong ability to model context. motivated by the recent success of llms, we try to investigate whether the rich context modeling ability of llms can be employed in visual speech processing and can mitigate the ambiguity of homophenes, especially focusing on two tasks, vsr and vst. to this end, in this paper, we propose a new framework named visual speech processing incorporated with llm (vsp-llm) that learns the seamless embedding of visual speech into the learned text space of llms. vsp-llm employs a self-supervised visual speech model to embed the input visual speech into phoneme-level representations, where the derived phonetic information can be effectively associated with text @xcite . moreover, to reduce the computational burden in training along with llms, we propose a novel deduplication method inspired by previous works @xcite @xcite . these studies have proven that speech unit-based deduplication lowers computational costs in audio-based tasks such as speechto-speech translation. building on this foundation, we expand the concept of unit-based deduplication to the visual modality. concretely, we employ visual speech units, the discretized representations of the features from a self-supervised model, as indicators for overlapped information between sequences. as the visual speech units can be regarded as pseudo-text @xcite , the visual speech features assigned to the same visual speech units are averaged to reduce the processing of redundant information and improve computational efficiency. through our analysis, we show that the sequence length can be reduced by approximately 50% using the proposed deduplication, with minimal performance degradation. finally, the proposed vsp-llm is jointly trained to perform vsr and vst with a single model which is the first explored in this paper. we show that by bringing the powerful context modeling ability into visual speech processing, we achieve state-of-the-art performances in both vsr and vst when using the lrs3 @xcite and muavic @xcite datasets as training data. additionally, our vsp-llm trained with just 30 hours of data outperforms the recent translation model used 433 hours of training data. the key contributions of this paper can be summarized as follows: 1) to the best of our knowledge, this is the first work to incorporate visual speech modeling with llms and achieve state-ofthe-art performances in vsr and vst. 2) this is the first to work to develop a unified visual speech processing model that can perform both vsr and vst with a single trained model. 3) we propose a novel visual speech deduplication that significantly improves computational efficiency. 4) we show that the proposed vsp-llm can perform multitasks with superior performances even in limited training resource situations, just with 30 hours of labeled data by outperforming the recent translation model. ## method figure 1 shows the overall framework of the proposed visual speech processing incorporated with llm (vsp-llm). it includes a visual encoder that embeds the input video into the input space of a pretrained llm, a visual speech unit based deduplication module that discards redundant information in contiguous frames, and an instruction embedding component that serves as a task specifier. in the following, we describe each component in detail. ## conclusion in this paper, we proposed a novel framework, visual speech processing with llms (vsp-llm), designed to leverage the context modeling ability of llms. through this framework, we built a unified model that can perform multi-tasks, vsr, and vst, with a single model. moreover, the proposed deduplication strategy reduces the redundant in-formation of visual speech representations based on pronunciation information modeled from visual speech units. through extensive experiments, we verified that the proposed deduplication method can reduce the visual sequence length by about 50% with minimal performance degradation. in addition, we validated the effectiveness of the vsp-llm by achieving a superior performance in the muavic benchmark with only 30 hours of labeled data.