MMS - LL a MA : Efficient LLM -based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens (2025)

Abstract: Audio-Visual Speech Recognition (AVSR) achieves robust speech recognition in noisy environments by combining auditory and visual information. However, recent Large Language Model (LLM) based AVSR systems incur high computational costs due to the high temporal resolution of audio-visual speech processed by LLMs. In this work, we introduce an efficient multimodal speech LLM framework that minimizes token length while preserving essential linguistic content. Our approach employs an early AV-fusion module for streamlined feature integration, an audio-visual speech Q-Former that dynamically allocates tokens based on input duration, and a refined query allocation strategy with a speech rate predictor to adjust token allocation according to speaking speed of each audio sample. Extensive experiments on the LRS3 dataset show that our method achieves state-of-the-art performance with a WER of 0.72% while using only 3.5 tokens per second. Moreover, our approach not only reduces token usage by 86% compared to the previous multimodal speech LLM framework, but also improves computational efficiency by reducing FLOPs by 35.7%. The code and models are available https://github.com/JeongHun0716/MMS-LLaMA .

## introduction in human communication, watching lip movements and listening to sounds are essential for understanding speech. these multimodal cues, which combine visual and auditory information, enable people to communicate effectively in bustling cafes, crowded streets, and noisy factories. thanks to these practical advantages and significant advances in deep learning, audio-visual speech recognition (avsr) technology has made remarkable progress through numerous research efforts @xcite @xcite @xcite @xcite . now, it is easy to find an avsr model that can accurately predict what you have said, even in noisy environments. rapid progress has been driven by large-scale audio-visual datasets @xcite , advanced neural architectures @xcite @xcite , improved multimodal learning strategies, including self-supervised learning @xcite and knowledge distillation using a pre-trained automatic speech recognition (asr) model @xcite ; carefully designed training methods @xcite ; and the utilization of large language models' (llms) language understanding capabilities as sentence predictors @xcite . among these, multimodal speech llm frameworks stand out by integrating llms for enhanced context modeling. however, despite these gain, they still suffer from high computational costs. this is largely due to multimodal speech tokens having a higher temporal resolution compared to text tokens, which forces the self-attention mechanism in each llm layer to process many more tokens, thereby significantly increasing the computational burden. to address these issues, we aim to develop an efficient multimodal speech llm framework (mms-llama) that minimizes the lenght of multimodal speech tokens while preserving their linguistic content. to achieve this, we construct this framework with three primary components: 1) early avfusion module shifts the fusion process to an earlier stage, prior to inputting multimodal speech tokens into the llm. 2) audio-visual speech q-former (av q-former) is designed to dynamically allocate the number of learnable queries according to the duration of audio-visual input, where queries are transformed into multimodal speech tokens. 3) going one step further, to more effectively adjust the number of multimodal tokens, we propose a speech rate predictor. it accounts for differences in speaking speed across each audio sample, enabling the assignment of more tokens to faster speech. reflecting the growing trend toward efficient multimodal models and leveraging insights from efficient multimodal llms that utilize minimal vision tokens during pre-fusion @xcite , we incorporate their effective pre-fusion approach into our multimodal speech llm framework. concurrently, we explore and integrate existing audiovisual speech fusion strategies into our early-av fusion module, which reduces the number of fused sequences by half. even though the number of fused sequences is reduced by half, their temporal resolution remains higher than that of text tokens. to minimize the audio-visual sequence length to similar level of text, av q-former is designed. since the language content in audio-visual inputs scales with their duration, a fixed-length output from a conventional q-former @xcite ) may be inefficient for compressing variable-length audio and visual modalities. to address this, we propose a novel query allocation strategy that adjusts the number of multimodal speech tokens based on the duration of the audio-visual input. using this dynamic strategy, we empirically determine the minimum number of multimodal speech tokens required for variable-length inputs and compress the sequence to a scale comparable to that of text tokens. finally, to further enhance efficiency, we refine our query allocation strategy by incorporating a speech rate predictor. even if two audio-visual inputs have the same duration, their language content may differ depending on the speaker's speaking rate. by taking this into account, we can assign more queries to faster speech segments, thereby generating a greater number of multimodal speech tokens and preserving their linguistic content. through extensive experiments, we demonstrate that using only 3.5 multimodal speech tokens per second can effectively preserve language content while maintaining performance. moreover, our approach achieves state-of-the-art performance with a word error rate (wer) of 0.72% on the lrs3 @xcite dataset. compared to the previous llm-based avsr model @xcite that uses 25 tokens per second, our method reduces token usage by 86%. additionally, our approach improves computational efficiency, reducing flops by 35.7%. ## method in this paper, our objective is to minimize the length of multimodal tokens while preserving their linguistic content, thereby enhancing the efficiency of our multimodal speech llm framework. recent works @xcite have demonstrated that llms can serve as effective mul-timodal speech learners by leveraging their context modeling capabilities. motivated by this finding, we adopt their framework as our baseline avsr model. as illustrated in figure 1 , the architecture consists of three main components: a visual encoder, an audio encoder, and an llm decoder that predicts sentences from multimodal tokens. building on these components, and with the goal of reducing the number of multimodal tokens while retaining essential linguistic information, we introduce three additional modules: the early av-fusion module, the av q-former, and the speech rate predictor. ## conclusion we have demonstrated that the proposed mms-llama, integrating efficient token compression strategies into multimodal speech llm frameworks can dramatically reduce computational costs while preserving high-level linguistic content. by incorporating an early av-fusion module, a dynamically adaptive av q-former, and a refined query allocation strategy with a speech rate predictor, we have achieved state-of-the-art avsr performance, attaining a wer of 0.72% while using only 3.5 multimodal speech tokens per second. moreover, our extensive experiments on the lrs3 dataset have confirmed that this method not only achieves a superior wer but also reduces flops by 35.7% compared to the previous llm-based avsr approach.