Multifaceted Evaluation of Audio-Visual Capability for MLLM s: Effectiveness, Efficiency, Generalizability and Robustness (2025)

Abstract: Multi-modal large language models (MLLMs) have recently achieved great success in processing and understanding information from diverse modalities (e.g., text, audio, and visual signals). Despite their growing popularity, there remains a lack of comprehensive evaluation measuring the audio-visual capabilities of these models, especially in diverse scenarios (e.g., distribution shifts and adversarial attacks). In this paper, we present a multifaceted evaluation of the audio-visual capability of MLLMs, focusing on four key dimensions: effectiveness, efficiency, generalizability, and robustness. Through extensive experiments, we find that MLLMs exhibit strong zero-shot and few-shot generalization abilities, enabling them to achieve great performance with limited data. However, their success relies heavily on the vision modality, which impairs performance when visual input is corrupted or missing. Additionally, while MLLMs are susceptible to adversarial samples, they demonstrate greater robustness compared to traditional models. The experimental results and our observations provide new insights into the audio-visual capabilities of MLLMs, highlighting areas for improvement and offering guidance for future research.

## introduction multi-modal large language models (mllms) @xcite @xcite @xcite @xcite have shown impressive performance in processing and understanding information from multiple modalities, such as text, image, and audio. the prevalent paradigm of mllms involves using modality-specific encoders @xcite to process individual modalities (e.g., image, video, and audio) † corresponding authors. among the modalities in the real world, text, vision, and audio are particularly important due to their prevalence and richness of information @xcite . therefore, evaluating the audio-visual capability of mllms is crucial for understanding their overall performance and potential applications in real-world scenarios @xcite . however, previous evaluation efforts @xcite @xcite have mostly focused on vision and language modalities, often ignoring the audio modality. this oversight limits our understanding of the full potential and limitations of mllms, especially in scenarios where audio information plays a critical role @xcite . for example, in autonomous driving, audio signals such as sirens and horns are crucial for safety @xcite . in multimedia content analysis, audio cues are essential for understanding context and emotions @xcite . compared to previous efforts involving only visual and linguistic modalities @xcite @xcite , the inclusion of the audio modalities poses several challenges. firstly, there are differences in the informativeness of different modalities @xcite @xcite . visual cues are often more informative (e.g., recog-nizing human actions or understanding locations), while audio signals can be more informative in rarer situations (e.g., detecting fire alarms or musical instruments). the multi-modal learning system may rely on the dominant modality (i.e., vision) while disregarding information from the other (i.e., audio) @xcite . secondly, the audio and visual modalities are complementary @xcite . when one modality is corrupted or missing, the other can provide supplementary information to aid scene understanding. the audio-visual llms should be able to leverage the complementary information from both modalities effectively. thirdly, the audio modality is noisier and less structured than the visual modality @xcite , as audio signals are often affected by background noise @xcite , reverberation @xcite , and other distortions @xcite . although there are some related works of audio-visual evaluation @xcite @xcite , they have mostly focused on effectiveness, whereas this work is more comprehensive, focusing on various aspects of mllms' ability. in this paper, we focus on evaluating the audiovisual capability of mllms. specifically, we aim to provide a comprehensive evaluation of their audio-visual capability across four key dimensions: ❶ effectiveness, measured by performance using audio and/or visual inputs. ❷ efficiency, which includes both data efficiency (how the models perform under limited data) and computational efficiency (e.g., model size, memory consumption, and inference speed). ❸ generalizability, focusing on performance under test-time distribution shifts. ❹ robustness, which measures resilience against adversarial perturbations. we conduct extensive experiments around the four aforementioned aspects with several observations. firstly, mllms are generally competitive in understanding audio-visual information, although they rely heavily on the visual modality. secondly, their over-reliance on the visual modality leads to poor performance when the video inputs are under test-time distribution shifts. thirdly, the mllms exhibit high data efficiency, achieving superior performance under limited data. however, they lag behind traditional models in terms of computational efficiency. fourthly, the complexity of language models in mllms makes them more robust under adversarial perturbations. we also provide addi-tional case studies to validate our observations. our contribution is summarized as follows: ## conclusion this paper systematically evaluates the audiovisual capabilities of mllms across four key dimensions: effectiveness, efficiency, generalizability, and robustness. the results show that mllms are generally effective in understanding audiovisual information, although they rely heavily on the visual modality, which leads to poor performance when video inputs are missing or undergo test-time distribution shifts. in addition, mllms exhibit high data efficiency with superior performance under limited data, but they lag behind in terms of computational efficiency. furthermore, mllms are more robust compared to traditional models against adversarial perturbations. these findings highlight the strengths and limitations of current mllms in handling audio-visual information, offering guidance for future research.