Investigating and Enhancing Vision-Audio Capability in Omnimodal Large Language Models (2025)

Abstract: Omnimodal Large Language Models (OLLMs) have shown significant progress in integrating vision and text, but still struggle with integrating vision and audio, often exhibiting suboptimal performance when processing audio queries compared to text queries. This disparity is primarily due to insufficient alignment between vision and audio modalities during training, leading to inadequate attention to visual information when using audio queries. To mitigate this issue, we propose a Self-Knowledge Distillation (Self-KD) training method where the vision-text component of the OLLM serves as the teacher and the vision-audio component as the student. This enables the model to process audio in a manner analogous to its text processing. Our experimental results demonstrate that Self-KD is an effective method for enhancing the vision-audio capabilities of OLLMs by learning from the vision-text components, which subsequently improves the interaction between audio and images and results in improved performance on multimodal tasks.

## introduction recent years have witnessed significant advancements in large language models (llms) @xcite @xcite , which have catalyzed the development of multimodal large language models (mllms) @xcite @xcite @xcite . this progress marks a paradigm shift in how machines understand and interact with the world, with omnimodal large language models (ollms) @xcite @xcite @xcite @xcite @xcite what's the name of the book on the top of the pile? (what's the name of the book on the top of the pile?) emerging as a new frontier. these models, exemplified by gpt-4o @xcite , demonstrate advanced capabilities in visual, linguistic, and auditory functionalities, promising more natural and comprehensive interactions. however, despite these advancements, a critical gap remains in the performance of ollms when processing vision-text versus vision-audio inputs. specifically, ollms often exhibit suboptimal performance on vision-audio tasks compared to their vision-language counterparts. for instance, replacing a text question with its audio equivalent can result in contradictory responses from the models. as illustrated in figure 1 , when the text question "what's the name of the book on the top of the pile?" is posed to megrez @xcite , the model accurately responds with "ariel". however, when the same question is converted into audio, it erroneously answers "plays pleasant". this inconsistency is prevalent across various ollms, indicating that the models exhibit different behaviors when processing vision-text and vision-audio inputs. to systematically evaluate this gap, we synthesize text questions from existing vision-language benchmarks into audio using text-to-speech (tts) technology. the results reveal that the vision-audio performance of ollms significantly lag behind their vision-text performance. notably, these incorrect audio responses, as illustrated in figure 1 , share a common thread: they are consistently image-relevant, despite being factually inaccurate. this observation implies that the models are processing both audio and visual cues but failing to synthesize them into correct answers. furthermore, we visualize the attention weights of ollms when processing input information and observe that the models show higher attention to query tokens in audio queries than in text queries, while exhibiting lower attention to vision tokens in audio queries compared to text queries. this indicates that ollms struggle to effectively integrate visual and audio information. it is hypothesized that this observation arises from a relative deficiency in the alignment between vision and audio compared to vision and text. to evaluate these alignments, we developed a new benchmark mmalign (see sec 3.2). the results confirm that the alignment between vision and audio is indeed weaker than that between vision and text. this discrepancy stems from the fact that during the alignment phase, ollm only aligned vision and text as well as audio and text, without directly aligning vision and audio. the model could only learn to process vision-audio inputs during the visionaudio sft phase. based on these results, we can conclude that conventional vision-audio sft alone is insufficient for enabling the model to effectively integrate vision and audio. to mitigate this issue, we propose a self-knowledge distillation (self-kd) training framework. in this framework, the vision-text component of the ollm serves as the teacher model, while the vision-audio component acts as the student model. unlike conventional vision-audio sft, self-kd uses the vision-text outputs of the model as soft labels to guide the training of the vision-audio component. after distillation, the student component learns the behavior of the teacher component, for instance, allocating more attention to vision tokens, thereby enhancing vision-audio performance. in summary, our contributions are as follows: ## evaluation of audio-vision capability for ollm currently, the evaluation of ollms focuses separately on their vision-language (vl) and audio capabilities, overlooking a holistic assessment of their vision-audio (va) capability. in this section, we first generate va benchmarks based on existing vl benchmarks and then conduct a comprehensive evaluation of ollms. ## why is ollm's audio-vision capability weaker? given that ollms exhibit inferior performance on vision-audio tasks compared to vision-text tasks, what factors contribute to this discrepancy? in this section, we first show that when processing vision-audio inputs, the attention weights of query tokens to vision tokens are lower than when processing vision-text inputs. we then build a new benchmark mmalign to evaluate the alignment between the audio modality and the image modality, as well as between the text modality and the vision modality within ollms. the results show that the alignment between audio and vision is significantly weaker than that between text and vision. finally, we discuss the connection between the model training process and vision-audio capability, suggesting that models need to enhance the integration of vision and audio during training. ## conclusions this paper investigates the issue of integration of audio and vision in ollms. we find that, for visual question answering tasks, performance with audio queries is significantly lower than with text queries. further analysis reveals that this disparity arises from insufficient alignment between images and audio during training, leading to inadequate attention to images when using audio queries. to address this, we propose a self-knowledge distillation training method, where the vision-text component serves as the teacher and the vision-audio component as the student. this approach aims to align the model's vision-audio capability with its vision-text capability. experimental results show that our method effectively improves the interaction between audio and images during model inference, outperforming existing baseline models in benchmark performance.