Can VLM s Actually See and Read? A Survey on Modality Collapse in Vision-Language Models (2025)

Abstract: Vision-language models (VLMs) integrate textual and visual information, enabling the model to process visual inputs and leverage visual information to generate predictions. Such models are demanding for tasks such as visual question answering, image captioning, and visual grounding. However, some recent work found that VLMs often rely heavily on textual information, ignoring visual information, but are still able to achieve competitive performance in vision-language (VL) tasks. This survey reviews modality collapse analysis work to provide insights into the reason for this unintended behavior. It also reviews probing studies for fine-grained vision-language understanding, presenting current findings on information encoded in VL representations and highlighting potential directions for future research.

## introduction integration of information from multiple sensory modalities, such as language and vision is crucial in forming a cohesive understanding of the world. humans naturally combine sensory inputs in a way that balances and enhances the contributions of each modality. this is called cross-modal integration, which allows humans to interpret complex environments effectively and make inferences that go beyond any single information source @xcite . despite the clear advantages of cross-modal integration observed in human cognition, many visionlanguage models (vlms) struggle with modality collapse problems and fail to achieve a similar balance @xcite @xcite . this problem arises when a model fails to utilize one modality (modality collapse) and only relies on another (modality dominance). modality collapse is when an unimodal model achieves similar accuracy on a vision-language task compared to a multimodal model, showing the other modality is not fully utilized @xcite @xcite . modality collapse can impact the reliability of vlms, especially in tasks requiring a fine-grained understanding of both vision and text. figure 1 shows examples where even powerful vlms fail on simple tasks due to modality collapse. previous works mainly focus on improving the model's performance and robustness through debiasing @xcite @xcite , increasing model size @xcite , and using more training data @xcite . nevertheless, the extent to which vlms utilize vision and language modalities and their limitations remain unclear. to guide further research in vlms, we collect and piece together existing knowledge about modality collapse in vlms to complete the puzzle, answering the following research questions. ## preliminary we group existing vision-language models (vlms) into four architectural categories based on their vision-language fusion strategies: single-stream, dual-stream, dual-encoder, and large visionlanguage models (lvlms). single-stream vlms refer to vlms that process text and image input with a single transformer encoder. examples in this category include vl-bert @xcite , visualbert @xcite , uniter @xcite , unicoder @xcite and oscar @xcite . this design is also referred to as early fusion, as text and visual inputs are combined at the input level. dual-stream vlms process text and image inputs independently using dedicated encoders, and then fuse their representations via a multimodal fusion module (e.g., co-attention). the fusion is performed in a shared transformer designed to jointly reason over both modalities. early dual-stream vlms include bert-based vlms such as vil-bert @xcite and lxmert @xcite . lvlms or large vision-language models extend pretrained large language models by incorporating visual inputs through an adapter (e.g., mlp or q-former). some lvlms (e.g., llava @xcite ) directly project image embeddings into the language model's input space, and some use a more complex network (e.g., q-former in blip-2 @xcite ) to perform lightweight vision-language fusion before passing the result to the language model. unlike dual-stream models, lvlms avoid deep co-attentional fusion and rely on the language model to generate outputs from injected visual context. dual encoder models like clip @xcite and align @xcite encode images and text separately, aligning them in a shared embedding space via contrastive learning. unlike other architectures, they perform no cross-modal fusion, making them efficient for retrieval tasks but less suited for generation or fine-grained reasoning. takeaway message: no, modality collapse often happens in vlms and text modality often dominates. though vlms showed outstanding performance on several vl tasks when tested on benchmarks, little is known about whether they really "see" and "read" the input image. in this section, we divide this research question into two parts: contributing factors to modality collapse, and methods to measure the contribution of different modalities. ## rq2: what are (not) encoded in vl representation? through the lens of probing studies takeaway message: vlms can encode basic linguistic structure, simple fine-grained information (one-object setting), but fail to encode rich compositional information foot_2 . following rq1, we see that vlms are often dominated by text modality, due to textual bias, task setup, and lack of fine-grained training objective. studies reviewed in rq1 reveal the limitations of vlms, showing that vlms perform relatively well on coarse-grained tasks but fail to utilize visual information on fine-grained tasks. in this section, we aim to understand what is encoded in vl representation, categorized into three categories: i) linguistic semantics, ii) visual content, and iii) vl compositional. ## future direction a review of existing studies shows that modality collapse is common in vlms. although coarsegrained visual information (e.g., distinguishing paired image-text samples from unmatched pairs) is encoded into vl representations and benefits downstream tasks, current vlms still struggle to encode fine-grained information (e.g., spatial relationships and attributes) and handle vl compositionally. to address these limitations and unlock the full potential of vlms, we believe there are several future research directions. ## conclusion this survey explores the issue of modality collapse in vlms, highlighting the tendency of existing models to rely more on text input than visual information. we review factors that may contribute to this imbalance, including dataset bias, model behavior, pretraining objectives, and task setup. the issue of modality collapse also underscores the importance of properly evaluating cross-modal interactions, particularly in assessing whether vlms genuinely and effectively utilize multimodal inputs. this paper is the first systematic review of modality collapse in vlms. we hope this survey will facilitate further research in this area.