LLM s Can Compensate for Deficiencies in Visual Representations (2025)

Abstract: Many vision-language models (VLMs) that prove very effective at a range of multimodal task, build on CLIP-based vision encoders, which are known to have various limitations. We investigate the hypothesis that the strong language backbone in VLMs compensates for possibly weak visual features by contextualizing or enriching them. Using three CLIP-based VLMs, we perform controlled self-attention ablations on a carefully designed probing task. Our findings show that despite known limitations, CLIP visual representations offer ready-to-read semantic information to the language decoder. However, in scenarios of reduced contextualization in the visual representations, the language decoder can largely compensate for the deficiency and recover performance. This suggests a dynamic division of labor in VLMs and motivates future architectures that offload more visual processing to the language decoder.

## introduction vision-language models (vlms) have made remarkable progress in recent years, with systems like magma @xcite , blip-2 @xcite , llava @xcite , and prismatic @xcite demonstrating strong performance for their time on key multimodal benchmarks. a common design choice across these models is the use of a frozen pretrained vision encoder, often based on clip @xcite , paired with a pre-trained language model, which is finetuned to map visual features into text. 1 despite the widespread use of clip, its limitations as a visual backbone are well-documented. clip representations have been shown to prioritize global over local features @xcite , to perform poorly in distinguishing objects which share high-level features @xcite , to lack fine-grained compositionality @xcite , and to exhibit quantity and size biases @xcite . many of these limitations have been attributed to the contrastive training objective that clip employs. nonetheless, many modern vlms that rely on the clip vision encoder perform surprisingly well, even on tasks requiring detailed visual understanding @xcite @xcite . this raises a fundamental question: how do vlms overcome the known limitations of clip's representations? one plausible hypothesis is that the language decoder-often much larger than the vision encoder and trained with rich linguistic supervision-plays a compensatory role, enriching or contextualizing the visual representations it receives. if true, this would suggest a more dynamic division of labor between vision and language components than currently believed. in this work, we investigate this hypothesis through a series of controlled self-attention blocking experiments @xcite on three vlms, each of which pairs clip with a distinct language model. we ask whether the language decoder in a vlm contributes to the enrichment of image features. as a diagnostic task, we focus on the identification of localized object parts (e.g., the ear of a cat or the stem of an apple). this fine-grained task requires extensive contextualization of the part region into the broader context of the object. we build a probe using segmentation annotations from the panoptic parts dataset @xcite , and apply logit lens analysis @xcite to inspect the intermediate representations across layers of the vlm. we iteratively block self-attention in the vision encoder, language decoder, or both, and evaluate the vlm's ability to maintain the identifiability of object parts in the relevant regions. our results yield three key findings: (1) when the decoder self-attention is disabled, part identifiability is largely preserved, suggesting that the self-attention in the decoder does not substantially enrich already-good visual features. (2) when both encoder and decoder self-attentions are blocked, part identifiability collapses, showing the importance of some form of context-based feature construction. (3) crucially, when only the encoder self-attention is blocked, part identifiability largely recovers-indicating that the decoder can compensate for deficiencies in visual representations when the need arises through contextualization. these findings challenge the assumption that visual semantic understanding is fully localized in the vision encoder of vlms. instead, they suggest an adaptive joint processing of image inputs by vlm components, where the language decoder can step in to compensate for a degraded input from the vision encoder. this has implications for future vlm architectures, which could actively offload more of the vision processing onto the language decoder, reducing the contribution of the vision encoder to only that which the decoder cannot recover. ## related work efforts in interpreting language models have led to the development of techniques that probe internal representations and decompose the mechanism behind next-token prediction. methods such as neuron attribution @xcite , causal tracing @xcite , attention knockout @xcite , and logit lens @xcite have improved our understanding of how information flows across tokens and model layers. these have also been extended to vlms to understand how visual and linguistic information interact. @xcite applied causal tracing to vlms and found that llava @xcite retrieves information from earlier layers in the language decoder compared to unimodal language models. building on this, @xcite used attention knockout and logit lens analysis to further decompose information flow in vlms, revealing a two-stage process: visual enrichment, where image features transfer to object tokens, and semantic refinement, where these features are interpreted through language. similarly, @xcite demonstrated that early layers transfer global visual information into question token representations, mid-layers inject question-relevant visual features into corresponding text positions, and later layers propagate this fused representation to the last token position for answer prediction. contrary to such grounding evidence, @xcite highlighted the dominant role of language priors in vlm's outputs, while @xcite found that in different settings llava either over-relies on text input or, conversely, exhibits strong visual grounding. while the studies above explore how vlms generate text from either visual input or a mix of visual and text input, there have been very few works that attempt to dissect the language model representation of the visual tokens as such. @xcite identified multimodal neurons in the language decoder that map visual inputs to corresponding linguistic concepts, highlighting that linear projections in vlms lack semantics on their own when interpreted using language vocabulary. most related to our focus, @xcite used visual token ablation and logit lens analysis to reveal that llava localizes object-specific information at the corresponding positional tokens and can align these with fine-grained semantic concepts beyond surface-level object categories. zooming in on the vision encoder itself, gandelsman et al. @xcite ) decomposed clip's visual embeddings into sparse text concepts and found that different attention heads specialize in distinct semantic properties (e.g., color, shape, location, etc). despite clip's strong zero-shot classification performance, various studies have highlighted its limitations, ranging from poor compositional reasoning and limited sensitivity to fine visual distinctions @xcite @xcite , to biases related to object quantity and size @xcite . meanwhile, @xcite showed that some perceived limitations in the clip visual representations in fact stem from the global pooling of features in similarity-based analysis and from its weak text encoder. in a similar vein, @xcite noted the inability of image-text metrics like clip-score @xcite to distinguish differences in object relations, attributes, or logical structure, often treating captions as a "bag of words." together, these studies tell a nuanced story of the mechanisms behind building visual representations, with their perceived and real limitations, and translating those into text. our work investigates how visual representations evolve through the layers of the language decoder and to what extent their deficiencies are resolved in the process. ## object part identification (opi) to investigate how vision-language models contextualize visual inputs, we propose a fine-grained probing task that requires extensive contextualization: identifying object parts in patch-level localized image regions. the complexity of this task arises from the fact that parts can often be overlooked in favor of the whole, while also being potentially unidentifiable in isolation of the object they belong to. the details of the probing task are described below and are visualized in figure 1 . task. opi assumes the availability of patch-level masks which localize object parts in an input image. a single patch can contain multiple parts and a single part can span multiple patches. probing the vlm consists of: (1) inducing a probability distribution from a vlm for every image patch, (2) extracting the rank of all part labels relevant to a patch, and (3) aggregating the ranks across all patches in the part-relevant. this results in one value per part which indicates how well the model identifies this part in this image. dataset. we derive the necessary annotations from the pascal panoptic parts (pascal-pp) dataset (de geus et al., 2021; meletis et al., 2020) which provides high-quality pixel-level segmentation annotations for 194 part classes spanning 16 object classes, e.g., dog-head, horse-tail, person-leg, etc. we focus on 7 out of the 16 object classes available in pascal-pp, specifically animal objects. this and other measures taken to reduce noise are outlined in appendix a. after filtering, we sample up to 100 images per class, where available. finally, we convert pixel-level segmentation masks into patch-level masks, effectively reducing the resolution of the masks. the final dataset contains 567 unique images and 2840 unique part regions. (see rank. prior to observing the rank of a label, the probability distribution is processed to aggregate all label aliases, e.g., "leg", "legs", "_leg", "_legs", under a single token, leg, summing up all their probabilities. 3 the rank of label l is obtained from the probability distribution p i as follows: ## conclusion & future work in this study, we investigated the internal mechanisms of vision-language models (vlms) by evaluating their ability to identify object parts under a range of controlled attention ablations. our findings reveal a nuanced and adaptive division of labor between the vision encoder and the language decoder, where an llm can compensate for some level of deficiency in the visual representations, recovering and enriching semantics that would otherwise be absent. these findings have several im-plications for future work. first, in more complex visual domains-where even fully contextualized visual features may fall short-the compensatory role of the llm may become even more critical: llms could serve as a fallback mechanism in scenarios where visual representations from the visual encoder alone are sufficiently inexpressive. second, our results suggest that future research could exploit the capabilities of llm further for contextualization within vlm architectures. specifically, disabling self-attention in certain layers of the vision encoder during pretraining-or rethinking the encoder-decoder interface entirely-could lead to more efficient models with deeper fusion of modalities from the ground up. it might be time to rethink vlm architectures: not as static pipelines, but as adaptive systems capable of redistributing computational burden between the vision and language components depending on the task.