What Do VLM s NOTICE ? A Mechanistic Interpretability Pipeline for G aussian-Noise-free Text-Image Corruption and Evaluation (2025)

Abstract: Vision-Language Models (VLMs) have gained prominence due to their success in solving complex cross-modal tasks. However, the internal mechanisms of VLMs, particularly the roles of cross-attention and self-attention in multimodal integration, are not fully understood. To address this gap, we introduce NOTICE, a Gaussian-Noise-free Text-Image Corruption and Evaluation pipeline for mechanistic interpretability in VLMs. NOTICE introduces Semantic Image Pairs (SIP) corruption, the first visual counterpart to Symmetric Token Replacement (STR) for text. Through NOTICE, we uncover a set of “universal attention heads” in BLIP and LLaVA that consistently contribute across different tasks and modalities. In BLIP, cross-attention heads implement object detection, object suppression, and outlier suppression, whereas important self-attention heads in LLaVA only perform outlier suppression. Notably, our findings reveal that cross-attention heads perform image-grounding, while self-attention in LLaVA heads do not, highlighting key differences in how VLM architectures handle multimodal learning.

## introduction vision-language models (vlms) have become central to both computer vision and natural language processing (nlp) due to their ability to complete complex multimodal tasks like visual recognition @xcite , visual question answering @xcite @xcite , and image captioning @xcite @xcite . many foundational vlms, such as blip (bootstrapping language-image pretraining) @xcite , use cross-attention to capture interactions between modalities @xcite @xcite , while others, like llava @xcite , successfully use self-attention for multimodal integration. despite recent works investigating the differences in cross-attention and self-attention in vlms @xcite @xcite , the specific role of different attention heads in vision-language is not fully understood, particularly in large vlms such as llava @xcite . in nlp, mechanistic interpretability (mi) has proven effective for unraveling the complex structure of llms by identifying pathways, or circuits, that enable behaviors like logical reasoning and indirect object identification @xcite . a core method in mi is causal mediation analysis (cma), which measures output changes by intervening on inputs and tracking effects. to apply cma in transformers, @xcite use activation patching, which identifies key model components for a given task by replacing the hidden states of a corrupt run with those from a clean run. a "clean run" refers to a standard forward pass on an input. a "corrupt" run either alters the input by swapping input tokens with semantically related tokens or applies gaussian noise to token embeddings so that the model can no longer produce the correct answer from the clean pass. while successful in nlp, mechanistic interpretability methods are understudied in vlms due to the need for an effective semantic-based corruption scheme for images. previous works applying activation patching to vlms only study blip, focus on multi-layer perceptron (mlp) layers in blip's decoder, and apply gaussian noise to image embeddings to create corrupt runs @xcite . however, multimodal interaction occurs in blip's image-grounded text encoder through the cross-attention layers, meaning findings from @xcite do not provide insights on how vlms integrate modalities. further, recent works have shown that creating corrupt runs by applying gaussian noise to token embeddings can produce illusory results @xcite . @xcite advocate for symmetric token replacement (str) that swaps tokens in a clean prompt with semantically related tokens to create a corrupt run as it keeps the corrupted run within distribution and provides more reliable insights. current approaches to image corruption lack an equivalent method to str that does not use gaussian noise for image corruption, leaving a gap in reliable multimodal analysis. we propose semantic image pairs (sip) corruption, which leverages existing datasets to create clean and corrupt images that vary by one semantic property (figure 1 ). for instance, in the svo-probes dataset (hendricks and nematzadeh, 2021), subject-verb-object triplets are already curated to vary by just one element (e.g., 'puppy, lying, grass' vs. 'goat, lying, grass'), allowing us to create "clean" and "corrupt" image pairs from the existing image labels (figure 1 panel 1). sip is not limited to curated datasets and can be extended to any image dataset with easily interchangeable attributes, such as cars, shapes, animals, etc. further, we show in section 4.1 that sip corruption can be performed using generative models to create corrupt images. our framework, notice (gaussian-noise-free text-image corruption and evaluation), is a mechanistic interpretability pipeline for vlms. no-tice uses our novel sip method to corrupt images and str to corrupt text. using notice, we uncover "universal cross-attention heads" in blip and "universal self-attention heads" in llava that consistently produce large patching effects across all tasks. we show that universal cross-attention heads implement human-interpretable functions categorized into three classes: object detection, object suppression, and outlier suppression. we find that both self-attention and cross-attention heads implement outlier suppression. however, only cross-attention heads in blip perform the imagegrounding functions of object detection and object suppression, highlighting key differences in how vlm architectures handle multimodal learning. multimodal models typically integrate their inputs through one of two main approaches: (1) early fusion @xcite , where self-attention processes combined image patches and text representations, or (2) cross-attention fusion @xcite , where queries from one modality interact with representations from the other. @xcite investigate cross-modal integration using input ablation techniques and conclude that self-attention models, such as visualbert, show less sensitivity to missing visual input compared to cross-attention models like vilbert, which exhibit more significant reliance on cross-modal interactions. @xcite finds that vision-text crossattention learns visual grounding of noun phrases into objects, while text-to-text attention captures low-level syntactic word relations. @xcite probes dual-stream vlms through tasks like image-sentence alignment and counting, revealing strong alignment performance but limited grounding of visual symbols, resulting in poor counting capabilities. however, there has yet to be a comprehensive study applying mechanistic interpretability techniques to investigate how recent architectures like blip and llava 1) use attention mechanisms to integrate multimodal information and 2) determine the specific functions that attention heads in these models learn to perform. ## results we begin our investigation by conducting modulewise activation patching to understand the global patterns of self-attention, cross-attention, and mlp modules under both image and text corruption. we demonstrate that sip image corruption produces results closely aligned with str text corruption and effectively highlights the role of middle-layer mlps, which have repeatedly proven crucial in a wide range of tasks @xcite . in contrast, gaussian noise image corruption only identifies the final mlp layers as producing a significant patching effect. building on this, we use sip to perform a fine-grained analysis of cross-attention heads in blip and self-attention heads in llava, the critical point of multimodal integration for each model. as we identify the most significant crossattention heads in blip and most significant selfattention heads in llava, we categorize them into three distinct functional roles, offering deeper insights into their contributions and behaviors. ## conclusion in this work, we study the roles of cross-attention and self-attention in multimodal integration. to accomplish this, we introduce notice, a gaussian-noise-free text-image corruption and evaluation pipeline for mechanistic interpretability in vlms. using our sip image corruption framework and str text corruption, notice enables causal me-diation analysis across both modalities. experiments across three tasks reveal "universal attention heads" with consistent patching effects across blip and llava. we show that cross-attention supports three functions: object detection, object suppression, and outlier suppression. however, self-attention primarily handles outlier suppression, indicating cross-attention's unique role in image grounding. while attention presents a plausible, but not necessarily faithful rationale for model prediction @xcite , our findings show patterns of attention heads that consistently, and causally, influence model logits. with the support notice, applications using vlms can benefit from more transparent and adaptable models, facilitating deeper insights and enhanced performance.