Cross-Modal Projection in Multimodal LLM s Doesn’t Really Project Visual Attributes to Textual Space (2024)

Abstract: Multimodal large language models (MLLMs) like LLaVA and GPT-4(V) enable general-purpose conversations about images with the language modality. As off-the-shelf MLLMs may have limited capabilities on images from domains like dermatology and agriculture, they must be fine-tuned to unlock domain-specific applications. The prevalent architecture of current open-source MLLMs comprises two major modules: an image-language (cross-modal) projection network and a large language model. It is desirable to understand the roles of these two modules in modeling domain-specific visual attributes to inform the design of future models and streamline the interpretability efforts on the current models. To this end, via experiments on 4 datasets and under 2 fine-tuning settings, we find that as the MLLM is fine-tuned, it indeed gains domain-specific visual capabilities, but the updates do not lead to the projection extracting relevant domain-specific visual attributes. Our results indicate that the domain-specific visual attributes are modeled by the LLM, even when only the projection is fine-tuned. Through this study, we offer a potential reinterpretation of the role of cross-modal projections in MLLM architectures.

## introduction the recent wave of advancements in large language models (llms) has equipped them with the ability to "see" images, leading to multimodal large language models (mllms) like llava @xcite , gpt-4(v) @xcite , and gemini @xcite . mllms unlock the potential to converse with visual data using language. however, existing mllms are trained and evaluated for general-purpose multimodal tasks like question-answering on natural images 1 @xcite , which limits their applicability in 1 we use 'natural images' or 'internet images' to refer to common images encountered on social media platforms and the web and contrast them with domain-specific images. ## discussion and implications existing literature on interpretability of neural networks has discussed the notion of "multimodal neurons" -neurons that trigger in response to particular concepts spanning disparate modalities @xcite @xcite . for instance, @xcite demonstrate that in the clip model, a single neuron could respond to the photographs, drawings, or images that relate to, let's say 'spiderman,' even though the input image may differ in terms of low-level visual attributes like color, edges, and corners. similarly, @xcite show that a specific neurons within a frozen text-only transformer are responsible for detecting visual concepts, let's say like 'horses,' in the input images that are projected to align with the text-only transformer. our study adds to this literature by showing that even the acquired abilities to detect visual attributes in an mllm are reliant on the llm parameters. notably, when the llm parameters are frozen, the cross-modal projection layer adapts to facilitate detection of visual attibutes in the llm without extracting domain-specific attributes. in other words, when the llm is frozen and the projection is finetuned, the projection parameters are updated to leverage the pre-existing domain-specific knowledge in the llm parameters. in the future, we aim to interpret the layer-& neuron-level contributions in llms towards acquired multimodal reasoning. 2023 )). future work could extend the current study to other projection schemes and models. beyond the adopted strategy of estimating the post-projection richness of image representations using an independent classifier, future work could also probe the mllm using concept bottleneck methods @xcite , or analyze mutual information between representations @xcite . finally, while outside the scope of the current work, a holistic evaluation of the mllm should focus on domain-specific capabilities as well as the general purpose capabilities. broader social impact: the authors do not foresee any negative social impacts of this specific work. however, we acknowledge that existing llms and mllms demonstrate different forms of biases @xcite that could be inherited in domain-specific variants. ## limitations and broader perspective in line with the ongoing effort towards mitigating social biases in deep neural networks, future efforts that aim to interface modality-specific reasoning with llms, should consider the additional biases that llms may introduce on top of the modalityspecific networks.