MMN euron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model (2024)

Abstract: Projecting visual features into word embedding space has become a significant fusion strategy adopted by Multimodal Large Language Models (MLLMs). However, its internal mechanisms have yet to be explored. Inspired by multilingual research, we identify domain-specific neurons in multimodal large language models. Specifically, we investigate the distribution of domain-specific neurons and the mechanism of how MLLMs process features from diverse domains. Furthermore, we propose a three-stage framework for language model modules in MLLMs when handling projected image features, and verify this hypothesis using logit lens. Extensive experiments indicate that while current MLLMs exhibit Visual Question Answering (VQA) capability, they may not fully utilize domain-specific information. Manipulating domain-specific neurons properly will result in a 10% change of accuracy at most, shedding light on the development of cross-domain, all-encompassing MLLMs in the future. The source code is available at https://anonymous.4open.science/r/MMNeuron.

## introduction neuron analysis, which interprets activation of neurons as the recall of learned knowledge in deep neural networks, has been widely adopted by researchers to understand the inner workings of models @xcite . prior studies have confirmed that certain neurons within deep neural networks play important roles in learning particular concepts @xcite @xcite , preserving factual knowledge @xcite @xcite as well as solving specific tasks @xcite . beyond enhancing model interpretability, current practical applications of neuron analysis include model distillation @xcite , knowledge editing @xcite , and * corresponding author controllable generation @xcite . central to such endeavors is the identification of neurons responsible for target scenarios. as illustrated in figure 1 @xcite , recent studies have focused on interpreting the multilingual capabilities of pre-trained large language models (llms) under the view of language-specific neurons, which are neurons uniquely responsible for particular languages. for instance, @xcite identified such neurons in pre-trained decoder-based language models, demonstrating that tampering with a few language-specific neurons significantly alters the occurrence probability of target language in text generation. similarly, @xcite detected language-specific neurons by measuring the significance of neurons when processing multilingual inputs and proposed a workflow of llms handling multilingual tasks. moreover, @xcite used language activation probability entropy (lape) to identify language-specific neurons, demonstrating that activating or deactivating certain neurons can change the language of the model's output. on the other hand, it has also been confirmed that neurons in text-only transformers can understand visual features extracted by a vision encoder @xcite . these findings have prompted an interesting question: do similar mechanisms exist in multimodal large language models (mllms) during the processing of features from different visual domains? as shown in figure 1 (b), we aim to apply the mechanism similar to multilingual neuron analysis @xcite to current representative open-source mllms, including llava-next @xcite and instructblip @xcite . the aforementioned models extract image features via a pre-trained vision encoder and project these features into the word embedding space. these post-projection visual features are concatenated with language features and fed into the model's llm module to generate text outputs. specifically, we investigate the activation patterns of neurons in mllms' feed-forward network (ffn) layers across corpora from five distinct domains, identifying less than 1% as domain-specific neurons. the datasets we utilized include lin-goqa @xcite , rs-vqa (hr) @xcite , pmc-vqa @xcite , docvqa @xcite and vqav2 @xcite , covering domains such as auto driving, remote sensing, medicine, document, and common scenes. figure 2 highlights the clustering and separation of image features across the domains. image examples of these domains can also be found in appendix b. based on our experiment results, we argue that differences exist among these visual domains and that the vision encoder and llm modules in mllms exhibit distinct pat-terns for these domains. furthermore, we propose a three-stage mechanism based on the distribution of domain-specific neurons among mllm's llm layers, where post-projection visual features are processed by llm. to validate our hypothesis, we employ logit lens (nostalgebraist, 2020) to decode the hidden states of llm's intermediate layers to visualize the feature transformation within transformer models @xcite . our main contributions are as follows: to the best of our knowledge, we are the first to investigate domain-specific neurons in the multimodal field, although there are already insightful discussions on visual representations in mllms @xcite . our findings can reveal the neuron-level similarity and distinction among these domains, offering insights to understand and enhance the cross-domain potential of current mllms. ## method in this section, we will introduce how to investigate the domain-specific neurons in mllms through domain activation probability entropy (dape). in section 3.1, we define the activation of neurons in vision-language models. in section 3.2, we introduce dape to reflect the specificity of neurons. furthermore, to verify how post-projection embeddings are processed within the language model, we decode the hidden states layer by layer with logit lens, as discussed in section 3.3. ## experiment in this section, we present empirical evaluation to elucidate the impact of domain-specific neurons, showing the potential mechanism of how mllms interpret image and language instructions. ## conclusion to explore the neuron-level domain-specific interpretation in current mllms, we propose mmneuron framework inspired by multilingual research. in particular, we first calculate the activation probabilities of neurons in llava-next and instruct-blip across five domains, identifying those with low domain dape scores as domain-specific neurons. by analyzing the distribution of domainspecific neurons and their influence on mllms, we find that the language model modules of mllms fail to fully utilize domain-specific information in vqa tasks. we further propose a three-stage framework that the language model module employs to handle projected visual features and corroborate it indirectly with logit lens. we envision that our work will shed light on the interpretability of current mllms, aiding the development of crossdomain, all-encompassing mllms in the future.