Attribution and Application of Multiple Neurons in Multimodal Large Language Models (2025)

Abstract: Multimodal Large Language Models (MLLMs) have demonstrated exceptional performance across various tasks. However, the internal mechanisms by which they interpret and integrate cross-modal information remain insufficiently understood. In this paper, to address the limitations of prior studies that could only identify neurons corresponding to single-token and rely on the vocabulary of LLMs, we propose a novel method to identify multimodal neurons in Transformer-based MLLMs. Then we introduce fuzzy set theory to model the complex relationship between neurons and semantic concepts and to characterize how multiple neurons collaboratively contribute to semantic concepts. Through both theoretical analysis and empirical validation, we demonstrate the effectiveness of our method and present some meaningful findings. Furthermore, by modulating neuron activation values based on the constructed fuzzy sets, we enhance performance on the Visual Question Answering (VQA) task, showing the practical value of our approach in downstream applications in MLLMs.

## introduction large language models (llms) have significantly advanced natural language processing, enabling breakthroughs in diverse natural language tasks @xcite @xcite . motivated by their success, researchers have further extended llms to multimodal domain, developing multimodal large language models (mllms) capable of integrating visual and textual modalities, achieving impressive results across multimodal understanding and generation @xcite @xcite . despite significant advances in llm interpretability research @xcite @xcite , the internal mechanisms by which mllms process and integrate * corresponding author: dong @xcite different modalities remain unclear. this opacity risks generating biased or hallucinatory content and hinders error traceability, undermining trust and escalating safety risks in critical applications, such as medical diagnostics @xcite or autonomous driving systems @xcite . hence, improving the explainability of mllms is essential for both advancing their development and ensuring the trustworthiness of ai. recent multimodal studies @xcite have focused on identifying neurons that are respond to both textual and visual inputs within transformer architectures, known as multimodal neurons. these neurons have learned visual features from images during training and are capable of influencing text generation. among existing methods, @xcite propose a gradient-based neuron attribution method to identify multimodal neurons within feedforward networks (ffns). @xcite define a contribution score based on activation outputs from ffns, enabling gradient-free identification of multimodal neurons in mllms. however, both methods rely on the next-token prediction for neuron attribution, restricting neuron identification to single-token concepts and limiting generalizability due to dependence on the vocabulary of llms. besides, previous methods only focus on the role of individual neurons, neglecting the complex mechanisms by which multiple neurons collaboratively encode different semantic concepts. moreover, the potential for broader application in real-world tasks remains largely unexplored. to address these challenges, we first propose a visual-semantic perturbation-based neuron attribution method. in this method, both original visual inputs and their counterparts with target semantic concepts removed are input into the model for caption generation. by comparing the neuron activation differences accumulated over all generated tokens between the two types of inputs, we can identify the neurons that are highly responsive to the semantic concepts. thus, our method can expand the scope of identification to include multitoken semantic concepts and avoid the influence of the vocabulary of llms. furthermore, inspired by prior work @xcite that investigate the function of neuron groups in convolutional neural networks (cnns), we believe that multimodal neurons in mllms also exhibit similar associations with semantic concepts-an individual neuron may contribute to multiple semantic concepts with varying degrees of activation, while the representation of semantic concepts often emerges from the coordinated activation of multiple neurons. to model this neuron-semantic relationship, we introduce fuzzy set theory @xcite . specifically, we construct a fuzzy set for each semantic concept, in which each neuron is assigned a membership degree according to its degree of affiliation with the set. this modeling approach systematically characterizes how multimodal neurons contribute to different concepts, offering a novel perspective on neuron activation mechanisms within mllms. based on the constructed fuzzy sets of multimodal neurons for semantic concepts, we further demonstrate their application in the visual question answering (vqa) task. by modulating the activation values of neurons in the fuzzy sets corresponding to the concepts relevant to each image-question pair, we strengthen the model's capacity to identify and interpret specific semantics within images, thereby improving vqa performance. further experimental results demonstrate that our method can be extended to identify text neurons representing interrogative words, which can then be applied in the vqa task, proving the generalizability of our approach. in summary, our contributions are three-fold: (1) we propose a new method for identifying multimodal neurons in mllms. (2) we introduce fuzzy set theory to systematically characterize the contributions of neurons across different semantic concepts. (3) we explore the application of multimodal neurons in the vqa task. ## method in this section, we first propose a perturbation approach to measure activation differences and identify neurons responsive to semantic concepts. we further leverage membership degrees from fuzzy set theory to quantify the semantic contribution of multimodal neurons. the overview of our method is illustrated in figure 1 . ## application: vqa to evaluate the practical benefits of our constructed fuzzy sets of multimodal neurons, we apply them to the vqa task. in typical vqa pipelines, the model takes an image and a question as input and outputs an answer. to improve model's semantic grounding during inference, we modulate the activation values of multimodal neurons corresponding to image-related semantics. to further capture question intent and facilitate cross-modal reasoning, we extend our method to the textual modality by identifying and modulating neurons associated with interrogative words (5w2h: what, why, who, which, where, how, how many). specifically, we construct paired inputs: one with the original image-question pair, and the other with the same image and a declarative sentence formed by replacing the interrogative word with the answer. we then record neuron activations over the textual tokens for both inputs and compute the differences between their mean activations. for each question type, we use 500 paired inputs and construct the corresponding neuron fuzzy sets, following the method described in section 3. other experimental settings follow those used for multimodal semantic neurons. on average, the seven fuzzy sets of interrogative neurons contain 1,022 neurons (0.29%) in llava-next and 1,588 neurons (0.48%) in janus-pro. the layer-wise distribution of these neurons is presented in appendix b.4. we conduct experiments on the coco-qa @xcite and vqa v2.0 @xcite datasets, both of which use images from mscoco. for each input pair, we select the multimodal neuron sets corresponding to the image semantics x and the interrogative neuron sets for the question type q. we then amplify their activations based on fuzzy membership degrees. let a(l, i) denote the original activation of neuron u(l, i), and let µ x (l, i), µ q (l, i) ∈ [0, 1] represent its membership degrees in the corresponding fuzzy sets. the amplified activation is computed as: ## conclusion in this paper, we propose a novel method to identify multimodal neurons in mllms and introduce fuzzy set theory to model their collaborative contributions to semantic concepts. we further successfully apply our method to the vqa task. extensive quantitative and qualitative experiments demonstrate the explanatory powers and practical value of our fuzzy set-based multimodal neurons, advancing interpretability research in mllms.