Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers (2024)

Abstract: Understanding the internal mechanisms by which multi-modal large language models (LLMs) interpret different modalities and integrate cross-modal representations is becoming increasingly critical for continuous improvements in both academia and industry. In this paper, we propose a novel method to identify key neurons for interpretability — how multi-modal LLMs bridge visual and textual concepts for captioning. Our method improves conventional works upon efficiency and applied range by removing needs of costly gradient computation. Based on those identified neurons, we further design a multi-modal knowledge editing method, beneficial to mitigate sensitive words or hallucination. For rationale of our design, we provide theoretical assumption. For empirical evaluation, we have conducted extensive quantitative and qualitative experiments. The results not only validate the effectiveness of our methods, but also offer insightful findings that highlight three key properties of multi-modal neurons: sensitivity, specificity and causal-effect, to shed light for future research.

## introduction recently, large language models (llms) have received much attention and become foundation models in many natural language processing applications @xcite @xcite . following the success, researchers in the area of computer vision have extended the input modality to both text and image, namely multi-modal llms, showing remarkable performance in various visual understanding tasks @xcite @xcite . however, the underlying mechanism of how multi-modal llms interpret different modalities of features beyond these tasks remains unclear. it hinders in-depth investigation and poses risks in model applications, such as producing misleading outputs without insight into decisions or propagating biases through automatic captions. there are two main types of methods on llms' interpretability. the first group targets probing various abilities through well-designed external tasks @xcite @xcite . another line of works, instead, attempt to reveal the internal states, by finding the processes of how llms understand and interpret textual inputs to form a response @xcite @xcite . among them, an interesting finding shows that llms' ability to understand textual information mainly comes from feed-forward networks (ffns). furthermore, @xcite identify key neurons from ffns, namely multi-modal neurons. these neurons play an important role in understanding images and generating textual descriptions. however, the identification process is inefficient and limited in applied range, due to costly gradient computation. besides, their theoretical rationale, empirical characteristics, and potential application remains under-exploration. to address the issues, we propose a novel method for multi-modal neurons identification. we define a contribution score based on the activation output in ffns, which is consistent with the probability distribution when predicting. as our method do not need access to the model gradients, we improve efficiency while ensuring effectiveness. based on the identified neurons, we further propose a multi-modal knowledge editing method as a potential application. we achieve the goal of editing a specific concept to another designative concept (e.g., in figure 1 (i), 'dog' is edited to 'mouse'), by changing the probability distribution of outputs. without additionally training the entire model or requiring access to model gradients, our proposed method facilitates a timely and resource-efficient llava: a large, fluffy white dog laying on a wooden deck. llava: a cat, likely a photograph, shows a part of a and a part of a, possibly a cat, in a close-up shot. llava: a large mouse with a white face and brown body is laying on a wooden deck. editing of a small portion of the model parameters. for empirical characteristics, we have designed metrics and conducted extensive experiments, which highlight three critical properties of multimodal neurons: (1) sensitivity ( §3.3). multi-modal neurons are sensitive to particular concepts. once they are activated by some regions of the input image, they are responsible for generating related textual concepts. more importantly, these neurons are invariant in visual translation to different inputs. (2) specificity ( §3.4). although different multi-modal neurons can be activated by the same concepts, they are selectively active for these concepts and hardly respond to others. (3) causal-effect ( §3.5). multimodal neurons and the associated concepts have causal-effect and are significantly susceptible. we perturb and edit the identified multi-modal neurons, which leads to significant changes in outputs. our contributions can be summarized as follows: ## method we first define neurons in the llm ( §2.1), and then define a contribution score for neurons identification ( §2.2). furthermore, we propose a multi-modal knowledge editing method based on identified neu-rons ( §2.3) and introduce several evaluation metrics to evaluate multi-modal neurons ( §2.4). ## conclusion we propose a new method to identify multi-modal neurons in transformer-based multi-modal llms. we also introduce a knowledge editing approach based on the identified neurons, which achieves a knowledge editing from a specific token to another designative token. we highlight three critical properties of multi-modal neurons by four welldesigned quantitative evaluation metrics through extensive experiments. both quantitative and qualitative experiments validate the explanatory powers of our multi-modal neurons. this work provides illuminating perspectives on multi-modal llms and stimulates additional explanatory artificial intelligence studies emphasizing model interpretability.