Knowledge Neurons in Pretrained Transformers (2022)

Abstract: Large-scale pretrained language models are surprisingly good at recalling factual knowledge presented in the training corpus. In this paper, we present preliminary studies on how factual knowledge is stored in pretrained Transformers by introducing the concept of knowledge neurons. Specifically, we examine the fill-in-the-blank cloze task for BERT. Given a relational fact, we propose a knowledge attribution method to identify the neurons that express the fact. We find that the activation of such knowledge neurons is positively correlated to the expression of their corresponding facts. In our case studies, we attempt to leverage knowledge neurons to edit (such as update, and erase) specific factual knowledge without fine-tuning. Our results shed light on understanding the storage of knowledge within pretrained Transformers.

## introduction large-scale pretrained transformers @xcite @xcite @xcite are usually learned with a language modeling objective on large-scale corpora, such as wikipedia, where exists oceans of factual knowledge. pretrained language models naturally play as a free-text knowledge base by predicting texts @xcite . @xcite and @xcite probe factual knowledge stored in pretrained language models by fillin-the-blank cloze queries. the evaluation shows that pretrained transformers have a strong ability to recall factual knowledge without any fine-tuning. @xcite use closed-book question answering to show that the larger a model is, the more knowledge it can store. however, most previous work focuses on evaluating the overall accuracy of text-form knowledge prediction. in this paper, we attempt to look deeper into pretrained transformers and investigate how factual knowledge is stored. as shown in figure 1 , we propose a knowledge attribution method to identify the neurons that express a relational fact, where such neurons are named knowledge neurons. specifically, we view feed-forward network (i.e., two-layer perceptron) modules in transformer as key-value memories @xcite . for the example in figure 1 , the hidden state is fed into the first linear layer and activates knowledge neurons; then, the second linear layer integrates the corresponding memory vectors. the key-value-memory nature @xcite inspires us to propose the knowledge attribution method, which identifies knowledge neurons in feed-forward networks by computing the contribution of each neuron to the knowledge prediction. extensive analysis shows that the activation of the identified knowledge neurons is positively correlated to the knowledge expression, which shows taking the activation of these neurons as weights, the second linear layer ffn (val) integrates value vectors through weighted sum. we hypothesize that knowledge neurons in the ffn module are responsible for expressing factual knowledge. the effectiveness of the proposed knowledge attribution method. first, suppressing and amplifying knowledge neurons notably affects the expression of the corresponding knowledge. second, we find that knowledge neurons of a fact tend to be activated more by corresponding knowledgeexpressing prompts. third, given the knowledge neurons of a fact, the top activating prompts retrieved from open-domain texts usually express the corresponding fact, while the bottom activating prompts do not express the correct relation. in our case studies, we try to leverage knowledge neurons to explicitly edit factual knowledge in pretrained transformers without any fine-tuning. we present two preliminary studies: updating facts, and erasing relations. after identifying the knowledge neurons, we perform a knowledge surgery for pretrained transformers by directly modifying the corresponding parameters in feed-forward networks. such surgery shows promising results, keeping a moderate influence on other knowledge. our contributions are summarized as follows: ## background: transformer transformer @xcite is one of the most popular and effective nlp architectures. a transformer encoder is stacked with l identical blocks. each transformer block mainly contains two modules: a self-attention module, and a feedforward network (abbreviated as ffn) module. let @xmath0, two modules can be formulated as follows: ## identifying knowledge neurons similar to @xcite , we view ffns in transformer as key-value memories as illustrated in figure 2 . we hypothesize that factual knowledge is stored in ffn memories and expressed by knowledge neurons. in this section, we propose a knowledge attribution method and a refining strategy to identify these knowledge neurons. ## case studies we present two preliminary studies to demonstrate the potential applications of knowledge neurons. we use the case studies as a proof of concept while leaving precise fact editing for future work. ## related work probing knowledge in pretrained models many pieces of previous work aim to measure knowledge stored in pretrained models. @xcite propose to retrieve knowledge in pretrained models (such as bert) using cloze queries. their experiments show that bert has a strong ability to recall factual knowledge without any finetuning. @xcite improve the cloze queries with mining-based and paraphrasing-based methods. @xcite propose the closedbook question answering to measure how much knowledge a pretrained model has stored in its parameters. @xcite measure and improve the consistency of pretrained models with respect to factual knowledge prediction. rather than examining only the model outputs, we provide an openthe-black-box analysis for the knowledge neurons in pretrained transformers. ## conclusion and future directions we propose an attribution method to identify knowledge neurons that express factual knowledge in pretrained transformers. we find that suppressing or amplifying the activation of knowledge neurons can accordingly affect the strength of knowledge expression. moreover, quantitative and qualitative analysis on open-domain texts shows that knowledge neurons tend to be activated by the corresponding knowledge-expressing prompts. in addition, we present two preliminary case studies that attempt to utilize knowledge neurons to update or erase knowledge in pretrained transformers. despite the effectiveness of identifying knowledge neurons, our current studies still have limitations. first, we examine knowledge neurons based on the fill-in-the-blank cloze task, while knowledge can be expressed in a more implicit way. it is an open question whether transformer can utilize stored knowledge in a generalized way, such as for reasoning. the interactions between knowledge neurons also remain under explored. second, we focus on factual knowledge for ease of evaluation, even though our method is also applicable for other types of knowledge. third, we use the single-word blank in cloze queries for simplicity, which requires multi-word extensions @xcite . besides, an interesting future direction is to figure out how knowledge neurons work in multilingual pretrained transformers @xcite @xcite .