BERT nesia: Investigating the capture and forgetting of knowledge in BERT (2020)

Abstract: Probing complex language models has recently revealed several insights into linguistic and semantic patterns found in the learned representations. In this paper, we probe BERT specifically to understand and measure the relational knowledge it captures. We utilize knowledge base completion tasks to probe every layer of pre-trained as well as fine-tuned BERT (ranking, question answering, NER). Our findings show that knowledge is not just contained in BERT’s final layers. Intermediate layers contribute a significant amount (17-60%) to the total knowledge found. Probing intermediate layers also reveals how different types of knowledge emerge at varying rates. When BERT is fine-tuned, relational knowledge is forgotten but the extent of forgetting is impacted by the fine-tuning objective but not the size of the dataset. We found that ranking models forget the least and retain more knowledge in their final layer.

## introduction large pre-trained language models like bert @xcite have heralded an imagenet moment for nlp foot_1 with not only significant improvements made to traditional tasks such as question answering and machine translation but also in the new areas such as knowledge base completion. bert has over 100 million parameters and essentially trades off transparency and interpretability for performance. loosely speaking, probing is a commonly used technique to better understand the inner workings of bert and other complex language models @xcite . probing, in general, is a procedure by which one tests for a specific pattern -like local syntax, long-range semantics or even compositional reasoningby constructing inputs whose expected output would not be possible to predict without the ability to detect that pattern. while a large body of work exists on probing bert for linguistic patterns and semantics, there is limited work on probing these models for the factual and relational knowledge they store. recently, @xcite probed bert and other language models for relational knowledge (e.g., trump is the president of the usa) in order to determine the potential of using language models as automatic knowledge bases. their approach converted queries in the knowledge base (kb) completion task of predicting arguments or relations from a kb triple into a natural language cloze task, e.g., @xcite is the president of the usa. this is done to make the query compatible with the pre-training masked language modeling (mlm) objective. they consequently showed that a reasonable amount of knowledge is captured in bert by considering multiple relation probes. however, there are some natural questions that arise from these promising investigations: is there more knowledge in bert than what is reported? what happens to relational knowledge when bert is fine-tuned for other tasks? is knowledge gained and lost through the layers? our contribution. in this paper, we study the emergence of knowledge through the layers in bert by devising a procedure to estimate knowledge contained in every layer and not just the last (as done by @xcite ). while this type of layer-by-layer probing has been conducted for syntactic, grammatical, and semantic patterns; knowledge probing has only been conducted on final layer representations. observing only the final layer (as we will show in our experiments) (i) underestimates the amount of knowledge and (ii) does not reveal how knowledge emerges. furthermore, we explore how knowledge is impacted when fine-tuning on knowledge-intensive tasks such as question answering and ranking. we list the key research questions we investigated and key findings corresponding to them: rq i: do intermediary layers capture knowledge not present in the last layer? (section 4.1) we find that a substantial amount of knowledge is stored in the intermediate layers (≈ 24% on average) rq ii: does all knowledge emerge at the same rate? do certain types of relational knowledge emerge more rapidly? (section 4.2) we find that not all relational knowledge is captured gradually through the layers with 15% of relationship types essentially doubling in the last layer and 7% of relationship types being maximally captured in an intermediate layer. rq iii: what is the impact of fine-tuning data on knowledge capture? (section 4.3) we find that the dataset size does not play a major role when the training objective is fixed as mlm. fine-tuning on a larger dataset does not lead to less forgetting. rq iv: what is the impact of the fine tuning objective on knowledge capture ? (section 4.4) fine tuning always causes forgetting. when the size of the dataset is fixed and training objective varies, the ranking model (rank-msmarco in our experiments) forgets less than the qa model. ## related work in this section, we survey previous work on probing language models (lms) with a particular focus on contextual embeddings learned by bert. probes have been designed for both static and contextualized word representations. static embeddings refer to non-contextual embeddings such as glove @xcite . for the static case, the reader can refer to this survey by @xcite . now we detail probing tasks for contextualized embeddings from language models. ## results in contrast to existing work, we want to analyze relation knowledge across layers to measure the total knowledge contained in bert and observe the evolution of relational knowledge through the layers. ## discussion and conclusion in this paper, we introduce a framework to probe all layers of bert for knowledge. we experimented on a variety of probes and fine-tuning tasks and found that bert contains more knowledge than was reported earlier. our experiments shed light on the hidden knowledge stored in bert and also some important implications to model building. since intermediate layers contain knowledge that is forgotten by the final layers to make way for task-specific knowledge, our probing procedure can more accurately characterize the knowledge stored. we show that factual knowledge, like syntactic and semantic patterns, is also forgotten at the last layers due to fine-tuning. however, the last layer can also make way for more domain specific knowledge when the fine tuning objective is the same as the pretraining objective (mlm) as observed in squad. interestingly, forgetting is not mitigated by larger datasets which potentially con-tain more factual knowledge (mlm-@xmath0. instead, we find that knowledge-intensive tasks like ranking do mitigate forgetting compared to span prediction. although the fine-tuned models always contain less knowledge, with significant (and expected) forgetting in the last layers, rank-msmarco remembers relatively more relationship types than bert (2% vs 7% forgotten) in its last layer ( our results have direct implications on the use of bert as a knowledge base. by effectively choosing layers to query and adopting early exiting strategies knowldge base completion can be improved. the performance of rank-msmarco also warrants further investigation into ranking models with different training objectives -pointwise (regression) vs pairwise vs listwise. more knowledge-intensive qa models like answer generation models may also show a similar trend as ranking tasks but require investigation. we also believe that our framework is well suited to studying variants of bert architecture and pretraining methods.