What Has Been Enhanced in my Knowledge-Enhanced Language Model? (2022)

Abstract: A number of knowledge integration (KI) methods have recently been proposed to incorporate external knowledge into pretrained language models (LMs). Even though knowledge-enhanced LMs (KELMs) outperform base LMs on knowledge-intensive tasks, the inner-workings of these KI methods are not well-understood. For instance, it is unclear which knowledge is effectively integrated into KELMs and which is not; and if such integration led to catastrophic forgetting of already learned knowledge. We show that existing model interpretation methods such as linear probes and prompts have some key limitations in answering these questions. Then, we revisit KI from an information-theoretic view and propose a new theoretically sound probe model called Graph Convolution Simulator (GCS) for KI interpretation. GCS is eventually quite simple – it uses graph attention on the corresponding knowledge graph for interpretation.We conduct various experiments to verify that GCS provides reasonable interpretation results for two well-known KELMs: ERNIE and K-Adapter. Our experiments reveal that only little knowledge is successfully integrated in these models, and simply increasing the size of the KI corpus may not lead to better KELMs.

## introduction pretrained language models (lms) have become the backbone of nlp. recent work has shown that linguistic knowledge is captured quite well in lms @xcite . however, lms are much worse at capturing factual knowledge about the world @xcite . this has led to the development of a number of knowledge integration (ki) methods which integrate external knowledge from knowledge graphs (kgs) into lms, leading to knowledge-enhanced language models such as ernie @xcite and k-adapters @xcite . even though enhanced lms perform better on knowledge-intensive tasks, there is little understanding of where this improvement comes from. which factual knowledge is successfully integrated into the lm, and which kind of knowledge is not, is not well-understood. as new knowledge is integrated in lms, old knowledge could be catastrophically forgotten @xcite . ki could also lead to a situation called catastrophic remembering @xcite , where the old knowledge could prevent the integration of new knowledge. our understanding about these issues is also limited. an intuitive way to understand the ki process could be to probe for factual knowledge in the base lm and the enhanced lm respectively, and compare their results for interpretation. however, we find that because of the high variance of classifierbased probes for kg prediction @xcite and the significant human effort required to verbalize knowledge graph facts into templates @xcite , we cannot easily extend existing probe methods to interpret knowledge-integration in lms ( §3). in this paper, we revisit the ki process and formulate it with an information-theoretic view ( §4). we measure the factual knowledge encoded in the lm by the mutual information (mi) between the model representations and the kg @xcite . now, the integration and forgetting of factual knowledge can be measured using the difference in mi between the kg and the base lm and the mi between the kg and the enhanced lm. based on this idea, we theoretically derive a transformation composed of graph convolutions to simulate and interpret this change in mi, leading to our proposed probe, graph convolution simulator (gcs) ( §5). the interpretation mechanism in gcs is quite different from existing probe methods as figure 1 : adaptations of two existing probe methods and gcs to interpret knowledge integration in language models. probes from left to right are: linear probe, prompt-based probe, and gcs. we can find that only gcs does not have the compare operation, which can avoid introducing extra noise of interpretation results. in our experiments ( §6), we verify that gcs can provide reasonable ki interpretation results. we show that: (a) results of gcs have significantly smaller variance compared to the linear probe, (b) dropping knowledge identified as non-integrated by gcs does not affect enhanced lms' performance, and (c) enhanced lms perform better on samples that include knowledge identified as integrated by gcs. in particular, we use gcs to understand two well-known knowledge-enhanced lms: ernie @xcite and k-adapter @xcite . our findings are listed as follows. knowledge graphs. we assume that factual knowledge for integration can be formulated as a kg @xmath0, e), where nodes v @xmath1, and edges in e represent relations between them. let n v i denote the set of neighbors of node v i , and t i denote the entity label corresponding to the node v i . further, let x @xmath2, and @xmath3. in this paper, we only consider nodes and edges in the kg and ignore other side information such as relation weights, directions and labels foot_2 . ## unsuitability of existing probes for ki classifier probes @xcite and prompt-based probes (e.g. the lama probe) @xcite are typically used to test for various kinds of knowledge in lms. classifier probes train simple (usually linear) classifiers to predict the linguistic property of interest and the probe accuracy is used for interpretation @xcite . however, simple classifiers are not powerful enough to make reliable predictions about large @xcite . 4 moreover, linear probes are also unable to provide reasonable insights for lms as they suffer from high variance. if we use them to probe two lms and compare their results for ki interpretation, variance of interpretation would further increase. we provide empirical evidence later in §6.1.1. prompting is another popular way to understand what factual knowledge lms know. prompts are designed to let lms solve text infilling problems, and the prompt output is then used for interpretation @xcite . however, people have to manually design templates for factual knowledge to be probed foot_4 , and the quality of templates is vital in the overall prompt accuracy @xcite . as ki methods often use large kgs for integration, it would be infeasible to write a large number of templates for all triples in kgs. to address these issues, we introduce the gcs model and its theoretical motivation behind. ## knowledge integration understanding first, we revisit ki in lms by formulating it in an information-theoretic way. then, we construct transformations to simulate and interpret the ki process. notations are summarized in appendix a. ## experiments we begin by reviewing ernie @xcite and k-adapter @xcite . knowledge is integrated in entity-wise manner in ernie, and triple-wise manner in k-adapter. ernie. ernie integrates knowledge into bert @xcite using a wikipedia corpus and wikidata kg. as no alignment is provided between sentences in the wikipedia corpus and entities in the wikipedia kg, ernie uses tagme @xcite to extract entity mentions in sentences and aligns them with corresponding entities in kgs. a new objective is designed for ki in addition to the standard mlm and nsp objectives: alignments in the input text are randomly masked, and the model is asked to select aligned entities from kgs. when ernie finds the aligned entity, its kg embedding obtained from @xcite is integrated into the hidden representations of bert. ## conclusion in this paper, through a series of theoretical results, we derived an information-theoretic probe that uses attention over knowledge graphs to interpret the knowledge integration process in lms. in our experiments, we verified our probe model and used it to understand what knowledge has been integrated in two existing knowledge-enhanced language models, leading to some new findings about these models. we hope that our probe model would aid in better understanding and informed design of knowledge integration approaches for lms. we have published the code and the demo to help users easily implement gcs for their own knowledgeenhanced lm interpretation.