The Effectiveness of Masked Language Modeling and Adapters for Factual Knowledge Injection (2022)

Abstract: This paper studies the problem of injecting factual knowledge into large pre-trained language models. We train adapter modules on parts of the ConceptNet knowledge graph using the masked language modeling objective and evaluate the success of the method by a series of probing experiments on the LAMA probe. Mean P@K curves for different configurations indicate that the technique is effective, increasing the performance on sub-sets of the LAMA probe for large values of k by adding as little as 2.1% additional parameters to the original models.

## introduction large pre-trained language models (plms) are difficult to interpret due to their complexity and large parameter size. this can partly be explained by the nature of popular training regimens, such as the masked language modelling objective, which encodes distributional knowledge. such regimens have proven effective for a range of downstream nlp tasks, but they also make it difficult to determine and validate the origin of whatever knowledge the models end up with. consequently, there have been multiple efforts to integrate structured information into plms @xcite @xcite . this has not only been motivated by the promise of better interpretability, but also the observation that there exist scenarios where we would want to stress information that might not be so easily encoded by modelling long range dependencies between fragments of text. this includes knowledge intensive tasks where employing the correct factual knowledge is crucial, for example within the medical domain @xcite and question answering @xcite . at the same time, there exist multiple structured sources that attempt to capture factual knowledge. these sources range from domain specific knowledge graphs for medical information @xcite , commonsense graphs like yago or conceptnet @xcite , to lexico-semantic networks like wordnet @xcite . in this paper, we attempt to inject the structured information found in the conceptnet knowledge graph @xcite into pre-trained language models. the injection is done by training relatively small neural networks, known as adapter modules @xcite , on subject-predicate-object triples. as in @xcite , we extract the triples using a random walk procedure and then translate them into natural language so that we can use masked language modeling as the training objective. the resulting adapters are injected into all layers of two popular pre-trained language models: bert base @xcite and roberta base @xcite . our code and data is made publicly available foot_0 . for the injection to be deemed effective, we argue that the adapter-injected models must be able to use the knowledge gained from the adapter training together with what the models learned during their initial pre-training. in order to quantitatively assess this, we evaluate our models in a zero-shot setting on the conceptnet subset of the lama probe @xcite . as conceptnet is the source for both our training corpus and the lama probe, we can better measure how much of the factual knowledge seen during adapter training the models can be expected to recall. ## related work combining structured information with language models is a standing problem in nlp. one approach to overcome this has been to combine knowledge graphs with plms, augmenting the distributional knowledge encoded in the models with the structured information found in the graphs @xcite , 2021; @xcite . within this approach, we find several uses of adapters. first introduced for nlp by @xcite , and popularized by the adapterhub framework @xcite , adapters are small neural networks injected into larger, often pre-trained models. during training the original model weights are kept static, and only the set of newly introduced weights from the adapter are adjusted. figure 1 illustrates the architecture proposed by @xcite and how it is injected into a transformer layer. the methodology in this paper is inspired by @xcite , who inject commonsense information and world knowledge into bert by using such adapter modules. as in our work, the adapters train with the masked language modeling objective over subject-predicate-object triples from the conceptnet graph, but they are evaluated on the glue benchmark @xcite . although the result are inconclusive for most of the tasks in glue, the injected models perform better than their base model counterparts on the world knowledge and commonsense categories of the diagnostic set. a similar approach is taken by @xcite . their k-dapter model has one adapter for factual knowledge, trained on aligned text triplets from wikipedia and wikidata, and one for linguistic knowledge, obtained via dependency parsing. results on knowledge-driven tasks, including relation classification, entity typing, and question answering, show that this setup improves performance, and furthermore, that k-adapter captures more versatile knowledge than roberta. in a more domain specific context, meng et al. ## experiments following lauscher et al. ( 2020 ), we use the same configuration for our adapter modules as in @xcite . we set the size of the adapter modules to 64, which implies a reduction factor of 12 from the original transformer layer size of 768 in bert base . this increases the total amount of parameters by 2.1%. we use gelu @xcite as the activation function inside the adapters, and the adam optimizer from (kingma and ba, 2017). we set the learning rate to 1e-4 with 10.000 warm-up steps and weight decay factor of 0.01. we allow the adapter to train for 100.000 optimization steps while freezing all the original transformer weights. the adapters are implemented using the adapter-transformers library @xcite . throughout the remainder of this paper, the resulting configuration is referred to as cn houlsby 100k in figures and as the houlsby configuration in text. the adapters train on the same subset of conceptnet as in @xcite . as this study was named retrograph, we refer to this particular set of predicate types as the retrograph predicate set. the predicates in this set are: antonymof, synonymof, isa and mannerof. subject-predicate-object triplets with one of these predicates in their middle position are extracted through a random traversal procedure foot_1 and then subsequently chained so that we get blocks of text in natural language on the following format: possible is a synonym of possibility. possibility is a concept. concept is a synonym of conception. conception is a synonym of fertilization. fertilization is a enrichment. enrichment is a gift. the corpus is processed using masked language modeling (mlm), parsed line for line with a mlm probability of 0.15, as in the original bert paper @xcite . we also experiment by training on the corpus by a maximum sequence length instead of line by line training. however, this did not affect the performance of the models in any significant way. ## results figure 2 shows the mean p@k curves for two language models, with and without an adapter. part (a) of the figure shows the result over all the predicate types present in the conceptnet split of lama (@xmath0. the injection of the adapter module decreases the performance of both bert and roberta for all values of k. however, the corpus with the retrograph predicate set that adapters trained on only includes one of these types. hence, there is little similarity between the two sets, and the reproduction of factual knowledge cannot be expected here. this also indicate that training on one set of predicate types does not improve the reproduction of facts on others. part (b) of figure 2, on the other hand, shows the same models and adapters, but with the probe restricted only to the isa predicate type -which is then present both in the training corpus and in the probe. in the corpus from @xcite , triples with this predicate type make up 23% of the total corpus (@xmath1. since both resources are extracted from con-ceptnet, we check the overlap between the masked tokens in the object position in lama and the object position in the triplets in the training set for the adapters. the actual percentage will depend on the random walk procedure, but for the sets used in figure 2 there is a 5.7% overlap between concepts. that is, approximately five percent of the concepts from lama that the models are expected to predict are also in the training corpus in some form, either with the same predicate type as in the probe, isa, or one of the others in the retrograph set. despite this, the injected models perform consistently better. as this performance gain is achieved by adding only 2.1% additional parameters to the original model, and without adjusting the original weights at all, we interpret the results as an indication that this method of knowledge injection is effective. ## changing the predicate set in order to further probe the effectiveness of the proposed method, we introduce a new corpus (@xmath2, but over a new set of predicate types, namely the same set of predicate types found in the conceptnet split of lama. by intuition, if the method is effective, the injected models should score higher on average over all these predicate types than their non-injected counterparts. a list of these predicate types can be found in appendix a. figure 3 compares the result of the injected models trained over our predicate set with that of the retrograph set and a plain bert model for different values of k. as can be seen from the p@k curves, models trained over our predicate set improve the performance on the full conceptnet split of the lama (@xmath3.39% for bert at large values of k. for @xmath4, where the model must guess the correct masked object "at first try", we see little difference. compared to the retrograph set, which has fewer predicate types, the difference in performance indicate that predicate type specificity is important (e.g subgraph quality). for this comparison, the overlap between the training corpus for the adapters and the full conceptnet split of lama is 36% on the object level, meaning that roughly one third of the concepts were seen during training in some form. this provides some evidence for the success of the knowledge injection. models are able to reproduce factual knowledge when queried over the lama probe, even though the phrasing of the questions in lama is different than the strict tripletstyle of the training corpus. ## conclusion and future work combining structured information and large pretrained language models is a standing problem in nlp research. in this work, we show that training adapter modules on triplets extracted from con-ceptnet using masked language modeling can help language models reproduce factual knowledge. experiments on the conceptnet split of the lama probe show that our adapter-injected models perform better in a zero-shot setting than non-injected models, having seen only a third of the relevant factual knowledge during pre-training in some form, encoded into only 2.1% of the total parameters of the total model. future work should investigate how this type of knowledge injection can augment language models on other types of tasks, such as language generation, multiple choice questions or natural language inference, which would require more fine-grained annotations of downstream tasks targeted at some form of knowledge. ruize wang, duyu tang, nan duan, zhongyu wei, xuanjing huang, jianshu ji, guihong cao, daxin jiang, and ming zhou. 2021. k-adapter: infusing knowledge into pre-trained models with adapters. in findings of the association for computational linguistics: acl-ijcnlp 2021, pages 1405-1418, online. association for computational linguistics. michihiro yasunaga, hongyu ren, antoine bosselut, percy liang, and jure leskovec. 2021. qa-gnn: reasoning with language models and knowledge graphs for question answering. in proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 535-546, online. association for computational linguistics. taolin zhang, zerui cai, chengyu wang, minghui qiu, bite yang, and xiaofeng he. 2021. smedbert: a knowledge-enhanced pre-trained language model with structured semantics for medical text mining. in proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pages 5882-5893, online. association for computational linguistics. xikun zhang, antoine bosselut, michihiro yasunaga, hongyu ren, percy liang, christopher d manning, and jure leskovec. 2022. greaselm: graph rea-soning enhanced language models. in international conference on learning representations.