LM - CORE : Language Models with Contextually Relevant External Knowledge (2022)

Abstract: Large transformer-based pre-trained language models have achieved impressive performance on a variety of knowledge-intensive tasks and can capture factual knowledge in their parameters. We argue that storing large amounts of knowledge in the model parameters is sub-optimal given the ever-growing amounts of knowledge and resource requirements. We posit that a more efficient alternative is to provide explicit access to contextually relevant structured knowledge to the model and train it to use that knowledge. We present LM-CORE – a general framework to achieve this– that allows decoupling of the language model training from the external knowledge source and allows the latter to be updated without affecting the already trained model. Experimental results show that LM-CORE, having access to external knowledge, achieves significant and robust outperformance over state-of-the-art knowledge-enhanced language models on knowledge probing tasks; can effectively handle knowledge updates; and performs well on two downstream tasks. We also present a thorough error analysis highlighting the successes and failures of LM-CORE. Our code and model checkpoints are publicly available.

## introduction large pre-trained language models (plms) @xcite @xcite have achieved state-of-the-art performance on a variety of nlp tasks. much of this success can be attributed to the significant semantic and syntactic information captured in the contextual representations learned by plms. in addition to applications requiring linguistic knowledge, plms have also been useful for a variety of tasks involving factual knowledge and it has been shown that models such as bert @xcite and t5 @xcite store significant world knowledge in their parameters @xcite . plms are typically fed a large amount of unstructured text which leads to the linguistic nuiances and world knowledge being captured in the model parameters. this implicit storage of the knowledge in the form of the parameter weights not only leads to poor interpretability while analyzing model predictions but also poses constraints on the amount of knowledge that can be stored. it is not practical to pack all the ever-evolving world knowledge in the language model parameters due to the great financial and environmental costs incurred by training of the plms. further, since the plms acquire knowledge from the text corpora they are trained on, they tend to become sensitive to the contextual and linguistic variations @xcite . moreover, plms do not contain explicit grounding to real world entities, and hence, often find it difficult to recall factual knowledge @xcite . for example, the model may not be able to recall correct information and successfully complete the sentence, "the birthplace of barack obama is ", if the lm has seen this fact in a different context during training (e.g., "barack obama was born in honolulu, hawaii."). large scale structured knowledge bases (kbs) such as yago @xcite and wikidata @xcite offer a rich resource of high quality structured knowledge that can provide the plms with explicit grounding to real world entities. consequently, efforts have been made to integrate factual knowledge into plms and create entity-enhanced language models @xcite @xcite @xcite . however, author creator location publisher occupation nationality (batman, creator, bob kane) (batman, creator, bill finger) (batman, publisher, dc comics) (american university, location, washington dc) (the american language, author, h. l. mencken) batman is a superhero who appears in american comic books published by dc comics. ## lm-core: knowledge retrieval and training framework task setting and overview: consider a language model l (such as bert and roberta) and a knowledge base @xmath0, r, @xmath1, @xmath2; @xmath3. here, we consider the knowledge base k as a set of triples such that each triple t hrt represents the relationship r between entities h and t. e is the set of all the entities, and r is the set of all the relationship types present in the knowledge base. given a text input x, the proposed lm-core framework retrieves a set of triples t @xmath4. the language model is then presented with the original input x and the contextually relevant knowledge in the form of t x and is trained to make predictions using this additional knowledge. we posit that the model essentially needs to learn relevant semantic associations between natural language input text and various relation types present in the knowledge base. identifying the correct relation types will help the model leverage the corresponding relevant facts in order to make an accurate prediction. this is accomplished via a modified masked language modeling (mlm) @xcite pre-training objective. figure 1 summarizes the complete workflow of our proposed lm-core framework and we describe the three main components in detail in the following sub-sections. ## experiments and discussions data sources and pre-processing: we create our pre-training corpus using the december 20, 2018 snapshot of english wikipedia that contains about 5.5m documents. processing all the articles following the masking strategy described in section 3.1 resulted in a total of ∼46.3m sentences with valid masks, from which we randomly sample sentences to create input examples. in order to illustrate the general nature of lm-core, we used two different plms as our lm encoders -bert-base (uncased) model and roberta-base (cased) model. we use yago and wikidata as two different knowledge bases giving us four variants of lm-core ({bert, roberta} × {yago, wikidata}). we use the english wikipedia version of yago 4 @xcite and preprocess it to obtain our retrieval corpus consisting of roughly 17m triples spanning over 4.9m entities and 131 unique relations. for wikidata, we used the wikidata5m version @xcite ) that consists of roughly 21m triples covering 821 unique relations and 4.8m entities. further details regarding retrieval corpus generation and processing can be found in the appendix (section a.2). for computing triple representations for retrieval (section 3.2), we concatenate the subject (head), relation, and object (tail) of triples and embed them using the sentence transformers @xcite and obtain the 768-dimensional embeddings (same as lm encoder dimensions). ## conclusion we presented lm-core, a framework to train language models with contextually relevant external knowledge. we show that having access to external knowledge leads to significant and robust outperformance over base language models and their knowledge enhanced versions on knowledge probing and two downstream tasks. we also showed how lm-core can handle knowledge updates and presented a thorough error analysis that helped us identify possible directions of future work. can be found in a.2.1 yago yago 4 foot_3 is in rdfs format. yago facts are derived from wikidata, however, all the entities are arranged in a taxonomy mapped to schema.org. we pre-process yago to remove triples involving relationships such as image, logo and url that point to meta-data such as images and other files. we also filter out triples that point to rdf literals or wikidata urls.