Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries (2021)

Abstract: Pretrained language models have been suggested as a possible alternative or complement to structured knowledge bases. However, this emerging LM-as-KB paradigm has so far only been considered in a very limited setting, which only allows handling 21k entities whose name is found in common LM vocabularies. Furthermore, a major benefit of this paradigm, i.e., querying the KB using natural language paraphrases, is underexplored. Here we formulate two basic requirements for treating LMs as KBs: (i) the ability to store a large number facts involving a large number of entities and (ii) the ability to query stored facts. We explore three entity representations that allow LMs to handle millions of entities and present a detailed case study on paraphrased querying of facts stored in LMs, thereby providing a proof-of-concept that language models can indeed serve as knowledge bases.

## introduction language models (lms) appear to memorize world knowledge facts during training. for example, bert @xcite correctly answers the query "paris is the capital of [mask]" with "france". this observation prompted @xcite to ask if lms can serve as an alternative or complement to structured knowledge bases (kbs), thereby introducing the idea of treating lms as kbs: during training, the lm encounters world knowledge facts expressed in its training data, some of which are stored in some form in the lm's parameters. after training, some of the stored facts can be recovered from the lm's parameters by means of a suitable natural language query (fig. 1 ). a lm with such a "built-in" kb is useful for knowledge-intensive tasks @xcite and question answering @xcite , and could improve natural language interfaces to structured data @xcite . however, this emerging lm-as-kb paradigm is faced with several foundational questions. first question: kbs contain millions of entities, while lm vocabulary size usually does not exceed 100k entries. how can millions of entities be represented in lms? @xcite circumvent this problem by only considering 21k entities whose canonical name corresponds to a single token in the lm's vocabulary, e.g., entities like "france" or "bert", but not "united kingdom" or "sesame street". hence, this approach cannot handle entities not contained in the vocabulary and a query like "bert is a character on [mask]" is not answerable in this simplified setting. to answer this first question, we compare three methods for scaling lm-as-kb to millions of entities: we find that, while all three entity representations allow lms to store millions of world-knowledge facts involving a large number of entities, each representation comes with different trade-offs: sym-bolic representation allows the most accurate storage, but is computationally expensive and requires entity-linked training data. surface representation is computationally efficient and does not require entity-linked data, but is less accurate, especially for longer entity names. continuous representation also requires entity-linked data, but is computationally more efficient than symbolic representation. second question: what is the capacity of lms for storing world knowledge? can a lm store, say, all relation triples contained in a kb like wikidata @xcite )? here we conduct experiments using synthetic data to study the scaling behaviour of current lm architectures. varying the number of trainable model parameters and recording the number of relation triples memorized at a given accuracy level, we find that, e.g., a transformer @xcite with 125 million parameters (12 layers of size 768), has the capacity to memorize 1 million wikidata relation triples with 95 percent accuracy or 5 million relation triples with 79 percent accuracy. assuming linear scaling, this finding suggests that larger lms with tens or hundreds of billions of parameters @xcite ) can be used to store sizable parts, if not all, of a kb like wikidata. third question: how robustly is world knowledge stored in lms? is the lm able to recall a fact even if the query is slightly different than what was memorized during training? for example, if the lm memorized "barack obama was born in hawaii" during training, can it answer queries like "barack obama is from [mask]" or "where was barack obama born? [mask]"? here we conduct experiments to measure how well the lm transfers knowledge from memorized statements to query variants, both in a zero-shot setting in which the model is not exposed to the target query variant during training, and a few shot setting, in which the model is finetuned on a small number of statements containing the target query variant. we observe zero-shot transfer in case of highly similar query variants, and see successful few-shot transfer after finetuning with 5 to 100 instances in case of less similar queries. this ability to handle soft, natural language queries, as opposed to hard, symbolic queries in a language like sql or sparql, is one of the key motivations for using lms as kbs. contributions. we formulate two requirements for treating lms as kbs: (i) the ability to store a large number of facts involving a large number of entities and (ii) the ability to query stored facts. after providing background on world knowledge in lms ( §2), we make the following contributions: foot_0 terminology. in this work we are interested in storing and retrieving world knowledge facts in and from a lm. world knowledge is knowledge pertaining to entities, such as barack obama. a fact is a piece of world knowledge that can be expressed with a concise natural language statement, such as the english sentence barack obama was born in hawaii, or with a relation triple, such as barack obama, wasbornin, hawaii . a relation triple, or relation for short, consists of a subject entity (barack obama), a predicate (wasbornin), and an object entity (hawaii). a knowledge base is a set of relations. knowledge bases, such as wikidata, typically contain thousands of predicates, millions of entities, and billions of relations. ## world knowledge in language models large pretrained lms have been the driver of recent progress in natural language processing @xcite @xcite . while the trend towards larger lms is likely to continue @xcite @xcite , it has limitations: (i) a model trained only on text lacks grounding in perception and experience and hence cannot learn meaning @xcite . (ii) reporting bias leads to certain knowledge rarely or never being expressed in text. for example, a lm will easily learn to associate the phrase "barack obama" with the phrase "u.s. president", but might less likely learn that he is a "human being", since the latter fact is rarely stated explicitly in text. in contrast, this type of knowledge is readily available in kbs. (iii) a large number of rare entities @xcite @xcite are, by definition, rarely mentioned, making it difficult for lms to acquire knowledge about this long tail of entities from text alone. these limitations have motivated efforts to explicitly foot_1 equip lms with world knowledge. ## entity representations how can millions of entities be represented in a lm? to answer our first question, we compare three types of entity representations: symbolic, surface form, and continuous. experimental setup. we evaluate entity representations by measuring how well they allow a lm to store and retrieve world knowledge facts. for example, if the lm's training data contains the statement "bert is a character on sesame street", the model should memorize this statement and recall the correct object sesame street when asked with a query like "bert is a character on @xcite ." synthetic data. it is not a priori clear how many facts a text from the lm's training data, say, a wikipedia article, expresses. since we want to precisely measure how well a lm can store and retrieve facts, we create synthetic data by generating statements from kb relations and then train the model to memorize these statements. using wikidata as kb, we first define two sets of entities: a smaller set consisting of the top 1 million wikidata entities according to node outdegree, and a larger set consisting of the roughly 6 million wikidata entities that have an entry in the english wikipedia. next, we manually create templates for the 100 most frequent wikidata predicates. for example, for the predicate p19 ("place of birth"), we create the template s was born in o and generate english statements by filling the s and o slots with entities from the sets defined above for which this relation holds. 4 to make queries for an object unique given subject and predicate, we arbitrarily select exactly one fact if there are multiple objects and discard the other facts. this process yields 5 million statements involving up to 1 million entities, and 10 million statements involving up to 6 million entities. these statements then serve as training instances, i.e., given the query "barack obama was born in [mask]", the model should predict hawaii. as our goal is to store facts in a lm, there is no distinction between training and test data. models and training. we consider two common lm architectures: lstms @xcite and transformers @xcite . for lstms, we compare two configurations; a randomly initialized two-layer lstm with layers size 256 (lstm 256) and one with layer size 1024 (lstm 1024). for transformers, we compare a pretrained roberta-base @xcite , and roberta without pretraining, i.e., a randomly initialized transformer of the same size. for consistent tokenization across all four models, we subword-tokenize statements with the roberta tokenizer. to store statements in a lm, we train until the model reaches 99 percent memorization accuracy, i.e., overfits the training data almost perfectly, or stop early if accuracy does not improve for 20 epochs. see appx. d for training details. ## lm capacity for storing facts we now turn to the second question, how model capacity scales with model size (fig. 7 , top). with a 12-layer transformer of layer size 96 or 192 (top subfigure, solid red and dashed green lines), memorization accuracy quickly drops as the number of facts to memorize increases. larger models can memorize more facts, but accuracy drops rather quickly, e.g., to 65 percent of 3 million facts memorized with a layer size of 384 (dotted orange line). assuming a desired memorization accuracy of 80 percent, we record the maximum number of facts a model of a given size can memorize at this level (fig. 7 , bottom). for the model sizes considered here, storage capacity appears to scale linearly, with a model of layer size 384 (55m parameters) storing one million facts and a model of layer size 960 (160m parameters) storing 7 million facts. apart from the number of facts to store, we hypothesize that successful storage depends on two more factors: the number of entities and the en- ## querying stored facts so far, we saw that it is possible to store millions of facts in a lm, by finetuning the model to predict the masked object of statements like barack obama was born in @xcite . however, given the large number of model parameters and training effort, mere storage is not a compelling achievement: the underlying relations, here barack obama, was-bornin, hawaii , can be stored more compactly and with perfect accuracy in a structured kb. one of the potential benefits of the lm-as-kb paradigm is the lm's ability to handle paraphrases. if the lm's representation of the statement above is sufficiently similar to its representation of queries like barack obama is from @xcite or where is barack obama from? @xcite , this similarity could allow transfer from the memorized statement to these unseen queries. is this soft querying of facts stored in a lm possible? we now conduct a controlled experiment to answer this question, expecting one of the following three outcomes: 1. rote memorization. the model memorizes statements with little or no abstraction, so that even small, meaning-preserving changes to the query prevent the model from recalling the correct object. 2. generic association. the model memorizes pairs of subject and object entities but disregards the predicate. for example, a model might predict hawaii whenever the query contains the phrase barack obama, regardless of context. this pathological behaviour could be especially prevalent if the distribution of object entities co-occurring with a given subject is dominated by one object. ## limitations and conclusions limitations. this work is not without limitations. we only use one kb in our experiments. arguably, as the largest publicly available source of world knowledge, wikidata is the most promising resource for equipping lms with such knowledge, but attempts to store a kb with different structure might result in different outcomes, since some types of graphs are easier to memorize for a lm than others (see appx. g). while we use language like "train a lm to memorize statements" for simplicity throughout this work, what we do in case of pretrained lms is more akin to adaptive pretraining @xcite . it is possible that integrating entity supervision directly into lm pretraining (févry et al., 2020) allows more efficient fact storage. our analysis was focused on entity representations and ignored the question of how to represent relation predicates or entire relation triples. here, relation learning @xcite and lm pretraining on fact-aligned corpora @xcite are avenues for future work. finally, we formulated the lm-as-kb paradigm in terms of storing and retrieving relation triples. while structured kbs such as wikidata consist of such triples and hence our experiments showing storage and retrieval of triples lms are sufficient as a proof-of-concept in principle, structured kbs allow more complex queries than the ones considered here, such as 1-to-n relations, multihop inference, queries involving numerical ranges, or facts qualified by time and location @xcite . conclusions. we gave a positive answer to @xcite 's question if language models can serve as knowledge bases. arguing that treating lms as kbs requires representing a large number of entities, storing a large number of facts, and the ability to query a fact with a variety of queries, we showed that current lm architectures fulfill these requirements when extended with a compo-nent for representing entities. in addition to the ability to handle paraphrased queries, we envision further benefits from the lm-as-kb paradigm. for example, the fact-memorization and paraphrasefinetuning setting introduced in section 5 allows precise control over which facts a lm learns.