How Can We Know What Language Models Know? (2020)

Abstract: Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA .

## introduction recent years have seen the primary role of language models (lms) transition from generating or evaluating the fluency of natural text @xcite @xcite to being a powerful tool for text understanding. this understanding has mainly been achieved through the use of language modeling as a pre-training task for feature extractors, where the hidden vectors learned through a language modeling objective are then used in * the first two authors contributed equally. interestingly, it is also becoming apparent that lms foot_0 themselves can be used as a tool for text understanding by formulating queries in natural language and either generating textual answers directly @xcite , or assessing multiple choices and picking the most likely one @xcite . for example, lms have been used to answer factoid questions @xcite , answer common sense queries @xcite , or extract factual knowledge about relations between entities @xcite . regardless of the end task, the knowledge contained in lms is probed by providing a prompt, and letting the lm either generate the continuation of a prefix (e.g., ''barack obama was born in ''), or predict missing words in a cloze-style template (e.g., ''barack obama is a by profession''). however, while this paradigm has been used to achieve a number of intriguing results regarding the knowledge expressed by lms, they usually rely on prompts that were manually created based on the intuition of the experimenter. these manually created prompts (e.g., ''barack obama was born in '') might be sub-optimal because lms might have learned target knowledge from substantially different contexts (e.g., ''the birth place of barack obama is honolulu, hawaii.'') during their training. thus it is quite possible that a fact that the lm does know cannot be retrieved due to the prompts not being effective queries for the fact. thus, existing results are simply a lower bound on the extent of knowledge contained in lms, and in fact, lms may be even more knowledgeable than these initial results indicate. in this paper we ask the question: ''how can we tighten this lower bound and get a more accurate estimate of the knowledge contained in state-of-the-art lms?'' this is interesting both scientifically, as a probe of the knowledge that lms contain, and from an engineering perspective, as it will result in higher recall when using lms as part of a knowledge extraction system. in particular, we focus on the setting of @xcite who examine extracting knowledge regarding the relations between entities (definitions in § 2). we propose two automatic methods to systematically improve the breadth and quality of the prompts used to query the existence of a relation ( § 3). specifically, as shown in figure 1 , these are mining-based methods inspired by previous relation extraction methods @xcite , and paraphrasing-based methods that take a seed prompt (either manually created or automatically mined), and paraphrase it into several other semantically similar expressions. further, because different prompts may work better when querying for different subjectobject pairs, we also investigate lightweight ensemble methods to combine the answers from different prompts together ( § 4). we experiment on the lama benchmark @xcite , which is an english-language benchmark devised to test the ability of lms to retrieve relations between entities ( § 5). we first demonstrate that improved prompts significantly improve accuracy on this task, with the one-best prompt extracted by our method raising accuracy from 31.1% to 34.1% on bert-base @xcite , with similar gains being obtained with bert-large as well. we further demonstrate that using a diversity of prompts through ensembling further improves accuracy to 39.6%. we perform extensive analysis and ablations, gleaning insights both about how to best query the knowledge stored in lms and about potential directions for incorporating knowledge into lms themselves. finally, we have released the resulting lm prompt and query archive (lpaqa) to facilitate future experiments on probing knowledge contained in lms. ## knowledge retrieval from lms retrieving factual knowledge from lms is quite different from querying standard declarative knowledge bases (kbs). in standard kbs, users formulate their information needs as a structured query defined by the kb schema and query language. for example, select ?y where {wd:q76 wdt:p19 ?y} is a sparql query to search the birth place of barack obama. in contrast, lms must be queried by natural language prompts, such as ''barack obama was born in '', and the word assigned the highest probability in the blank will be returned as the answer. unlike deterministic queries on kbs, this provides no guarantees of correctness or success. while the idea of prompts is common to methods for extracting many varieties of knowledge from lms, in this paper we specifically follow the formulation of @xcite , where factual knowledge is in the form of triples x, r, y . here x indicates the subject, y indicates the object, and r is their corresponding relation. to query the lm, r is associated with a cloze-style prompt t r consisting of a sequence of tokens, two of which are placeholders for subjects and objects (e.g., ''x plays at y position''). the existence of the fact in the lm is assessed by replacing x with the surface form of the subject, and letting the model predict the missing object (e.g., ''lebron james plays at position''): foot_2 ŷ = arg max in previous work @xcite @xcite , t r has been a single manually defined prompt based on the intuition of the experimenter. as noted in the introduction, this method has no guarantee of being optimal, and thus we propose methods that learn effective prompts from a small set of training data consisting of gold subject-object pairs for each relation. ## prompt generation first, we tackle prompt generation: the task of generating a set of prompts {t r,i } t @xmath0, where at least some of the prompts effectively trigger lms to predict ground-truth objects. we employ two practical methods to either mine prompt candidates from a large corpus ( § 3.1) or diversify a seed prompt through paraphrasing ( § 3.2). ## prompt selection and ensembling in the previous section, we described methods to generate a set of candidate prompts {t r,i } t @xmath1. each of these prompts may be more or less effective at eliciting knowledge from the lm, and thus it is necessary to decide how to use these generated prompts at test time. in this section, we describe three methods to do so. ## omitted design elements finally, in addition to the elements of our main proposed methodology in § 3 and § 4, we experimented with a few additional methods that did not prove highly effective, and thus were omitted from our final design. we briefly describe these below, along with cursory experimental results. ## related work much work has focused on understanding the internal representations in neural nlp models @xcite , either by using extrinsic probing tasks to examine whether certain linguistic properties can be predicted from those representations @xcite @xcite , or by ablations to the models to investigate how behavior varies @xcite . for contextualized representations in particular, a broad suite of nlp tasks are used to analyze both syntactic and semantic properties, providing evidence that contextualized representations encode linguistic knowledge in different layers @xcite @xcite @xcite . different from analyses probing the representations themselves, our work follows @xcite ; @xcite in probing for factual knowledge. they use manually defined prompts, which may be under-estimating the true performance obtainable by lms. concurrently to this work, @xcite made a similar observation that using different prompts can help better extract relational knowledge from lms, but they use models explicitly trained for relation extraction whereas our methods examine the knowledge included in lms without any additional training. orthogonally, some previous works integrate external knowledge bases so that the language generation process is explicitly conditioned on symbolic knowledge @xcite @xcite . similar extensions have been applied to pre-trained lms like bert, where contextualized representations are enhanced with entity embeddings @xcite @xcite . in contrast, we focus on better knowledge retrieval through prompts from lms as-is, without modifying them. ## conclusion in this paper, we examined the importance of the prompts used in retrieving factual knowledge from language models. we propose mining-based and paraphrasing-based methods to systematically generate diverse prompts to query specific pieces of relational knowledge. those prompts, when combined together, improve factual knowledge retrieval accuracy by 8%, outperforming manually designed prompts by a large margin. our analysis indicates that lms are indeed more knowledgeable than initially indicated by previous results, but they are also quite sensitive to how we query them. this indicates potential future directions such as (1) more robust lms that can be queried in different ways but still return similar results, (2) methods to incorporate factual knowledge in lms, and (3) further improvements in optimizing methods to query lms for knowledge. finally, we have released all our learned prompts to the community as the lm prompt and query archive (lpaqa), available at: https://github.com/jzbjyb/lpaqa .