Factual Probing Is [ MASK ]: Learning vs. Learning to Recall (2021)

Abstract: Petroni et al. (2019) demonstrated that it is possible to retrieve world facts from a pre-trained language model by expressing them as cloze-style prompts and interpret the model’s prediction accuracy as a lower bound on the amount of factual information it encodes. Subsequent work has attempted to tighten the estimate by searching for better prompts, using a disjoint set of facts as training data. In this work, we make two complementary contributions to better understand these factual probing techniques. First, we propose OptiPrompt, a novel and efficient method which directly optimizes in continuous embedding space. We find this simple method is able to predict an additional 6.4% of facts in the LAMA benchmark. Second, we raise a more important question: Can we really interpret these probing results as a lower bound? Is it possible that these prompt-search methods learn from the training data too? We find, somewhat surprisingly, that the training data used by these methods contains certain regularities of the underlying fact distribution, and all the existing prompt methods, including ours, are able to exploit them for better fact prediction. We conduct a set of control experiments to disentangle “learning” from “learning to recall”, providing a more detailed picture of what different prompts can reveal about pre-trained language models.

## introduction pre-trained language models like bert are optimized to predict the distribution of words in an internet corpus @xcite . naturally, this distribution encodes information about world facts. recently, researchers have taken an interest in measuring how much factual information language models acquire from pre-training. @xcite formally define this project in the lama ## can we trust optimized prompts? our factual probing results confirm that op-tiprompt is an effective approach, outperforming the best previous method by 6.4% on the lama benchmark. however, can we conclude that bert encodes 6.4% more facts than was previously known? our prompts, like lpaqa and autoprompt, are optimized on in-distribution wikidata relations, which raises the possibility that they exploit some regularities in the underlying fact distribution. in this section we aim to answer two questions. first, are there patterns in the wikidata fact distribution that statistical model could theoretically exploit to predict unseen facts? second, are optimized prompts capable of exploiting these patterns in practice? ## discussion our experiments show that optiprompt is an effective optimization algorithm, outperforming prior work at the task of eliciting facts from a pretrained language model. however, our results are complicated by the fact that any data-driven optimization can find prompts that encode new information from the training data. this leaves open the question of which method we should select if we are interested in factual probing. continuous vs. discrete prompts we find that both continuous and discrete optimization are capable of finding prompts that exploit the training data. even when the prompt is discrete, it is rarely clear why a prompt elicits a particular prediction. 11 hence, we believe that continuous prompting is more preferable, because it is easier and more efficient to optimize, and makes better predictions (in both easy and hard subsets). on the other hand, one drawback of optiprompt (which is shared by autoprompt) is that we need white-box access to the lm to compute the gradients. discrete prompts will still be necessary in cases where the model parameters are not available, for example in the case of very large language models that are provided over an api. learning vs. learning to recall regardless of how we choose to optimize prompts, it remains difficult to say why a model made a particular prediction-whether it was learned from training data or encoded in the lm. some avenues for future work might be to consider techniques for attributing predictions to specific training instances, with the goal of developing a causal understanding of how facts are acquired during pre-training or prompt optimization. more generally, our real goal is to understand how pre-trained language models learn and represent information. prompt-based probing might provide some insight into this question, but we hope that future research will eventually be able to provide more mechanistic explanations for neural network behavior. for example, it would be interesting to understand how information about entities is laid out in neural network parameters and later retrieved in response to an input prompt. ## related work our work follows from the line of factual probing experiments initiated by @xcite , who introduced the lama benchmark for clozestyle factual probing. subsequent work on lama has introduced data-driven methods for optimizing prompts @xcite . @xcite point out that many facts in lama can be predicted using lexical clues, and they introduce a new benchmark, lama-uhn, that is less susceptible to these heuristics. our work follows these projects by introducing (a) more effective techniques for optimizing prompts, and (b) a more comprehensive approach for accounting for the role of train/test overlap. concurrently with this work, other authors explore continuous prompt optimization: @xcite use an encoder to map a manually written prompt to a sequence of continuous vectors, which are then replaced with the discrete tokens that are nearby in embedding space; @xcite propose prefix-tuning, which fine-tunes the left-most hidden representations in auto-regressive language models; @xcite use an lstm to generate a sequence of prompt vectors. prompting has been explored more generally as a method for achieving "few-shot" learning with language models @xcite @xcite . linguistic probing is an extensive area of research that we do not attempt to summarize here (see @xcite for an overview). our work is most related to recent proposals about how to measure whether a probe is extracting information from a representation or learning to predict the annotation from probe training data. these include random baselines @xcite and information-theoretic measurements @xcite . we adopt the notion of control functions from @xcite . our study also relates to a larger category of work diagnosing "shortcut learning" @xcite in neural nlp models. @xcite discover that models like bert are often "right for the wrong reason", exploiting shallow heuristics rather than underlying linguistic structure, and similar effects have been discovered in many other tasks @xcite . ## conclusion we introduce optiprompt, an effective continuous method for optimizing prompts. applied to factual probing, optiprompt outperforms the best previous prompt method by 6.4% on the lama benchmark. we find that the typical training data used for prompt optimization reveals useful information about the underlying task distribution, to the point that search algorithms can find prompts that recover "facts" even from a randomly initialized model. by comparing the predictions of different prompt methods across our different controls we can form a more detailed understanding of how different prompts behave and what they can reveal about pre-trained language models.