Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal View (2022)

Abstract: Prompt-based probing has been widely used in evaluating the abilities of pretrained language models (PLMs). Unfortunately, recent studies have discovered such an evaluation may be inaccurate, inconsistent and unreliable. Furthermore, the lack of understanding its inner workings, combined with its wide applicability, has the potential to lead to unforeseen risks for evaluating and applying PLMs in real-world applications. To discover, understand and quantify the risks, this paper investigates the prompt-based probing from a causal view, highlights three critical biases which could induce biased results and conclusions, and proposes to conduct debiasing via causal intervention. This paper provides valuable insights for the design of unbiased datasets, better probing frameworks and more reliable evaluations of pretrained language models. Furthermore, our conclusions also echo that we need to rethink the criteria for identifying better pretrained language models.

## introduction during the past few years, the great success of pretrained language models (plms) @xcite @xcite raises extensive attention about evaluating what knowledge do plms actually entail. one of the most popular approaches is promptbased probing @xcite @xcite @xcite , which assesses whether plms are knowledgable for a specific task by querying plms with task-specific prompts. for example, to evaluate whether bert knows the birthplace of michael jordan, we could query bert with "michael jordan was born in [mask]". recent studies often construct promptbased probing datasets, and take plms' performance on these datasets as their abilities for the corresponding tasks. such a probing evaluation has been wildly used in many benchmarks such as superglue @xcite , lama @xcite , olmpics @xcite , lm diagnostics @xcite , cat @xcite , x-factr @xcite , biolama @xcite , etc. unfortunately, recent studies have found that evaluating plms via prompt-based probing could be inaccurate, inconsistent, and unreliable. for example, @xcite finds that the performance may be overestimated because many instances can be easily predicted by only relying on surface form shortcuts. @xcite shows that semantically equivalent prompts may result in quite different predictions. @xcite demonstrates that plms often generate unreliable predictions which are prompt-related but not knowledge-related. in these cases, the risks of blindly using promptbased probing to evaluate plms, without understanding its inherent vulnerabilities, are significant. such biased evaluations will make us overestimate or underestimate the real capabilities of plms, mislead our understanding of models, and result in wrong conclusions. therefore, to reach a trustworthy evaluation of plms, it is necessary to dive into the probing criteria and understand the following two critical questions: 1) what biases exist in current evaluation criteria via prompt-based probing? 2) where do these biases come from? to this end, we compared plm evaluation via prompt-based probing with conventional evaluation criteria in machine learning. figure 1 shows their divergences. conventional evaluations aim to evaluate different hypotheses (e.g., algorithms or model structures) for a specific task. the tested hypotheses are raised independently of the training/test data generation. however, this independence no longer sustains in prompt-based probing. there exist more complicated implicit connections between pretrained models, probing data, and prompts, mainly due to the bundled pretraining data with specific plms. these unaware connections serve as invisible hands that can even dominate the evaluation criteria from both linguistic and task aspects. from the linguistic aspect, because pretraining data, probing data and prompts are all expressed in the form of natural language, there exist inevitable linguistic correlations which can mislead evaluations. from the task aspect, the pretraining data and the probing data are often sampled from correlated distributions. such invisible task distributional correlations may significantly bias the evaluation. for example, wikipedia is a widely used pretraining corpus, and many probing data are also sampled from wikipedia or its extensions such as yago, dbpedia or wikidata @xcite @xcite . as a result, such task distributional correlations will inevitably confound evaluations via domain overlapping, answer leakage, knowledge coverage, etc. to theoretically identify how these correlations lead to biases, we revisit the prompt-based probing from a causal view. specifically, we describe the evaluation procedure using a structural causal model @xcite (scm), which is shown in figure 2a . based on the scm, we find that the linguistic correlation and the task distributional correlation correspond to three backdoor paths in figure 2b -d, which lead to three critical biases: we further propose to conduct causal intervention via backdoor adjustments, which can reduce bias and ensure a more accurate, consistent and reliable probing under given assumptions. note that this paper not intends to create a "universal correct" probing criteria, but to remind the underlying invisible risks, to understand how spurious correlations lead to biases, and to provide a causal toolkit for debiasing probing under specific assumptions. besides, we believe that our discoveries not only exist in prompt-based probing, but will also influence all prompt-based applications to pretrained language models. consequently, our conclusions echo that we need to rethink the criteria for identifying better pretrained language models with the above-mentioned biases. generally, the main contributions of this paper are: ## structural causal model for factual knowledge probing in this section, we formulate the scm for factual knowledge probing procedure and describe the key variables and causal relations. the scm is shown in figure 2a , which contains 11 key variables: 1) pretraining corpus distribution d a ; 2) pretraining corpus c, e.g., webtext for gpt2, wikipedia for bert; 3) pretrained language model m ; 4) linguistic distribution l, which guides how a concept is verbalized into natural language expression, e.g., relation to prompt, entity to mention; 5) relation r, e.g., birthplace, capital, each relation corresponds to a probing task; 6) verbalized prompt p for each relation , e.g, x was born in y; 7) taskspecific predictor i, which is a plm combined with a prompt, e.g., <bert, was born @xmath0; 8) probing data distribution d b , e.g., fact distribution in wikidata; 9) sampled probing data t such as lama, which are sampled entity pairs (e.g., <q41421, @xmath1; 10) verbalized instances x, (e.g., <michael jordan, @xmath2, @xmath3; 11) performance e of the predictor i on x. the causal paths of the prompt-based probing evaluation contains: the path {m, p } → @xmath4, where the predictor i is first derived by combining plm m and prompt p , and then the performance e is estimated by applying predictor i on verbalized instances x. to evaluate plms' ability on fact extraction, we need to estimate p(e|do(@xmath5, @xmath6. such true causal effects are represented by the path @xmath7. unfortunately, there exist three backdoor paths between pretrained language model m and performance e, as shown in figure 2b-d . these spurious correlations make the observation correlation between m and e cannot represent the true causal effects of m on e, and will inevitably lead to biased evaluations. in the following, we identify three critical biases in the prompt-based probing evaluation and describe the manifestations, causes, and casual interventions for each bias. ## prompt preference bias in prompt-based probing, the predictor of a specific task (e.g., the knowledge extractor of relation birthplace) is a plm m combined with a prompt p (e.g., bert + was born in). however, plms are pretrained on specific text corpus, therefore will inevitably prefer prompts sharing the same linguistic regularity with their pretraining corpus. such implicit prompt preference will confound the true causal effects of plms on evaluation performance, i.e., the performance will be affected by both the task ability of plms and the preference fitness of a prompt. in the following, we investigate prompt preference bias via causal analysis. ## instance verbalization bias apart from the prompt preference bias, the underlying linguistic correlation can also induce bias in the instance verbalization process. specifically, an instance in probing data can be verbalized into different natural language expressions (e.g., verbalize relation mention prediction capital of america chicago the u.s. washington china beijing cathay bangkok birthplace einstein berlin albert einstein vienna isaac newton london sir isaac newton town q30 in wikidata into america or the u.s.), and different plms may prefer different verbalizations due to mention coverage, expression preference, etc. this will lead to instance verbalization bias. ## sample disparity bias besides the biases induced by linguistic correlations, the distributional correlations between pretraining corpus and task-specific probing data can also introduce sample disparity bias. that is, the performance difference between different plms may due to the sample disparity of their pretraining corpus, rather than their ability divergence. in conventional evaluation, the evaluated hypotheses are independent of the train/test data generation, and all the hypotheses are evaluated on training data and test data generated from the same distribution. therefore, the impact of correlations between training data and test data is transparent, controllable, and equal for all the hypotheses. by contrast, in prompt-based probing, each plm is bundled with a unique pretraining corpus, the correlation between pretraining corpus distribution and probing data distribution cannot be quantified. in the following we investigate this sample disparity bias in detail. ## bias elimination via causal intervention this section describes how to eliminate the abovementioned biases by blocking their corresponding backdoor paths. according to the backdoor criterion in section 2.1, we need to choose a set of variables z that can block every path containing an arrow into m between m and e. since the linguistic distribution l, pretraining corpus distribution d a and probing data distribution d b are unobservable, we choose @xmath8, x} as the variable set for blocking all backdoor paths between (m, e) in the scm by conducting backdoor adjustment: equation 2 provides an intuitive solution. to eliminate the biases stemming from the spurious correlations between pretraining corpus, probing data and prompts, we need to consider the natural distribution of prompts and verbalized probing data regardless of other factors. consequently, the overall causal effects between plm and evaluation result are the weighted averaged effects on all valid prompts and probing data. unfortunately, the exact distribution of p(x, p) is intractable , which needs to iterate over all valid prompts and all verbalized probing data. to address this problem, we propose a sampling-based approximation. specifically, given a specific assumption about p(x, p) (we assume uniform distribution in this paper without the loss of generality), we sample k p prompts for each relation and k x kinds of verbalization for each instance according to p(x, p), and then these samples are used to estimate the true causal effects between m and e according to equation 2. to verify whether causal intervention can improve the evaluation consistency and robustness, we conduct backdoor adjustment experiments on 8 different plms. we randomly sample 1000 subsets with 20 relations from lama, and observe whether the evaluation conclusions were consistent and stable across the 1000 evaluation runtimes. specifically, we use rank consistency as the evaluation metric, which measures the percentage of the most popular rank of each model in 1000 runtimes. for example, if bert ranks at 3 rd place in 800 of the 1000 runtimes, then the rank consistency of bert will be 80%. the above results verify that causal intervention is an effective technique to boost the stability of evaluation, and reach more consistent conclusions. ## related work prompt-based probing prompt-based probing is popular in recent years @xcite for probing factual knowledge @xcite @xcite , commonsense knowledge @xcite , semantic knowledge @xcite @xcite and syntactic knowledge @xcite in plms. and a series of prompt-tuning studies consider optimizing prompts on training datasets with better performance but may undermine interpretability @xcite @xcite @xcite @xcite . because such prompt-tuning operations will introduce additional parameters and more unknown correlations, this paper does not take prompt-tuning into our scm, delegate this to future work. ## conclusions and discussions this paper investigates the critical biases and quantifies their risks in the widely used prompt-based probing evaluation, including prompt preference bias, instance verbalization bias, and sample disparity bias. a causal analysis framework is proposed to provide a unified framework for bias identification, interpretation and elimination with a theoretical guarantee. our studies can promote the understanding of prompt-based probing, remind the risks of current unreliable evaluations, guide the design of unbiased datasets, better probing frameworks, and more reliable evaluations, and push the bias analysis from empirical to theoretical. another benefit of this paper is to remind the evaluation criteria shifts from conventional machine learning algorithms to pretrained language models. as we demonstrate in figure 1 , in conventional evaluation, the evaluated hypotheses (e.g., algorithms, architectures) are raised independently of the train/test dataset generation, where the impact of correlations between training data and test data is transparent, controllable, and equal for all the hypotheses. however, in evaluations of pretrained language models, the pretraining corpus is bundled with the model architecture. in this case, it is significant to distinguish what you need to assess (architecture, corpus, or both), as well as the potential risks raised by the correlations between pretraining corpus and test data, which most current benchmarks have ignored. consequently, this paper echoes that it is necessary to rethink the criteria for identifying better pretrained language models, especially under the prompt-based paradigm. in the future, we would like to extend our causal analysis framework to fit prompt-tuning based probing criteria and all plm-based evaluations.