Prompting is not a substitute for probability measurements in large language models (2023)

Abstract: Prompting is now a dominant method for evaluating the linguistic knowledge of large language models (LLMs). While other methods directly read out models’ probability distributions over strings, prompting requires models to access this internal information by processing linguistic input, thereby implicitly testing a new type of emergent ability: metalinguistic judgment. In this study, we compare metalinguistic prompting and direct probability measurements as ways of measuring models’ linguistic knowledge. Broadly, we find that LLMs’ metalinguistic judgments are inferior to quantities directly derived from representations. Furthermore, consistency gets worse as the prompt query diverges from direct measurements of next-word probabilities. Our findings suggest that negative results relying on metalinguistic prompts cannot be taken as conclusive evidence that an LLM lacks a particular linguistic generalization. Our results also highlight the value that is lost with the move to closed APIs where access to probability distributions is limited.

## introduction few technologies have been as exciting -and divisive -for language science as large language models (llms). llms are capable of incredibly sophisticated linguistic behaviors, which emerge through statistical learning with massive amounts of text data and highly expressive, domain-agnostic learning architectures. on the one hand, the success of these models has sparked a growing movement to treat them as candidate models of human language acquisition and processing (e.g., @xcite @xcite @xcite even claims that they "refute" chomsky's approach to language. on the other hand, linguists have highlighted shortcomings of current models that make them unsuitable as cognitive theories (e.g., @xcite @xcite @xcite . no matter their theoretical position, researchers need a way to assess the capabilities of llms in order to substantiate such claims. the fundamental unit of llm computation is p (token|context), which, in principle, can be directly read out from a model by accessing its output layer of vocabulary logits. the distribution that this implies over word strings reflects the model's linguistic generalizations: that is, a generative model of the language seen during training, which can be used to evaluate the likelihood of previously unseen strings. direct measurements of model-derived string probabilities have revealed capabilities such as syntactic generalizations (e.g., @xcite @xcite , semantic plausibility judgments @xcite , and certain coherence inferences @xcite . recently, there has been a growing trend to use prompting to evaluate llms' capabilities. prompting (popularized by @xcite enables end-to-end interaction with models through natural language, and can be done entirely through inference (i.e., without gradient updates). this method has revealed new classes of emergent abilities in llms, such as arithmetic, instruction-following, and grounded conceptual mappings @xcite @xcite , as well as the ability to render sentence acceptability judgments @xcite . from a sociological angle, prompting has also made llm evaluations more accessible for domain experts in linguistics and cognitive science, sparking new discussion about the limitations of llms' abilities @xcite @xcite . for example, @xcite prompt gpt-3 to produce grammaticality judgments of infrequent linguistic constructions, and conclude that the model "show[s] a critical lack of understanding even of high-frequency words" (p. 1). similarly, @xcite argues that "llms are poor theories of human linguistic cognition" (p. 2) based on prompting chatgpt to compare the well-formedness of two english sentences. as these examples illustrate, researchers have been making substantial theoretical claims based on prompting. however, there is an important caveat to using prompts to evaluate models' linguistic knowledge. prompt-based methods test not only whether a model represents the generalization of interest (e.g., a certain ordering of probabilities), but also whether the model can report the outcome of applying the generalization to the sentence in the prompt. in this way, prompting implicitly tests a new type of emergent ability -metalinguistic judgment -which has not yet been systematically explored as a way to evaluate model capabilities. to demonstrate the difference between direct probability measurements and metalinguistic prompting, consider the case of english subjectverb agreement. a direct approach might compare the probability assigned by the model to singular and plural verbs, given a particular subject noun phrase (e.g., @xcite . for example, we might compare p ("is") and p ("are"), conditioned on the prefix given by (1-a). in contrast, a prompting approach might present a sentence prefix, and pose a question about this linguistic content. for example, we might compare p ("is") and p ("are"), conditioned on the prompt in (1-b). a model that perfectly performs the metalinguistic task posed in prompt (1-b) should have identical probability distributions over the next word given (1-a) and (1-b). however, there is no guarantee that a model's response to a metalinguistic prompt will match its underlying internal representations. in light of this, how should we interpret models' responses to metalinguistic prompts? how do these responses correspond to models' internal representations? and when should we use metalinguistic prompts as opposed to direct measurements? these questions are becoming increasingly important, as prompting plays a growing role in the debate about llms as models of human language processing. in this study, we do not intend to take a stance on this theoretical debate. rather, our goal is to evaluate the validity of metalinguistic prompting as a way of measuring llms' internal knowledge. we pose two research questions: (1) how well do models perform under direct and metalinguistic evaluation methods? and (2) how consistent are the metalinguistic methods with the direct method? we investigate these questions through four experiments, covering a range of tasks and linguistic domains. our findings (and their supporting figures) are summarized below: taken together, our findings suggest that negative results relying on metalinguistic prompts cannot be taken as conclusive evidence that an llm lacks a particular linguistic generalization. these findings suggest a possible basis for a competenceperformance distinction in llms: namely, the distinction between the information encoded in a model's isolated-sentence string probability distribution versus the model's behavioral responses to prompts. we discuss this topic in greater detail in section 6.1. our results also highlight the value that is lost as researchers move toward interacting with llms through closed apis, where access to models' underlying probability distributions is limited. indeed, only two days before acceptance of this paper to emnlp, the ability to obtain arbitrary token logprobabilities from gpt-3.5-turbo-instruct was removed from the openai api, reinforcing the timeliness of the issue. we urge future research to clearly motivate the evaluation methods used to assess llm abilities, and highlight the importance of developing and using open-source models with access to internal probabilities. 2 related work our study is related to model calibration, or the problem of estimating predictive uncertainty (e.g., @xcite @xcite . most relevant to our work, @xcite perform a broadscale evaluation of "honesty" in lms. in particular, they analyze models' "truthfulness" and "selfknowledge", which are conceptually similar to our ground-truth and internal consistency evaluations. similarly, @xcite find that chatbots' expressions of confidence and doubt are poorly calibrated with the likelihood that the models' responses are actually correct. these approaches differ from ours in that they analyze models' metacognition by annotating verbalized expressions (e.g., "i don't know, but..."), or by using models to evaluate the correctness of their own generated answers. other studies have also demonstrated that models can respond to prompts in unexpected ways @xcite @xcite @xcite , in some cases achieving successful outcomes even when prompts are misleading or incoherent. similarly, @xcite find that explanations elicited through chain-of-thought prompting @xcite can systematically misrepresent the underlying reasons and causes for a model's prediction. @xcite also demonstrate how llms' responses to prompts are highly sensitive to word probabilities, due to their original training objectives. @xcite also evaluate llms' "metalinguistic" abilities, but in a different sense of the word. they investigate whether llms can perform theoretical analyses of the structures and regularities of linguistic expressions -for example, drawing the syntactic tree diagram of a given sentence in valid l a t e x code. beguš et al.'s tested abilities are metalinguistic in the sense that they require judgments about linguistic objects given linguistic input. in contrast, our study investigates whether models can access and report their internal probability distributions through linguistic prompts. ## results we now return to our main research questions, laid out in the introduction: (1) how well does each evaluation method perform on each task? (2) how consistent are the metalinguistic evaluation methods with the direct evaluation method? we address these in sections 5.1 and 5.2, respectively. ## discussion in this study, we compared metalinguistic prompting and direct measurements as ways of evaluating llms' linguistic knowledge. broadly, we find that metalinguistic judgments are inferior to direct measurements of token-or sentence-level probabilities. we also find evidence that minimal-pair comparisons help reveal models' generalization capacities. we do not intend to claim that prompting should be categorically dispreferred in favor of other evaluation methods. prompting is useful for eliciting open-ended responses, such as chain-of-thought reasoning. metalinguistic prompting could also be used to ask questions that would be challenging to translate into direct probability measurements (e.g., "which of the following two sentences is more semantically plausible, but less syntactically wellformed?"). in addition, prompting lowers the technical barrier for domain experts to conduct llm evaluations, which could contribute to the development of more robust behavioral benchmarks. what do our findings mean for researchers interested in the linguistic abilities of llms? while there are valid reasons to prefer both prompting meta question simple meta instruct meta question complex word prediction (exp 1) word comparison (exp 2) sentence judgment (exp 3a) sentence comparison (exp 3b) 0.79 0.78 0.63 0.5 0.5 0.38 0.2 0.2 0.2 0.23 0.24 0.22 more distant task more distant prompt 1.0 0.5 0.0 0.5 1.0 figure 3: internal consistency: correlation between metalinguistic and direct responses gets weaker as prompts become less direct. pearson r correlation between response magnitudes (averaged over models and datasets) measured by direct prompts versus each metalinguistic prompt. see appendix c for more details. and direct probability measurements, these methods will not necessarily generate consistent results. more specifically, our findings suggest that negative results relying on metalinguistic prompts cannot be taken as conclusive. while our paper focuses on the value of direct probability measurements for evaluating linguistic generalizations, other endeavors in cognitive science and machine learning also rely on access to llm probabilities. for example, token probabilities enable multiple-choice evaluation and beam search, as well as investigating mode collapse (janus, 2022) and obtaining quantities for bayesian inference (e.g., @xcite @xcite . 8 thus, the value that is lost as we move toward closed apis extends beyond linguistic analysis, further underscoring the importance of using and developing llms with open access to internal probabilities.