Selective- LAMA : Selective Prediction for Confidence-Aware Evaluation of Language Models (2023)

Abstract: Recent studies have suggested that neural language models learn and store a large amount of facts and commonsense knowledge from training data. The ability of language models to restore such knowledge is often evaluated via zero-shot cloze-style QA tasks. However, such evaluations rely only on prediction accuracy without punishing the systems for their mistakes, e.g., simply guessing or hallucinating likely answers. Selective prediction is a more informative evaluation framework that takes the confidence of predictions into account. Under the selective prediction setting, a model is evaluated not only by the number of correct predictions, but also by the ability to filter out dubious predictions by estimating the confidence of individual predictions. Such confidence-aware evaluation is crucial for determining whether to trust zero-shot predictions of language models. In this paper, we apply the selective prediction setting to an existing benchmark, LAMA probe, and conduct extensive experiments with recent neural language models and different confidence functions. We empirically show that our Selective-LAMA evaluation is more robust to the effect of simple guesses than the conventional accuracy-based evaluation. Our evaluation reveals the importance of the choice of confidence functions by showing that simply relying on token probabilities is not always the best choice. Further analysis shows that various confidence functions exhibit different preferences over predicted tokens for a given context.

## introduction recently, knowledge stored in pre-trained language models has been intensively investigated. many studies have suggested that language models trained on a large amount of textual corpora, such as bert @xcite and gpt @xcite , store both linguistic knowledge @xcite and factual and commonsense knowledge @xcite during training. however, this knowledge is embedded in the parameters of these language models and thus is difficult to interpret, in contrast to symbolic knowledge bases, which allows us to inspect and edit stored facts explicitly. @xcite proposed a benchmark task, the lama probe, that aims at evaluating the amount of relational knowledge, such as commonsense knowledge and facts, which is stored in a language model. in lama probe, a relational fact is converted into a cloze statement (query) and then given to a language model as a fill-in-the-blank question. if the language model fills in the blank with the correct answer, the model is considered to possess "knowledge" of the relation. according to @xcite s experiments, the bert language model @xcite has a comparable performance to a supervised relation extraction baseline, with precision ranging from 10.5 to 32.3 depending on the dataset type. however, in many applications, we are concerned not only with the amount of the knowledge extracted from a language model, but also with its reliability. this is because large pre-trained language models are known to fluently generate "facts" that they have never seen @xcite @xcite . therefore, it is crucial to know when we can trust the output of a language model. the lama probe framework does not cover this issue, as it always forces the model to output an answer for all instances, regardless of whether the model really "knows" the answer to a query. this means that it implicitly trusts all outputs of a language model to the same degree. figure 1 shows an example suggesting that a pre-trained language model is not always using its knowledge for prediction. the figure shows the distribution of predicted tokens for a particular relation in the original lama probe benchmark @xcite . we can see that three tokens account for more than half of the wrong predictions. this indicates that the model has a bias which it acquired during training, probably due to the input template used, rather than using actual questionspecific knowledge about individual facts. to address this issue, we apply the selective prediction @xcite setting to the lama probe and propose a new evaluation framework, selective-lama, to evaluate both the amount of knowledge in a pre-trained language model and the model's ability to estimate the reliability of its prediction. selective prediction is a framework by which a system can choose whether to output the individual predictions of a model based on the prediction results. specifically, we consider the selection with guaranteed risk control setting @xcite , where the system computes confidence scores of individual predictions to determine whether it outputs the predictions. a system is evaluated by the number of predictions it can make while maintaining a risk of error below a certain level. to achieve high performance, a system is required not only to answer many questions correctly, but also to accurately estimate the model's confidence about individual facts and determine when the system should not answer a question. in this paper, we focus on masked language models and address the following research questions: ## selective prediction under the selective prediction setting @xcite , a selective classifier determines whether a system should output the prediction of the model. we consider a classification problem from an input space x to a set of labels y. a selective classifier (f, g) consists of an original classification model f : @xmath0: @xmath1, 1}. given an input example @xmath2, a selection function determines whether the system outputs the prediction f (x) ∈ y: geifman and el-yaniv (2017) introduced the selection with guaranteed risk (sgr) setting, which uses a confidence-based selection function: this setting allows a user to adjust the error risk generated by the system by appropriately setting the value of β. specifically, increasing β decreases the number of cases predicted by the system while reducing the risk of making a wrong prediction. under the sgr setting, there is a risk-coverage trade-off between the risk (n pred -n corr )/n pred that a selective classifier will make a wrong prediction and the coverage n pred /n of the predictions made by the system. here, n, n pred , and n corr denote the number of all examples, predicted examples, and correct predictions, respectively. the performance of a selective classifier is evaluated based on the auc of the risk-coverage curve (rc-auc) obtained by changing β in the selection function (2). a smaller rc-auc value indicates a lower risk of making a wrong prediction. in practice, the threshold will be determined by the level of risk acceptable to the users. ## related work in nlp, the reliability of the model responses has been discussed mainly in the field of question answering. estimating the confidence of an answer is critical in quiz competitions, such as jeopardy, since the system has to decide when to answer the questions @xcite . @xcite recently introduced a selective prediction setting to question answering tasks and then evaluated the performance of the models on out-ofdomain questions. @xcite addressed a similar problem, but focused on a calibration of the model prediction on qa tasks. while they focused on extractive or multiple-choice qa tasks where a limited number of candidate answers are available, our focus is on the knowledge probing of language models where the candidate answer is the entire vocabulary and, thus, false positives are more frequent. several studies have addressed the reliability issue of pre-trained language models as a calibration problem; the goal of these studies is to train a wellcalibrated language model that makes accurate confidence estimation. desai and durrett (2020) investigate the calibration level of pre-trained language models, focusing on bert @xcite and roberta @xcite . they evaluate the "out-of-the-box" performance of these models without post-processing, as well as the performance of post-hoc calibration methods (e.g., temperature scaling and label smoothing). @xcite proposed regularization methods to better calibrate pre-trained language models. both studies assume access to (at least in-domain) training data of the target tasks on which parameterized calibration models can be trained. in contrast, our study primarily aims to explore better signals in pre-trained language models to estimate the knowledge they store. thus, we focus on methods that do not require additional training data or an external knowledge source. although training-based methods (e.g., temperature scaling) have the potential to achieve better performance in terms of calibration, optimal parameters vary depending on models and tasks, especially when evaluated in out-of-domain datasets @xcite . in our experiments, all queries have at least one correct answer. therefore, when a model cannot answer a question correctly, this implies that it did not acquire the correct knowledge during training or that its knowledge was not elicited by the natural language query because of a sub-optimal prompt @xcite . however, there are also cases where the question is essentially impossible to answer due to ambiguity @xcite or false presupposition @xcite . an investigation of such cases remains a direction of future research. ## conclusion in this paper, we introduced the selective prediction setting to the lama probe benchmark to evaluate both the amount of relational knowledge stored in a language model and the ability of the models to effectively filter out unconfident predictions. we compared different confidence functions that can be calculated using only the model parameters and the output information. the experimental results are summarized as follows: future studies will include a detailed analysis of the relationship between tasks, models, and confidence scores. moreover, more sophisticated methods will be explored to ensure the reliability of language model predictions under various tasks. the code for our work is attached as supplementary material.