How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering (2021)

Abstract: Recent works have shown that language models (LM) capture different types of knowledge regarding facts or common sense. However, because no model is perfect, they still fail to provide appropriate answers in many cases. In this paper, we ask the question, “How can we know when language models know, with confidence, the answer to a particular query?” We examine this question from the point of view of calibration, the property of a probabilistic model’s predicted probabilities actually being well correlated with the probabilities of correctness. We examine three strong generative models—T5, BART, and GPT-2—and study whether their probabilities on QA tasks are well calibrated, finding the answer is a relatively emphatic no. We then examine methods to calibrate such models to make their confidence scores correlate better with the likelihood of correctness through fine-tuning, post-hoc probability modification, or adjustment of the predicted outputs or inputs. Experiments on a diverse range of datasets demonstrate the effectiveness of our methods. We also perform analysis to study the strengths and limitations of these methods, shedding light on further improvements that may be made in methods for calibrating LMs. We have released the code at https://github.com/jzbjyb/lm-calibration .

## introduction language models (lms; @xcite @xcite learn to model the probability distribution of text, and in doing so capture information about various aspects of the syntax or semantics of the language at hand. recent works have presented intriguing results demonstrating that modern large-scale lms also capture a significant amount of knowledge, includwe apply the largest publicly available lms, t5, bart, and gpt-2, over a wide range of question answering (qa) datasets @xcite covering diverse domains. we first observe that despite the models' high performance (e.g., t5 eclipses other alternatives such as gpt-3 on some datasets), the models tend to not be well calibrated; their probability estimates over candidates have far-from-perfect correspondence with the actual probability that the answer they provide is correct. some examples of this are demonstrated in the ''original'' column of to alleviate this problem, we propose methods to make lms' confidence scores correlate better with the likelihood of model prediction being correct. we examined both fine-tuning methods that modify lms' parameters and post-hoc methods that keep lms fixed and only manipulate the confidence values or inputs. specifically, we fine-tune the lm using softmax-or margin-based objective functions based on multiple candidate answers. for post-hoc calibration, we examine temperature-based scaling and feature-based decision trees that take prediction probability and input-related features as input and produce calibrated confidence @xcite @xcite . we also study the sensitivity of lms' confidence estimation with respect to language variation by paraphrasing candidate answers and augmenting questions using retrieved context. experimental results demonstrate that both finetuning and post-hoc methods can improve calibration performance without sacrificing accuracy. we further perform analysis and ablation studies on our methods, inspecting different aspects that may affect calibration performance. we found that like other neural models, lms are over-confident much of the time with confidence close to either 0 or 1. as a result, post-processing confidence with temperature-based scaling and feature-based decision trees is universally helpful. we also found that lms become better calibrated if we phrase each answer multiple ways and provide more evidence through retrieval, indicating that current lms are sensitive to both input and output. ## lm-based question answering lms are now a ubiquitous tool in not only natural language generation, but also natural language understanding (nlu), where they are largely used for unsupervised representation learning in pretrained models such as bert @xcite . however, recent work has demonstrated that lms can also be used as-is to solve nlu tasks, by predicting the missing words in cloze-style questions @xcite , or by predicting the continuation to prompts @xcite . previous works that purport to calibrate lms @xcite @xcite mainly focus on the former use case, using representations learned by lms to predict target classes (for tasks such as natural language inference, part-of-speech tagging, or text classification) or identify answer spans (for tasks such as extractive qa). in contrast, we focus on the latter case, calibrating lms themselves by treating them as natural language generators that predict the next words given a particular input. to make our observations and conclusions as general as possible, we experiment over a diverse range of qa datasets with broad domain coverage format datasets and domains multi-choice arc (science @xcite ), ai2 science questions (science @xcite ), openbookqa (science @xcite ), winogrande (commonsense @xcite ), commonsenseqa (commonsense @xcite ), mctest (fictional stories @xcite ), piqa (physical @xcite ), siqa (social @xcite ), race (english comprehension @xcite ), qasc (science @xcite ), mt-test (mixed @xcite )) extractive squad 1.1 (wikipedia @xcite ), squad 2 (wikipedia @xcite ), newsqa (news @xcite ), quoref (wikipedia @xcite ), ropes (situation understanding @xcite ) specifically, we focus on two varieties of qa: multiple-choice and extractive, with examples shown in extractive qa for extractive qa, inputs x to lms are questions concatenated with context passages from which the answer must be extracted. in this case, every span within the passage is a candidate answer in i(x). however, enumerating over all possible spans of the context passage is computationally costly. thus, we follow jagannatha and yu (2020) in using a manageable set of candidate outputs to perform calibration. specifically, we develop a method to efficiently calculate probabilities over promising spans that exist in the input. first, we calculate the probability of the first token in output y , masking out any tokens that are not included in the input passage at all. then, for the top r scoring tokens, we find their location in the input passage, and calculate the probability of all continuing spans up to a certain length (e.g., 20 tokens). we finally keep the top k spans as candidates i(x) and use all candidates to calculate the probability in a manner similar to that of multiple-choice qa. ## background on calibration a model is considered well calibrated if the confidence estimates of its predictions are well-aligned with the actual probability of the answer being correct. given an input x and true output y , a model output ŷ , and a probability p n ( ŷ |x) calculated over this output, a perfectly calibrated model satisfies the following condition: in practice, we approximate this probability by bucketing predictions into m disjoint equally sized interval bins based on confidence. @xcite examined the calibration properties of neural network classifiers, and proposed a widely used measure of calibration called expected calibration error (ece), which is a weighted average of the discrepancy between each bucket's accuracy and confidence: unfortunately, we found that state-of-the-art lm-based methods for question answering (such as the unifiedqa model of @xcite ) were extraordinarily poorly calibrated, with the normalized probability estimates barely being correlated with the likelihood of the outputs being correct. for the two examples in ## calibrating lms for question answering our calibration methods can be grouped into two categories: methods that fine-tune lms and posthoc methods that keep lms fixed and only manipulate confidence or inputs. ## conclusion in this paper, we examine the problem of calibration in lms used for qa tasks. we first note that despite the impressive performance state-of-the-art lm-based qa models tend to be poorly calibrated in their probability estimates. to alleviate this problem, we attempted several methods to either fine-tune the lms, or adjust the confidence by post-processing raw probabilities, augmenting inputs, or paraphrasing candidate answers. experimental results demonstrate the effectiveness of these methods. further analysis reveals the challenges of this problem, shedding light on future work on calibrating lms. some future directions could be developing calibration methods for lms on a more finegrained level than simply holistic calibration across the entire dataset. for example, there has been significant interest in how models perform across diverse subsets of the entire training data @xcite and how they reflect dataset biases @xcite , and the interaction of model confidence with these phenomena is of significant interest. it is also interesting to investigate the effect of calibration on users or downstream tasks. for instance, providing users with model confidences can influence downstream decisions @xcite , and users may want to adjust required confidence thresholds on critical domains (e.g., health, safety, medicine). all of these are interesting paths of inquiry for future research.