A Close Look into the Calibration of Pre-trained Language Models (2023)

Abstract: Pre-trained language models (PLMs) may fail in giving reliable estimates of their predictive uncertainty. We take a close look into this problem, aiming to answer two questions: (1) Do PLMs learn to become calibrated in the training process? (2) How effective are existing calibration methods? For the first question, we conduct fine-grained control experiments to study the dynamic change in PLMs’ calibration performance in training. We consider six factors as control variables, including dataset difficulty, available training samples, training steps, the number of tunable parameters, model scale, and pretraining. We observe a consistent change in calibration performance across six factors. We find that PLMs don’t learn to become calibrated in training, evidenced by the continual increase in confidence, no matter whether the predictions are correct or not. We highlight that our finding somewhat contradicts two established conclusions: (a) Larger PLMs are more calibrated; (b) Pretraining improves model calibration. Next, we study the effectiveness of existing calibration methods in mitigating the overconfidence issue. Besides unlearnable calibration methods (e.g., label smoothing), we adapt and extend two recently proposed learnable methods that directly collect data to train models to have reasonable confidence estimations. Experimental results show that learnable methods significantly reduce PLMs’ confidence in wrong predictions.

## introduction pre-trained language models (plms) are successful in many downstream tasks regarding performance @xcite . in high-stake applications, it's equally essential for plms to possess a sense of calibration @xcite ., * equal contribution 0.0 0.8 1.6 2.4 3.2 4.0 steps (×10 3 ) 0 20 40 60 80 100 acc conf ece cerrpos cerrneg under-fitted over-fitted 2019). however, the confidence scores (a.k.a, predictive probability) of existing deep neural networks cannot serve as reliable estimates of their uncertainty @xcite , and a deep understanding of plms calibration is lacking. in this paper, we give a systematical analysis of plms calibration. we consider two questions about plms calibration: (1) do plms learn to become calibrated in the training process? (2) how effective are existing calibration methods? we first introduce the metrics we adopt for calibration performance evaluation. the most widely used calibration metric ece @xcite ) is considered. it measures the difference between confidence and accuracy by portioning samples into various confidence zones. to give a more comprehensive and practical calibration evaluation, we provide an application-driven perspective, describing two undesirable situations in practice: (1) correct predictions (positive) are rejected due to low confidence; for the first question, we consider the influ-ence of six factors on plms' calibration performance, including dataset difficulty, available training samples, training steps, the number of tunable parameters, model scale, and pretraining. some of them are overlooked in previous empirical studies @xcite @xcite . we motivate to conduct finegrained control experiments to study the dynamic change in plms' calibration performance in training through manipulating control variables. we empirically observe an overall consistent change in calibration performance across six factors. all six factors influence plms' fitness on the training distribution. this results in two states of plms considering calibration performance, namely under-fitted and over-fitted states (see fig. 1 ). in the under-fitted state, plms' performance and confidence increase at different speeds when more fitted on the training distribution. in the over-fitted state, plms' confidence continues to increase steadily with little change in performance. we find evidence that plms don't learn to become calibrated in training: plms' confidence in their predictions continues to increase when more fitted on the distribution (e.g., more tunable parameters, training longer). this results in two miscalibration behaviors: (1) increasing ece in the latter over-fitted state, and (2) continually increasing confidence in wrong predictions, indicating that plms mostly don't know "what they don't know". we highlight our finding presents contradictory views with the two established conclusions: (a) larger plms show better calibration @xcite ; (b) pretraining improves model calibration @xcite . we identify that the inconsistency lies in: (1) the difficulty of evaluation datasets: the performance doesn't saturate in the considered datasets (e.g., big-bench @xcite ). thus, the evaluation is on the under-fitted state, leaving the miscalibration behavior in the over-fitted state unobserved; (2) evaluation metrics: previous work doesn't measure the confidence in wrong predictions, overlooking the fact that models are becoming more confident in wrong predictions when scaling larger and employing pretraining. thus, we find that the main issue of plms calibration lies in their overconfidence in wrong predictions, which cannot be trivially solved by increasing the model scale. so we consider the ef-fectiveness of existing calibration methods in mitigating the overconfidence issue. we partition existing calibration methods into unlearnable and learnable groups. unlearnable methods heuristically manipulate the original confidence in predictions (e.g., label smoothing). learnable methods directly collect data and train models to give reasonable confidence scores in their predictions. namely, an extra calibration task is introduced, which aims to extract features from samples and models' preceding performance to predict whether models' predictions are correct or not. in our experiments, we identify the superiority of learnable methods compared to unlearnable ones, considering both in-distribution (id) and out-of-distribution (ood) settings. this is characterized by a sharp decrease in their confidence in wrong predictions when using learnable methods, indicating that they significantly mitigate the overconfidence issue. moreover, learnable methods can maintain a reasonable increase in cerr pos , holding consistent correlations between the drop in confidence and performance under distribution shifts. this shows the difference from unlearnable methods, which take effect by roughly imposing confidence regularization on models' predictions (e.g., label smoothing), resulting in almost the same amount of increase in cerr pos with the decrease in cerr neg . to further understand learnable calibration methods, we consider the influence of more data and larger model scales for the calibration task, the adopted model for the calibration task, and the data distribution, on plms' calibration performance. we highlight three findings: (1) more data and larger model scales for the calibration task both play significant positive roles in plms' calibration performance; (2) plms can be trained to give their uncertainty. this finding is consistent with the concurrent work @xcite . further, we provide an extension to this conclusion. we find that using an extrinsic predictive model can achieve comparable results, given the same calibration training data. thus, we identify that the success of this paradigm essentially lies in the learnable attribute of the calibration task, instead of the plms' self-checking process; (3) plms' calibration performance under distribution shifts depends on the evaluation datasets chosen. previous work shows that plms exhibit degraded calibration performance under distribution shifts @xcite . we find that this conclusion is reversed when the id datasets are harder and plms achieve better performance on ood datasets. the concrete arguments and explanations are detailed in appendix e. ## evaluation metrics for basic evaluation, we report accuracy (acc) and average confidence score (conf) on the testing set. for calibration evaluation, we report ece using equal-mass binning and 100 bins following @xcite . besides, we provide an application-driven perspective to evaluate model calibration, aiming to quantify two unsatisfied scenarios due to miscalibration in practice: (1) correct predictions (positive) are rejected due to low confidence; (2) wrong predictions (negative) are accepted due to high confidence. specifically, we consider the average confidence in correct predictions conf pos and wrong predictions conf neg respectively. for unified comparison, we report two calibration error (cerr) cases, cerr @xmath0. in principle, we expect calibrated models to have both low cerr pos and cerr neg , indicating that they reasonably assign high confidence in correction predictions and low confidence in wrong predictions. ## conclusion we take a close look into plms calibration, motivating to answer two central questions: (1) do plms learn to become calibrated in the training process? (2) how effective are existing calibration methods? we present a comprehensive empirical study, including the analysis of various decisive factors and concrete calibration methods. besides the findings that support existing conclu-sions, we also provide extensions or contradictory arguments to some established conclusions.