Making Pre-trained Language Models both Task-solvers and Self-calibrators (2023)

Abstract: Pre-trained language models (PLMs) serve as backbones for various real-world systems. For high-stake applications, it’s equally essential to have reasonable confidence estimations in predictions. While the vanilla confidence scores of PLMs can already be effectively utilized, PLMs consistently become overconfident in their wrong predictions, which is not desirable in practice. Previous work shows that introducing an extra calibration task can mitigate this issue. The basic idea involves acquiring additional data to train models in predicting the confidence of their initial predictions. However, it only demonstrates the feasibility of this kind of method, assuming that there are abundant extra available samples for the introduced calibration task. In this work, we consider the practical scenario that we need to effectively utilize training samples to make PLMs both task-solvers and self-calibrators. Three challenges are presented, including limited training samples, data imbalance, and distribution shifts. We first conduct pilot experiments to quantify various decisive factors in the calibration task. Based on the empirical analysis results, we propose a training algorithm LM-TOAST to tackle the challenges. Experimental results show that LM-TOAST can effectively utilize the training data to make PLMs have reasonable confidence estimations while maintaining the original task performance. Further, we consider three downstream applications, namely selective classification, adversarial defense, and model cascading, to show the practical usefulness of LM-TOAST.

## introduction we have witnessed the great success of pre-trained language models (plms) over the past few years in various tasks @xcite . nowadays, real-world natural language processing (nlp) systems are mostly built upon plms to effectively utilize their strong capacities @xcite . beyond the performance evaluation, an essential requirement in high-stake applications for plms is to assign reasonable confidence to their predictions. this can enable the decision-makers to better handle the low-confident predictions, e.g., directly abstain to give predictions or transfer the input to human experts. the original predictive probability of plms can be effectively utilized for ranking predictions. this simple strategy can reasonably give relatively higher confidence to correct predictions compared to the wrong ones @xcite . however, relying on the vanilla confidence scores cannot well distinguish between correct and wrong predictions. plms consistently assign high confidence in their predictions, no matter correct or not @xcite . this results in a large number of wrong predictions distributed in the high-confident zone (see figure 1 ). the direct undesirable consequence is the false acceptance of wrong but high-confident predictions. besides, previous work avoids the issue of selecting a concrete confidence threshold by using hyperparameter-free metrics (e.g., auroc) in relevant tasks (e.g., selective classification). but in practice, the small gap between confidence in correct and wrong predictions may cause large performance variance due to the manually chosen threshold. existing work shows that an extra calibration task can be taken as a remedy @xcite . the calibration task uses extra samples to train models to have reasonable confidence estimations. however, previous work considers ideal situations to demonstrate the feasibility, assuming access to a large number of unused labeled samples, typically from the validation set. in practice, the samples in the validation dataset may be too small to guarantee good calibration performance. besides, relying on the validation samples for the calibration task training causes data leakage, which may result in unreliable performance estimation when adopting the validation dataset to choose hyper-parameters. in practice, we need to effectively utilize training samples for both the original and the calibration tasks training. three challenges are presented: • limited training samples: how to effectively utilize the training samples to increase the calibration task performance while maintaining the original task performance? then we employ data down-sampling, adversarial data augmentation, and consistent training to tackle the challenges of data imbalance and distribution shifts. note that lm-toast can be applied to all classification tasks to improve confidence estimations. experimental results show that lm-toast can increase the discrimination between correct and wrong predictions, evidenced by the fine-grained order ranked by confidence and a sharp difference in average confidence. further, we show that the good self-calibration performance of lm-toast can be transferred to downstream applications. we consider three downstream tasks, including selective classification @xcite , adversarial defense @xcite , and model cascading @xcite . experimental results demonstrate the practical significance of lm-toast in these applications. ## pilot experiments and analysis we conduct pilot experiments to quantify the influence of several decisive factors in the calibration task, which can help for a better design of various components in the training algorithm. specifically, we consider the number of training samples, dataset imbalance, and input "features". the concrete experimental settings are described in appendix c. ## method based on our empirical analysis summarized in appendix c, we motivate a practical training algorithm to make plms both task-solvers and training data subset-1 subset-2 subset-k . . . k-split divide training set held-out set training calibration data annotate & add repeat k times calibration data down-sampled calibration data down-sample majority class augmented data augment minority class down-sampled calibration data training data normal task training calibration task training consistent training augmented data self-calibrators (lm-toast ). lm-toast can be divided into three stages (see figure 4): (1) generate calibration training data from the given training dataset; (2) post-process generated calibration training data; (3) multi-task training using the original training data and the processed calibration training data. we follow the notations in sec. 2.1. generate calibration training data. we propose the k-fold cross-annotation to generate the calibration dataset from the original training samples. we first split the original training dataset into k subsets equally, and perform k-rounds annotation. for each round, we leave one subset out and train the model on the remaining k-1 subsets. then we use the trained model to annotate the held-out set. specifically, for each sample in the held-out set, we obtain the model's prediction and compare it with the golden label. a binary annotated label is obtained, indicating whether the prediction is correct or not. after k-rounds annotation, we can generate a calibration training dataset d * with the size equal to the original training dataset's size. we empirically set @xmath0. we justify this setting in the further analysis of lm-toast (see appendix d). note that due to the strong capacity of plms, there exists a significant data imbalance issue in d * , where positive cases dominate the distribution. thus, post-processing the generated calibration data is needed to make full use of d * . ## experiments we conduct experiments to demonstrate the effectiveness of lm-toast in confidence estimations. we run all experiments three times and report both the average performance and the standard variance. ## applications we consider applying lm-toast for three tasks, namely selective classification @xcite yaniv, 2017), adversarial defense @xcite , and model cascading @xcite . the baseline methods are the same as in sec. 5 ## conclusion we present the task-agnostic lm-toast to make plms have reasonable confidence estimations while maintaining the original task performance. we also show that its good self-calibration can be transferred to downstream applications.