On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliency (2022)

Abstract: A well-calibrated neural model produces confidence (probability outputs) closely approximated by the expected accuracy. While prior studies have shown that mixup training as a data augmentation technique can improve model calibration on image classification tasks, little is known about using mixup for model calibration on natural language understanding (NLU) tasks. In this paper, we explore mixup for model calibration on several NLU tasks and propose a novel mixup strategy for pre-trained language models that improves model calibration further. Our proposed mixup is guided by both the Area Under the Margin (AUM) statistic (Pleiss et al., 2020) and the saliency map of each sample (Simonyan et al., 2013). Moreover, we combine our mixup strategy with model miscalibration correction techniques (i.e., label smoothing and temperature scaling) and provide detailed analyses of their impact on our proposed mixup. We focus on systematically designing experiments on three NLU tasks: natural language inference, paraphrase detection, and commonsense reasoning. Our method achieves the lowest expected calibration error compared to strong baselines on both in-domain and out-of-domain test samples while maintaining competitive accuracy.

## introduction training a well-calibrated classifier that produces a match between confidence (the probability output that a model assigns to a prediction) and correctness (accuracy), is important in modern neural networks. as an example, if an ai-based application knows what it does not know, or in other words, the chance that the current prediction is wrong, a human is more helpful to correct the error. however, many works reveal that current deep neural networks are prone to over-confidence, which implies that the models' confidence is not reliable @xcite . this is a critical issue on the deployment of ai-based user applications such as the healthcare domain @xcite or safety-critical domain @xcite due to the problem of prediction trustworthiness. recently, the study of calibration on neural network models especially on natural language processing tasks has started to receive attention. to overcome the problem of miscalibration, numerous suggestions on how to address it have been proposed. for example, @xcite revealed that using temperature scaling before the final softmax layer reduces calibration errors. @xcite , @xcite , and @xcite found that label smoothing and its variants yield better calibration for neural machine translation. @xcite also reported that the aforementioned miscalibration correction methods can be applied to calibrate pre-trained language models which are often miscalibrated potentially due to over-parameterization. mixup @xcite is a data augmentation method for deep neural networks in which additional samples are generated during training by combining random pairs of training inputs and their associated labels. while simple to implement, mixup has been shown to improve both predictive performance and model calibration, particularly on image classification tasks due to its regularization effect through data augmentation @xcite . the recent success of mixup on image classification has led to the development of various mixup strategies for nlu especially those that use hidden state representations @xcite @xcite @xcite . however, most prior works on nlu focus on performance improvement using mixup rather than model calibration. despite its benefits for calibration, a mixup for correcting miscalibrated predictions is still an under-explored topic in nlu. while @xcite explored bert @xcite cali-bration using mixup for both in-domain and outof-domain, they only focused on generating mixup samples by utilizing the distance between instances in the feature space. in contrast, we propose a novel mixup method, in which we first leverage the behavior of a model on individual samples during training (training dynamics), which can reveal samples with distinct pronounced characteristics-whether they are easy-to-learn or hard-to-learn/ambiguous for the model, and then we generate mixup samples by mixing easy-to-learn with hard-to-learn/ambiguous samples according to their similarity/dissimilarity provided by saliency maps. saliency maps capture how much each data portion contributes to the final classification decision of a sample @xcite . intuitively, easy-to-learn samples help with model optimization, whereas hard-to-learn or potentially ambiguous samples are essential for learning since they are the most challenging for the model @xcite , and mixing them using saliency maps can yield better calibrated models (more realistic model confidence), e.g., mixing easy-to-learn with hard-tolearn/ambiguous samples by similarity in saliency maps can benefit in-domain calibration and by dissimilarity can benefit out-of-domain calibration. to monitor training dynamics, we use the area under the margin (aum) statistic @xcite which measures how different a true label for a sample is compared to a model's beliefs at each epoch and is calculated as the average difference between the logit values for a sample's assigned class and its highest non-assigned class across training epochs. moreover, we combine our mixup with wellknown miscalibration correction methods such as label smoothing and temperature scaling @xcite to investigate their impact on our proposed mixup. we conduct a comprehensive set of experiments using bert @xcite and roberta @xcite to show the efficacy of our mixup approach by testing on three nlu tasks: natural language inference, paraphrase detection, and commonsense reasoning. we achieve the lowest expected calibration error (ece) without accuracy drops in comparison with strong baseline methods. our contributions are as follows: model calibration calibration on nlu tasks has been widely studied in related literature. nguyen and o'connor (2015) provided the method of how to analyze the calibration of non-neural nlp models. guo et al. (2017) examined the calibration of modern deep neural networks and revealed that techniques such as temperature scaling and dropout affect the calibration on binary/multiclass classification tasks. wang et al. (2020b) investigated the calibration of neural machine translation models and found that inference suffers from serious miscalibration. jagannatha and yu (2020) demonstrated that neural networks show high calibration error on structured predictions such as ner, pos, and qa, and proposed to use a binary class forecaster to calibrate the predictor confidence for a defined output entity of interest. desai and durrett (2020) explored pre-trained language models' calibration in combination with temperature scaling and label smoothing both on in-domain and out-of-domain datasets. jung et al. ( 2020 ) jointly optimized two objectives (a cross-entropy loss and a calibration loss) and directly penalized the difference between the predicted and the true posterior probabilities dynamically over the training steps. @xcite obtained better calibration on natural language understanding tasks by augmenting and training the classifier jointly with an energybased model using noise-contrastive estimation. mixup mixup @xcite is a method for data augmentation in which additional samples are generated during training by convexly combining random pairs and their associated labels, and aims to alleviate overfitting. @xcite showed that manipulating hidden representations rather than manipulating input-level features on mixup results in better regularization effects due to the fact that it encourages the neural network to focus more on representations of the real training examples in a low dimensional subspace. many works have empirically noticed regularization effects that improve model performance on deep neural networks. for example, @xcite explored the nlu specific mixup strategy by using sentence and word embeddings on cnns and lstms to add performance gains in supervised text classification. @xcite proposed mixup for semi-supervised learning in which labeled and unlabeled samples are interpolated with their hidden representations to improve the performance of text classification. @xcite explored mixup for sequence labeling tasks with active learning to improve the performance of supervised sequence labeling tasks. @xcite proposed mixup that interpolates every instance in a mini-batch to boost the performance of nlu tasks on the pre-trained language model roberta @xcite . similar to us, @xcite explored mixup by incorporating saliency signals to generate augmented samples. precisely, they use saliency signals to select a span of text from one sample to be replaced with another text span from another sample. however, in contrast, our method first divides data samples into two categories (easyto-learn and hard-to-learn/ambiguous categories) according to their aum @xcite distribution monitored over training epochs and then uses saliency to find the most similar/dissimilar samples across these two data categories. recently, several works started to explore mixup for nlu model calibration. for example, @xcite investigated the impact of mixup for model calibration of nlu but only explored in-domain settings with simple deep learning architecture such as cnns. @xcite explored bert calibration using mixup as a regularization component on in-domain and out-of-domain. however, their mixup method only relied on the feature space distance between samples. in contrast, we explore a novel mixup method in which we categorize the training samples into two sets using aum @xcite and combine samples across these two sets based on saliency signals, for in-domain and out-of-domain model calibration. ## conclusion we proposed a novel mixup guided by the area under the margins (aum) and saliency maps to mitigate the miscalibration of pre-trained language models bert and roberta. we showed that our proposed mixup method achieves the lowest expected calibration errors (eces) for both pretrained language models on various types of natural language understanding tasks, for both in-domain and out-of-domain data. for future work, we will enhance our proposed mixup further, focusing not only on model calibration but also on performance gains. exploring different saliency maps for computing sample similarity/disimilarity (and its degree) is another interesting future direction.