Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data (2020)

Abstract: Fine-tuned pre-trained language models can suffer from severe miscalibration for both in-distribution and out-of-distribution (OOD) data due to over-parameterization. To mitigate this issue, we propose a regularized fine-tuning method. Our method introduces two types of regularization for better calibration: (1) On-manifold regularization, which generates pseudo on-manifold samples through interpolation within the data manifold. Augmented training with these pseudo samples imposes a smoothness regularization to improve in-distribution calibration. (2) Off-manifold regularization, which encourages the model to output uniform distributions for pseudo off-manifold samples to address the over-confidence issue for OOD data. Our experiments demonstrate that the proposed method outperforms existing calibration methods for text classification in terms of expectation calibration error, misclassification detection, and OOD detection on six datasets. Our code can be found at https://github.com/Lingkai-Kong/Calibrated-BERT-Fine-Tuning .

## introduction pre-trained language models have recently brought the natural language processing (nlp) community into the transfer learning era. the transfer learning framework consists of two stages, where we first pre-train a large-scale language model, (e.g., bert @xcite , roberta @xcite , albert @xcite and t5 @xcite ) on a large text corpus and then finetune it on downstream tasks. such a fine-tuning approach has achieved sota performance in many nlp benchmarks @xcite . many applications, however, require trustworthy predictions that need to be not only accurate but also well calibrated. in particular, a wellcalibrated model should produce reliable confi-figure 1 : the reliability diagrams on in-distribution data (the first row) and the histograms of the model confidence on out-of-distribution (ood) data (the second row) of cnn @xcite and fine-tuned bert-mlp classifier @xcite . though bert improves classification accuracy, it makes over-confident predictions for both in-distribution and ood data. unfortunately, @xcite have shown that due to over-parameterization, deep convolutional neural networks are often miscalibrated. our experimental investigation further corroborates that fine-tuned language models can suffer from miscalibration even more for nlp tasks. as shown in figure 1 , we present the calibration of a bert-mlp model for a text classification task on the 20ng dataset. specifically, we train a textcnn @xcite and a bert-mlp using 20ng 15 (the first 15 categories of 20ng) and then evaluate them on both in-distribution and ood data. the first row plots their reliability diagrams @xcite on the test set of 20ng 15 . though bert improves the classification accuracy from 83.9% to 87.4%, it also increases the expected calibration error (ece, see more details in section 2) from 4.0% to 9.5%. this indicates that bert-mlp is much more miscalibrated for in-distribution data. the second row plots the histograms of the model confidence, i.e., the maximum output probability, on the test set of 20ng 5 (the unseen 5 categories of 20ng). while it is desirable to produce low probabilities for these unseen classes, bert-mlp produces wrong yet overconfident predictions for such ood data. such an aggravation of miscalibration is due to the even more significant over-parameterization of these language models. at the pre-training stage, they are trained on a huge amount of unlabeled data in an unsupervised manner, e.g., t5 is pre-trained on 745 gb text. to capture rich semantic and syntactic information from such a large corpus, the language models are designed to have enormous capacity, e.g., t5 has about 11 billion parameters. at the fine-tuning stage, however, only limited labeled data are available in the downstream tasks. with the extremely high capacity, these models can easily overfit training data likelihood and be over-confident in their predictions. to fight against miscalibration, a natural option is to apply a calibration method such as temperature scaling @xcite in a post-processing step. however, temperature scaling only learns a single parameter to rescale all the logits, which is not flexible and insufficient. moreover, it cannot improve out-of-distribution calibration. a second option is to mitigate miscalibration during training using regularization. for example, @xcite propose an entropy regularizer to prevent over-confidence, but it can needlessly hurt legitimate high confident predictions. a third option is to use bayesian neural networks @xcite , which treat model parameters as probability distributions to represent model uncertainty explicitly. however, these bayesian approaches are often prohibitive, as the priors of the model parameters are difficult to specify, and exact inference is intractable, which can also lead to unreliable uncertainty estimates. we propose a regularization approach to addressing miscalibration for fine-tuning pre-trained language models from a data augmentation perspective. we propose two new regularizers using pseudo samples both on and off the data manifold to mitigate data scarcity and prevent over-confident predictions. specifically, our method imposes two types of regularization for better calibration during fine-tuning: (1) on-manifold regularization: we first generate on-manifold samples by interpolating the training data and their corresponding labels along the direction learned from hidden feature space; training over such augmented on-manifold data introduces a smoothness constraint within the data manifold to improve the model calibration for in-distribution data. (2) off-manifold regularization: we generate off-manifold samples by adding relatively large perturbations along the directions that point outward the data manifold; we penalize the negative entropy of the output distribution for such off-manifold samples to address the overconfidence issue for ood data. we evaluate our proposed model calibration method on six text classification datasets. for indistribution data, we measure ece and the performance of misclassification detection. for outof-distribution data, we measure the performance of ood detection. our experiments show that our method outperforms existing state-of-the-art methods in both settings, and meanwhile maintains competitive classification accuracy. we summarize our contribution as follows: (1) we propose a general calibration framework, which can be applied to pre-trained language model finetuning, as well as other deep neural network-based prediction problems. (2) the proposed method adopts on-and off-manifold regularization from a data augmentation perspective to improve calibration for both in-distribution and ood data. (3) we conduct comprehensive experiments showing that our method outperforms existing calibration methods in terms of ece, miscalssification detec-tion and ood detection on six text classification datasets. ## preliminaries we describe model calibration for both indistribution and out-of-distribution data. calibration for in-distribution data: for indistribution data, a well-calibrated model is expected to output prediction confidence comparable to its classification accuracy. for example, given 100 data points with their prediction confidence 0.6, we expect 60 of them to be correctly classified. more precisely, for a data point x, we denote by y (x) the ground truth label, ŷ (x) the label predicted by the model, and p (x) the output probability associated with the predicted label. the calibration error of the predictive model for a given confidence @xmath0, 1) is defined as: as (1) involves population quantities, we usually adopt empirical approximations @xcite to estimate the calibration error. specifically, we partition all data points into m bins of equal size according to their prediction confidences. let b m denote the bin with prediction confidences bounded between ℓ m and u m . then, for any @xmath1, u m ), we define the empirical calibration error as: to evaluate the overall calibration error of the predictive model, we can futher take a weighted average of the calibration errors of all bins, which is also known as the expected calibration error (ece) @xcite defined as: ## calibrated fine-tuning via manifold smoothing we consider n data points of the target task @xmath2, y i )} n @xmath3, where x i 's denote the input embedding of the sentence and y i 's are the associated onehot labels. let f (•) denote the feature extraction layers (e.g., bert); let g(•) denote the task-specific layer; and let θ denote all parameters of f and g. we propose to optimize the following objective at the fine-tuning stage: ## experiments to evaluate calibration performance for indistribution data, we measure the expected calibration error (ece) and the misclassification detection score. for out-of-distribution data, we measure the ood detection score. we detect the misclassified and ood samples by model confidence, which is the output probability associated with the predicted label p (x). specifically, we setup a confidence threshold τ ∈ [0, 1], and take the samples with confidence below the threshold, i.e., p (x) < τ , as the misclassified or ood samples. we can compute the detection f 1 score for every τ : f 1 (τ ), and obtain a calibration curve (f 1 (τ ) vs. τ ). then, we set τ upper as the upper bound of the confidence threshold, since a well calibrated model should provide probabilities that reflect the true likelihood and it is not reasonable to use a large τ to detect them. we use the empirical normalized bounded area under the calibration curve (nbaucc) as the overall detection score: ## conclusion we have proposed a regularization method to mitigate miscalibration of fine-tuned language models from a data augmentation perspective. our method imposes two new regularizers using generated on-and off-manifold samples to improve both in-distribution and out-of-distribution calibration. extensive experiments on six datasets demonstrate that our method outperforms state-of-the-art calibration methods in terms of expected calibration error, misclassification detection and ood detection.