P RESERVING P RE - TRAINED F EATURES H ELPS C ALIBRATE F INE - TUNED L ANGUAGE M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Large pre-trained language models (PLMs) have demonstrated strong performance on natural language understanding (NLU) tasks through ﬁne-tuning.
However, ﬁne-tuned models still suffer from overconﬁdent predictions, especially in out-of-domain settings.
In this paper, we tackle the problem of calibrating ﬁnetuned language models.
We demonstrate that the PLMs are well-calibrated on the masked language modeling task with robust predictive conﬁdence under domain shift, yet the ﬁne-tuned models fail to retain such property due to catastrophic forgetting, which impacts the calibration on the downstream classiﬁcation task.
In light of these observations, we evaluate the calibration of several methods that preserve pre-trained features and show that preserving pre-trained features can improve the calibration of ﬁne-tuned language models.
Among these methods, our proposed method that encourages the ﬁne-tuned model to learn generative representations with auxiliary language modeling objective achieves competitive accuracy and the lowest expected calibration error compared to several strong baselines under both in-domain and out-of-domain settings on three downstream
NLU tasks.
I NTRODUCTION
Fine-tuning pre-trained language models (PLMs) is a dominating paradigm for natural language understanding (NLU) with state-of-the-art results for a variety of NLU tasks (Peters et al., 2018;
Devlin et al., 2019; Liu et al., 2019; He et al., 2021a).
The powerful ﬁne-tuned language models have been experimented with for decision-making in real-world applications such as the healthcare domain (He et al., 2020) and safety-critical domain (Sandagiri et al., 2020), where the classiﬁcation networks need to be highly accurate and provide calibrated conﬁdence for their predictions to improve the safety and trustiness of the models (Guo et al., 2017).
For example, suppose a medical language inference LM that predicts the disease given the description of symptoms is well-calibrated, i.e., the model’s posterior probabilities (or conﬁdence) align well with the true correctness likelihood.
In that case, the wrong predictions can be easier to detect and correct by human doctors by given low predictive conﬁdence.
However, as with other modern neural networks, the ﬁne-tuned LMs are shown to suffer from overconﬁdence (Desai & Durrett, 2020; Jiang et al., 2021), which creates obstacles and concerns for their deployment in real-world applications.
Uncertainty estimation of ﬁne-tuned models is challenging due to the small amount of available data for ﬁne-tuning.
especially under out-of-domain settings (Desai & Durrett, 2020; Guo et al., 2021).
While prior works illustrate that simple calibration techniques such as temperature scaling and label smoothing are not sufﬁcient to calibrate the ﬁne-tuned LMs under both in-domain (ID) and outof-domain (OD) settings (Desai & Durrett, 2020; Park & Caragea, 2022), several approaches with strong regularization have been developed to calibrate the ﬁne-tuned model on NLU tasks, including knowledge distillation from deep ensembles (Guo et al., 2021), stochastic network architectures (Fan et al., 2020; Zhang et al., 2021), and Mixup (Park & Caragea, 2022).
However, these existing works mostly utilize general calibration methods for supervised learning, while speciﬁc properties of the pre-training & ﬁne-tuning paradigm are still largely neglected.
In this work, we tackle the calibration of ﬁne-tuned models from a different angle.
Through a carefully designed empirical study on the calibration of both pre-trained and ﬁne-tuned models, we ﬁrst observe that PLMs themselves are actually well-calibrated on the masked language modeling
(MLM) task and robust to higher levels of perturbation to the inputs, which suggests the PLMs can model the predictive uncertainty well across different domains.
However, the pre-trained features are only used as initialization and distorted by the fully discriminative ﬁne-tuning, known as catastrophic forgetting (McCloskey & Cohen, 1989; Kirkpatrick et al., 2017; Howard & Ruder, 2018).
Such forgetting makes the ﬁne-tuned language models fail to hold proper predictive conﬁdence toward the OD and outlier samples, which results in miscalibration on the downstream tasks.
Based on the observations, we hypothesize that preserving the pre-trained features helps calibrate the ﬁne-tuned
LMs.
To validate this hypothesis, we experiment with a series of methods that preserve the pre-trained features: (1) Parameter-efﬁcient tuning, (2) Regularization with pre-trained weight, (3) Joint learning with MLM loss.
The empirical results in both ID and OD settings on three NLU tasks, including natural language inference, paraphrase detection, and commonsense reasoning, demonstrate that these methods outperform vanilla ﬁne-tuning in calibration, especially under out-of-domain settings.
Among these methods, our proposed method that utilizes the MLM objective to maintain the consistency between the pre-trained and ﬁne-tuned models achieves the lowest expected calibration error and competitive accuracy compared to existing calibration methods in both ID and OD settings on three NLU tasks, showing that preserving the pre-trained features is an effective regularization for improving the calibration of ﬁne-tuned LMs.
P RELIMINARIES 2. M ASKED L ANGUAGE M ODELS
Masked language models generally consist of a transformer-based text encoder f φ parameterized by φ and a linear language modeling head g θ parameterized by θ .
In the pre-training phase, the model handles the masked language modeling task (Devlin et al., 2019).
Assume we have unsupervised sequence inputs x sampled from large-scale corpora p u ( x ) .
A subset of x is ﬁrst masked by a corruption function or distribution.
Denote the indices of the masked tokens as M , the set of masked tokens as x M , and the observed unmasked input as x \ M .
The model is trained to recover the masked tokens x M .
In particular, the masked language model ﬁrst uses the text encoder to get a hidden representation of the input, denoted as f φ ( x \ M ) .
Then the language modeling head g θ with softmax function is applied to f φ ( x \ M ) to obtain a conditional categorical distribution p mlm ( x i | x \ M ) over the vocabulary V for each masked position i ∈ M .
The masked language modeling objective is:
[Equation]
In the ﬁne-tuning phase, assume we have samples in the form of ( x , y ) sampled from data distribution p d , where x corresponds to the text input, and y corresponds to the label.
For classiﬁcation tasks, a task-speciﬁc head h ϕ that parameterized by ϕ is applied on the hidden representation of input to obtain the logit for each class.
The predictive posterior distribution q ( y | x ) is given by the logits after softmax operation.
In standard ﬁne-tuning, the pre-trained encoder f φ and the task-speciﬁc head h ϕ are jointly optimized using the cross-entropy loss:
[Equation]
which is also known as full ﬁne-tuning (Full-FT).
2. C ONFIDENCE C ALIBRATION
The framework of conﬁdence calibration under the supervised classiﬁcation setting can be expressed as a joint distribution P (ˆ y, ˆ p ) over the label prediction ˆ y ∈ |Y| and the corresponding conﬁdence ˆ p ∈ [0 , 1] .
A perfectly calibrated model holds P (ˆ y =
y | ˆ p = p ) = p .
One way to evaluate calibration through ﬁnite samples is expected calibration error, i.e., ECE (Naeini et al., 2015; Guo et al., 2017).
The model’s predictive conﬁdences are ﬁrst grouped into M equal-sized bins.
Denote B m as the indices of samples whose conﬁdences are in the interval ( m − 1 M , mM ] .
Suppose we have N samples, the ECE is calculated by the weighted average of the difference between conﬁdence and accuracy in each bin:
[Equation]
In this work, we set M = 10 following Desai & Durrett (2020).
A C LOSER L OOK TO THE P RE - TRAINED AND F INE - TUNED L ANGUAGE M ODELS IN C ALIBRATION
In this section, we explore the connection between the pre-trained and ﬁne-tuned language models in terms of calibration by examining: (1) The calibration of the pre-trained language models themselves.
(2) How ﬁne-tuning affects calibration on the downstream classiﬁcation tasks.
First, we evaluate the calibration of PLMs on the MLM task to investigate whether the pre-trained models hold calibrated and robust predictive conﬁdences p mlm ( x | x \ M ) after pre-training with large corpora, which may beneﬁt the calibration on downstream tasks.
Then we illustrate the effect of ﬁne-tuning on the pre-trained features and study how the changes in the pre-trained features during ﬁne-tuning affect the model’s predictive conﬁdence q ( y | x ) on downstream classiﬁcation tasks.
[Equation] [Table] [Equation] [Table]
[Equation]
of on task.
We use the test split of seven datasets with different masking probabilities.
For each dataset, we sample the masked inputs with p mask ﬁve times and report the averaged ECE.
We also report the corresponding standard deviation in subscripts.
[Figure] [Table] [Figure] [Table]
[Figure]
Setup:
We ﬁrst evaluate the calibration of the pre-trained RoBERTa BASE (Liu et al., 2019) on the
MLM task.
We use the test split of the WikiText-103 (Merity et al., 2016) and six datasets from different domains (see §5.1 and A.1 for details) as sequence inputs for masked language modeling.
The inputs are masked and corrupted with three levels of 15%, 30%, and 50% mask probability with the same approach (i.e., 80-10-10 strategy) as in the pre-training phase (Devlin et al., 2019).
We also visualize the hidden representation of some sampled inputs from different datasets given by the pre-trained language model using t-SNE (Van der Maaten & Hinton, 2008).
Results and Analysis:
As shown in Table 1 and Figure 1, the pre-trained language model is relatively well-calibrated across different domains on the MLM task, where the model needs to recover the corrupted position with the options of | V | .
Moreover, as the mask probability grows higher value than in the pre-training phase, the ECE of the PLM increases only by a relatively small amount, which indicates that the PLM’s predictive conﬁdence p mlm ( x | x \ M ) on the MLM task is robust to higher corruption levels.
Figure 2 demonstrates that the hidden representations of 50% masked inputs have shifted signiﬁcantly from the original input, and the PLM can still make calibrated predictions to the original inputs.
Intuitively, the calibrated conﬁdence on the MLM task suggests that the pre-trained features of PLMs are good at modeling the samples under large domain shifts, which may beneﬁt the calibration of the downstream classiﬁcation task under OD settings.
However, this property is less likely to be retained by the ﬁne-tuned LM due to catastrophic forgetting caused by full ﬁne-tuning with the discriminative objective (Howard & Ruder, 2018).
[Figure] [Figure]
3. H OW F INE - TUNING A FFECTS C ALIBRATION ON THE D OWNSTREAM T ASKS
Setup: To study why ﬁne-tuned LMs are miscalibrated under the OD settings, we focus on the
LM ﬁne-tuned on the QQP dataset, which is a typical failure case that the ﬁne-tuned model exhibits dramatic disparity in ECE on ID and OD settings, as shown in Figure 3.
We ﬁne-tune the pre-trained RoBERTa BASE on the QQP training set following the default conﬁguration of the Huggingface Transformers library (Wolf et al., 2020).
Compared to the pre-trained model, we visualize the hidden representations of the same inputs with §3.1 given by the ﬁne-tuned model.
Besides, we evaluate the average conﬁdence of the ﬁne-tuned model’s prediction on several datasets, including the in-domain QQP validation set, the out-of-domain TwitterPPDB validation set, and the outlier
WikiText-103 validation set that does not hold any particular attributes of the downstream classiﬁcation task.
Results
and Analysis:
As shown in Figure 2, compared to the pre-trained models, ﬁne-tuning changes the hidden representation of the LM in two ways: (1) For the inputs within the same domain, ﬁne-tuning enlarges the difference of the corresponding hidden representations, which aligns with the quantitative results of the previous work (Zhou & Srikumar, 2022).
(2) For the inputs across different domains, ﬁne-tuning makes the hidden representations from different domains much harder to distinguish by projecting them to a simpler data manifold, which causes the ﬁne-tuned model fails to give proper predictive conﬁdence for OD and outlier samples.
As shown in Figure 3, the average predictive conﬁdence of the ID, OD, and outlier validation sets increases as the training step increases.
The gap between the average predictive conﬁdence and the correctness likelihood (i.e., classiﬁcation accuracy) in the OD setting is relatively larger than in the
ID setting, which results in a larger OD ECE.
More crucially, the average conﬁdence of the outlier samples is higher during the whole training process than both ID and OD settings and increases to nearly 100% after three training epochs, which can not be ﬁxed by simple techniques such as temperature scaling and early stopping.
Ideally, the model should be uncertain about the outlier samples signiﬁcantly deviating from the training samples.
However, the ﬁne-tuned LM exhibits overconﬁdence toward the OD and outlier samples, which implies that strong regularization methods are needed to improve the conﬁdence modeling for OD and outlier samples.
Based on the observations that the pre-trained features of the PLMs can model the predictive uncertainty well across different domains and are distorted by the ﬁne-tuned LMs in §3.1, we hypothesize that preserving the pre-trained features of PLMs helps the ﬁne-tuned LMs better model the predictive conﬁdence and improve calibration on downstream classiﬁcation tasks.
M ETHODS
To validate the hypothesis that preserving the pre-trained features helps the calibration of ﬁne-tuned
LMs, we examine some existing methods that can preserve the pre-trained features in different ways compared to vanilla ﬁne-tuning and evaluate their performance in calibration in §5.
4. P ARAMATER -E FFICIENT T UNING
Parameter-efﬁcient tuning is a series of ﬁne-tuning methods which keep the pre-trained parameters of the text encoder
φ frozen and update only a small number of extra parameters φ ∆ and the task-speciﬁc head h ϕ while preserving competitive performance with Full-FT.
Since the pre-trained knowledge is encoded to the model’s parameters and there are only a small amount of extra parameters, parameter-efﬁcient tuning methods are able to preserve the pre-trained features.
In this work, we choose three mainstream parameter efﬁcient tuning methods: (1) Adapter (Houlsby et al., 2019), which adds a light-weight bottleneck module after the output of each sub-layer in the transformer block; (2) LoRA
(Hu et al., 2021), which updates the attention weight matrix using low-rank reparameterization; (3) Preﬁx Tuning (Li & Liang, 2021), which prepends tunable preﬁx vectors to keys and values of the multi-head attention layers.
4. R EGULARIZATION WITH P RE - TRAINED W EIGHT
Introducing regularization terms using the pre-trained weight can also better leverage the pre-trained features during ﬁne-tuning.
In this work, we adopt two common regularization techniques:
Weight decay towards pre-trained weight: Traditional weight decay methods add a regularization term λ 2 || w || 2 that penalizes large weights to improve generalization (Krogh & Hertz, 1991), where λ is a regularization coefﬁcient.
As an alternative, performing weight decay towards the pre-trained weight w 0 by adding λ 2 || w − w
|| 2 to the task loss function is shown by previous works as an effective way to mitigate catastrophic forgetting caused by ﬁne-tuning (Wiese et al., 2017) and can improve the performance of the downstream task (Chen et al., 2020).
Mixout: To explicitly prevent the deviation from the pre-trained weight w 0 , Lee et al.
propose Mixout that stochastic replaces the model parameters to their pre-trained counterparts with probability p at each training iteration, which has been shown to improve the stability of ﬁne-tuning and enhance the performance of the ﬁne-tuned LMs on downstream task.
4. J OINT L EARNING WITH MLM O BJECTIVE
Besides ﬁxing or constraining the model parameters to the pre-trained counterpart, we can also enforce the consistency between the pre-trained and ﬁne-tuned LMs by utilizing the MLM objective.
Previous works have demonstrated that using the MLM objective before or during the ﬁne-tuning process yields better performance on the downstream tasks (Sun et al., 2019; Wiedemann et al., 2020; Ma et al., 2021).
In this work, in order to better preserve the pre-trained features, we jointly optimize the MLM objective while ﬁne-tuning:
[Equation]
where α is the scaling factor of the MLM loss.
In addition to introducing the MLM objective, we propose three simple techniques to further strengthen the connection between the pre-trained model and our ﬁne-tuned model:
Utilizing corpus of the pre-training phase: The dataset for the MLM task is not required to be labeled as for the downstream task, so we can choose any text dataset.
In this work, we present two criteria for selecting p u ( x ) of the MLM task.
The ﬁrst one uses the dataset of downstream tasks, referred to as JL-D.
The second one introduces the corpus of the pre-training phase, referred to as JL-P.
We suppose that using the corpus in the pre-training phase is helpful for preserving the pre-trained features.
Distillation from the pre-trained model: Knowledge distillation (Hinton et al., 2015) has been shown to be an effective technique to reap the beneﬁts of a powerful teacher model.
In addition to preserving the raw quality, Guo et al.
(2021) illustrate that the calibration performance of the teacher model can be distilled into the student model and propose to distill from an ensemble of ﬁne-tuned LMs for better calibration.
Inspired by this, we perform knowledge distillation from the pre-trained language model when performing the MLM task.
Specifically, instead of calculating the MLM loss using the original text as hard labels, we use the
KL-divergence D KL ( p mlm ( x i | x \ M ; φ 0 , θ 0 ) (cid:107) p mlm ( x i | x \ M ; φ, θ )) between the predictive distribution p mlm ( x i
| x \ M ; φ 0 , θ 0 ) of the pre-trained language model and our model.
Regularization on the contextualized representation: Zhou & Srikumar (2022) validate that ﬁnetuning enlarges the distance between samples with different labels.
This behavior may increase the deviation of the ﬁne-tuned model from the pre-trained model.
To address this problem, we introduce a heuristic regularization adding the L 2 norm of the contextualized representation to L joint with regularization coefﬁcient β .
Following previous works (Desai & Durrett, 2020; Park & Caragea, 2022), we also apply label smoothing (Szegedy et al., 2016) on the classiﬁcation task, which can mitigate overconﬁdent predictions by distributing a σ fraction of probability mass of the ground-truth label equally to other non-ground-truth classes.
E XPERIMENTS 5. G ENERAL S ETUP
Datasets: We conduct experiments on three natural language understanding tasks: natural language inference (NLI), paraphrase detection (PD), and commonsense reasoning (CR).
Each task consists of a pair of in-domain (ID) and out-of-domain (OD) datasets.
Speciﬁcally, SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) are ID and OD datasets for NLI; QQP (Shankar et al., 2017) and TwitterPPDB (Lan et al., 2017) are ID and OD datasets for PD; SWAG (Zellers al.,
We use the same train/validation/test split for those six datasets published by Desai & Durrett (2020).
For each task, we ﬁne-tune the model using the ID training set and evaluate the model’s performance with both ID and OD test sets.
We use WikiText-
(Merity et al., 2016) as the corpus of the pre-training phase for JL-P.
The detailed statistics for each dataset can be found in Appendix A.1.
Setup:
For parameter-efﬁcient tuning methods, we use the default hyperparameter conﬁguration provided by OpenDelta (Ding et al., 2022) library for all three methods and conduct a grid search for learning rates on different tasks.
The detailed setup for each method can be found in Appendix A.2.1.
For all ﬁne-tuning methods, we set a learning rate of 1e-5 and batch size of 32 across all three tasks as Desai & Durrett (2020) does 1 .
For weight decay towards pre-trained weight (Pre-trained
Weight Decay), we adopt the implementation of the RecAdam (Chen et al., 2020), which integrates the quadratic penalty between the model parameters and the pre-trained parameters into the Adam optimizer (Kingma & Ba, 2015).
We use the constant annealing function of RecAdam and tune the balancing coefﬁcient λ pwd for the classiﬁcation objective and the pre-trained weight decay term.
For Mixout, we tune the mixout probability p mixout .
For joint learning methods, we use a Bernoulli corruption distribution 2 .
We conduct hyperparameter search for the mask probability p mask , the scaling factor α mlm of the MLM loss, and the regularization coefﬁcient β L 2 on the contextualized representation.
We also tune the hyperparameter σ ls for label smoothing (LS).
We search all the hyperparameters on the validation set of each task independently.
Completed setup details for each method on each task can be found in Appendix A.2.
All experiments are run for 3 training epochs and are deployed on a single NVIDIA
A40 48G GPU within 3 hours to ﬁne-tune a single model.
Evaluation:
In this section, we use pre-trained RoBERTa BASE (Liu et al., 2019) for all experiments.
For each NLU task, we report accuracy and expected calibration error (ECE) on both ID and OD test sets.
We evaluate “out-of-the-box” calibration of our method, which does not apply any post-hoc calibration methods such as temperature scaling (Guo et al., 2017).
5. M AIN R ESULTS
We summarize our results in Table 2.
Our baselines include full ﬁne-tuning and two advanced methods based on the pre-trained language models: Bayesian Attention Belief Networks (BABN) (Zhang et al., 2021) and Mixup (Park & Caragea, 2022).
We copied the results of the same setting from the original paper.
We also report our full ﬁne-tuning runs following the default behavior of the training scripts provided by the Huggingface Transformers library.
We present our experimental results for
Pre-trained Weight Decay (PWD), Mixout, JL-D and JL-P 3 .
As shown in Table
[Table]
full ﬁne-tuned models generally have low in-domain ECE compared to outof-domain, which illustrates that the ﬁne-tuned language models tend to be overconﬁdent under OD settings.
BABN exhibits better generalization performance than deterministic methods
[Table]
but it has a limited effect on the OD calibration.
Mixup signiﬁcantly lowers the ECE under both ID and OD settings while preserving comparable accuracy to vanilla ﬁne-tuning
[Table] [Table]
The experimental results show that all methods that preserve the pre-trained features in different ways are generally more calibrated than full ﬁne-tuning, especially under the OD settings, which matches our expectations that pre-trained features are helpful to better model the predictive conﬁdence for OD samples.
In addition to achieving competitive performance, parameter-efﬁcient tuning methods have advantages in calibration over full ﬁne-tuning.
Fine-tuning with regularization to pre-trained models in the parameter space also signiﬁcantly improves calibration.
However, our empirical experiments show that this requires maintaining a relatively high constraint strength to the pre-trained weights throughout the ﬁne-tuning process, which leads to a large loss of raw quality in some cases, e.g., Mixout on QQP/TwitterPPDB.
Notably, the proposed joint learning (JL) methods outperform previous calibration methods in ECE across three tasks simultaneously under both ID and OD settings, which suggests that it may be more effective to encourage ﬁne-tuned models to be consistent with pre-trained models in the function space.
In addition to being well-calibrated, JL models achieve the best accuracy in both ID and OD settings on NLI and CR tasks and maintain accuracy within < 1% drop compared to vanilla ﬁnetuning on the PD task.
We also highlight that the JL models have relatively low standard deviations for both accuracy and ECE compared to other methods.
5. P RESERVING P RE - TRAINED F EATURES
ALIBRATE
[Figure]
[Table]
[Figure] [Figure]
As shown in Figure
[Figure]
compared to full ﬁne-tuning
[Figure]
the hidden representations for OD samples of the methods described in §3.2 are more consistent with the PLMs, showing that they are able to preserve the pre-trained features and can mitigate catastrophic forgetting.
The representations for outlier samples are more distinguishable from the OD samples, as the pre-trained model does.
Figureillustrates that preserving the pre-trained features helps calibrate the ﬁne-tuned models by mitigating the overconﬁdent tendency to the OD and outlier samples discussed in §3.2.
Speciﬁcally, parameterefﬁcient tuning, pre-trained weight decay
[Figure]
and JL-D slightly alleviate the overconﬁdence toward the
OD and outlier samples, while JL-P and Mixout signiﬁcantly improve the ﬁne-tuned models’ ability to model the predictive conﬁdence for OD and outlier samples.
Among these methods, JL-P with knowledge distillation is shown to be the most effective regularization that can achieve low ECE and competitive raw quality at the same time.
Nevertheless, it requires access to the corpus of the pre-training phase, which may not be available in some cases.
There is clearly more room for further improve these methods.
Mixout exhibits a promising ability to model the conﬁdence of OD and outlier samples properly and improve the OD generalization (e.g., the result on HellaSWAG) by taking advantage of the pre-trained models.
However, Mixout fails to balance the preservation of pre-trained features and the learning of downstream tasks in some cases, which leads to low accuracy and high ECE.
Figure 4 demonstrates that the outlier samples can be easily distinguished from the hidden representations of the JL-D models, but they do not provide as reasonable predictive conﬁdence as JL-P, as shown in Figure 7.
We leave these questions for future work.
R ELATED W ORK
Uncertainty estimation of PLMs.
Previous works have demonstrated that PLMs can be beneﬁcial for improving the robustness and calibration on downstream tasks compared to non-pre-trained models (Hendrycks et al., 2020; Desai & Durrett, 2020).
However, PLMs can still fail to model their predictive uncertainty on downstream tasks.
For example, Desai & Durrett (2020); Kong et al.
(2020); Guo et al.
(2021) have shown that ﬁne-tuned masked language models (e.g., BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019)) are overconﬁdent on text classiﬁcation tasks, while Jiang et al.
(2021) has shown that powerful generative PLMs (e.g., T5 (Raffel et al., 2020), BART (Lewis et al., 2020), and
GPT-2 (Radford et al., 2019)) are poorly calibrated on QA tasks.
In this work, we study a speciﬁc failure case of ﬁne-tuned LM in which the models are overconﬁdent of the OD and outlier samples due to catastrophic forgetting.
Calibrating ﬁne-tuned LMs.
Several approaches have been developed to calibrate the ﬁne-tuned
LMs on NLU tasks.
For instance, Desai & Durrett (2020) demonstrate that temperature scaling and label smoothing can improve the calibration of the models in ID and OD settings, respectively.
He et al.
(2021b) introduce a new discriminative objective under the noise contrastive estimation (NCE) framework to jointly train an energy-based model deﬁned on the classiﬁer, which leads to better ID calibration.
Fan et al.
; Zhang et al.
(2021) model the attention weights as random variables and design a series of methods to optimize the stochastic attention layer with variational inference, which yields better performance in accuracy and calibration compared to vanilla deterministic attention layers.
There are also several works adopting Mixup (Zhang et al., 2018) for calibrating ﬁne-tuned language models.
For example, Kong et al.
(2020) use Mixup as a regularization to mitigate overconﬁdence for OD data, and Park & Caragea (2022) mixup two categorized training sets guided by Area Under the Margin and Saliency.
The MixUp-based methods can effectively improve calibration in both ID and OD settings.
We tackle the problem of calibrating the ﬁne-tuned LMs from a new perspective by focusing speciﬁcally on the pre-training & ﬁne-tuning paradigm and validate that preserving the pre-trained features is an effective way to improve the ﬁne-tuned LMs’ calibration.
Beneﬁt from mitigating catastrophic forgetting.
Previous works have shown that mitigating catastrophic forgetting of PLMs can be helpful for various aspects of downstream tasks.
For example,
Chen et al.
; Lee et al.
(2020) show that constraining the models’ parameters closer to the pre-trained ones can improve the training stability and performance of ﬁne-tuned LMs on downstream tasks.
Xie et al.
(2021) validate that standard ﬁne-tuning can destroy the output structure of pre-trained generative denoiser such as BART and show that preserving pre-trained features via lightweight ﬁne-tuning can improve out-of-distribution generalization on downstream generation tasks.
Dong et al.
(2021) show that the pre-trained features of PLMs are beneﬁcial for a robust objective model and improve the adversarial robustness of ﬁne-tuned language models by maximizing the mutual information between the hidden representation of the pre-trained and ﬁne-tuned models during the whole ﬁne-tuning process.
Our work makes a complementary contribution that the calibration of ﬁne-tuned LMs can be improved by mitigating catastrophic forgetting, which helps the ﬁne-tuned LMs yield more reasonable predictive conﬁdence toward OD and outlier samples.
C ONCLUSIONS
In this work, we show that the ﬁne-tuned LMs suffer from overconﬁdence caused by forgetting the pre-trained features.
Our experimental results validate that preserving the pre-trained features can better calibrate the ﬁne-tuned LMs.
We hope our work can draw more attention to the deeper exploitation of the pre-trained features learned by PLMs and contribute to building safe and reliable
NLP systems for real-world applications.
R EPRODUCIBILITY S TATEMENT
We have provided detailed setup for all experiments in §3.1, §3.2, §5.1, A.1, and A.2, we submit our codes as the supplementary material.
The information we provided is sufﬁcient to reproduce our results.
A A PPENDIX A. D ATASET D ETAILS
The general information of in-domain and out-of-domain datasets of the three NLU tasks are shown below:
Natural Language Inference:
Stanford Natural Language Inference (SNLI) (Bowman et al., 2015) requires the model to learn the textual entailment by predicting the relationship between given premise and hypothesis among entailment , contradiction , or neutral .
Multi-Genre Natural Language
Inference (MNLI)
(Williams et al., 2018) shares the same task form as SNLI, but the samples are from more diverse domains than SNLI.
Paraphrase Detection: Quora Question Pairs (QQP) (Shankar et al., 2017) contains question pairs from Quora.
The model needs to discriminate whether the given pairs are semantically equivalent.
TwitterPPDB (Lan et al., 2017) is the out-of-domain dataset that collects sentence pairs shared with the same URLs.
Commonsense Reasoning: Situations With Adversarial Generations (SWAG)
(Zellers et al., 2018) is a common sense reasoning task that requires the model to choose the most plausible continuation of a sentence given four possible candidates.
HellaSWAG (Zellers et al., 2019) is designed as a more challenging commonsense reasoning task for the pre-trained language models built with Adversarial
Filtering.
The statistic of the datasets for both MLM and NLU tasks are shown in Table 3.
For the datasets of NLU tasks (SNLI/MNLI, QQP/TwitterPPDB, SWAG/HellaSWAG), we use the published version by Desai & Durrett (2020).
For WikiText-103, we use the version provided by Huggingface
Datasets (Lhoest et al., 2021) library.
Note that the training splits of the OD datasets (MNLI, TwitterPPDB, HellaSWAG) are not used.
[Table]
Labels
SNLI (Bowman et al., 2015) 549, 4, 4,
MNLI (Williams et al., 2018) 392, 4, 4,
QQP (Shankar et al., 2017) 363, 20, 20,
TwitterPPDB (Lan et al., 2017) 46, 5, 5,
SWAG (Zellers et al., 2018) 73, 10, 10,
HellaSWAG (Zellers et al., 2019) 39, 5, 5,
WikiText-103 (Merity et al., 2016) 1,801,
[Table]
A. S ETUPS A.2. S ETUP FOR F ULL F INE - TUNING AND P ARAMETER -E FFICIENT T UNING
We conduct the experiments with the Huggingface Transformers library (Wolf et al., 2020).
For parameter-efﬁcient tuning methods, we use the implementations of OpenDelta (Ding et al., 2022) library for the three parameter-efﬁcient tuning methods (Adapter, LoRA, Preﬁx Tuning).
We use the same default hyperparameters provided by the OpenDelta library for each method across all three tasks 4 .
All ﬁne-tuning methods are trained with the AdamW optimizer (Loshchilov & Hutter, 2019).
For full ﬁne-tuning, we use a learning rate of 1e-5 across all tasks.
For parameter-efﬁcient tuning methods, we search the learning rate among { 1e-5, 2e-5, 5e-5, 1e-4, 2e-4, 5e-4 } .
We use the one that has the best ID accuracy on the validation set, which is the widespread application scenario for the parameter-efﬁcient tuning methods.
Table 4 shows the learning rate used to ﬁne-tune all the methods.
For other training hyperparameters, we follow the default setup of the Huggingface Trans-
formers library 5 .
In speciﬁc, we set a batch size of 32, a maximum sequence length of 256, and a weight decay of 0.1.
We use a linear decay learning rate scheduler without warmup and do not apply gradient clipping.
Note that the reported baselines for full ﬁne-tuning and Mixup (Desai & Durrett, 2020; Park & Caragea, 2022) in Table 2 do not use learning rate scheduler and use a gradient clip of 1.0 compared to our runs.
There are also some minor differences, such as the padding strategy for input texts between our ﬁne-tuning setup and theirs.
[Table]
A.2. S ETUP FOR THE J OINT L EARNING M ETHOD
We conduct hyperparameter search with the ID/OD validation set for the models that joint learning with the MLM objective (JL).
Speciﬁcally, we set a learning rate of 1e-5 and a batch size of 32 across all three tasks as Desai & Durrett (2020) does, except for ﬁne-tuning JL-P with label smoothing on SWAG, where we use a larger learning rate of 5e-5.
For other training parameters, we adopt the same setup described in A.2.1 except for ﬁne-tuning JL-P on QQP, where we do not use a learning rate scheduler.
For the hyperparameter of the JL models, we search the scaling factor of the MLM loss α mlm ∈ { 0 . 1 , 0 . 3 , 0 .
5 , 1 , 2 , 3 , 4 , 5 } , the coefﬁcient of the regularization term on contextualized representation β L 2 ∈ { 1e-9, 1e-8, 1e-7, 1e-6, 1e-5, 1e-4 } , the masking probability for a sentence p mask ∈ { 0 . 05 , 0 .
5 , 0 . 6 } , and the hyperparameter of label smoothing σ ls ∈ { 0 . 01 , 0 .
03 , 0 . 05 } for each task.
We also search the maximum sequence length of the MLM task for JL-P and the batch size of MLM tasks for both JL-D and JL-P.
In detail, we use a batch size for the MLM task of 32 on NLI and PD tasks and a batch size for the MLM task of 8 on the CR task.
The maximum sequence lengths of the MLM task for JL-P are set to 32/32/64 for NLI, PD, and CR tasks, respectively.
Training a single model for 3 epochs can be done in two hours with a single NVIDIA A40 48G GPU.
We present the detailed hyperparameter setup of each method on each task in Table 5.
[Table]
JL-P + LS 0.
1e0.
A.2. S ETUP FOR R EGULARIZATION WITH P RE - TRAINED W EIGHT
Pre-trained Weight Decay: We tune the balancing coefﬁcient of the classiﬁcation loss and the weight decay term λ pwd ∈ { 0 . 1 , 0 . 3 , 0 . 5 , 0 .
and use λ pwd = 0 .
, QQP, and SWAG, respectively.
We use the default coefﬁcient on the quadratic penalty of the ofﬁcial implementation of RecAdam (Chen et al., 2020).
Mixout: We tune the mixout probability p mixout ∈ { 0 . 1 , 0 . 3 , 0 . 5 , 0 .
and use p mixout = 0 . 9 for all three NLU task.
For other training parameters, we use the same default setup described in A.2.2.
A.2. C OMPARISON OF T RAINING T IME
We compare the training time of different methods using a single NVIDIA A40 48G GPU.
We use the full ﬁne-tuning as a baseline (1x).
The time cost of 3 training epochs using full ﬁne-tuning is 1/0.8/0.3 GPU hours for SNLI/QQP/SWAG.
We present the time cost of different methods in
Table 6.
[Table]
A.
A DDITIONAL E XPERIMENTAL R ESULTS
A.3.
A BLATION S TUDY FOR THE J OINT L EARNING M ETHOD
[Figure]
Effect of the MLM objective: As shown in Figure 6, compared to vanilla ﬁne-tuning ( α mlm = 0 ), introducing the MLM objective lowers the ECE effectively in both ID and OD settings with a relatively small effect on accuracy.
However, as the weight of the MLM loss becomes too large compared to the classiﬁcation loss, the performance of the NLU task will be apparently damaged.
For the overconﬁdence toward the OD samples and outlier samples illustrated in §3.2, as shown in
Figure 7, can be mitigated by using a relatively small magnitude of α mlm , while increasing it can have a more signiﬁcant effect.
Effect of introducing corpus of pre-training phase: From Table 2 and Figure 6, we observe that performing the MLM task with the corpus of the pre-training phase has lower OD ECE on all three tasks.
This conﬁrms our belief that using the corpus of the pre-training phase can preserve more helpful features from the PLMs.
We also notice that sometimes JL-P degrades the ID calibration (e.g., calibration on SWAG in Table 2), however, applying label smoothing on downstream tasks can relieve this negative effect.
An interesting observation is that although the hidden representations given by both JL-D and JL-P can distinguish outlier samples easily, the JL-D holds higher conﬁdence toward outlier samples than JL-P. As shown in Figure 7, compared to JL-P, the JL-D are more conﬁdent in samples from the BookCorpus dataset, which are not seen by both JL-P and JL-D in the ﬁne-tuning phase for a fair comparison, which suggests utilizing the pre-training phase’s corpora that are more diverse than the training datasets are helpful for the ﬁne-tuned LMs to model the conﬁdence of the outlier samples better.
Effect of knowledge distillation from the pre-trained model:
Table 7 shows that applying knowledge distillation has limited effect on JL-D but enhances the accuracy and ECE of JL-P in most cases.
We hypothesize that performing the MLM task with the corpus that is distinct from the downstream datasets can hurt relatively more performance on the downstream tasks, and distilling from the pretrained model’s predictive distribution might be a smoother and more effective regularization.
Effect of regularization on the contextualized representation: As shown in Figure
[Figure]
the introduced heuristic regularization term could improve the ECE of JL-D models under both ID and OD settings by smoothing the models’ predictive conﬁdence.
The effect of this regularization term is similar to applying temperature scaling under the ID validation set
[Figure]
where large magnitudes can lead to overly conservative prediction conﬁdence.
We also ﬁnd that a proper choice of β L 2 can marginally beneﬁt the accuracy under ID and OD settings.
However
[Figure] the effect of this term on Full-FT models is limited.
[Figure]
[Figure]
[Figure]
[Table]
A.3. S AMPLING FROM J OINT L EARNING M ODELS
Since the JL model is encouraged to learn both discriminative and generative representations, we can perform conditional generation with the MLM head g φ and classiﬁer h ϕ .
In this work, we choose
Mask-Predict (Ghazvininejad et al., 2019) as our basic sampling algorithm, which can incorporate the rejection sampling framework to generate samples from the conditional distribution p ( x | y ∗ ) given the desired label y ∗ .
We show the detailed sampling process in Alg 1.
Table 8 shows the samples generated by JL-D models with the sampling algorithm described above.
The JL-D models are able to generate text with the given labels, which can be seen as a diagnostic for the model.
For example, the models prefer to generate contradictory hypotheses by changing the objective’s entity or adjectives with high conﬁdence and tend to copy the preﬁx to generate positive textual entailment, which may expose some spurious correlations that the models rely on when performing the NLU tasks (Tu et al., 2020).
[Table]
Two
[Table]
gym.
[Table]
Two men
[Table]
playing basketball.
[ Entailment, 96% ]
Two men are jumping around.
[ Contradiction, 90% ]
Two men are jumping outside.
[ Natural, 44% ]
Two men are playing basketball.
Text Preﬁx: a blue and gray race car driving on a dirt track.
[Table]
A race car is driving on a dirt track.
[ Entailment, 98% ]
A race car is driving on a dirt track
[ Contradiction, 99% ]
A race car is parked on a dirt track
[ Natural, 66% ]
A race car is racing on a dirt track
Text Preﬁx:
A boy poses in karate form and uniform.
A
boy is
practicing
karate
[ Entailment, 75% ]
A
boy
is practicing karate.
[ Contradiction, 2% ]
A boy is in a costume.
[ Natural, 25% ]
A boy is practicing karate.
Text Preﬁx:
Four females wearing helments are riding on an ATV. Four females are wearing helments on an ATV.
[ Entailment, 98% ]
Four females are wearing helments on an ATV.
[ Contradiction, 99% ]
Four females are riding on a horse in the park.
[ Natural, 99% ]
Four females are riding a ATV in the desert.
Text Preﬁx:
How does cloud computing work?
How does cloud computing in India work?
[ Non-Paraphrase, 95% ]
How does Google think cloud computing work?
[ Paraphrase, 74% ]
How does I understand cloud computing work?
Text Preﬁx:
Do you think time travel is possible?
Do you think space and time travel is possible?
[ Non-Paraphrase, 73% ]
Do you think gravity can make time travel possible?
[ Paraphrase, 98% ]
Do you think it is possible to time travel?
Text Preﬁx:
What is a good data analysis book? What is a good free data analysis book?
[ Non-Paraphrase, 78% ]
What is a good book for analysis books?
[ Paraphrase, 67% ]
What is the best free data analysis book?
Text Preﬁx:
Feeling bored.
What do I do?
What to do to get bored?
[ Non-Paraphrase, 97% ]
What should I do to myself?
[ Paraphrase, 28% ]
What should I start to do?