The impact of domain shift on the calibration of ﬁne-tuned models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Transfer learning has become a standard technique in computer vision and naturallanguage processing thanks to the fact that it often substantially improves performance on downstream tasks.
Recent work by Hendrycks et al.
[12] demonstratedthat using a pre-trained model can also signiﬁcantly improve a model’s calibration,i.e. how well the model’s conﬁdence estimates correspond to the probability ofits prediction being correct.
In this paper, we provide some nuance to the claimthat pre-training improves calibration by demonstrating that this beneﬁcial effectdiminishes when there is a domain shift between the pre-training and ﬁne-tuningtasks.
Introduction
Transfer learning has become a central technique in the application of deep neural networks in bothcomputer vision (CV) and natural language processing (NLP).
Models like BiT [14], SimCLR [2],and ResNets [9] pre-trained on ImageNet [23] have achieved state-of-the-art performance on variousvision benchmarks, while NLP models like BERT [7], RoBERTa [16], and T5 [21] have demonstratedexcellent performance on the GLUE [26] and SuperGLUE [25] benchmarks and beyond.
Previousworks have also argued that using a pre-trained model can result in faster convergence and bettergeneralization [22, 18].
Work by Hendrycks et al.
[12] points out that pre-training can also help to improve the calibrationof the models, i.e. how well the model’s conﬁdence estimates correspond to the probability of itsprediction being correct.
Quantifying the calibration of a model can be invaluable in determining howmuch to trust its predictions.
Speciﬁcally, Hendrycks et al.
[12] took models pre-trained on ImageNetand ﬁne-tuned them on CIFAR-10, CIFAR-100, and Tiny Imagenet.
These downstream tasks havea close resemblance to the pre-training task and are relatively large (with at least 60000 trainingsamples).
In practice, the performance of ﬁne-tuned models can be greatly affected by the domainshift between the pre-training task and the downstream task.
Recent work by Desai and Durrett [6]shows that calibration error can be greatly affected when there is a distribution shift between thetraining and testing data.
In this paper, we focus on the question: How does domain shift betweenthe pre-training task and downstream task change a model’s calibration?
Most past work on transferlearning has evaluated these factors in terms of their downstream task performance; to the best of ourknowledge, there has been limited study of their impact on model calibration.
In this work, we, therefore, undertake an empirical study of how varying amounts of domain shiftbetween the pre-training and downstream tasks can change the ﬁne-tuned model’s calibration.
Ourstudy includes experiments on NLP with BERT [7] and RoBERTa [16] in addition to CV experimentsusing BiT [14] and an ImageNet-pre-trained ResNet [23, 10].
We consider downstream tasks withvarying amounts of domain shift: the QQP [3], PubMed-RCT [5], XNLI-Swahili [4] and XNLIGerman [4] datasets for NLP, and the CIFAR-100 and CheXpert [13] datasets for CV Experiments.
Our results show that pre-training leads to lower calibration error in no or mild domain shift, but thebeneﬁcial effect of pre-training on calibration decreases in extreme domain shift cases.
This insightwill help practitioners make more informed design decisions when calibration of the ﬁne-tuned modelis a major concern.
The rest of the paper is structured as follows: section 2 provides background calibration error anddomain shift.
section 3 describes our experimental setup and results.
We conclude in section 4 with asummary of our ﬁndings and suggestions for future work.
Background
Our focus in this paper is on measuring the impact of transfer learning (i.e. starting from a pre-trainedmodel before ﬁne-tuning on a downstream task) on a model’s calibration error (i.e. how well amodel’s conﬁdence correlates with the probability of it being correct).
The main experimental factorwe consider is the domain shift, which in the context of transfer learning refers to a mismatch betweenthe type of data used for pre-training and ﬁne-tuning.
In the following subsections, we provide somebackground on calibration and domain shift before delving into our empirical study.
2.1 Calibration Error
A model is considered to be well-calibrated if the conﬁdence estimates it gives its predictions arewell-aligned with the actual probability of the predictions being correct.
Given X (input to themodel), ground-truth output Y , and the predicted probability of the model for input X denoted by
P out ( ˆ Y | X ) , a perfectly calibrated model satisﬁes the following condition
[Equation]
In practice, the above condition is impossible to evaluate directly.
Instead, calibration is typicallyapproximated by discretizing probability intervals into a ﬁxed number of bins.
Then, the predictionsare assigned to bins, and calibration error is approximated by computing the difference between thefraction of samples in the bin which are classiﬁed correctly (referred to as accuracy) and the mean ofprobabilities in the bin (referred to as conﬁdence).
ECE can be computed as follow
[Equation]
Here n i refers to the number of data points in bin i , N is the total number of data points in the dataset,acc ( i ) is the accuracy of bin i , and conf ( i ) is conﬁdence estimates of bin i .
As noted by Nixonet al.
[19], comparing the calibration of different models using the expected calibration error can beconﬂated by the models’ accuracy.
Nixon et al.
[19] therefore introduced a new metric for measuringthe calibration error called Adaptive Calibration Error (ACE).
This metric uses an adaptive binningstrategy where the bin interval are spaced such that each bin contains an equal number of predictions.
ACE can be computed using the equation deﬁned below
[Equation]
Here acc ( r, k ) and conf ( r, k ) are the accuracy and conﬁdence of the adaptive calibration range r forclass k , K are the total number of classes in the classiﬁcation problem and R is the total number ofranges.
2.2 Domain Shift
In the context of transfer learning, domain shift means there is a mismatch between the pre-trainingand downstream tasks.
For example, if a model was pre-trained on ImageNet [23] (which contains adiverse set of natural images) and was ﬁne-tuned on CheXpert (a dataset of chest x-ray images) [13],it is intuitively clear that the data seen during ﬁne-tuning is substantially different from data used forpre-training.
It follows that the features learned during pre-training might not provide substantialbeneﬁts for ﬁne-tuning.
While domain shift is often deﬁned intuitively, some methods have been proposed to attempt tomeasure it quantitatively (e.g. [8, 27, 1]).
For example, a simple approach proposed by Ganin et al.[8] is to train a classiﬁer to distinguish between data from each of the datasets.
If the classiﬁer canreach perfect accuracy, there might be a dramatic domain shift; if it can only attain chance accuracythe datasets might be highly similar.
Instead of relying on any quantitative approximations of domainshift, in this work, we divide domain shifts into three intuitively-deﬁned categories: minimal, mild,and extreme.
Experimental Setting and Results
We now present our main experimental results to measure the impact of domain shift on the calibrationof ﬁne-tuned models.
To improve the robustness of our ﬁndings, we measure calibration in manysettings (across different modalities, models, and datasets).
In the following subsections, we describeour experimental setup and main ﬁndings.
We report ACE in the main text of the paper because of itsbeneﬁts over ECE but you can also ﬁnd ECE values in the Appendix.
3.1 Setup
Natural Language Processing We experimented with the BERT [7] and RoBERTa [16] pre-trainedmodels.
BERT and RoBERTa are both pre-trained Transformer encoder [24] models that can beapplied to text classiﬁcation and span labeling problems.
A full description of these architectures andtheir pre-training is out of the scope of this paper; we refer to the original sources of each model foradditional details [7, 16].
To study the impact of domain shift, we consider the following downstreamtasks: Quora Question Pairs (QQP) [3], PubMed-RCT [5], and the Swahili and German task from
XNLI [4].
All downstream task datasets have at least 100,000 samples, we report the results for 1000samples in the main text of the paper and also provide results for 100 samples in the Appendix.
Wespeciﬁcally chose a low data regime (< 1000 samples for training) for our experiments because Heet al. [11] pointed out that the performance of the pre-trained model is similar to the performanceof models trained from scratch in the high data regime.
When subsampling downstream tasks, wealways choose data in such a way that the resulting dataset is class-balanced.
On each downstreamtask, we ﬁne-tuned BERT and RoBERTa for either 40 epochs or until it reached 100 % train accuracyusing a default learning rate of 2e-5.
Computer Vision We used BiT [14] and a ResNet [9] pre-trained on Imagenet [23].
BiT is acollection of pre-trained ResNet models that were designed to perform well on a wide variety ofdownstream tasks using the same ﬁne-tuning formula.
Our Imagenet-pre-trained ResNet correspondsto an incredibly common choice for vision-based transfer learning.
To study the impact of domainshift, we used CIFAR-100 [15] and the Pleural Effusion detection task from CheXpert [13].
CIFAR100 is essentially a subset of the classes from Imagenet, constituting almost no domain shift, whereas
CheXpert focuses on the entirely different task of classifying chest x-rays.
As in the NLP experiments,we report the results for 1000 samples per dataset in the main text of the paper and also provide resultsfor 100 samples in the Appendix.
For BiT experiments, we follow the recommended “BiT-Hyperrule”,where we use an initial learning rate of 0.003 and decay the learning rate every 100 steps (For BiT
CIFAR-100 Experiments we use a batch size of 256 while for CheXpert we use a batch size of 32).
For ResNet experiments, we use an initial learning rate of 0.01 and use an exponential decay learningrate schedule with γ = 0 .
975 and batch size of 32 for both CIFAR-100 and CheXpert experiments.
For BiT and Resnet, we train until 9,000 steps or 100 % training accuracy is reached.
We used A
GPU for all the experiments conducted in the paper.
3.2 Findings
First, we conﬁrm one of the main results of Hendrycks et al.
[12]: pre-training can signiﬁcantlyimprove a ﬁne-tuned model’s calibration.
This effect was apparent across all modalities and modelswe considered.
For example, in table 1, pre-trained BERT-Base and RoBERTa-Base models alwaysattained a lower calibration error on QQP than BERT-Base and RoBERTa-Base models trained fromscratch.
We also observe that the performance (accuracy) of pre-trained models is signiﬁcantly betterthan the performance of models trained from scratch.
Under mild domain shift (PubMed-RCT) thecalibration error of the pretrained model is still signiﬁcantly lower than the model that was trained [Table] [Table]
from scratch. From table 1 and table 2 it is clear that under no domain shift or mild domain shiftcases we typically observe 2-3 × lower ACE than models trained from scratch.
However, this effectdoes not necessarily hold when there is a signiﬁcant domain mismatch.
As seen e.g. table 1, whenﬁne-tuning on XNLI-Swahili or German, using a pre-trained BERT-Base and RoBERTa-Base onlyslightly improves calibration.
table 2 (right-most column) when ﬁne-tuning on CheXpert, usingpre-trained BiT-M-50x1 leads to higher calibration error than the model trained from scratch.
It istempting to explain this effect by conjecturing that extreme domain shift makes the solution foundduring pre-training about as useful as a random initialization, but the accuracy of pre-trained modelsis often better under extreme domain shift.
This supports recent ﬁndings suggesting that pre-trainingcan be beneﬁcial for downstream performance even when there is an extreme domain shift betweenthe pretraining and downstream task.
Conclusion
This paper investigated how domain shifts between the pretraining and downstream tasks can affectthe calibration error of models.
In contrast to prior work, [12], we ﬁnd that extreme domain shiftcan decrease pre-training’s downstream beneﬁts on calibration.
Our ﬁndings provide signiﬁcantlymore nuances to the conventional wisdom that “pre-training helps calibration”.
We hope that theseinsights help guide practitioners towards making more informed decisions when the ﬁne-tunedmodel’s calibration is a major concern and that our work helps prompt future work on improving thecalibration of ﬁne-tuned models under domain shift.