The impact of domain shift on the calibration of ﬁne-tuned models
Abstract
Transfer learning has become a standard technique in computer vision and natural language processing thanks to the fact that it often substantially improves performance on downstream tasks.
Recent work by Hendrycks et al.
[12] demonstrated that using a pre-trained model can also signiﬁcantly improve a model’s calibration, i e. how well the model’s conﬁdence estimates correspond to the probability of its prediction being correct.
In this paper, we provide some nuance to the claim that pre-training improves calibration by demonstrating that this beneﬁcial effect diminishes when there is a domain shift between the pre-training and ﬁne-tuning tasks.
Introduction
Transfer learning has become a central technique in the application of deep neural networks in both computer vision (CV) and natural language processing (NLP).
Models like BiT [14], SimCLR [2], and ResNets [9] pre-trained on ImageNet [23] have achieved state-of-the-art performance on various vision benchmarks, while NLP models like BERT [7], RoBERTa [16], and T5 [21] have demonstrated excellent performance on the GLUE [26] and SuperGLUE [25] benchmarks and beyond.
Previous works have also argued that using a pre-trained model can result in faster convergence and better generalization [22, 18].
Work by Hendrycks et al.
[12] points out that pre-training can also help to improve the calibration of the models, i.e. how well the model’s conﬁdence estimates correspond to the probability of its prediction being correct.
Quantifying the calibration of a model can be invaluable in determining how much to trust its predictions.
Speciﬁcally, Hendrycks et al.
[12] took models pre-trained on ImageNet and ﬁne-tuned them on CIFAR-10, CIFAR-100, and Tiny Imagenet.
These downstream tasks have a close resemblance to the pre-training task and are relatively large (with at least 60000 training samples).
In practice, the performance of ﬁne-tuned models can be greatly affected by the domain shift between the pre-training task and the downstream task.
Recent work by Desai and Durrett [6] shows that calibration error can be greatly affected when there is a distribution shift between the training and testing data.
In this paper, we focus on the question: How does domain shift between the pre-training task and downstream task change a model’s calibration?
Most past work on transfer learning has evaluated these factors in terms of their downstream task performance; to the best of our knowledge, there has been limited study of their impact on model calibration.
In this work, we, therefore, undertake an empirical study of how varying amounts of domain shift between the pre-training and downstream tasks can change the ﬁne-tuned model’s calibration.
Our study includes experiments on NLP with BERT [7] and RoBERTa [16] in addition to CV experiments using BiT [14] and an ImageNet-pre-trained ResNet [23, 10].
We consider downstream tasks with varying amounts of domain shift: the QQP [3], PubMed-RCT [5], XNLI-Swahili [4] and XNLIGerman [4] datasets for NLP, and the CIFAR-100 and CheXpert [13] datasets for CV Experiments.
Our results show that pre-training leads to lower calibration error in no or mild domain shift, but the beneﬁcial effect of pre-training on calibration decreases in extreme domain shift cases.
This insight will help practitioners make more informed design decisions when calibration of the ﬁne-tuned model is a major concern.
The rest of the paper is structured as follows: section 2 provides background calibration error and domain shift.
section 3 describes our experimental setup and results.
We conclude in section 4 with a summary of our ﬁndings and suggestions for future work.
Background
Our focus in this paper is on measuring the impact of transfer learning (i.e. starting from a pre-trained model before ﬁne-tuning on a downstream task) on a model’s calibration error (i.e. how well a model’s conﬁdence correlates with the probability of it being correct).
The main experimental factor we consider is the domain shift, which in the context of transfer learning refers to a mismatch between the type of data used for pre-training and ﬁne-tuning.
In the following subsections, we provide some background on calibration and domain shift before delving into our empirical study.
2.1 Calibration Error
A model is considered to be well-calibrated if the conﬁdence estimates it gives its predictions are well-aligned with the actual probability of the predictions being correct.
Given X (input to the model), ground-truth output Y , and the predicted probability of the model for input X denoted by
P out ( ˆ Y | X ) , a perfectly calibrated model satisﬁes the following condition
[Equation]
In practice, the above condition is impossible to evaluate directly.
Instead, calibration is typically approximated by discretizing probability intervals into a ﬁxed number of bins.
Then, the predictions are assigned to bins, and calibration error is approximated by computing the difference between the fraction of samples in the bin which are classiﬁed correctly (referred to as accuracy) and the mean of probabilities in the bin (referred to as conﬁdence).
ECE can be computed as follow
[Equation]
Here n i refers to the number of data points in bin i , N is the total number of data points in the dataset, acc ( i ) is the accuracy of bin i , and conf ( i ) is conﬁdence estimates of bin i .
As noted by Nixon et al.
[19], comparing the calibration of different models using the expected calibration error can be conﬂated by the models’ accuracy.
Nixon et al.
[19] therefore introduced a new metric for measuring the calibration error called Adaptive Calibration Error (ACE).
This metric uses an adaptive binning strategy where the bin interval are spaced such that each bin contains an equal number of predictions.
ACE can be computed using the equation deﬁned below
[Equation]
Here acc ( r, k ) and conf ( r, k ) are the accuracy and conﬁdence of the adaptive calibration range r for class k , K are the total number of classes in the classiﬁcation problem and R is the total number of ranges.
2.2 Domain Shift
In the context of transfer learning, domain shift means there is a mismatch between the pre-training and downstream tasks.
For example, if a model was pre-trained on ImageNet [23] (which contains a diverse set of natural images) and was ﬁne-tuned on CheXpert (a dataset of chest x-ray images) [13], it is intuitively clear that the data seen during ﬁne-tuning is substantially different from data used for pre-training.
It follows that the features learned during pre-training might not provide substantial beneﬁts for ﬁne-tuning.
While domain shift is often deﬁned intuitively, some methods have been proposed to attempt to measure it quantitatively (e.g. [8, 27, 1]).
For example, a simple approach proposed by Ganin et al. [8] is to train a classiﬁer to distinguish between data from each of the datasets.
If the classiﬁer can reach perfect accuracy, there might be a dramatic domain shift; if it can only attain chance accuracy the datasets might be highly similar.
Instead of relying on any quantitative approximations of domain shift, in this work, we divide domain shifts into three intuitively-deﬁned categories: minimal, mild, and extreme.
Experimental Setting and Results
We now present our main experimental results to measure the impact of domain shift on the calibration of ﬁne-tuned models.
To improve the robustness of our ﬁndings, we measure calibration in many settings (across different modalities, models, and datasets).
In the following subsections, we describe our experimental setup and main ﬁndings.
We report ACE in the main text of the paper because of its beneﬁts over ECE but you can also ﬁnd ECE values in the Appendix.
3.1 Setup
Natural Language Processing We experimented with the BERT [7] and RoBERTa [16] pre-trained models.
BERT and RoBERTa are both pre-trained Transformer encoder [24] models that can be applied to text classiﬁcation and span labeling problems.
A full description of these architectures and their pre-training is out of the scope of this paper; we refer to the original sources of each model for additional details [7, 16].
To study the impact of domain shift, we consider the following downstream tasks: Quora Question Pairs (QQP) [3], PubMed-RCT [5], and the Swahili and German task from
XNLI [4].
All downstream task datasets have at least 100,000 samples, we report the results forsamples in the main text of the paper and also provide results for 100 samples in the Appendix.
We speciﬁcally chose a low data regime (< 1000 samples for training) for our experiments because He et al.
[11] pointed out that the performance of the pre-trained model is similar to the performance of models trained from scratch in the high data regime.
When subsampling downstream tasks, we always choose data in such a way that the resulting dataset is class-balanced.
On each downstream task, we ﬁne-tuned BERT and RoBERTa for either 40 epochs or until it reached 100 % train accuracy using a default learning rate of 2e-5.
Computer Vision We used BiT [14] and a ResNet [9] pre-trained on Imagenet [23].
BiT is a collection of pre-trained ResNet models that were designed to perform well on a wide variety of downstream tasks using the same ﬁne-tuning formula.
Our Imagenet-pre-trained ResNet corresponds to an incredibly common choice for vision-based transfer learning.
To study the impact of domain shift, we used CIFAR-100 [15] and the Pleural Effusion detection task from CheXpert [13].
CIFAR100 is essentially a subset of the classes from Imagenet, constituting almost no domain shift, whereas
CheXpert focuses on the entirely different task of classifying chest x-rays.
As in the NLP experiments, we report the results for 1000 samples per dataset in the main text of the paper and also provide results for 100 samples in the Appendix.
For BiT experiments, we follow the recommended “BiT-Hyperrule”, where we use an initial learning rate of 0.003 and decay the learning rate every 100 steps (For BiT
CIFAR-100 Experiments we use a batch size of 256 while for CheXpert we use a batch size of 32).
For ResNet experiments, we use an initial learning rate of 0.01 and use an exponential decay learning rate schedule with γ = 0 .
975 and batch size of 32 for both CIFAR-100 and CheXpert experiments.
For BiT and Resnet, we train until 9,000 steps or 100 % training accuracy is reached.
We used [-A
-]A
GPU for all the experiments conducted in the paper.
3.2 Findings
First, we conﬁrm one of the main results of Hendrycks et al.
[12]: pre-training can signiﬁcantly improve a ﬁne-tuned model’s calibration.
This effect was apparent across all modalities and models we considered.
For example, in table 1, pre-trained BERT-Base and RoBERTa-Base models always attained a lower calibration error on QQP than BERT-Base and RoBERTa-Base models trained from scratch.
We also observe that the performance (accuracy) of pre-trained models is signiﬁcantly better than the performance of models trained from scratch.
Under mild domain shift (PubMed-RCT) the calibration error of the pretrained model is still signiﬁcantly lower than the model that was trained
[Table] [Table]
from scratch.
From table 1 and table 2 it is clear that under no domain shift or mild domain shift cases we typically observe 2-3 × lower ACE than models trained from scratch.
However, this effect does not necessarily hold when there is a signiﬁcant domain mismatch.
As seen e.g. table 1, when ﬁne-tuning on XNLI-Swahili or German, using a pre-trained BERT-Base and RoBERTa-Base only slightly improves calibration.
table 2 (right-most column) when ﬁne-tuning on CheXpert, using pre-trained BiT-M-50x1 leads to higher calibration error than the model trained from scratch.
It is tempting to explain this effect by conjecturing that extreme domain shift makes the solution found during pre-training about as useful as a random initialization, but the accuracy of pre-trained models is often better under extreme domain shift.
This supports recent ﬁndings suggesting that pre-training can be beneﬁcial for downstream performance even when there is an extreme domain shift between the pretraining and downstream task.
Conclusion
This paper investigated how domain shifts between the pretraining and downstream tasks can affect the calibration error of models.
In contrast to prior work, [12], we ﬁnd that extreme domain shift can decrease pre-training’s downstream beneﬁts on calibration.
Our ﬁndings provide signiﬁcantly more nuances to the conventional wisdom that “pre-training helps calibration”.
We hope that these insights help guide practitioners towards making more informed decisions when the ﬁne-tuned model’s calibration is a major concern and that our work helps prompt future work on improving the calibration of ﬁne-tuned models under domain shift.