T HE E VOLUTION OF O UT - OF -D ISTRIBUTION R OBUSTNESS T HROUGHOUT F INE -T UNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Although machine learning models typically experience a drop in performance on out-of-distribution data, accuracies on in- versus out-of-distribution data are widely observed to follow a single linear trend when evaluated across a testbed of models.
Models that are more accurate on the out-of-distribution data relative to this baseline exhibit “effective robustness” and are exceedingly rare.
Identifying such models, and understanding their properties, is key to improving out-of-distribution performance.
We conduct a thorough empirical investigation of effective robustness during ﬁne-tuning and surprisingly ﬁnd that models pre-trained on larger datasets exhibit effective robustness during training that vanishes at convergence.
We study how properties of the data inﬂuence effective robustness, and we show that it increases with the larger size, more diversity, and higher example difﬁculty of the dataset.
We also ﬁnd that models that display effective robustness are able to correctly classify 10% of the examples that no other current testbed model gets correct.
Finally, we discuss several strategies for scaling effective robustness to the high-accuracy regime to improve the out-of-distribution accuracy of state-of-the-art models.
I NTRODUCTION
The ability to generalize to data not seen during training is essential for the widespread trust and adoption of machine learning models.
In practical applications of machine learning, we typically train and ﬁne-tune on a dataset which are in-distribution (ID) with each other, but when deployed the model will face shifts from this distribution, and empirically the vast majority of models show a signiﬁcant drop in performance from ID data to out-of-distribution (OOD) data (Quiñonero-Candela et al., 2009; Recht et al., 2019; Biggio & Roli, 2018; Szegedy et al., 2013; Hendrycks & Dietterich, 2019; Azulay & Weiss, 2018; Shankar et al., 2019; Gu et al., 2019; Torralba & Efros, 2011).
Common examples include data captured in a different environment, like time of day or geographical location (Koh et al., 2020); noise or small corruptions of the input data (Hendrycks & Dietterich, 2019; Geirhos et al., 2018); or adversarial examples created to explicitly fool neural networks into making incorrect predictions (Szegedy et al., 2013; Biggio & Roli, 2018).
Although the performance gap on OOD shifts is pervasive, it follows an intriguing pattern: there is a clear linear relationship between a model’s ﬁnal performance on ID and OOD data (Taori et al., 2020; Recht et al., 2019; Yadav & Bottou, 2019; Miller et al., 2020).
In other words, given a model’s performance on an ID test set, a linear ﬁt can accurately predict what the performance drop will be on the OOD test set.
This also implies that a certain amount of improvement on OOD accuracy can be explained by an improvement on ID accuracy.
The linear relationship holds across a wide range of models and is well-established for several robustness benchmarks, including image classiﬁcation on both synthetic and natural distribution shifts (Recht et al., 2018; Yadav & Bottou, 2019;
Recht et al., 2019; Taori et al., 2020), 2D object detection (Shankar et al., 2019), and question-answer models in
NLP (Miller et al., 2020).
Since even the highest-performing models will still have a gap between
ID and OOD accuracy, the linear relationship reveals that our current methods are insufﬁcient for addressing OOD shift.
Models which lie above the linear ﬁt are said to exhibit effective robustness (ER) (Taori et al., 2020), which measures the model’s
OOD accuracy relative to the ﬁt (see Figure 1(a) for an illustration).
Models with nonzero ER deviate in their OOD behavior in a manner that is qualitatively and
[Figure]
quantitatively different from what we can currently achieve.
We seek to thoroughly investigate when the pre-training and ﬁne-tuning paradigm gives rise to models with nonzero ER, and understand factors that control this behaviour.
Models with high ER ( > 1% ) are exceedingly rare.
First, the most notable example is the recently proposed zero-shot CLIP model (Radford et al., 2021b), and subsequent work (Wortsman et al., 2021) has found that weight-space ensembling of zero-shot and ﬁne-tuned CLIP-style models can improve both ID and OOD accuracy and produce models with high ER.
Second, only a handful of the
ImageNet models evaluated in Taori et al.
(2020) were found to have non-zero ER.
Though most of these effectively robust models identiﬁed so far were pre-trained on large datasets, the majority of models pre-trained this way exhibit no ER, and we currently do not know when additional data helps; what effect the choice of architecture has; as well as the effects of dataset and distribution shift on these ﬁndings.
In this work, we present an empirical study of the evolution of ER throughout ﬁne-tuning.
By studying the evolution, we ﬁnd intriguing properties that are missed by focusing only on models at convergence: pre-trained models exhibit ER, while randomly-initialized models do not (see Figure 1).
Summary of Contributions:
• (Section 5.1, 5.3)
We identify pre-trained, ﬁne-tuned models as an entire class of effectively robust models (that match the ER of CLIP (Radford et al., 2021a)) and investigate how details such as model size, dataset size, and example difﬁculty inﬂuence ER.
We ﬁnd that the vanishing of ER at convergence depends on the distribution shift, and on some datasets pre-trained, ﬁne-tuned models may still exhibit non-zero ER at convergence.
• (Section 5.4) We analyze properties of pre-trained models at the peak of their ER, including particular metrics deﬁned in previous theoretical work (Mania & Sra, 2020).
We ﬁnd that effectively robust models make remarkably dissimilar predictions compared to standard models, and are able to correctly classify 10% of the examples that no other model gets correct.
• (Section 5.5)
We ﬁnd that pre-trained models gradually lose their ER throughout ﬁne-tuning, even as both the ID and OOD accuracies of the model simultaneously increase.
We discuss several potential solutions to mitigate this problem, but ﬁnd that none of them are able to maintain high ER at high ID accuracy.
R ELATED W ORK
Here we review some prior ﬁndings that are key for understanding the context of our results.
Linear trends under distribution shift.
In recent replication efforts, researchers carefully recreated new test sets for the CIFAR-10 and ImageNet classiﬁcation benchmarks (Recht et al., 2018; 2019).
Despite following the original collection procedures as closely as possible, some shift was introduced.
Evaluating an extensive testbed of trained models on the original and new tests revealed a clear relationship between accuracy on the original and new tests that is well-captured by a linear ﬁt with positive slope.
Follow-up work also found linear trends for new test sets on MNIST (Yadav & Bottou, 2019) and the SQUAD question-answer dataset (Miller et al., 2020), geographic distribution shifts for 3D Object Detection in self-driving cars (Sun et al., 2019), and synthetic distribution shifts (Taori et al., 2020).
We ﬁnd that models pre-trained on larger and more diverse datasets can break the linear trend of accuracy under distribution shift in the middle of the ﬁne-tuning process, and we investigate factors that affect this.
Examples of effectively robust models .
Models that lie off the linear trend ( effectively robust models) are historically extremely rare.
In a recent extensive empirical study, researchers evaluated 204 trained ImageNet models spanning a wide range of architectures and training techniques on several popular natural and synthetic robustness benchmarks for ImageNet (Taori et al., 2020).
Across several natural robustness benchmarks, the main outliers with positive ER and high accuracy on
ImageNet were all models that were pre-trained on larger and more diverse data than the ImageNet training set.
They included a ResNet152 model trained on 11,000 ImageNet classes (Wu, 2016), several ResNeXt models trained on 1 billion images from Instagram (Mahajan et al., 2018), and
the
EfﬁcientNet-L2 (NoisyStudent) model trained on a Google-internal JFT-300M dataset of 300 million images (Xie et al., 2020).
However, not all models that were pre-trained on larger datasets showed effective robustness.
Another recent work, (Radford et al., 2021a), observed that zero-shot CLIP classiﬁers have larger
ER than CLIP models that had been ﬁne-tuned to the downstream task.
While the CLIP model was pre-trained with a contrastive loss that combined components from natural language processing and image classiﬁcation, our results show that pre-trained ImageNet models that are evaluated in a zero-shot manner also have high ER, and, similar to the CLIP model, these image classiﬁcation models lose their ER once they are ﬁne-tuned to the downstream task.
Since the completion of this paper, subsequent work (Wortsman et al., 2021) has proposed weight-space ensembles for ﬁne-tuning (WiSE-FT) that interpolates between zero-shot and ﬁne-tuned weights to ﬁnd models with high ER as well as high ID and OOD accuracy.
This method is currently limited to CLIP-style models, and in this work we mostly focus on traditional supervised image classiﬁcation models.
Model similarity.
Recent work offers a theoretical explanation, relying on an assumption of model similarity (Mania & Sra, 2020), for why classiﬁer accuracies follow a linear trend under distribution shift.
Empirically, Mania et al.
(2019) observed that the labeling assignments across a wide range of ImageNet models are signiﬁcantly more similar to each other than would be expected by chance, which suggests that the size of the class of function approximators that neural networks learn is smaller than what may be expected a priori.
Under the assumption of model similarity, Mania et al.
(2019) proves that models’ accuracies, when evaluated on two distributions, must be approximately collinear, unless the size of the distribution shift is large in a certain sense.
In our work, we identify and analyze models that are not collinear with other models when evaluated on two distributions, and we ﬁnd that such models break the assumption of model similarity sufﬁcient to prove the existence of the linear ﬁt.
E FFECTIVE R OBUSTNESS
The linear ﬁt to testbed model accuracies on ID and OOD data provides a baseline for how improvements in OOD accuracy are tied to improvements in ID accuracy based on current methods.
Effective robustness (ER), a metric for robustness proposed in Taori et al.
(2020), measures the difference between a model’s
OOD accuracy and that predicted from the baseline.
(See Figure 1(a) for an illustration.)
[Figure]
To deﬁne ER, we ﬁx an ID and OOD test set and let β ( x ) be the baseline predicted OOD test set accuracy for a given accuracy x on the ID test set.
Given a set of models evaluated on both the ID and OOD test sets, β can be computed by performing a log-linear ﬁt between the models’ accuracies on the two test sets.
Similar to Taori et al.
(2020), we empirically ﬁnd that the best linear ﬁt comes from rescaling the accuracies using the logit function, logit ( α )
= log (cid:16) α 1 − α (cid:17) , before performing the linear ﬁt.
In Figure 2 we show the log-linear ﬁt for CIFAR-10 vs. CIFAR-10.1 in logit-space as well as transformed back to linear-space.
See Appendix A for more details.
The effective robustness of a model f is then deﬁned as
[Equation]
where acc in ( f ) and acc out ( f ) represent the model’s accuracies on the ID and OOD test sets, respectively.
Graphically, ER represents the distance above the testbed line to a model’s
OOD accuracy, as shown in Figure 1(a).
We note that ER is distinct from the absolute accuracy a model can achieve on the OOD data.
For instance, the effectively robust model (green point) in Figure 1(a) still has low absolute OOD accuracy relative to other models.
An ideal model achieves both high ER and high absolute OOD accuracy.
E XPERIMENTAL SETUP
In this section, we discuss which ﬁne-tuning datasets, pre-trained models, robustness benchmarks, and linear ﬁts we use in our experimental evaluation.
Fine-tuning datasets.
We evaluate ER throughout ﬁne-tuning (see Appendix B.2 for our deﬁnition of ﬁne-tuning) on both ImageNet and CIFAR-10 since researchers commonly ﬁne-tune pre-trained models to these popular benchmarks.
For CIFAR-10 in particular, there are several widely available pre-trained ImageNet models that we can easily transfer to the CIFAR-10 dataset.
Pre-trained models.
The majority of the pre-trained models we study are Big Transfer (BiT) models (Kolesnikov et al., 2020), which use a ResNet-v2 architecture (He et al., 2016b), with Group
Normalization (Wu & He, 2018) and Weight Standardization (Qiao et al., 2019).
The BiT models come in three different ﬂavors: BiT-S, BiT-M and BiT-L, where S, M and L indicate if the pre-training was done on ILSVRC-2012 (which we denote as ImageNet-1k or just ImageNet) (Russakovsky et al., 2015), ImageNet-21k (Deng et al., 2009), or JFT (Sun et al., 2017), respectively.
In addition, these models may be further ﬁne-tuned, and "1k", "21k" or "JFT" indicate the dataset it was ﬁne-tuned on.
For example, BiT-M-R152x4-1k is the ResNet-152x4 model that was pre-trained on ImageNet21k ("M") and ﬁne-tuned on ImageNet-1k ("1k").
We also study a series of BiT models that were pre-trained in Djolonga et al.
(2021) on different amounts of data from JFT.
[Table]
We also study six ImageNet pre-trained P Y T ORCH (Paszke et al., 2019) models: AlexNet (Krizhevsky, 2014), ResNet-18, -50 and -
(He et al., 2016a), VGG-11 with Batch Normalization (Simonyan &
Zisserman, 2014), and WideResNet-50-2 (Zagoruyko & Komodakis, 2016).
Robustness benchmarks.
Both CIFAR-10 and ImageNet have several well-established robustness benchmarks.
We choose to focus on naturally occurring distribution shifts, rather than synthetic shifts which modify images, because they are more realistic and because there are several robustness interventions that work well on synthetic shifts but do not transfer to natural distribution shifts Taori et al.
Thus, for CIFAR-10, we evaluate robustness on CIFAR-10.1, which was created in the replication study of Recht et al.
and is one of the few natural distribution shift benchmarks available for it; and for ImageNet, we use the ImageNetV2 (Recht et al., 2019), ObjectNet (Barbu et al., 2019), and ImageNet-R (Hendrycks et al., 2020) test sets.
Linear ﬁts.
Measuring ER requires establishing a linear relationship between a model’s accuracy on
ID and OOD data.
To deﬁne the linear relationship between the original test accuracy and the OOD test accuracy on CIFAR-10 and ImageNet, we require a testbed of models that span the range from low to high accuracy.
For ImageNet, we use the publicly available results from Recht et al.
(2019), and
for CIFAR-10 we use the results from Recht et al.
R ESULTS
We overview our main experimental results.
First, we establish that pre-trained models exhibit ER during ﬁne-tuning, and we examine how properties of the pre-training setup, such as model size, dataset size, and example difﬁculty, inﬂuence the magnitude of the ER.
Finally, we explore ideas that attempt to maintain high ER at the end of ﬁne-tuning.
5. P RE -T RAINED M ODELS H AVE H IGH E FFECTIVE R OBUSTNESS
We ﬁnd that models pre-trained on large and diverse datasets exhibit high effective robustness during ﬁne-tuning, while randomly initialized models do not at any point during training (Figure 3).
As described below, this observation is invariant to the choice of robustness benchmark (CIFAR-10.1,
ImageNet-V2, ObjectNet, and ImageNet-R); model architecture; and the details of the ﬁne-tuning, such as learning rate, optimization algorithm, type of ﬁne-tuning (last-layer versus whole model), and loss function.
CIFAR-10.
We evaluate several architectures, including AlexNet, ResNet-18, ResNet-50, ResNet152 and VGG-11-BN, throughout ﬁne-tuning on CIFAR-10 using both ImageNet pre-trained initialization and random initialization.
At every epoch of training, we measure ER using CIFAR-10.1 as the OOD test set and CIFAR-10 as the ID test set.
Figure 3(a) plots the ER (%) versus CIFARtest accuracy (%) throughout ﬁne-tuning for all of the ImageNet pre-trained architectures, as well as the randomly initialized architecture that achieved the maximum ER (for the full comparison see
Appendix 10).
We see that all of the ImageNet pre-trained models have signiﬁcantly higher maximum
ER than either the testbed models or the best performing randomly initialized model.
Moreover, we observe that ER generally increases throughout ﬁne-tuning, peaks, and then gradually disappears towards the end of ﬁne-tuning.
ImageNet.
For models ﬁne-tuned on ImageNet, we measure ER for JFT pre-trained models at every epoch of training using either ImageNetV2, ObjectNet, or ImageNet-R as the OOD test set and
ImageNet as the ID test set.
We report ImageNet top-1 accuracy, and for ImageNet-R and ObjectNet, we compute accuracy over the respective subset of classes for the two benchmarks.
Figure 3(b) shows the ImageNetV2 ER, and Figures 3(c) and 3(d) shows ER on ImageNet-R and ObjectNet respectively.
In all cases, the pre-trained model reaches a maximum ER during ﬁne-tuning which then decreases as we continue ﬁne tuning.
For the ObjectNet and ImageNet-R evaluation, we ﬁne-tune a BiT-L-R152x4-JFT model on ImageNet and evaluate the same model on ImageNetV2, ObjectNet, and ImageNet-R.
Though the model’s
ImageNet accuracy is the same in all cases, for ImageNetV2, the ER vanishes (OOD accuracy drops to the baseline set by the testbed models), while for ImageNet-R and ObjectNet there is still a non-zero
[Figure]
1 while Figures (b,c,d) show the ER on ImageNetV2, ImageNet-R, and ObjectNet, respectively.
For each plot, we also show the ER and accuracy achieved by converged testbed models in blue with 95% Clopper Pearson conﬁdence intervals.
For (a) CIFAR-10.1 and (b) ImageNetV2, we trained a randomly initialized version of all the model architectures and we report the model with the highest maximum ER during training (solid black curve).
We observe that the pre-trained models all exhibit high ER in the middle of ﬁne-tuning, which eventually decreases as ﬁne-tuning proceeds.
In contrast, the randomly initialized models do not have signiﬁcant ER relative to the testbed models.
ER at the end of training.
This observation is consistent with Taori et al.
(2020), which found that converged pre-trained models are more likely to exhibit nonzero ER on ObjectNet or ImageNet-R than ImageNetV2.
Model architecture.
We ﬁnd that pre-trained models with larger architectures exhibit higher ER.
For ImageNetV2 in Figure 3(b), we evaluate multiple BiT ResNets with increasing depth and width (R150x1, R101x3, R152x4) that were pre-trained on JFT, and we ﬁnd that models with more parameters have a higher peak in ER.
Similarly, for CIFAR-10.1, Figure 3(a) shows that larger, more recent architectures such as the Wide-ResNet-50-2 exhibit higher effective robustness than smaller architectures like AlexNet.
Moreover, Figure 4(b) shows that max ER on CIFAR-10.1 increases for
BiT models of increasing depth and width as we vary the pre-training dataset.
Details of training.
We performed a detailed study of a variety of components of the optimization process, and we found that the ER is not sensitive to these details.
See Appendix C for variations in learning rate, loss function, type of optimization (full model vs. last layer)
. 5. Z ERO -S HOT E VALUATION
Recent work (Radford et al., 2021a) introduced the CLIP architecture that allowed for zero-shot evaluation on image classiﬁcation datasets like ImageNet and the ImageNet robustness benchmarks.
The zero-shot CLIP models has high amounts of ER, but it is unclear if the ER comes from the new and very large dataset used for training, the natural language model component, the contrastive training objective, or its zero-shot evaluation.
In Figure 3(b) we show the ER of the best zero-shot and linear probe CLIP models from Radford et al.
(2021a).
We also show results from zero-shot evaluation on a BiT model that was pre-trained on JFT.
This is done by creating a mapping between similar classes of JFT and ImageNet.
While the BiT model has about 5% lower ImageNet topaccuracy, it matches the amount of ER that the CLIP model has.
This is surprising, and suggests that the zero-shot component of CLIP plays a signiﬁcant role in the high value of ER CLIP achieves.
[Figure]
on the C-score Jiang et al.
(2020) of CIFAR-10 and train on either the 5000 easiest, hardest or random images in the training set.
Training on harder examples yields more ER for both pre-trained and randomly initialized models, but lower CIFAR-10 test accuracy than training on random examples.
5. T HE I MPACT OF D ATA ON E FFECTIVE R OBUSTNESS
We ﬁnd that the maximum ER depends on both the pre-training dataset and the data the model is ﬁne-tuned on.
As we discuss below, larger and more diverse pre-training datasets, as well as more difﬁcult ﬁne-tuning examples, all increase the magnitude of the ER peak.
Dataset size and diversity.
In Figure 4(a), we pre-train models on randomly sampled fractions of JFT.
We observe that maximum ER increases as the amount of pre-training data increases but eventually plateaus for the largest dataset sizes.
We suspect the reason for the plateau is the ﬁxed number of iterations used across those experiments, which eventually serves a bottleneck for performance improvements as reported by prior work (Hernandez et al., 2021; Nakkiran et al., 2020).
Figure 4(b) shows that maximum ER increases as we pre-train on larger and more diverse datasets.
We also ﬁnd that pre-training ﬁrst on a larger dataset and then switching to a smaller dataset leads to lower ER during the ﬁnal ﬁne-tuning task than a setup that only pre-trains on the larger dataset.
To see this, consider the BiT-M-1k and BiT-M-21k in Figure 4(b).
Both models were initially pre-trained on ImageNet-21k (as denoted by the ‘M’), but the BiT-M-1k model was additionally ﬁne-tuned on
ImageNet-1k as part of the pre-training procedure.
We then evaluated the ER of both models as we ﬁne-tuned them to CIFAR-10.
Figure 4(b) shows that the extra ﬁne-tuning step on ImageNet-1k gives about a 1% drop in ER relative to the BiT-M-21k model.
Example difﬁculty.
The ER during training also depends on the data the model is being ﬁne-tuned on.
In Figure 4(c) we select the 5000 easiest, hardest or random examples from CIFAR-10 training set and only ﬁne-tune on these examples.
The difﬁculty of each example is measured by the C-score (Jiang et al., 2020).
Training on easier (harder) examples gives less (more) ER, but training on only the hardest or easiest examples gives lower ﬁnal accuracy than training on the examples of random difﬁculty.
See Appendix C.9 for more details.
5. A NALYSIS AT M AXIMUM E FFECTIVE R OBUSTNESS
To understand the differences between models that exhibit ER and testbed models, we compare the predictions from the former – both at the peak ER and at the end of training – and the latter.
Dominance probability.
Recent work (Mania & Sra, 2020) showed that the linear trend between the
ID and OOD accuracies will arise if two assumptions are satisﬁed: low dominance probability and a speciﬁc notion of distributional closeness.
Dominance probability (Mania & Sra, 2020) is a similarity
6 µ i − µ j 0 .0 .0 .0 .0 .0 .
D o m i n a n ce p r o b a b ili t
i e s
Dominance
[Figure]
C
[Figure] [Figure]
[Figure] [Figure]
metric between the predictions of two models, measuring what fraction of examples the model with lower overall accuracy gets correct that the model with higher overall accuracy gets wrong.
(See formal deﬁnition in Appendix C.10.)
In this work, we ﬁnd that the dominance probability assumption does not hold for models with ER, but that the assumption of distributional closeness still holds.
In Figure 5, we see that the dominance probability of a pre-trained BiT model early-stopped at its maximum ER is shifted relative to the models in the testbed, while the pre-trained BiT model that was ﬁne-tuned to convergence (and has low ER) has dominance probability similar to the models in the testbed.
We also ﬁnd that the dominance probability for the pre-trained BiT model is very similar to that of a zero-shot CLIP model (see Appendix B.3 details on ﬁne-tuning).
Even though the ﬁne-tuned CLIP model has close to zero ER (see Figure 3(b)), we see in Figure 5(a) and 5(b) that its dominance probability is also higher than that of standard testbed models.
Unlike the BiT JFT pre-trained model, the CLIP model was pre-trained with contrastive learning and text embeddings, and we hypothesize that these elements may contribute to this effect.
Examples that only effectively robust models get right.
Models that have high ER are able to correctly classify images that no other testbed model gets correct.
We select the 4.8% of images that none of the testbed models get correct, and we ﬁnd that the ER model with the best ID performance gets 10% of these examples correct, i.e. 237 examples, divided among 198 different classes, with a maximum number of three examples in each class.
After manual inspection of all examples, we do not see any pattern among the 237 images.
Remarks.
Since testbed models all have relatively low pairwise dominance probabilities, there are a set of easy examples correctly classiﬁed by all models, but harder examples are only classiﬁed correctly by increasingly higher accuracy models.
This suggests that images can be placed on a scale of difﬁculty, and that this notion of difﬁculty is shared by all the testbed models.
In contrast, the high dominance probability of the effectively robust models suggests that they have a different notion of difﬁculty compared to that of the testbed models, i.e. some images that are difﬁcult for testbed models are not as difﬁcult for effectively robust models.
As further evidence, effectively robust models are able to classify some images correctly that no other testbed model gets correct.
We hypothesize that the different notion of difﬁculty of the effectively robust models comes from pre-training on larger and more diverse data, which could expose them to examples that are dissimilar from those in the ﬁne-tuning dataset.
5. M AINTAINING E FFECTIVE R OBUSTNESS AT H IGH A CCURACY
In the previous sections, we have seen that models pre-trained on larger and more diverse datasets exhibit robustness properties during ﬁne-tuning, but that, depending on the choice of OOD dataset, the ER can vanish as the model converges towards the end of training.
In this section we explore several ideas for ways to maintain the high ER as the model reaches high accuracies, but ultimately we ﬁnd that none of the techniques are able to avoid the loss of ER.
Replay buffer.
Deep neural networks that are subject to training on two distinct tasks sequentially commonly forget learned information about the ﬁrst task after training on the second, a phenomenon known as catastrophic forgetting (Kirkpatrick et al., 2017; Toneva et al., 2019).
A possible hypothesis, therefore, is that retaining learned information about the pre-training dataset – for instance, by maintaining high accuracy on this dataset during the course of ﬁne-tuning to downstream tasks – may have a beneﬁcial effect on ER.
We explore this possibility by implementing a popular technique for mitigating catastrophic forgetting – the use of a “replay buffer" (Ratcliff, 1990; Rolnick et al., 2019).
Without a replay buffer, we ﬁnd that a ResNet-18 model pre-trained on ImageNet and ﬁne-tuned on
CIFAR-10 has a drop in ImageNet test accuracy from 69% to 39% when early stopped at 92% CIFAR10 test accuracy.
When simultaneously ﬁne-tuning on CIFAR-10 with a replay buffer, we retain 63%
ImageNet test accuracy, but we observe no improvement in retained ER.
See Appendix C.11 for more experimental details and results.
Mapping between different class sets.
Using the same constructed map of classes used for zeroshot evaluation (c.f. Section 5.2 and Appendix C.7), we can use the target classes of the pre-training dataset for ﬁne-tuning.
We use the mapping from CIFAR-10 to ImageNet classes, and we use this for ﬁne-tuning on CIFAR-10 initializing with the pre-trained prediction head.
First, we explored ﬁne tuning on just CIFAR-10 images with the 1000 ImageNet target classes.
Second, we used a replay buffer for combined ﬁne-tuning on CIFAR-10 and ImageNet using the same prediction head.
In both setups we see no signiﬁcant change in ER at the end of ﬁne-tuning.
See Appendix C.12 for further details on the experiments and results.
Example difﬁculty.
In Section 5.3 we saw that training on harder examples during ﬁne-tuning increases the ER, but prohibits reaching high ﬁnal accuracy.
We study different training schedules where we vary the difﬁculty of examples during training, but we ﬁnd that none of the proposed methods are sufﬁcient to maintain high ER and reach high test accuracy at the end of training.
C ONCLUSION
Machine learning models currently experience an undesirable yet predictable drop in accuracy on
OOD shifts.
The exceptions from this trend – models with ER – experience less of a drop in OOD accuracy compared to standard testbed models at the same ID accuracy.
The ultimate goal is to produce ER models at state-of-the-art accuracies; however, none of the known models with high ER are in the high-accuracy regime, and the models with the highest OOD accuracy have no ER.
One way to approach this problem is to understand why some models have ER so that we can replicate this behavior for state-of-the-art models and decrease the OOD performance gap.
In this work, we demonstrate that pre-trained models in the middle of ﬁne-tuning, as well as zero-shot pre-trained models, represent an entire class of models that exhibit high amounts of ER.
With this rich set of effectively robust models, we take some initial steps towards understanding what is unique about them that causes them to outperform testbed models on OOD shifts.
Our results suggest that properties of the pre-training or ﬁne-tuning datasets, such as the size, diversity, or difﬁculty, cause effectively robust models to learn an entirely different difﬁculty scale than that of standard testbed models.
Moreover, because ER models make different predictions than standard models and are able to correctly classify examples that no standard models get right, these models are valuable for applications such as model ensembling, which requires prediction diversity.
From our work, we know that pre-training provides a clear beneﬁt to ER, but that ﬁne-tuning on the
ID task reduces ER.
The future challenge lies in constructing mechanisms to enhance ER in the high accuracy regime.
We hope our initial implementations of and investigations into such techniques will be a launching point for researchers for further experimentation.
A S CALING FOR L INEAR F IT
When modeling the linear relationship between the original and new test set for CIFAR-10 and
ImageNet, applying a nonlinear scaling to the accuracy α has shown to improve the linear ﬁt, especially when considering both very high and low performing models.
Recht et al.
(2018; 2019) applies a probit scaling to the accuracies, and Taori et al.
(2020) also considers a logit scaling.
Probit scaling uses the probit function
[Equation]
where Φ − 1 ( α ) is the inverse of the cumulative distribution function of the standard normal distribution; the logit scaling uses the logit function
[Equation]
We compare the goodness of the ﬁt for each of the scalings in Table 2.
We conclude that the logit scaling gives the best linear ﬁt for both CIFAR-10 and ImageNet and will use this scaling throughout this work.
In Figure 2 we show the linear ﬁt in a logit-scaled space and transformed back to linear space for CIFAR-10.
[Table]
Numerical ﬁt values.
All linear ﬁts throughout this work were computed by ﬁrst transforming to logit space (see Section A) and then ﬁtting using scipy.stats.linregress .
Eq.
4 gives the functional form for our ﬁt, and the numerical values are listed in Table 3.
[Equation] [Equation]
[Table] [Table]
B E XPERIMENTAL S ETUP B. P RE - TRAINED I MAGE C LASSIFICATION M ODELS
We conduct our experiments using a diverse set of model architectures and sizes to make sure the observed phenomenon are general and not tied to any particular choice.
We study two groups of models.
ImageNet pre-trained.
The ﬁrst group consists of AlexNet Krizhevsky (2014), ResNet-18, ResNet50, ResNet-152 He et al.
(2016a), VGG11 BN Simonyan & Zisserman (2014), and WideResNet-50Zagoruyko & Komodakis (2016), where we use the P Y T ORCH Paszke et al.
(2019) implementation and weights coming from torchvision.models v0.8.2 where the models were pre-trained on
ImageNet.
When ﬁne-tuning pre-trained ImageNet models on CIFAR-10, we always replace the original classiﬁcation layer with a 10-class randomly initialized classiﬁcation layer.
Big Transfer (BiT).
The second group of models consists of a series of models from Big Transfer (BiT) Kolesnikov et al.
The BiT models all use a ResNet-v2 architecture He et al.
(2016b), except that they replace all Batch Normalization Ioffe & Szegedy (2015) layers with Group Normalization Wu & He (2018) and use Weight Standardization Qiao et al.
in all convolutional layers.
Following Kolesnikov et al.
(2020), we label the ResNet architectures by their depth and the widening factor for the hidden layers, for example R50x1, R101x3 and R152x4, which are the sizes we will consider in this work.
The BiT model come in three different ﬂavors: BiT-S, BiT-M and BiT-L, where S, M and L indicate if the pre-training was done on ILSVRC-2012 (which we denote as ImageNet-1k or just ImageNet)
Russakovsky et al. (2015), ImageNet-21k Deng et al.
(2009) or JFT Sun et al.
(2017), respectively.
In addition, these models may be further ﬁne-tuned, and "1k", "21k" or "JFT" indicates the dataset it was ﬁne-tuned on.
For example, BiT-M-R152x4-1k is the ResNet-152x4 model that was pre-trained on ImageNet-21k ("M") and ﬁne-tuned on ImageNet-1k ("1k").
We also consider a series of BiT models that were pre-trained by Djolonga et al.
(2021) on different random fractions of the JFT dataset with no additional ﬁne-tuning in Section 5.3.
B. F INE -T UNING
Fine-tuning refers to a training setup where the weights learned by training a model on one data distribution are reused when training the model on a second data distribution.
Fine–tuning can either be end-to-end, where the weights learned on the ﬁrst distribution are only used for initialization when training on the second distribution, or last-layer, where only the last layer weights are updated when training on the second distribution.
Fine-tuning can occur in multiple rounds using multiple successive data distributions.
Typically, the earlier training datasets are larger in size and class sets and more diverse.
B. CLIP MODEL
Our CLIP zero-shot model uses the publicly released pre-trained weights for the ViT-B-32 model from https://github.com/OpenAI/CLIP .
Following the methodology from Radford et al.
(2021a), we ﬁne-tune the zero-shot CLIP model using a logistic regression classiﬁer optimized with
L-BFGS, and we determine the L2 regularization strength λ using a hyperparameter sweep on the validation sets with ten logarithmically-spaced values between 10 − 6 and 10 7 .
B. D ATA P REPROCESSING
CIFAR-10.
In our experiments on CIFAR-10, we are interested in testing models that were pretrained on ImageNet-like datasets.
Since the ImageNet images are typically preprocessed to 224x during training, we also preprocess the CIFAR-10/10.1 images by resizing from 32x32 to 224x pixels.
To keep the experimental setup the same, we also resize the CIFAR-10/10.1 images to 224x pixels when using randomly initialized models.
2 All CIFAR images are normalized using the mean =
[0 . 4914 , 0 . 4822 , 0 . 4465] and std = [0 . 2023 , 0 . 1994 , 0 . 2010] .
When evaluating on a test set, or when extracting the features from the penultimate ResNet layer when training only the last layer, we do not apply any preprocessing beyond the upsampling and normalization.
When training the full model on the CIFAR-10 training set (with image size 224x224), we additionally apply RandomCrop with padding=28 and RandomHorizontalFlip .
ImageNet.
The ImageNet training images are preprocessed with RandomResizedCrop topixels and RandomHorizontalFlip , and the test images are resized to 256 pixels and preprocessed with CenterCrop to 224 pixels.
Both are normalized with mean= [0 . 485 , 0 .
456 , 0 . 406] and std= [0 . 229 , 0 . [Figure]
B. C ONFIDENCE I NTERVALS , A VERAGING R UNS , AND E RROR B ARS
Testbed conﬁdence intervals.
For the ImageNet and CIFAR testbed models, described in Section 4, we report conﬁdence intervals on the accuracies calculated using Clopper-Pearson intervals at conﬁdence level 95% Recht et al.
Averaging runs.
For all measurements of ER that were averaged over multiple runs, we report the binned average value, which we calculate using scipy.stats.binned_statistic usingbins.
Error bars.
For most of the results throughout this work, we have left out the error bars to make the plots more readable.
Here, we show the typical error bars for models ﬁne-tuned on ImageNet and
CIFAR-10 in Figure 6.
The reported error bars are the standard deviation from the binned average across 5 runs.
When we are measuring the maximum ER, like in Figure 4(a), we have the choice between using the standard deviation in the bin with maximum ER, or the maximum ER across all the bins.
Since the number of data points in each bin can vary, we observe that the standard deviation in the maximum
ER bin has more variation across different runs.
So, to be conservative, we report the maximum
ER across all the bins.
A comparison between the two choices of standard deviation can be seen in
Figure 7.
B. O PTIMIZER AND H YPERPARAMETERS
Here we give the details used in the training of all our results.
For all models we ran ﬁve runs and all were trained using stochastic gradient descent with momentum 0.9 and weight decay 10 − 4 , unless otherwise speciﬁed.
The default settings to train the full P Y T ORCH models (see Section B.1) was using batch size 64 for 250 epochs using learning rates [0 . 1 , 0 . 01] for the randomly initialized model, and for 100 epochs using learning rates [0 . 01 , 0 . 001 , 0 . 0001] for the pre-trained models.
The main results shown in the main text and the appendix are using learning rate 0.01 and 0.001 for the randomly initialized and pretrained model, respectively, but we have checked that the general conclusions are the same across the different learning rates.
When training the BiT models, we extract the features from the penultimate layer and only train the prediction head (except for the randomly initialized BiT models where we train the full model).
Unless otherwise speciﬁed, the models are trained using learning rate 0.001 with batch size 32768 for 100 epochs with no weight decay. [Figure] a [Figure]
E ﬀ [Figure] t i v e R
[Figure]
bu s t n e
[Figure]
[Figure]
C A DDITIONAL E XPERIMENT D ETAILS AND R ESULTS
This section describes we additional experimental results and details of training not described in the main paper.
C. P RE - TRAINED VERSUS R ANDOMLY I NITIALIZED M ODELS
In Section 5.1 we discussed how pre-trained models exhibit ER during ﬁne-tuning, while randomly initialized models have no ER during training.
Here we add additional experimental details that did not ﬁt into the main paper.
ImageNet.
In Figure 8(a) we show the ER during training on ImageNet for a BiT-L-R101x3 model that was pre-trained on JFT in comparison to a randomly initialized model of the same architecture.
The pre-trained model is averaged across 5 runs and is only training the last layer.
For the randomly initialized model we train the full model, but due to the high computational cost, we only perform one run.
The pre-trained model is using hyper-parameters as described in Appendix B.6 while the randomly initialized model was trained following the setup as described in Kolesnikov et al.
CIFAR-10.
In Figure 8(b) we show the exact same experiment, as described in the previous paragraph, on CIFAR-10.
Again, the randomly initialized model is only run a single time due to the computational cost.
For CIFAR-10 we also perform extensive comparisons between several different architectures where we either use pre-trained or randomly initialized weights.
The results are shown in Figure 10 and the hyper-parameters are as described in Appendix B.6.
C. I MAGE N ET R OBUSTNESS B ENCHMARKS
For models that are ﬁne-tuned on ImageNet, we study the ER on ImageNetV2, ImageNet-R and
ObjectNet.
We train one BiT-L-R152x4_JFT model on ImageNetV2, and evaluate the same model on all three robustness benchmarks.
The ER for ImageNet-R and ObjectNet was shown in Figure 3(c) and 3(d).
In Figure 9 we show the same data as in Figure 3(c) and 3(d), but in term of accuracies in stead of ER.
While the ER on ImageNetV2 goes to zero at the end of ﬁne-tuning, the ER for
ImageNet-R and ObjectNet remains non-zero.
As discussed in Section 5.1, this is consistent with previous observations in the literature.
Besides observing that the accuracy on ImageNet-R and
ObjectNet stops at around 60%, we do not know why this is happening for these two datasets but not for ImageNetV2. [Figure]
[Figure]
model that is ﬁne-tuned on ImageNet.
Where the ImageNetV2 ER goes to zero at the end of ﬁne-tuning, the pre-trained model still has non-zero ER.
C. A RCHITECTURE I NDEPENDENCE
We observe ER of pre-trained models across different model architectures in Figure 10, and none of these architectures have any signiﬁcant ER when randomly initialized.
We used P YTORCH models (c.f.
Section B.1) pre-trained on ImageNet and replaced the prediction head with a randomly initialized layer for CIFAR-10.
Both the pre-trained and randomly initialzed models were trained with SGD with learning rates 0.01, 0.001 and 0.0001, and batch size 64, momentum 0.9, and weight-decay 5 · 10 − 4 .
We show the results from averaging ﬁve runs using learning rate 0.001 in Figure 10, but the results are similar using the other learning rates (learning rate dependence is studied in Appendix C.5).
The randomly initialized models were trained for 250 epochs, while the pre-trained models were trained for 100 epochs.
C. L OSS FUNCTION
In this section we show that our observations about ER are not unique to models trained using cross-entropy loss, and we get the same behaviour using mean square error (MSE). [Figure]
[Figure] For evaluation of the MSE loss, we compute the MSE between the output logits and a one-hot target vector.
We train the model using stochastic gradient descent with constant learning rate 0.
and batch size 4096 for 100 epochs.
In Figure 11 we compare the ER duing ﬁne-tuning of the last layer of a JFT pre-trained BiT-L-R152x model where one is trained using MSE and the other is trained using cross-entropy loss.
In this speciﬁc experiment, the MSE trained model has higher ER at its maximum, but due to the other similarities between the evolution we do not believe this would always be the case.
C. L EARNING R ATE
The observed ER during ﬁne tuning is largely independent of the learning rate used during optimization.
In Figure 12 we show the ER during training on CIFAR-10 for a pre-trained and randomly initialized Wide-ResNet-50-2 using the same hyper-parameters as in Appendix C.3.
[Figure]
C. F ULL M ODEL VS . L AST L AYER F INE -T UNING
The ER during ﬁne-tuning of pre-trained models is largely insensitive to whether the full network is trained or if we only train the last layer.
In Figure 13 we compare the ER of full model versus last layer ﬁne-tuning for the six P YTORCH models described in Appendix B.1.
We start both last layer and full model ﬁne-tuning with the same pre-trained weights for that architecture.
For training of the full model, we use SGD with learning rate 0.001, and batch size 64, momentum 0.9, and weight-decay 5 · 10 − 4 for 250 epochs.
When training only the last layer, we use
SGD with learning rate 0.0001, and batch size 4096, and without momentum or weight-decay, and we trained for 1000 epochs.
We experimented with learning rate schedulers, but saw no signiﬁcant change.
Despite the difference in training and hyper-parameters, we observe that both have about the same amount of ER.
The biggest observed difference is that the only last-layer trained models don’t reach the same ﬁnal accuracy as when training the full model, which is to be expected as the last-layer model cannot change its pre-learnt features.
C. Z ERO -S HOT E VALUATION
Zero-shot evaluations on traditional image classiﬁers is made possible with hand curated maps between the classes of one dataset to another.
To map from CIFAR-10 to ImageNet, we utilize the mapping from ImageNet to CIFAR-10 classes used in CINIC-10 Darlow et al.
and we utilize a previously constructed map from ImageNet to JFT to evaluate the performance of BiT-L models on
ImageNet in a zero-shot manner.
These maps were used for zero-shot evaluation on ImageNet and ImageNetV2 in Figure 1 and discussed in Section 5.2.
The map from CIFAR-10 to ImageNet was used in Appendix C.12.
In this section we investigate different ways of combining the multiple logits corresponding to each
CIFAR-10 class to make predictions using a model pre-trained on ImageNet.
Given a CIFARclass, we collect the corresponding ImageNet softmax scores and combine the probabilities using three different functions: max, mean and sum.
We show the results in Figure 14, where we see that most of the zero-shot models have some amount of ER, but are not signiﬁcantly more robust than pre-trained models in the middle of ﬁne-tuning.
We see no signiﬁcant difference between the three combination functions we test, and we use the max as the default in all other plots throughout our work.
C. D ATASET S IZE AND D IVERSITY
In Section 5.3 we summarized our experiments where we vary the pre-training dataset for different ﬁxed BiT architectures.
Here, we will include additional experimental details and observations.
[Figure] [Figure] some amount of ER. All models in this section was ﬁne-tuned using SGD with learning rate 0.001, batch size 4096, with no momentum or weight decay, and we trained for 1000 epochs.
JFT dataset size.
Using a series of BiT-R101x3 that were pre-trained on different randomly sampled fractions of the JFT-300M dataset Sun et al.
, we show that the maximum ER increases with the pre-training dataset size.
Since the pre-training images were randomly sampled from one dataset, it gives us some control for the dataset diversity as opposed to our experiments where we compare models trained on ImageNet-1k, ImageNet-21k and JFT.
However, since the models were trained on different amounts of data, we cannot rule out any effects from the pre-training optimization on the maximum ER.
Different pre-training datasets.
When using different pre-training datasets, we ﬁnd that larger and more diverse datasets give higher maximum ER during ﬁne-tuning on CIFAR-10.
We show the ER during ﬁne-tuning for three different BiT model sizes in Figure 17.
For R152x
and R-101x3 we see that the BiT-L models that were pre-trained on JFT has the highest ER and the
BiT-S models that were pre-trained on ImageNet-1k has the lowest.
We can also make an interesting observation by looking at the two BiT-M models: The BiT-M-21k model (pre-trained on ImageNet21k) has higher ER than the BiT-M-1k model (pre-trained on ImageNet-21k and then ﬁne-tuned on
[Figure]
[Figure]
ImageNet-1k).
In other words, ﬁne-tuning on the smaller and less diverse ImageNet-1k dataset before training on CIFAR-10 gives about a 1 percentage point drop in maximum ER.
While the observed trend that the maximum ER is higher for larger and more diverse pre-training datasets seems very robust, we observe a deviation from this trend for the BiT-S-R50x1 model.
The
BiT-S-R50x1 model has higher maximum ER than the BiT-L-R50x1 and BiT-M-R50x1 models.
Also, it has higher ER than the comparable model with larger model architecture (BiT-S-R101x
and
BiT-S-R152x4), thus breaking yet another trend.
However, we do not know if this is caused by an interesting phase transition when using the combination of the smallest pre-training dataset and the smallest model, or if it is simply caused by some part of the pre-training optimization that we do not have access to.
Model size.
In Figure 18 we compare the ER during training for three different BiT architecture sizes for ﬁxed pre-training dataset.
For all but the BiT-S model (as discussed above), we ﬁnd that the
ER increases with increased model size.
This is a plausible outcome since bigger models can build more expressive hidden representations.
[Figure] [Figure]
E XAMPLE
IFFICULTY
In this section we explore the ways in which the ﬁne-tuning dataset affects the ER on an OOD dataset.
We study a ResNet-18 model that is either randomly initialized or pre-trained on ImageNet.
Both sets of models are trained with a single run for 100 epochs using batch size 64, and the pre-trained and randomly initialized models use learning rate 0.001 and 0.01, respectively.
Fine-tuning with easy, random or hard examples.
Using the C-score Jiang et al.
(2020) as a metric for example difﬁculty, we select 5 , 000 of the easiest, hardest or random examples while maintaining class balance (i.e. 500 examples from each class).
In Figure 4(c) we show the ER during training, and we make two observations: First, training on more difﬁcult examples gives higher ER than when training on easy or random examples.
Second, the models that train on only hard examples reaches a lower ID accuracy than the easy or random models.
Phasing out easy examples during ﬁne-tuning Since the ﬁne-tuning dataset affects the ER during ﬁne-tuning, there might be a way to choose what examples from the ﬁne-tuning dataset to train on that achieves both high ID accuracy and high ER.
In Figure 19 we show the result from experiments on randomly initialized and pre-trained models where we start the ﬁne-tuning process with the full
CIFAR-10 training set, and we gradually phase out the easy examples until there are only 1000,or 10000 examples left in the last epoch of training.
However, as we can see this does not seem to have any effect on the ER.
An alternative approach we attempted was switching between random, easy and hard, or ﬁrst ﬁnetuning on all the data and switching to only the hard examples at the end.
Generally, we found that when a model that reached high ID accuracy was being further ﬁne-tuned on difﬁcult examples, the
ER does go up, but the ID accuracy also goes down.
The gradual phase-out described in the previous section was the most consistent way to reach high ID accuracy while only training on hard examples by the end of training.
[Figure]
We show this effect for four different BiT architecture sizes for three different ﬁxed pre-training datasets.
[Figure]
C. D OMINANCE PROBABILITY
The dominance probability Mania & Sra (2020) compares the predictions of two models and is the probability that the lower accuracy model correctly classiﬁes a given example which the higher accuracy model incorrectly classiﬁes.
Formal deﬁnition Mania & Sra (2020).
Let X be a covariate space, Y be a discrete label space, and P be a probability distribution over X × Y .
Suppose model f maps X → Y and has accuracy µ =
E
P [ 1 ( f ( x ) = y )] .
Given any two models f i and f j with µ i ≤ µ j , the dominance probability is given by
[Equation]
Pairwise dominance probability comparison.
We compute the pairwise dominance probabilities on the ImageNet validation set for the following set of models: the ImageNet testbed models, two
BiT-L models (BiT-L-R152x4 early stopped at maximum ER, and the same model at end of training when the model has converged), and our zero-shot and ﬁne-tuned CLIP models (see Appendix B.3).
Figure 21 shows a heatmap of the full set of pairwise dominance probabilities (a smaller version appears in Figure 5 in the main paper).
The models are sorted by increasing ImageNet validation accuracy and the lighter colors in the heatmap correspond to models with higher dominance probabilities.
Dominance probability is not symmetric, but to make the heatmap readable using either the row or column models as the more accurate model, we mirror the dominance probabilities across the y = x axis.
We observe that the models with the highest dominance probabilities, the BiT-L-R152x4 Max ER model and the CLIP zero shot model, also have the highest effective robustness.
However, the CLIP
L-BFGS model has lower effective robustness than the BiT-L-R152x5 converged model ( 0% versus 1%, respectively), but slightly higher average dominance probability.
This suggests that effective robustness alone is not the sole inﬂuencer of dominance probability, and some component of the
CLIP pre-training, such as the natural language supervision, contrastive learning objective, or large and diverse pre-training dataset, must contribute to the high dominance probabilities for even the converged CLIP model.
Distributional closeness.
To characterize distributional closeness, Mania & Sra (2020) also introduce another metric which we call the “triplet-probabilities", measuring how close the predictions between a triplet of models are (see Mania & Sra (2020) for a formal deﬁnition and discussion).
In Figure 20, we show the triplet-probabilities for the same set of models as for the dominance probabilities in the last section.
We observe that the effectively robust models do not violate the notion of distributional closeness as deﬁned by the triplet probabilities, i.e. the majority of the triplet probabilities for the effectively robust models are in the same cone as the standard testbed models.
Discussion.
Mania & Sra (2020) proved that models’ accuracies are collinear under distribution shift under the assumptions of (a) low dominance probabilities and (b) distributional closeness. Conversely, if we see deviations from the linear ﬁt, at least one of these assumtions must be violated. Oour results show that effectively robust models (which do not exhibit collinear accuracies under distribution shift) break the dominance probability assumption rather than the distributional closeness assumption.
We note that ImageNet and ImageNet-V2 are explicitly constructed as to be similar, so for other OOD test sets one might expect that the distributional closeness will also be violated.
[Table]
n
[Table]
l a
[Table]
e
[Table]
f pn
[Table]
a
[Table]
e n a s n e
[Table]
l a
[Table]
e t f
B i
[Table]
n v e r
[Table]
e
[Table]
n
[Table]
n e
[Table]
a
[Table]
e
[Table]
i
[Table]
n
[Table]
e
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
nasnetamobileinceptionv2tfnasnetmobiletfbninceptionvgg16bnresnet34vgg19vgg16vgg13bnmobilenetv1tfvgg19tfvgg16tfvgg11bnvgg13inceptionv1tfresnet18vgg11CLIP
[Figure]
C. R EPLAY B UFFER : R EPLAYING I MAGES FROM THE P RE - TRAINED D ATASET
Our experiments using a replay buffer discussed in Section 5.5 were conducted on a ResNet-18 model that was pre-trained on ImageNet.
As a baseline, we use the pre-trained ResNet-18 model ﬁne-tuned on only CIFAR-10 in Appendix C.3.
Next, we ﬁne-tune the pre-trained ResNet-18 model on both CIFAR-10 and ImageNet using batch size 256 and learning rate 0.001 for 100 epochs.
The replay buffer model uses two prediction heads, one for ImageNet and one for CIFAR-10.
For ImageNet, we kept the original 1 , 000 class prediction head from the pre-trained model, and the CIFAR-10 prediction head was randomly initialized.
During ﬁne-tuning, we varied the ratio of ImageNet to CIFAR-10 images to be 1, 10, or 100.
The results are shown in Figure 22.
We observe that none of the replay buffer models are able to maintain high ER at high ID accuracy.
This is despite the fact that such a training procedure still maintains – by design – a high accuracy on the pre-training dataset.
Before any ﬁne-tuning, the top-1 accuracy of the pre-trained ResNet-18 on the ImageNet test set is 69%.
This drops to 39% during ﬁne-tuning on CIFAR-10 when early stopped at 92% CIFAR-10 test accuracy; however, with the use of a replay buffer, the ﬁnal ImageNet test accuracy is 63% at the end of ﬁne-tuning on CIFAR-10.
This is intriguing since it demonstrates that a model that is forced to maintain high accuracy on a different OOD dataset during the course of ﬁne-tuning on the target dataset does not do better on the (closer)
OOD dataset of interest.
For this particular setup, one might have imagined that a model that continues to perform well on ImageNet during ﬁne-tuning on CIFAR-10 has learned more general properties of the image data manifold than it otherwise would have, and that this might be beneﬁcial in ER on CIFAR-10.1.
In conclusion, using a replay buffer does help prevent catastrophic forgetting, but it does not prevent the ER on CIFAR-10 from disappearing towards the end of ﬁne-tuning.
[Figure]
C. C LASS M APPING
Instead of starting from a randomly initialized prediction head when ﬁne-tuning a pre-trained model on CIFAR-10, we can use a pre-trained prediction head if we map the ten CIFAR classes to the 1 , 000 ImageNet classes.
For the map from CIFAR-10 to ImageNet, we use the map as deﬁned by
Darlow et al.
(2018) in the construction of the CINIC-10 dataset.
With this setup, we conduct several experiments to test if there is any beneﬁt from the pre-trained prediction head in maintaining ER at high ID accuracies.
Since each CIFAR-10 class can map to multiple ImageNet classes, we use a cross-entropy loss where we select the max logit across the multi-target classes.
We train the last layer of a BiT-M-R152x4_1k
[Figure]
Replay buffer with class mapping.
Since CIFAR-10 and ImageNet now have the same target classes, we can repeat the buffer replay experiments using the same prediction head for both datasets.
In Figure 23(a) we show the results from training using the same number of images from CIFARand ImageNet, and also with 100 times more ImageNet images, but we see no improvement in maintaining high ER.
Weight regularization.
To test if any of the beneﬁt of the pre-training comes from the pre-trained prediction head, we introduce a L2 regularization term to keep the weights close to the pre-trained weights of the prediction head.
We introduce a pre-factor λ to the L2 regularization in the loss, but we see no improvement in ER from small λ despite a small reduction in ID accuracy relative to the results in Figure 23(a).
For large λ , the L2 regularization dominates, and we do not even reach high
ID accuracies.
D S HAPE OF THE E FFECTIVE R OBUSTNESS C URVE
In our experiments, we observe that the ER of pre-trained models starts out near zero, increases during training, reaches a peak, and then diminishes towards the end of training.
Interestingly, we ﬁnd that the existence of the peak is in part caused by the linear ﬁt in logit space and the deﬁnition of ER.
To understand this, we plot the ER of the y = x line (i.e. the line “Same accuracy ( y = x )" in Figure 1 and 2) along with the ER of pre-trained BiT models in the left plot of
Figure 24 (CIFAR-10) and 25 (ImageNet).
The ER of the y = x
curve also has the same observed shape with a peak.
Hence, to isolate the effect of deviating from the linear relationship from the natural shape of any ER line we also show the ER as a fraction of the accuracy gap between the linear ﬁt and the y = x line in the right plot of Figure 24 for CIFAR-10 and Figure 25 for ImageNet.
We observe that these curves do not have the same maximum as observed in the ER, and instead they have a mostly decreasing trend during ﬁne-tuning.
D. M IXED C LASSIFIER
Due to the convex shape of the testbed model accuracies (without the logit scaling), one way to construct a model that has ER is to randomly sample predictions from a low accuracy testbed model with probability α and from a high accuracy testbed model with probability 1 − α .
We call this model a mixed classiﬁer with parameter α .
As α varies from 0 to 1, the accuracies on the ID and OOD
[Figure] [Figure]
test sets traces out a straight line between the accuracy of the random and highest performing model, which will have ER due to the convexity of the logit-scaled linear ﬁt.
While mixed classiﬁers hold theoretical interest and should be analyzed more in future work, their use in practical applications is limited since we cannot construct a mixed classiﬁer that has both high ER and high ID accuracy.