A N E MPIRICAL S TUDY OF P RE - TRAINED V ISION M ODELS ON O UT - OF - DISTRIBUTION G ENERALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Generalizing to out-of-distribution (OOD) data – that is, data from domains unseen during training – is a key challenge in modern machine learning, which has only recently received much attention.
Some existing approaches propose leveraging larger models and pre-training on larger datasets.
In this paper, we provide new insights in applying these approaches.
Concretely, we show that larger models and larger datasets need to be simultaneously leveraged to improve OOD performance on image classiﬁcation.
Moreover, we show that using smaller learning rates during ﬁne-tuning is critical to achieving good results, contrary to popular intuition that larger learning rates generalize better when training from scratch.
We show that strategies that improve in-distribution accuracy may, counter-intuitively, lead to poor OOD performance despite strong in-distribution performance.
Our insights culminate to a method that achieves state-of-the-art results on a number of OOD generalization benchmark tasks, often by a signiﬁcant margin
I NTRODUCTION
Most machine learning (ML) models assume that test data is drawn from the same distribution as training data.
However, this assumption does not hold in many real-world applications.
As a result, ML models often fail to generalize to out-of-distribution (OOD) data encountered during their deployment and suffer from signiﬁcant performance drops compared with the model performance on in-distribution (ID) data (Qui˜nonero-Candela et al., 2009; Torralba & Efros, 2011).
For example, common distribution shifts prevalent during test time include variation in locations (Koh et al., 2021) and weather (Volk et al., 2019), noise and blur corruptions (Hendrycks & Dietterich, 2018), and small adversarial perturbations (Szegedy et al., 2013).
As ML models are increasingly deployed in safety-critical applications, it is becoming ever more critical to ensure strong OOD generalization for such models, i.e., the models robustly generalizing to relevant OOD data not seen during training.
While this problem is indeed difﬁcult since the goal is to generalize to data that are not seen during training, there have been a handful of methods recently proposed to improve OOD generalization.
Some methods propose specialized training methods, such as simulating OOD data during training (Li et al., 2018a), learning invariant representations (Arjovsky et al., 2019), and performing adversarial data augmentation (Volpi et al., 2018).
Intriguingly, Gulrajani & Lopez-Paz (2020) conducted an extensive empirical evaluation on domain generalization benchmark datasets, and demonstrated that classical empirical risk minimization (ERM) approach achieves nearly state-of-the-art OOD generalization performance compared with these specialized methods.
On the other hand, most of the approaches apply small or medium size networks that are usually pre-trained on the ImageNet-1k dataset, such as pre-trained ResNet50 (Gulrajani & Lopez-Paz, 2020).
On the other hand, recent works ﬁnd that pre-training on larger and more diverse data is one of the most effective paths toward generalizing to out-of-distribution data on ImageNet (Taori et al., 2020).
In this paper, we systematically investigate the importance of pre-trained models for OOD generalization.
Speciﬁcally, we conduct extensive experiments on models with different model sizes that are pre-trained on large datasets.
The pre-trained models are then ﬁne-tuned on the training data for the underlying task.
Instead of focusing on achieving state-of-the-art results on benchmark
OOD datasets, our empirical study aims to develop a better understanding of the critical role that pre-trained models along with different design choices for ﬁne-tuning such models play in ensuring
[Figure]
good OOD generalization.
This provides novel insights towards closing the gap between ID and
OOD generalization for future research.
The main contributions of our work are as follows:
• When leveraging larger models for OOD generalization, we ﬁnd that both large pre-training dataset sizes and large model sizes are critical.
Missing either one of the two components may hurt OOD generalization.
• We show that using a small learning rate generalizes better for OOD when we leverage a pretrained model.
This is a complementary argument to Li et al.
(2019) that suggested to use a large learning rate for better generalization in the case of no pre-training.
• We show cases where improving in-distribution performance actually leads to worse OOD performance suggesting that ID performance is not a reliable indicator of OOD performance.
• Our insights culminate in a method that achieves SOTA, often by a signiﬁcant margin, on a number of OOD generalization benchmark tasks including PACS, VLCS, and Ofﬁce-Home.
P RELIMINARIES AND E XPERIMENTAL S ETUP
We begin this section by introducing the out-of-distribution generalization problem, with the primary focus on a special case of this broader issue, namely domain generalization .
Subsequently, we describe the experimental setup adopted in our empirical study and various parameter choices we make throughout the paper.
In this paper, we study a general multi-class classiﬁcation setting, with all input instances and their labels belonging to X and Y :
= { 1 , . . .
, K } , respectively.
Let D denote the training dataset which may potentially comprises data belong to k different training domains {D j } j ∈ [ k ]:= { 1 ,...,k } , i.e., D = ∪ j ∈ [ k ] D j .
We assume that the training data from the j -th domain D j = { ( x ji , y ji ) } i ∈ [ n j ] ⊂ X × Y is sampled from the distribution P j id , i.e., ( x ji , y ji ) ∼ P j id .
At test time, we evaluate a trained model for both its in-distribution and out-of-distribution performance.
The in-distribution performance is evaluated on a test dataset T id that consists of instances belonging to the domains encountered in the training dataset D .
On the other hand, we utilize an OOD test dataset T ood from an unseen domain with distribution P ood to assess the model’s
OOD performance.
Given a family of candidate classiﬁers F = { f : X → Y} and the underlying training dataset D , we primarily employ standard empirical risk minimization (ERM) (Vapnik, 1998) to learn a classiﬁer ˆ f as follows:
[Equation]
where (cid:96) ( · , · ) denotes the cross-entropy loss function.
For a classiﬁer f , we deﬁne its in-distribution accuracy Acc id ,f and out-of-distribution accuracy Acc ood ,f as follows:
[Equation]
where 1 {·} denotes the standard indicator function.
Often, models that achieve large in-distribution accuracy Acc id only achieve relatively small out-of-distribution accuracy Acc ood , i.e., Acc id (cid:29)
Acc ood
Torralba & Efros, 2011; Hendrycks & Gimpel, 2016; Gulrajani & Lopez-Paz, 2020).
Thus, under the domain generalization problem, one particularly focuses on designing training methods that result in classiﬁer with good performance on both out-of-distribution data and in-distribution data.
Pre-trained models.
We mainly focus on ﬁne-tuning pre-trained models on the training dataset D by using ERM.
We explore four classes of pre-trained models for OOD generalization: (1).
ResNetbased models (He et al., 2016a; Xie et al., 2017) pre-trained on ImageNet (Russakovsky et al., 2015) (ResNet50, ResNext50-32x4d, and ResNext101-32x8d); (2).
(BiTm)-ResNet-v2-based models (He et al., 2016b) pre-trained on ImageNet-21k (Deng et al., 2009) (ResNetV2-50x1, ResNetV2-50x3, and
ResNetV2-101x1) (Kolesnikov et al., 2020), where group normalization (Wu & He, 2018) and weight standardization (Qiao et al., 2019) are used in ResNetV2; (3).
(SWSL)-ResNet-based semi-weakly supervised ImageNet models pre-trained on IG-1B-Targeted data (Yalniz et al., 2019); and (4).
Vision transformer (ViT) pre-trained on ImageNet-21k (Dosovitskiy et al., 2020).
A detailed description of these pre-trained models can be found in Table 3 (in Appendix).
OOD datasets.
In our experiments, we use four vision datasets used to benchmark domain generalization algorithms (Gulrajani & Lopez-Paz, 2020): (1).
PACS dataset (Li et al., 2017); (2).
Ofﬁce-Home dataset (Venkateswara et al., 2017); (3).
VLCS dataset (Fang et al., 2013); and (4).
TerraIncognita dataset (Beery et al., 2018).
Each of these datasets contains 4 different domains.
We train the models on 3 domains and treat the examples from the remaining domain as the out-ofdistribution data T ood .
For the three training domains, we use 80% data as training dataset and the remaining 20% data for evaluation.
We add the test domain information after the dataset to specify the test domain, for example, PACS (S) means the training domains are ‘P’, ‘A’, and ‘C’ and the test (OOD) domain is ‘S’.
Fine-tuning and model selection.
We use stochastic gradient descent (SGD) with a momentum of 0.9 for ﬁne-tuning all pre-trained models considered in this paper.
The default weight decay for SGD is set to be 0.
We use a cosine learning rate decay (Loshchilov & Hutter, 2016) as learning rate scheduler for SGD.
For initial learning rates η we compare the following set { 0 . 05 , 0 . 02 , 0 . 01 , 0 .
For evaluation, we pick the ﬁve checkpoints from each model with highest in-distribution accuracy.
We then compute the average OOD accuracy of these ﬁve checkpoints.
M AIN RESULTS
We present our main experimental results in this section.
First, we highlight the importance of models pre-trained on large and diverse datasets for OOD generalization.
Next, we investigate the effect of ﬁne-tuning learning rates, especially for models pre-trained on diverse datasets.
Then, we perform a systematic examination of several components of pre-trained models for OOD generalization, including model size, pre-training dataset, and model architecture.
Finally, we evaluate whether techniques used for improving ID accuracy can also enhance OOD generalization.
3. I MPORTANCE OF A BETTER PRE - TRAINED MODEL
Models pre-trained on more diverse datasets have been shown to achieve better OOD generalization on real-world distribution shifts (Taori et al., 2020;
Hendrycks et al., 2020a).
As a warm-up for understanding the properties of pre-trained models on OOD data, we ﬁrst study the OOD generalization performance of a ResNet-based model that is pre-trained on a large and diverse pre-training dataset.
In particular, we focus on an SWSL-ResNext101-32x4d model pre-trained on the IG-1B-Targeted data (Yalniz et al., 2019), which is a much larger and more diverse dataset than ImageNet.
The ﬁne-tuning approach described in Section 2 with different learning rate signiﬁcantly outperforms the baseline results from Gulrajani & Lopez-Paz (2020) (cf.
Table 1).
This shows that a better pretrained model indeed improves OOD generalization without using specialized algorithms for domain
[Table]
[Figure]
(h)
[Figure]
(L
[Figure]
generalization.
For example, the OOD accuracy improves from 79 .
1% to 91 .
3% on PACS (S), and 62 .
7% to 76 .
4% on Ofﬁce-Home (A).
Overall, Table 1 suggests that using a larger model pre-trained on more data can be very effective for better OOD generalization.
Next, we conduct more detailed experiments to better understand the OOD generalization performance of various pre-trained models.
3. E FFECT OF FINE - TUNE LEARNING RATE
Given the fact that simply ﬁne-tuning the SWSL model leads to compelling improvement in the OOD generalization, we now take a closer look at models trained with different learning rates.
We evaluate models trained with different learning rates on both ID and OOD test data.
Figure 2 summarizes the results for ﬁne-tuning SWSL-ResNext101-32x4d with different learning rates.
In Figure 2(a)-2(d), each point in the plot corresponds to a model trained with a distinct learning rate.
As mentioned in
Section 2, we select models that achieves the top-5 ID accuracy, and depict the standard deviation of both Acc id and Acc ood for each model.
We only show results for models that achieve > 95% training accuracy to better compare model performance.
Regularization effect of small learning rate.
Based on Figure 2, our main ﬁnding is that the ﬁne-tuning learning rate plays a key role in determining both ID and OOD accuracy.
In particular, we
[Figure]
This is different from the fact that larger learning rates generalize better when the model is directly trained on the training data D without a pre-training phase (Li et al., 2019; Lewkowycz et al., 2020; Nakkiran, 2020). We also observe similar behavior with SWSL-ResNext101-32x8d models, where smaller learning rates lead to better OOD performance (see Figure 9 in Appendix). Intriguingly, we do not observe this phenomenon for models pre-trained on ImageNet.
There, as shown in Figurein Appendix, when the ID accuracy is similar, a model trained with larger learning rate achieves better OOD generalization (e.g., see Figure 12(e) and 12(f)).
To summarize, our results indicates that when models are pre-trained on more diverse datasets, smaller learning rates can lead to better OOD generalization.
To obtain a better understanding of the effect of learning rate, we also study the OOD accuracy during training.
In Figure 3, we visualize the OOD accuracy vs. training loss as measured every 100 SGD iterations for models trained with three different learning rates η ∈ { 0 . 01 , 0 .
Note that all three learning rates eventually achieve similar training loss, but the models trained with smaller learning rates have better OOD generalization.
Figure 3 also suggests that models trained with larger learning rates cannot achieve similar OOD accuracy by using early stopping.
Meanwhile, the ﬁgure also conﬁrms the regularization effect of small learning rates for ﬁne-tuning pre-trained models.
Overall, we ﬁnd that the learning rate is a key parameter for achieving good OOD generalization.
While different learning rates may not affect ID accuracy much, OOD accuracy can be very sensitive to the choice of learning rate.
Our results also highlight a limitation of performing model selection based on ID accuracy as models with similar ID accuracy may have very different OOD performance.
3. P RE - TRAINING FOR BETTER OOD GENERALIZATION
Now, we systematically explore the role that pre-trained models play in improving OOD generalization.
Speciﬁcally, we study three aspects of pre-trained models: pre-training dataset, model architecture, and model size.
Furthermore, we compare the models trained from scratch (i.e., without pre-training) with pre-trained models with respect to OOD generalization.
Effect of pre-training dataset.
We study the OOD performance of models pre-trained on different datasets, inlcuding ResNext101-32x8d pre-trained on ImageNet, BiTm-ResNetV2-50x3 pretrained on ImageNet-21k, and SWSL-ResNext101-32x8d pre-trained on IG-1B-Targeted.
The SWSL model and BiTm model achieve similar Top-1 accuracies on ImageNet, 84 .
2% and 84 .
0% , respectively, and the standard ResNext101-32x8d achieves a slightly worse in terms of Top-1 accuracy of 79 .
The results for this comparison are summarized in Figure 4 and Figure 15 (in Appendix).
Our ﬁrst observation is that the pre-training dataset has a big impact on OOD generalization performance.
With same architecture and model size, the SWSL pre-trained model consistently outperforms the standard ImageNet pre-trained model across.
Secondly, we ﬁnd that SWSL generally performs the best among all three models as it is pre-trained on the largest and the most diverse pre-training dataset.
Furthermore, we ﬁnd that the BiTm model pre-trained on ImageNet-21k also generally
[Figure] [Figure]
performs better than the ResNext model pre-trained on ImageNet (except for the PACS dataset, where both models have similar OOD performance).
Another interesting ﬁnding is that the OOD accuracy can be signiﬁcantly improved by chosing the right ﬁne-tuning learning rate, while the improvement in the ID accuracy is small.
For example, in Figure 4(a), the ID accuracy improves from 98% to 99% , whereas the OOD accuracy increases ∼ 10% .
Our results suggest that the “ accuracy on the line ” phenomenon (Miller et al., 2021) does not always hold for the benchmark datasets used in domain generalization.
This also echoes the observation from D’Amour et al.
(2020), that OOD generalization can be very different among models with similar ID accuracies.
Effect of model architecture: ViTs vs. CNNs.
We now investigate the role that the model architecture plays in ensuring good OOD generalization.
Compared with convolutional neural networks (CNNs), the recently proposed Vision Transformers (ViT) achieves similar or even better performance on image classiﬁcation tasks (Dosovitskiy et al., 2020).
This raises the question whether vision transformers behave differently from CNNs in terms of their OOD performance?
Towards this, we explore three BiT-ResNetV2 pre-trained models (Kolesnikov et al., 2020) and four ViT pre-trained models (Dosovitskiy et al., 2020).
In particular, we compare BiTm-ResNetV2- { 50 x 1 , 101 x 1 , 50 x 3 } with ViT- { small-patch32, small-patch16, base-patch32, base-patch16 } .
We present the comparison between ViTs and BiTms on OOD benchmarks in Figure 5 and Figure 16 (in Appendix).
We ﬁnd that the OOD generalization accuracy of ViT models is similar to that of BiTm models.
In some cases, BiTs slightly outperform ViTs on OOD generalization, for example, results shown in
Figure 5(b), 5(c), 16(a).
Since both classes of models are pre-trained on the same pre-training dataset (i.e., ImageNet-21k) and achieve similar ImageNet Top-1 accuracies ( 84 .
5% for ViTs vs. 84 .
0% for
BiTms)
, our results indicate that replacing convolution operation with self-attention operation does not bring additional beneﬁts in terms of OOD generalization for the settings we consider in this paper.
Effect of model size.
It is evident from Figure 4, where both the baseline model (i.e., ResNet50) and ResNext101-32x8d are pre-trained on the ImageNet-1k dataset, that increasing the model [Figure] size from ResNet50 to ResNext101-32x8d can improve OOD generalization.
For example, OOD generalization is improved from 78 .
0% to 84 .
9% in Figure 4(a).
Motivated by these promising results, we conduct experiments to understand the role of model size for OOD generalization.
We consider increasing model size on three classes of models, including ResNe(x)t pre-trained on ImageNet-1k,
BiTm-ResNetV2 pre-trained on ImageNet-21k, and ResNe(x)t pre-trained on IG-1B-Targeted.
We investigate three model sizes for each class of pre-trained models and the results are summarized in
Figure 6 and Table 4-6 (in Appendix).
For all three classes of models, we ﬁnd that increasing the model size can improve OOD generalization in many settings.
Overall, this shows that model size is a crucial design choice for improving OOD generalization.
Furthermore, we note that SWSL-ResNet50 is pre-trained on a larger and more diverse dataset than BiTm-ResNetV2-50x3 and both models achieve ∼ 100% training accuracy, but
SWSL-ResNet50 has lower OOD accuracy than BiTm-ResNetV2-50x3 on OfﬁceHome (C).
This suggests that both the pre-training dataset and the model size play key role in determining the OOD generalization, and ideally it is preferable to employ larger models pre-trained on larger and more diverse pre-training data.
OOD performance of models trained from random initialization.
To further validate the importance of pre-training data along with the model size for better OOD generalization, we train models with increasing model sizes from random initialization, i.e., without employing a pre-training phase.
Our results for this experiment (cf.
Table 8) show that larger models do not signiﬁcantly improve the
OOD generalization without the use of pre-training.
For example, increasing ResNext50-32x4d to
ResNext101-32x8d only improves the OOD accuracy on OfﬁceHome (C) by less than 2% .
Thus, our results suggest that without pre-training, only increasing model size is not very effective in improving the OOD performance.
3. A NALYSIS OF TECHNIQUES USED FOR IMPROVING ID ACCURACY
In this subsection, we examine various techniques that have been shown to improve ID accuracy to assess their utility in achieving good OOD generalization.
First, we study the impact of training data size on the OOD generalization, as increasing the number of training samples is an effective approach to improve ID generalization.
Then we evaluate four techniques in our OOD setting, including label smoothing (Szegedy et al., 2016), AutoAugment (Cubuk et al., 2018), PatchGaussian (Lopes et al., 2019), and Sharpness-Aware Minimization (SAM) (Foret et al., 2020).
Utility of more training data.
We consider ﬁne-tuning with datsets of four different sizes, i.e., 100% , 50% , 25% , and 12 .
5% of the total training data for a given benchmark.
Our results in Figureshow that increasing the training data from 12 .
5% to 100% does not signiﬁcantly improve OOD generalization.
In fact, on VLCS (S), a better OOD generalization is realized when we utilize less training data to train a ResNext101-32x8d model (cf.
Figure 7(c)).
This suggests that increasing ID training samples is not as effective as using larger and better pre-trained models.
Methods for improving ID accuracy.
We ﬁnd that applying the methods listed in Table 2 does not signiﬁcantly improve the OOD generalization across four datasets compared with scaling model size and pre-training dataset size, and utilizing augmentations/regularization can potentially even hurt [Figure] [Table] OOD generalization.
For example, applying PatchGaussian decreases OOD accuracy on Terra (L46) from 43 .
7% to 13 .
1% and 9 .
7% with σ = 1 . 0 and 0 .
5 , respectively.
In constrast, PatchGaussian has very minimal impact on the ID performance, where it achieves Acc id = 95 .
8% and Acc id = 95 .
6% , respectively; ERM attains
Acc id = 95 .
Contrary to our results, Lopes et al.
(2019) notice that PatchGaussian can improve both the clean (ID) accuracy and robustness to common corruptions (OOD accuracy).
This suggests that one should employing augmentation/regularization techniques carefully so as to not harm OOD generalization as a side effect.
On the other hand, SAM with parameter 2 ρ = 0 .
02 improves the OOD generalization on three benchmarks.
Overall, we ﬁnd that methods used for improving ID accuracy do not necessarily improve OOD accuracy, when compared with the simple ERM-based ﬁne-tuning approach.
R ELATED W ORK
Domain adaptation and domain generalization.
Ben-David et al.
(2010) proposed H ∆ Hdivergence and applied it to develop error bounds on new test data distribution that are different from the training distribution.
Motivated by the theoretical results in Ben-David et al.
(2010), a large body of work was devoted to learning domain-invariant representations for domain adaptation where unlabeled data from the target domain are available during training (Ajakan et al., 2014;
Ganin et al., 2016; Tzeng et al., 2017; Zhao et al., 2018).
Different from the domain adaptation problem, no information from the target domain is available in the domain generalization problem.
Recent works proposed various algorithms to improve domain generalization, including learning invariant representations across training domains (Li et al., 2018b; Arjovsky et al., 2019), distributionally robust optimization Sagawa et al.
(2020), data augmentation (Volpi et al., 2018;
Zhou et al., 2020), and causal framework (Heinze-Deml & Meinshausen, 2017; Mahajan et al., 2021).
Zhou et al.
Wang et al.
(2021) provide a more detailed review of the topic of domain generalization.
OOD generalization and model robustness.
Recent works developed new datasets to evaluate model robustness to out-of-distribution data, including ImageNet-V2 (Recht et al., 2019), CIFAR10.1 (Recht et al., 2018), ImageNet-C and CIFAR-10-C (Hendrycks & Dietterich, 2018), ImageNetR (Hendrycks et al., 2020a), WILDS (Koh et al., 2021), etc.
A line of work proposed methods to improve model robustness to commons corruptions (Lopes et al., 2019; Xie et al., 2020; Hendrycks et al., 2020b; Calian et al., 2021; Yi et al., 2021). Xie et al.
(2021) investigated how to leverage auxiliary information and pre-training to improve OOD generalization.
On multiple datasets, the researchers have observed a near linear correlation between the OOD accuracy and the ID accuracy (Recht et al., 2019; Miller et al., 2020; Mania & Sra, 2020; Miller et al., 2021).
However, Taori et al.
(2020) found that most approaches, including the ones that improve robustness to synthetic distribution shifts, do not improve model robustness to natural distribution shifts, except for the models that are pre-trained on larger datasets.
Similar observation has been made in Hendrycks et al.
(2020a).
Our work focuses on the domain generalization benchmark datasets as well as how to perform ﬁne-tuning one various pre-trained models for better OOD generalization.
D ISCUSSION AND F UTURE W ORK
Pre-training is one of the most effective approaches for improving model performance in a wide range of machine learning tasks.
In this work, we perform an empirical study of ﬁne-tuning a diverse set of pre-trained models and evaluate their OOD generalization.
We ﬁnd that simply ﬁne-tuning larger models pre-trained on more (diverse) data can signiﬁcantly improve model performance on
OOD data.
Additionally, and we also identify the regularization effect of small learning rates that is important for achieving better OOD generalization.
Further, through extensive experimentation, we demonstrate that, while relying on pre-trained models, model size and pre-training dataset play a key role in ensuring good OOD generalization.
We hope our results can further inspire future research on bridging the gap between in-distribution accuracy and out-of-distribution accuracy.
There are multiple interesting immediate directions to explore in future work that we discuss next.
Scaling model size and pre-training dataset size.
Our empirical results indicate that model size and the pre-training dataset size are two essential factors for improving OOD performance.
Even though we primarily focus on image classiﬁcation benchmarks, similar behavior has been observed in NLP domain (Brown et al., 2020), where increasing model size leads to monotonic improvements in zero-shot performance on unseen tasks for example.
It is worth noting that while increasing model size or pre-training dataset size only marginally improves the in-distribution accuracy, the increase in
OOD accuracy can be much larger.
It is an interesting direction to explore the differences in ID and
OOD generalization in other domains.
How to perform model selection?
Our study indicates that models trained via different methods can exhibit a large variation in terms of their OOD performance, even when their in-distribution accuracy is very similar, e.g., see Figure 4(c) and Figure 5(c).
We thus believe that the development of better model selection approaches for OOD generalization will be a key direction of future work.
Limitations of ERM.
Despite impressive results for OOD generalization, we ﬁnd that ERM-based ﬁnetuning of pre-trained models is unlikely to close the in-distribution to out-of-distribution generalization gap , even when the model size or the pre-training dataset becomes much larger.
To generalize to unseen domains, it might be necessary to have access to extra information about the OOD dataset (e.g., unlabeled OOD data as available in the domain adaptation setting (Pan & Yang, 2009)) and/or design novel algorithms (e.g., updating the model parameters during test time (Sun et al., 2020)).
Reproducibility Statement.
In Section 2 and Appendix A.2, We provide detailed descriptions on the implementations used in this paper, including about the pre-trained models, datasets, models selection, hyperparameters, and how to train the models.
A A PPENDIX A. O THER R ELATED W ORKS
Pre-training and ﬁne-tuning. Li et al.
(2020) examined how to select hyperparameters for ﬁnetuning on a various benchmark vision datasets.
Dodge et al.
(2020) performed an extensive study on ﬁne-tuning pre-trained language models on NLP benchmarks, and investigated the impact of weight initialization and training data order on model performance.
Kaplan et al.
(2020) studied the scaling laws, including pre-training dataset size, model size, and compute budget for training, for language models.
They found that increasing model size for pre-training can improve model performance on
OOD data.
More recently, Zhai et al.
(2021) explored how to improve ViT model performance on
ImageNet and few-shot learning tasks through scaling the model size and data size
. A. A DDITIONAL E XPERIMENTAL D ETAILS
Implementation details.
Our implementation is mainly adapted from DomainBed (developed in
Gulrajani & Lopez-Paz (2020)).
Before we ﬁne-tune the pre-trained models, we replace the ﬁnal linear layer of the pre-trained model with a random initialized linear layer with output dimension equals to the number of classes for the dataset.
Most of the pre-trained models are from Py T orch
Im age M odels ( timm ).
The default training iteration is 5 , 000 .
For the training from scratch setting (i.e., without pre-training), we set the training iteration as 10 , 000 and weight decay as 10 − 5 .
During the ﬁne-tuning, we save model checkpoint every 100 iterations and select ﬁve checkpoints that achieves the top-5 in-distribution accuracies.
The input size of the ViT models is (3 , 224 , 224) .
In
Table 3, we summarize the top-1/top-5 ImageNet accuracies of pre-trained models used in this paper.
[Table]
A. A DDITIONAL E XPERIMENTAL R ESULTS
In this subsection [Table] we present additional experimental results in Section [Table]
Effect of ﬁne-tune learning rate in Section 3.2.
In addition to Figure 2 and Figure [Table] we provide more experimental results on the effect of ﬁne-tune learning rate.
For ﬁne-tuning results with different learning rates, we present more results for ﬁne-tuning SWSL-ResNext101-32x4d in Figure 8, results for ﬁne-tuning SWSL-ResNext101-32x8d in Figure 9, results for ﬁne-tuning SWSL-ResNext5032x4d in Figure 10, results for ﬁne-tuning BiTm-ResNetV2-50x3 in Figure 11, results for ﬁne-tuning
ResNext101-32x8d (ImageNet pre-trained) in Figure 12, and results for ﬁne-tuning ResNext5032x4d (ImageNet pre-trained) in Figure 13.
Meanwhile, in Figure A.3, we provide more results on visualizing the OOD accuracy vs. training loss for SWSL-ResNext101-32x4d.
Effect of pre-training dataset.
In Figure 15, we provide additional experimental results on studying the effect of pre-training dataset.
Effect of model architecture: ViTs vs. CNNs.
In Figure 16, we provide additional experimental results on comparing the performance of ViTs and CNNs.
Effect of model size.
We provide additional results on comparing models with different model sizes in Table 4 (ImageNet-1k pre-trained models), Table 5 (BiTm pre-trained models (Kolesnikov et al., 2020)), Table 6 (SWSL pre-trained models (Yalniz et al., 2019)),
Table 7 (ViT pre-trained models (Dosovitskiy et al., 2020)), and Table 8 (without pre-training).
Utility of more training data.
In Figure 17, we provide additional experimental results on investigating the impact of the number of training samples on the TerraIncognita dataset, including the
ID/OOD accuracy results.
Methods for improving ID accuracy.
In Table 9, we provide the in-distribution accuracy evaluations of the methods described in Table 2.
Also, in Table 10 and 11, we provide additional results on the ID/OOD accuracy evaluations of different methods (on more datasets). [Figure] [Figure] [Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Figure]
[Figure] [Figure]
[Figure] [Figure]
[Table] [Table]
[Table] [Table]
[Table]
Method
PACS (S)
Ofﬁce (A)
VLCS (S)
Terra (L38)
ERM (in Table 1) 99.
Label Smoothing ( 0 . 1 ) 99. Label Smoothing ( 0 . 2 ) 99.
AutoAugment 99.
PatchGaussian ( 1 . 0 ) 99. PatchGaussian ( 0 . 5 ) 99.
SAM ( 0 . 02 ) 99. SAM ( 0 . 05 ) 99. [Figure] [Figure]
[Figure] (L [Figure] [Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]