An Empirical Study on Distribution Shift Robustness From the Perspective of Pre-Training and Data Augmentation
Abstract
The performance of machine learning models under distribution shift has been the focus of the community in recent years.
Most of current methods have been proposed to improve the robustness to distribution shift from the algorithmic perspective, i.e., designing better training algorithms to help the generalization in shifted test distributions.
This paper studies the distribution shift problem from the perspective of pre-training and data augmentation, two important factors in the practice of deep learning that have not been systematically investigated by existing work.
By evaluating seven pre-trained models, including ResNets [1] and ViT’s [2] with self-supervision and supervision mode, on five important distributionshift datasets, from WILDS [3] and DomainBed [4] benchmarks, with five different learning algorithms, we provide the first comprehensive empirical study focusing on pre-training and data augmentation.
With our empirical result obtained from 1,330 models, we provide the following main observations: 1) ERM combined with data augmentation can achieve state-of-the-art performance if we choose a proper pre-trained model respecting the data property; 2) specialized algorithms further improve the robustness on top of ERM when handling a specific type of distribution shift, e.g., GroupDRO [5] for spurious correlation and CORAL [6] for large-scale out-of-distribution data; 3) Comparing different pre-training modes, architectures and data sizes, we provide novel observations about pre-training on distribution shift, which sheds light on designing or selecting pre-training strategy for different kinds of distribution shifts.
In summary, our empirical study provides a comprehensive baseline for a wide range of pre-training models fine-tuned with data augmentation, which potentially inspires research exploiting the power of pre-training and data augmentation in the future of distribution shift study.
Introduction
Machine learning (ML) has received much success in many computer vision applications such as image classification.
However, it heavily depends on an in-distribution (ID) assumption that the training data and testing data are identically and independently from the same data distribution.
Unfortunately, this ID assumption may be hardly satisfied in practice, which leads to distribution [Table] shifts.
In the past several years, there has been a line of work proposed to study distribution shifts from different perspectives [7, 5, 8, 6].
Arjovsky et al.
[7] proposed Invariant Risk Minimization (IRM) to learn invariant prediction across multiple training environments, which does not need the assumption of i.i.d.
in training and testing.
Follow-up works of IRM consider both theoretical and empirical aspects (e.g., see [9, 10, 11, 12]).
On the other hand, robust optimization is a popular technique for distribution shift problems, which minimizes the worst-case loss.
Examples include
Distributionally Robust Optimization (DRO) [13, 14, 5], minimax Risk Extrapolation (MM-REx) [8], and Heterogeneous Risk Minimization (HRM) [15].
However, most existing methods focus on developing new specialized learning algorithms to improve the generalization under distribution shifts, without considering deep learning (DL) characteristics such as training procedure and model architecture which are the important components to performance improvement in deep learning.
In the paper, we empirically investigate the importance of pre-training and model architectures for generalization under distribution shift, with a focus on image classification tasks.
To this end, we consider Empirical Risk Minimization (ERM) [22], the most commonly-used learning paradigm in
DL.
We identify the source of naturally arising distribution shifts and study the impact of pre-training, which is carried out on a well-controlled training set without distribution shifts such as ImageNet [23], on downstream datasets with distribution shifts when fine-tuning with data augmentations.
The contributions of our paper are:
We provide the first extensive empirical study on how pre-training and data augmentation affect robustness to distribution shifts on various computer vision datasets.
We consider seven pre-training models (including self-supervised and supervised ResNets and ViT’s), three data augmentations (including a group-aware mixup), and five datasets with distribution shifts (e
g., spurious correlation, subpopulation shift and OOD generalization), totaling 1,330 trained models.
We find that specialized algorithms such as GroupDRO [5] and CORAL [6] for distribution shifts achieve the state-of-the-art performance with proper pre-trained models on some datasets, while ERM with data augmentation is also a quite competitive baseline.
Comparing different pre-training strategies, models and data sizes, we summarize key observations about the impact of pre-training on distribution shifts, and provide practical tips on selecting pre-trained models under different types of distribution shifts.  Related Work Out-of-Distribution Generalization.
The OOD generalization task assumes that the generation processes of training and test samples conditioned on the target label are different.
For example, the environment or background of objects changes between training and test (WILDS-Camelyon,
WILDS-iWildCam, ImageNet-C [24]) or the visual features of objects are changed during test (DomainNet, ImageNet-A [25]).
Various methods have been proposed to improve the generalization under OOD, including domain adversarial learning [26, 27] and domain feature aligning [6, 28]. We propose to study the performance of different OOD generalization algorithms including the popular method CORAL [26], when different pre-trained models and data augmentation tricks are used. Similar to the finding for spurious correlation, different pre-trained models have substantially different impacts on OOD generalization. Key observations and tips about OOD generalization are provided based on our empirical results.
Please refer to the survey [29] for more methods on OOD generalization.
The Impact of Pre-Training and Data Augmentation.
[30] summarizes a framework for distribution shift, consisting of spurious correlation, low-data shift and unseen data, and evaluates the performance of representation learning, data augmentation and neural architectures.
However, [30] analyzed those
[Table]
[Table] factors independently and only evaluated the pre-trained ResNet50. In contrast, our paper investigates the de facto training setting in distribution shift problem, i
[Table]
e., fine-tuning a pre-trained model on a target task, and we comprehensively evaluate the performances of different pre-training models and neural architectures when combined with data augmentation and fine-tuning.
[4] focuses on the role of data augmentation on domain generalization, where the visual features of objects are shifted during test such as DomainNet [31] (Fig. 1), and draws a similar conclusion that ERM is a strong baseline in their context.
Our work uses the largest dataset in [4] as representative of domain shift and studies the impact of pre-training in addition to the data augmentation.
[32] provides a nice benchmark on distribution shift consisting of different levels of semantic hierarchies with ImageNet [23].
We do not evaluate on [32] because our study focuses on the performance of pre-trained models on ImageNet in various downstream applications, instead of the subpopulation shift in ImageNet data.
[33] studies the domain generalization from the perspective of pre-training.
The major differences are: 1) our paper has a focus on both pre-trained models and data augmentation, especially mixup and the proposed GroupMix, while [33] only considers pre-trained models; 2) we evaluate not only supervised pre-trained models but also self-supervised pre-trained ViT and ResNet models, including both MoCo and MAE training, while [33] only considers (semi) supervised pre-trained models; 3) our paper evaluates on the WILDS benchmark in addition to the domain generalization data.
Similar to our GroupMix, [16] proposes to use a selective mix-up augmentation to interpolate samples with the same label or the same environment.
The difference is that we investigate the performance of
GroupMix on a variety of pre-trained models to reveal the impact of pre-training on the GroupMix.
[34] also studies the impact of pre-training on robustness to distribution shift, where they assume there is an OOD training set for fine-tuning the classification layer.
In our paper, there is no such a strong assumption and we not only consider pre-training but also data augmentation in the fine-tuning.
Experiment Settings and Results
Our experiment uses seven pre-trained models, i.e., MoCo-ViT, MoCo-R50 [35], MAE pre-trained [36] on ImageNet-1k (IN-1k), Supervised ViT and R50 pre-trained on ImageNet-1k and ImageNet21k (IN-21k).
ERM, Mixup, GroupDRO, group-based Mixup and CORAL are evaluated on four
WILDS datasets (Waterbirds [5], Camelyon [37], FMoW [38], iWildCam [39]) and the large-scale domain generalization dataset DomainNet [31].
See the Appendix for details of experiment.
We next report the experimental results.
Data augmentation (DA), MixUp and GroupMix are included as the generalized ERM algorithm since they are often considered as general data augmentation methods in DL and simple to implement.
Thus, we denote the DA, mixup and GroupMix as general [Table] [Table]
DA in the following content.
In our experiment, DA is used with ERM (mixup and GroupMix),
GroupDRO and CORAL to make a fair comparison.
On four WILDS datasets and DomainNet, we report the result of fine-tuning seven pre-trained models with three different algorithms in Tabs.
The bold numbers indicate the algorithm with the best performance when fixing the pre-trained model, while the green number denotes the best pre-trained model for each algorithm.
Next we analyze the empirical result for the five datasets respectively.
Conclusion: Key Observations and Tips
[Table]
Self-supervised or supervised pretraining?
Fig.
1 shows the accuracy of seven pre-trained models, averaged over learning algorithms.
For spurious correlation and subpopulation shift (Waterbirds and FMoW), self-supervised pre-training has a clear benefit over the supervised pre-training, when using the
IN-1k.
It indicates that the high-level features learned in supervised training are not robust to spurious correlation and subpopulation shift.
On Camelyon, MAE is better than the remaining models including ViT and R50 trained on IN-21k, since the tissue images only contain low-level features.
In contrast, the self-supervised model has not much benefit over supervised ones on iWildCam and DomainNet for ViT models.
The OOD generalization in object recognition may need high-level features from the labels in pre-training.
For
ResNet, when the pre-training data is insufficient, MoCo-R50 is better than Sup.
R50 on IN-1k.
[Figure]
t r i c
[Figure]
A vg
[Figure] [Table]
the best performance, while on iWildCam and DomainNet, the supervised pre-trained R50 on IN-21 achieves a better result than the ViT counterpart.
On Waterbirds, ViT’s benefit is quite substantial when pre-trained on IN-21k, indicating that with more fine-grained labels, the spurious correlation problem can be alleviated as a result of narrowed ambiguity.
On OOD generalization datasets (Camelyon, iWildCam and
DomainNet), an interesting phenomenon is that ViT is more data-efficient in supervised pre-training than R50: ViT outperforms R50 with a large margin using IN-1k while the benefit is washed out if using the larger IN-21k.
Larger pre-training dataset leads to stronger robustness?
Comparing the performance of different pretraining sizes (corresponding to marker sizes in Fig. 1), the larger dataset IN-21k substantially improves the performance of IN-1k in most cases.
Thus, it is validated that generalization under distribution shift in a downstream can be achieved by pre-training on a standard large-scale dataset.
Which learning algorithm is best?
Tab.
7 shows the averaged accuracy of each algorithm over pre-trained models.
GroupDRO is the best training method on average when evaluated on Waterbirds and Camelyon, where the data size is quite small and the difficulty of classification is not high (both are binary classification task).
In contrast, on large-scale datasets such as iWildCam and DomainNet,
GroupDRO is the worse one compared with others, indicating that GroupDRO is specialized for small-scale data.
CORAL is quite competitive on iWildCam and DomainNet but not on small datasets such as Waterbirds.
The possible reason is that CORAL relies on estimated means and variances of features, whose stability and accuracy will be affected if the data is scarce.
ERM with GroupMix is a strong baseline on Waterbirds and FMoW, showing the importance of using group-information in spurious correlation with spurious correlation and subpopulation shift.
On Camelyon, iWildCam and
DomainNet, ERM with general DA is also quite competitive, ranking the first or second on the three datasets.
Tips.
Finally, from these observations, we provide tips for practitioners for improving robustness to distribution shift:
For OOD generalization and spurious correlation in object recognition, use a supervised pretrained model with as much pre-training data as possible.
When the pre-training data is limited, self-supervised pre-trained models can be better.
If the target data is small, use algorithms respecting the group information such as GroupDRO and GroupMix.
If the target data is large, CORAL and ERM with general DA are quite competitive.
For distribution shift in dense image classification, use a patch-based self-supervised pre-trained model, e.g., MAE.
Acknowledgement
This work was supported by Alibaba Group through Alibaba Research Intern Program, the Fundamental Research Funds for the Central University of China (DUT No. 82232031) and a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No.
CityU 11215820).
A An Overview of Distribution Shift
We summarize two common sources of distribution shift in practice.
The first source is the bias in data, as a result of real-world bias.
Assuming the data distribution is p ( x , y ) where y is an attribute vector, the conditional distribution of one attribute given another p ( y i | y j ) may be high, indicating a spurious correlation between y i and y j , e.g., the background and bird species in WILDS-Waterbirds.
Some marginal distributions of p ( y i ) may be relatively small or large, leading to a subpopulation shift, e.g., the size of African images in WILDS-FMoW.
The second source is the scarcity of data, i.e., the dataset cannot contain all possible images from the support of p ( x , y ) as a result of the high dimensionality of the continuous space.
In other words, the training set contains data from a certain distribution p tr ( x | y ) but the test set have a different distribution p te ( x | y ) ̸
= p tr ( x | y ) , e.g.,
WILDS-iWildCam [3] and DomainNet [31].
Note that these two types of distribution shifts could be conceptually addressed by deliberately sampling a balanced dataset to avoid spurious correlation and subpopulation shift, and collecting all possible data from the distribution support.
But in reality, we cannot obtain such a perfect dataset to train the model in many real-world applications.
A pre-trained model on a balanced large-scale dataset is often used as the initialization when training on such downstream tasks, because the pre-trained model learns good representations and benefits the optimization.
Our work mainly investigates the impact of pre-training on distribution shift, and aims to answer the following question: How does the pre-training, i.e., training on the standard large-scale dataset that is more carefully curated, affect the fine-tuning result on downstream datasets with distribution shift?
To this end, we study different aspects of pre-training including training strategies, neural architectures and pre-training data sizes in our paper and summarize our findings in
Section 4.
B Experimental Settings
In this section we discuss our experiment settings, including datasets, pre-trained models, and learning algorithms.
B.1 Datasets
To have a comprehensive evaluation on the effect of pre-training and data augmentation, we use five datasets that represent different types of distribution shifts.
A general description of the five datasets follows.
For more details about the datasets, see the original papers [3] and [31].
WILDS-Waterbirds [3, 5] is a bird classification task and has two kinds of birds in two types of environments.
Specifically, bird images from CUB [40] are cropped and pasted to scene images from
Places [41].
The training set of Waterbirds has 3,498 landbirds on land, 1,057 waterbirds on water, 184 landbirds on water and 56 waterbirds on land, inducing a spurious correlation in the data: the environment label (land/water) is correlated with the target label (waterbird/landbird).
If the classical
ERM is used, the environment features will be exploited and the performance of minor groups will be substantially worse than of major groups.
Following the common practice [5, 42, 43, 44, 8], we use the worse-group (WG) accuracy as a metric for robustness to spurious correlation and report the average accuracy weighted by the training group size.
As in WILDS benchmark [3], we use the input resolution of 224 × 224, the training epoch of 200 and batch size of 128 in this paper.
We search the learning rate from { 0.000001,0.000002, 0.000004, 0.000008, 0.00001, 0.00002, 0.00004, 0.00008 } and weight decay (WD) from { 0.0,1e-3,1e-2,1e-1,1.0 } , α of mixup from { 0.1,0.3,1.0 } .
For GroupMix, we search the constant C from { 1.0,3.0,10.0,30.0 } .
For GroupDRO, the constant
C DRO is searched from { 1.0,2.0,4.0,8.0,16.0 } and the step size η q is fixed as 0.01 for all datasets.
For CORAL, the λ is searched from { 0.3,1.0,3.0 } .
Thus, in the hyperparameter search stage, we train 960 models.
WILDS-FMoW [3, 38] is a satellite imagery data consisting of dense objects in the image.
We use the satellite images to predict land usage as in [3, 45, 46].
The images are divided into five groups according to their regions (Asia, Europe, Africa, Americas and Oceania).
Due to historical and economic reasons, developing regions have fewer satellite images than developed regions.
So in the
FMoW data, the Africa region (1,582 training images) has much fewer images than Europe (34, training images) or Americas (20,973 training images).
With the unbalanced groups and the fact that image features often vary across different continents, training with classical ERM may have a lower accuracy in minor groups compared with major groups, and has the risk of learning biased models with discriminatory prediction.
The input resolution is set as 224 × 224, the training epoch as 50 and batch size as 72.
We fix the WD as 0.0 following existing practice and search the learning rate from { 0.00001, 0.0001, 0.001 } and the α from { 0.1,0.3,1.0 } .
We do not use the adjustment GroupDRO on this dataset and the following datasets.
For GroupMix, we search C from { 3.0,10.0,30.0,100.0 } .
The λ in CORAL is searched from { 0.3,1.0,3.0 } .
We trained 60 models for hyperparameter search on
FMoW.
WILDS-Camelyon [3, 37] contains tissue slide images from five hospitals, and the task is to predict the presence of tumor tissue.
The ways to collect and process the slide images differ among hospitals, while it is often desired that the trained model on limited data from a few hospitals generalizes to unseen data from a novel environment.
Three hospitals are used for training, one hospital is for validation and the other is for test.
The input resolution is set as 96 × 96, the training epoch asand batch size as 168.
The WD and learning rate are searched from { 0.000001, 0.000003, 0.00001, 0.00003, 0.0001, 0.0003, 0.001, 0.003, 0.01 } and { 0.0,1e-5,1e-4,1e-3,1e-2 } .
The α is searched from { 0.1,0.3,1.0 } .
For GroupMix, C is searched from { 3.0,10.0,30.0,100.0 } .
For CORAL, the λ is searched from { 0.3,1.0,3.0 } .
So we trained 900 models in total for Camelyon for hyperparameter search.
WILDS-iWildCam [3, 39] is collected by cameras in the wild, and the task is to predict wild animal species.
The data are split into 323 groups according to the location of cameras since the environment has an impact on the image features.
The trained model on certain groups is expected to generalize to
OOD, i.e., images from unseen or new cameras.
As in [3], we use the Macro F1 score as the metric of the performance on iWildCam.
The input resolution is set as 448 × 448, the training epoch asand batch size as 24.
The learning rate is searched.
The WD is fixed as 0.0, according to existing experiment in [3] and the learning rate is searched from { 0.000001, 0.000002, 0.000004, 0.000008, 0.00001, 0.00002, 0.00004, 0.00008,0.0001 } .
The α in mixup is searched from { 0.1,0.2,0.4,1.0 } .
For GroupMix, we search the constant C from { 1.0,3.0,10.0,30.0 } .
For CORAL, the λ is searched from { 0.3,1.0,3.0 } .
In the hyperparameter search stage, we train 225 models.
DomainNet [31] is often used to test domain generalization or adaptation, and consists of six domains that have the same categories but different image styles, e.g., painting, real and clipart.
There are several other domain generalization datasets but we use the DomainNet because it is the largest dataset among its counterparts, e.g., PACS [47], VLCS [48], OfficeHome [49] and TerraIncognita [50].
We follow the common practice [4] to train a model on five domains and evaluate the performance on the held-out domain.
Thus, we train six models since we repeat the process to test on each domain, and report the average results.
Following [4], the input resolution is set as 224 × 224 and the training lasts for 5000 update steps.
The batch size is 32 for each environment, resulting in a 32*5= samples for the whole batch.
We search the learning rate from { 0.00001, 0.0001, 0.001 } , WD from { 0.0,1e-5,1e-4 } , α from { 0.1,0.3,1.0 } .
The C in GroupMix is searched from { 1.0,3.0,10.0 } , the λ in
CORAL is searched from { 0.3,1.0,3.0 } .
This amounts to 810 trained models in this stage.
Finally, we use the PyTorch [51] code from WILDS [3] and DomainBed [4] benchmark.
All of our experiment is run on a single Nvidia-V100 GPU with 32GB.
B.2 Pre-Trained Models
The benchmarks in WILDS and DomainNet use a pre-trained ResNet [1] on ImageNet-1k [23] as the initialization for training.
Specifically, Waterbirds and iWildCam use ResNet50 [1], FMoW and Camelyon use DenseNet121 [52].
In a domain generalization benchmark [4], the pre-trained
ResNet50 is the default model for DomainNet.
In this work, the following seven pre-trained models are evaluated to study the impact of pre-training strategies, neural architectures and pre-training data size on the distribution shifts.
MoCo-R50 and MoCo-ViT-B/16 [35, 53] are self-supervised pre-trained models using contrastive learning to learn generic visual features.
We use pre-trained ResNet50 (R50) and ViT-B/16 from [53].
MAE -ViT-B/16 [36] uses a masked image modeling task for self-supervised pre-training of the patch-based transformer architecture.
We use the ViT-B/16 version of MAE.
Supervised ViT-B/ 16 and R50 that are trained on ImageNet-1k (IN-1k) and ImageNet-21k (IN-21k) are evaluated.
There are 1,281,167 images and 1,000 classes in IN-1k, and 14,197,122 images and 21,841 classes in IN-21k.
We use ViT-IN-1k and ViT-IN-21k model from [54].
The R50-IN-21k model is from [55].
Since all pre-trained ViT models have the same architecture, i.e., ViT-B/16, we use ViT for short in this paper.
By comparing the performance of models pre-trained on 1k and 21k, we can investigate the impact of pre-training data scale on downstream distribution shifts.
B.3 Learning Algorithms
ERM is the classical objective function used in ML.
We note that in stochastic gradient descent, the
ERM is related to batch sampling.
On Waterbirds and DomainNet, we follow the common practice [5, 4] to use the weighted sampling so that each batch has the same number of samples for all groups.
On other datasets, the standard batch sampling is used.
Data augmentation .
On Waterbirds, we use the standard data augmentation, i.e., random resizing and cropping, and random horizontal flipping.
On Camelyon and FMoW, we use the 2-level of randaug [56].
On iWildCam, the 1-step randaug is used.
On DomainNet, we use the standard data augmentation in Waterbirds, along with color jittering and random grayscale.
Mixup [57] optimizes a convex hull of training samples to improve the generalization, by augmenting the training data with random combination of input images and labels.
Specifically, two samples xand x 2 with labels y 1 and y 2 will generate a new training sample as follows
[Equation]
where λ is sampled from a Beta distribution with parameters α and β .
GroupDRO [5].
GroupDRO takes { ( x i , y i , g i ) } as the input, where the g i is the group label for x i , and updates the network by minimizing a group-weighted loss.
Specifically, the group weight of j th group is updated as
[Equation]
where the η q is the step size for group weight update, N j ( t + 1) is the number of samples from j th group in the current batch.
For GroupDRO with adjustment, there is an extra term C DRO / √ n j added in η q l j ( t + 1) of Equation 2.
We use the shorthand notation l j ( t + 1) for the j th group’s loss in the batch.
Then the loss is weighted in a group-wise way,
[Equation]
GroupMix We add the group information in the mixup step,
[Equation]
The default GroupMix in our paper is to set g ( b ) 1 = g ( b )
= g ( x )
[Equation]
= 1 and g ( [Equation] [Equation] 2 is related to group size, i.e., the number of training samples in a group.
Assume the number of training samples of j th group is n j , the group weight is defined as the output of a softmax function g j = softmax ( C/ √ n j ) where the C is a hyperparameter.
So the summation of group weights is one and smaller groups have larger weights.
As in the main paper, we use the function g ( · ) for x to denote the group weight of x .
We also investigate three variants of GroupMix, i.e., 2) g ( b )
= γg ( x 1 ) and g ( b )= γg ( x 2 ) , keep other four weights to be 1.
The γ > 0 is a constant hyperparameter to control the parameters in Beta distribution.
To keep a stable sampling for two input images from the same group, we fix the α, β to be 0.1 if two images are from the same group.
This variant uses the property of Beta distribution to give a larger λ for an image from minor groups. 3) g ( b )=
γg ( x 1 ) and g ( b )= γg ( x 2 ) , g ( x )=
g ( x )= 1 .
and g ( l ) 1 = g ( x 1 ) , g ( l ) 2 = g ( x 2 ) .
Then normalize each sample’s weight so that the summation of weight for a batch is 1.
The difference between this method and 2) is that the label is multiplied by group weight and the coefficient/weight for losses is normalized.
4) g ( x )=
g {+( x )= g ( b ) 
 = g ( b ) 2 = 1 , g ( l ) 1 = g ( x 1 ) , g ( l ) 
+}
= g ( x 2 ) and normalize the loss weights as in the third variant.
This is the normalized version of the default GroupMix.
Waterbirds
FMoW
Camelyon
[Table]
, Camelyon and iWildCam.
[Table]
CORAL [6] aligns the feature means and covariances of different groups by minimizing the squared
L2/Frobenius norm of the difference in means/covariances, i.e., [Equation] where the j and k denote two groups and the mean and covariance are computed from the features of two groups, i.e., the last backbone layer’s output. B.4 Training Settings
We use the official training/validation/testing split for WILDS datasets.
For DomainNet, if a domain is used for training, then the data is randomly split into 80% and 20% as training and validation set following [4].
As the de facto setting in WILDS and DomainBed benchmark, the training starts with a pre-trained model and optimizes all parameters with the training data.
In other words, we study the performance of fine-tuning all parameters of a pre-trained model on distribution-shift datasets.
The training epochs/steps are set as the same in the WILDS [3] or DomainBed [4] benchmark.
On
WILDS datasets, we use the official (out-of-distribution) validation set to do the model selection.
On
DomainNet, we use the training domain validation set in the model selection, as suggested by [4].
In Table 8, the optimizer is the same as the benchmark.
In all other experiments, AdamW [58] is used as the optimizer.
We search the hyperparameters for each setting by the grid search, where the parameter space is determined by either existing training practice or our pilot study on those datasets and models.
For the search space for each dataset, please see the appendix.
In total, we trained about 3000 models during the hyperparameter searching.
Following the common practice [3, 4], on
WILDS datasets, we report the mean and standard deviation of 5 trials; on DomainNet, we report the result of 3 trials.
The model selection (early-stopping) uses the default validation ood set, i.e., choose the model with the best validation accuracy.
Different from the original WILDS benchmark, the hyperparameters are chosen based on the accuracy of the test ood set, instead of the validation ood set.
C More Experimental Result
Tab. 8 shows the performance of ERM with DA, mixup and GroupMix, compared on several WILDS baselines using the pre-trained models.
On Camelyon, we use a pre-trained DenseNet121, which is different from the default setting of WILDS benchmark.
Except for Waterbirds, ERM with general
DA achieves the best result.
On Waterbirds, GroupMix substantially improves ERM with other types of data augmentation.
For the result of ERM on DomainNet, [4] draws a similar conclusion that
ERM with data augmentation is a quite strong baseline on all popular domain generalization datasets.
[Table]
ERM (Sup., R50, IN-21k) 82.9(1.4) 96.0(0.5) 37.2(2.9) 55.1(1.3)
GroupDRO (MoCo, ViT-B, IN-1k) 86.7(0.8) 90.1(0.9) 35.2(1.6) 55.5(1.0)
GroupDRO (MoCo, R50, IN-1k) 88.1(0.9) 90.6(0.6) 37.2(1.1) 54.5(0.7)
GroupDRO (MAE, ViT-B, IN-1k) 87.4(1.0) 88.6(0.8) 37.9(1.4) 57.8(0.6)
GroupDRO (Sup., ViT-B, IN-1k) 87.0(1.1) 88.8(1.4) 35.8(1.8) 56.5(0.4)
GroupDRO (Sup., R50, IN-1k) 87.6(0.4) 88.5(0.7) 36.1(1.6) 52.0(0.7)
GroupDRO (Sup., ViT-B, IN-21k) 92.6(0.5) 93.2(0.5) 39.0(1.0) 59.2(0.5)
GroupDRO (Sup., R50, IN-21k) 86.6(1.1) 93.5(0.3) 37.1(1.9) 55.9(0.7)
CORAL (MoCo, ViT-B, IN-1k) 83.3(1.9) 93.1(1.2) 35.8(1.7) 53.5(1.7)
CORAL (MoCo, R50, IN-1k) 83.
[Table]
[Table] [Table]
[Table] [Table]
CORAL (MAE, ViT-B, IN-1k)
[Table]
[Table] [Table]
[Table] [Table]
CORAL (Sup., ViT-B, IN-1k)
[Table] [Table]
[Table] [Table]
CORAL (Sup., R50, IN-1k)
[Table] [Table]
CORAL (Sup., ViT-B, IN-21k)
[Table] [Table]
CORAL (Sup., R50, IN-21k)
[Table] [Table]
Mixup (MoCo, ViT-B, IN-1k)
[Table]
Mixup (MoCo, R50, IN-1k)
[Table] [Table]
Mixup (MAE, ViT-B, IN-1k)
[Table] [Table]
[Table] [Table]
Mixup (Sup., ViT-B, IN-1k)
[Table] [Table]
[Table] [Table]
[Table] [Table]
21k)
[Table] [Table]
[Table] [Table]
(MAE,
[Table] [Table]
[Table]
(Sup., R50, IN-1k)
[Table]
(Sup.
[Table] [Table]
GroupMix (Sup., R50, IN-21k)
[Table]
C.1 Variants of GroupMix
We evaluate four variants of GroupMix on Waterbirds. In addition to original V1, we consider the following 3 variants: 2) the weight g ( b ) 1 , g ( b ) 2 is proportional to g ( x 1 ) , g ( x 2 ) , which gives larger λ to a sample from minor groups, and keep other weights 1; 3) ( g ( b ) 1 , g ( b ) 2 ) ∝ ( g ( x 1 ) , g ( x 2 )) , g ( x )= g ( x )= 1 . 0 , g ( l ) 1 = g ( x 1 ) , g ( l ) 2 = g ( x 2 ) and normalize the loss weight so that the weight sums to 1 as in ERM; 4) g ( b )=
g ( b )=
g ( x )=
g ( x )= 1 . 0 , g ( l ) 1 = g ( x 1 ) , g ( l ) 2 = g ( x 2 ) and normalize the weight as in 3).
Tab.
9 shows the result of four variants, where the GroupMix-V3 is the best one in terms of worse-case accuracy and matches the performance of GroupDRO.
It indicates that the group-condition sampling can further improve the performance of GroupMix.
Moreover, all versions of GroupMix have a higher average accuracy than GroupDRO, because mixup alleviates the overfitting in training. [Table] [Table]
clipart infograph painting quickdraw
[Table]
D The full experimental results
Tab. 10 shows the full result of Waterbirds and FMoW in our paper, including the averaged accuracy.
On Waterbirds, it is worth noting that the averaged accuracy of GroupMix is generally higher than that of GroupDRO.
On FMoW, even though the worse-group accuracy of CORAL is higher than
GroupMix, the average accuracy of GroupMix is significantly higher than CORAL’s.
In conclusion,
ERM with GroupMix is quite competitive in both worse-group accuracy and averaged performance.
Tab.
11 shows the result of WILDS with ID and OOD Macro F1.
To compare the difference between
ID and OOD validation set based model selection, we report the result using ID validation set in the model selection in Tab.
11.B. We observe that CORAL has the drawback of sacrificing the ID accuracy to achieve a high OOD accuracy, whil GroupMix does not reduce the ID accuracy compared with DA and Mixup.
Thus, the ERM with data augmentation which uses group information is a strong baseline in both ID and OOD distribution.
The table also shows that the two model selection methods do not differ at a significant level.
For example, the top-1 models in Tab.
11.A and B overlap
[Table]
[Table] [Table]
R Pearson =0.24, p=0.
i
W il d
C a m 74 76 78 80 82 84 86 2832 3438
R Pearson =0.81, p=0.
R Pearson =0.77, p=0.
R Pearson =0.8, p=0.
R Pearson =0.52, p=0.
R Pearson =0.5, p=0.
D o m a i n N e t 74 76 78 80 82
R Pearson =0.63, p=0.
R Pearson =0.74, p=0.
R Pearson =0.69, p=0.
R Pearson =0.76, p=0.
R Pearson =0.83, p=0.
in most learning algorithms.
So the ID validation set based model selection will not change our general conclusion in the main paper.
Tab.
12 reports the result of 6 domain generalization accuracy for each experimental setting.
It is interesting that on the two most challenging domains, i.e., infograph and quickdraw, ViT and
R50 supervised pre-trained on IN-21k have quite different performance.
Sup-ViT-IN21k is better at generalizing to quickdraw, the most difficult domain, while Sup-R50-IN21k is better at infograph.
This phenomenon indicates that the performance in domain generalization can be further improved by using an ensemble of different neural architectures.
Fig.2 shows the result of linear regression between ImageNet (IN) [23] test accuracy of a pre-trained model and its performance in a target task with distribution shift.
For self-supervised models, we use the linear probing result as the IN accuracy as reported in their paper [36, 59, 35].
Fig.2 shows all the regression analyses of learning algorithms and datasets.
For Waterbirds, the correlation between
IN accuracy and downstream performance is statistically significant for GroupMix and CORAL.
On iWildCam and DomainNet, the correlation is more obvious than on Waterbirds.
However, on FMoW and Camelyon, there is no significant correlation between the two performances.
This phenemenon further validates our hypothesis that for object recognition task, increasing the performance on the standard dataset (IN) is helpful for downstream tasks under distribution shift.
But the benefit of IN performance is no longer valid if the downstream task has quite different visual features such as dense images in FMoW and Camelyon.