I N S EARCH OF L OST D OMAIN G ENERALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The goal of domain generalization algorithms is to predict well on distributions different from those seen during training.
While a myriad of domain generalization algorithms exist, inconsistencies in experimental conditions—datasets, network architectures, and model selection criteria—render fair comparisons difﬁcult.
The goal of this paper is to understand how useful domain generalization algorithms are in realistic settings.
As a ﬁrst step, we realize that model selection is non-trivial for domain generalization tasks, and we argue that algorithms without a model selection criterion remain incomplete.
Next we implement D OMAIN B ED , a testbed for domain generalization including seven benchmarks, fourteen algorithms, and three model selection criteria.
When conducting extensive experiments using D OMAIN B ED we ﬁnd that when carefully implemented and tuned, ERM outperforms the state-of-the-art in terms of average performance.
Furthermore, no algorithm included in D OMAIN B ED outperforms ERM by more than one point when evaluated under the same experimental conditions.
We hope that the release of D OMAIN B ED , alongside contributions from fellow researchers, will streamline reproducible and rigorous advances in domain generalization.
I NTRODUCTION
Machine learning systems often fail to generalize out-of-distribution , crashing in spectacular ways when tested outside the domain of training examples (Torralba and Efros, 2011).
The overreliance of learning systems on the training distribution manifests widely.
For instance, self-driving car systems struggle to perform under conditions different to those of training, including variations in light (Dai and Van Gool, 2018), weather (Volk et al., 2019), and object poses (Alcorn et al., 2019).
As another example, systems trained on medical data collected in one hospital do not generalize to other health centers (Castro et al., 2019; AlBadawy et al., 2018; Perone et al., 2019; Heaven, 2020).
Arjovsky et al.
(2019) suggest that failing to generalize out-of-distribution is failing to capture the causal factors of variation in data, clinging instead to easier-to-ﬁt spurious correlations prone to change across domains.
Examples of spurious correlations commonly absorbed by learning machines include racial biases (Stock and Cisse, 2018), texture statistics (Geirhos et al., 2018), and object backgrounds (Beery et al., 2018).
Alas, the capricious behaviour of machine learning systems out-of-distribution is a roadblock to their deployment in critical applications.
Aware of this problem, the research community has spent signiﬁcant efforts during the last decade to develop algorithms able to generalize out-of-distribution.
In particular, the literature in Domain
Generalization (DG) assumes access to multiple datasets during training, each of them containing examples about the same task, but collected under a different domain or experimental condition (Blanchard et al., 2011; Muandet et al., 2013).
The goal of DG algorithms is to incorporate the invariances across these training domains into a classiﬁer, in hopes that such invariances will also hold in novel test domains.
Different DG solutions assume different types of invariances, and propose algorithms to estimate them from data.
Despite the enormous importance of DG, the literature is scattered: a plethora of different algorithms appear yearly, each of them evaluated under different datasets, neural network architectures, and model selection criteria.
Borrowing from the success of standardized computer vision benchmarks such as ImageNet (Russakovsky et al., 2015), the purpose of this work is to perform a rigorous comparison of DG algorithms, as well as to open-source our software for anyone to replicate and extend our analyses.
This manuscript investigates the question: How useful are different DG algorithms when evaluated in a consistent and realistic setting?
[Table]
To answer this question
[Table]
we implement and tune fourteen DG algorithms carefully [Table] to compare them across seven benchmark datasets and three model selection criteria.
There are three major takeaways from our investigations:
• Claim 1: A careful implementation of ERM outperforms the state-of-the-art in terms of average performance across common benchmarks (Table 1, full list in Appendix A.5).
• Claim 2: When implementing fourteen DG algorithms in a consistent and realistic setting, no competitor outperforms ERM by more than one point (Table 3).
• Claim 3: Model selection is non-
trivial for DG, yet affects results (Table 3).
As such, we argue that DG algorithms should specify their own model selection criteria.
As a result of our research, we release D OMAIN B ED , a framework to streamline rigorous and reproducible experimentation in DG.
Using D OMAIN B ED , adding a new algorithm or dataset is a matter of a few lines of code.
A single command runs all experiments, performs all model selections, and auto-generates all the result tables included in this work.
D OMAIN B ED is a living project: we welcome pull requests from fellow researchers to update the available algorithms, datasets, model selection criteria, and result tables.
Section 2 kicks off our exposition with a review of the DG setup.
Section 3 discusses the difﬁculties of model selection in DG and makes recommendations for a path forward.
Section 4 introduces
D OMAIN B ED , describing the features included in the initial release.
Section 5 discusses the experimental results of running the entire D OMAIN B ED suite, illustrating the competitive performance of ERM and the importance of model selection criteria.
Finally, Section 6 offers our view on future research directions in DG.
Appendix A reviews one hundred articles spanning a decade of research in DG, summarizing the experimental performance of over thirty algorithms.
T HE PROBLEM OF DOMAIN GENERALIZATION
The goal of supervised learning is to predict values y ∈ Y of a target random variable Y , given values x ∈ X of an input random variable X .
Predictions ˆ y = f
( x ) about x originate from a predictor f : X → Y .
We often decompose predictors as f = w ◦ φ , where we call φ : X → H the
[Table]
featurizer, and w : H → Y the classiﬁer.
To solve the prediction task we collect the training dataset
D = { ( x i , y i ) } ni =1 , which contains identically and independently distributed (i.i.d.)
examples from the joint probability distribution P ( X, Y ) .
Given a loss function (cid:96) :
Y × Y → [0 , ∞ ) measuring prediction error, supervised learning seeks the predictor minimizing the risk E ( x,y ) ∼ P [ (cid:96) ( f ( x ) , y )] .
Since we only have access to the data distribution P ( X, Y ) via the dataset D , we instead search a predictor minimizing the empirical risk 1 n (cid:80) ni =1 (cid:96) ( f ( x i ) , y i )
(Vapnik, 1998).
The rest of this paper studies the problem of Domain Generalization (DG) , an extension of supervised learning where training datasets from multiple domains (or environments) are available to train our predictor (Blanchard et al., 2011).
Each domain d produces a dataset D d = { ( x di , y di ) } n d d = containing i.i.d.
examples from some probability distribution P ( X d , Y d ) , for all training domains d ∈ { 1 , . . .
, d tr } .
The goal of DG is out-of-distribution generalization : learning a predictor able to perform well at some unseen test domain d tr + 1 .
Since no data about the test domain is available during training, we must assume the existence of statistical invariances across training and testing domains, and incorporate such invariances (but nothing else) into our predictor.
The type of invariance assumed, as well as how to estimate it from the training datasets, varies between DG algorithms.
We review a hundred articles in DG spanning a decade of research and thirty algorithms in Appendix A.5.
DG differs from unsupervised domain adaptation.
In the latter, unlabeled data from the test domain is available during training (Pan and Yang, 2009; Patel et al., 2015; Wilson and Cook, 2018).
Tablecompares different machine learning setups to highlight the nature of DG problems.
The causality literature refers to DG as learning from multiple environments (Peters et al., 2016; Arjovsky et al., 2019).
Although challenging, the DG framework can capture some of the difﬁculty of real prediction problems, where unforeseen distributional discrepancies between training and testing data are surely expected.
At the same time, the framework can be limiting: in many real world scenarios there may be external variables informing about task relatedness (space, time, annotations) that the DG framework ignores.
M ODEL SELECTION AS PART OF THE LEARNING PROBLEM
Here we discuss issues surrounding model selection (choosing hyperparameters, training checkpoints, architecture variants) in DG and make speciﬁc recommendations for a path forward.
Because we lack access to a validation set identically distributed to the test data, model selection in DG is not as straightforward as in supervised learning.
Some works adopt heuristic strategies whose behavior is not well-studied, while others simply omit a description of how to choose hyperparameters.
This leaves open the possibility that hyperparameters were chosen using the test data, which is not methodologically sound.
Differences in results arising from inconsistent tuning practices may be misattributed to the algorithms under study, complicating fair assessments.
We believe that much of the confusion surrounding model selection in DG arises from treating it as merely a question of experimental design.
To the contrary, model selection requires making theoretical assumptions about how the test data relates to the training data.
Different DG algorithms make different assumptions, and it is not clear a priori which ones are correct, or how they inﬂuence the model selection criterion.
Indeed, choosing reasonable assumptions is at the heart of DG research.
Therefore, a DG algorithm without a strategy to choose its hyperparameters should be regarded as incomplete.
Recommendation 1 A DG algorithm should be responsible for specifying a model selection method.
While algorithms without well-justiﬁed model selection methods are incomplete, they may be useful stepping-stones in a research agenda.
In this case, instead of using an ad-hoc model selection method, we can evaluate incomplete algorithms by considering an oracle model selection method , where we select hyperparameters using some data from the test domain.
Of course, it is important to avoid invalid comparisons between oracle results and baselines tuned without an oracle method.
Also, unless we restrict access to the test domain data somehow, we risk obtaining meaningless results (we could just train on such test domain data using supervised learning).
Recommendation 2 Researchers should disclaim any oracle-selection results as such and specify policies to limit access to the test domain.
3. T HREE MODEL SELECTION METHODS FOR DG
Having made broad recommendations, we review and justify three model selection criteria for DG.
Appendix B.3 illustrates these with an speciﬁc example.
Training-domain validation We split each training domain into training and validation subsets.
We train models using the training subsets, and choose the model maximizing the accuracy on the union of validation subsets.
This strategy assumes that the training and test examples follow similar distributions.
For example, Ben-David et al.
(2010) bound the test error of a classiﬁer with the divergence between training and test domains.
Leave-one-domain-out validation Given d tr training domains, we train d tr models with equal hyperparameters, each holding one of the training domains out.
We evaluate each model on its held-out domain, and average the accuracies of these d tr models over their held-out domains.
Finally, we choose the model maximizing this average accuracy, retrained on all d tr domains.
This strategy assumes that training and test domains follow a meta-distribution over domains, and that our goal is to maximize the expected performance under this meta-distribution.
Note that leaving k > 1 domains out would increase greatly the number of experiments, and introduces a hyperparameter k .
Test-domain validation (oracle)
We choose the model maximizing the accuracy on a validation set that follows the distribution of the test domain.
Following our earlier recommendation to limit test domain access, we allow one query (the last checkpoint) per choice of hyperparameters, disallowing early stopping.
Recall that this is not a valid benchmarking methodology.
Oracle-based results can be either optimistic, because we select models using the test distribution, or pessimistic, because the query limit reduces the number of considered hyperparameters.
We also tried limiting the size of the oracle test set instead of the number of queries, but this led to unacceptably high variance.
3. C ONSIDERATIONS FROM THE LITERATURE
Some references in prior work discuss additional strategies to choose hyperparameters in DG.
For instance, Krueger et al.
(2020, Appendix B.1) suggest choosing hyperparameters to maximize the performance across all domains of an external dataset.
This “leave-one-dataset out” is akin to the second strategy outlined above.
Albuquerque et al.
(2019, Section 5.3.2) suggest performing model selection based on the loss function (which often incorporates an algorithm-speciﬁc regularizer), and
D’Innocente and Caputo (2018, Section 3) derive an strategy speciﬁc to their algorithm.
Finally, tools from differential privacy enable multiple reuses of a validation set (Dwork et al., 2015), which could be a tool to control the power of test-domain validation (oracle).
D OMAIN B ED : A P Y T ORCH TESTBED FOR DOMAIN GENERALIZATION
At the heart of our large scale experimentation is D OMAIN B ED , a PyTorch (Paszke et al., 2019) testbed to streamline reproducible and rigorous research in DG:
ANONYMIZED .
The initial release comprises fourteen algorithms, seven datasets, and three model selection methods (those described in Section 3), as well as the infrastructure to run all the experiments and generate all the L A TEX tables below with a single command.
The ﬁrst version of D OMAIN B ED focuses on image classiﬁcation, leaving for future work other types of tasks.
D OMAIN B ED is a living project: together with pull requests from collaborators, we continuously update the above repository with new algorithms, datasets, and result tables.
As illustrated in Appendix B.5, adding a new algorithm or dataset to D OMAIN B ED is a matter of a few lines of code.
Algorithms
D OMAIN B ED currently includes fourteen algorithms chosen based on their impact over the years, their published performance, and a desire to include varied DG strategies.
These are
Empirical Risk Minimization ( ERM , Vapnik (1998)), Group Distributionally Robust Optimization ( GroupDRO , Sagawa et al.
, Inter-domain Mixup ( Mixup , Xu et al.
Yan et al.
Wang et al.
(2020b)),
Meta-Learning for Domain Generalization ( MLDG , Li et al.
(2018a)), DomainAdversarial Neural Networks ( DANN , Ganin et al.
Class-conditional DANN ( C-DANN ,
Li et al.
(2018d))
, Deep CORrelation ALignment ( CORAL , Sun and Saenko (2016)), Maximum
Mean Discrepancy ( MMD , Li et al.
(2018b)),
Invariant Risk Minimization ( IRM Arjovsky et al.
(2019)), Adaptive Risk Minimization ( ARM , Zhang et al.
Marginal Transfer Learning ( MTL ,
Blanchard et al.
(2011; 2017)), Style-Agnostic Networks ( SagNet , Nam et al.
and Representation Self Challenging ( RSC , Huang et al.
Appendix B.1 describes these algorithms, and
Appendix B.4 lists their network architectures and hyperparameter search distributions.
Datasets
D OMAIN B ED currently includes downloaders and loaders for seven standard DG image classiﬁcation benchmarks.
These are Colored MNIST (Arjovsky et al., 2019), Rotated MNIST (Ghifary et al., 2015), PACS (Li et al., 2017), VLCS (Fang et al., 2013), OfﬁceHome (Venkateswara et al., 2017), Terra Incognita (Beery et al., 2018), and DomainNet (Peng et al., 2019).
The datasets based on MNIST are “synthetic” since changes across domains are well understood (colors and rotations).
The rest of the datasets are “real” since domains vary in unknown ways.
Appendix B. describes these datasets.
Implementation choices We highlight three implementation choices made towards a consistent and realistic evaluation setting.
First, whereas prior work is inconsistent in its choice of network architecture, we ﬁnetune ResNet-50 models (He et al., 2016) pretrained on ImageNet for all nonMNIST experiments.
We note that recent state-of-the-art results (Balaji et al., 2018; Nam et al., 2019; Huang et al., 2020) also use ResNet-50 models.
Second, for all non-MNIST datasets, we augment training data using the following protocol: crops of random size and aspect ratio, resizing to 224 × 224 pixels, random horizontal ﬂips, random color jitter, grayscaling the image with 10% probability, and normalization using the ImageNet channel statistics.
This augmentation protocol is increasingly standard in state-of-the-art DG work (Nam et al., 2019;
Huang et al., 2020; Krueger et al., 2020; Carlucci et al., 2019a; Zhou et al., 2020; Dou et al., 2019; Hendrycks et al., 2020;
Wang et al., 2020a; Seo et al., 2020; Chattopadhyay et al., 2020).
We use no augmentation for MNIST-based datasets.
Third, and for RotatedMNIST, we divide all the digits evenly among domains, instead of replicating the same 1000 digits to construct all domains.
We deviate from standard practice for two reasons: using the same digits across training and test domains leaks test data, and reducing the amount of training data complicates the task in an unrealistic way.
E XPERIMENTS
We run experiments for all algorithms, datasets, and model selection criteria shipped in D OMAIN B ED .
We consider all conﬁgurations of a dataset where we hide one domain for testing, resulting in the training of 58,000 models.
To generate the following results, we simply run sweep.py at commit 0xANONYMIZED from D OMAIN B ED ’s repository.
Hyperparameter search For each algorithm and test domain, we conduct a random search (Bergstra and Bengio, 2012) of 20 trials over a joint distribution of all hyperparameters (Appendix B.4).
Appendix C.4 shows that running more than 20 trials does not improve our results signiﬁcantly.
We use each model selection criterion to select amongst the 20 models from the random search.
We split the data from each domain into 80% and 20% splits.
We use the larger splits for training and ﬁnal evaluation, and the smaller splits to select hyperparameters (for an illustration, see Appendix B.3).
All hyperparameters are optimized anew for each algorithm and test domain, including hyperparameters like learning rates which are common to multiple algorithms.
Standard error bars While some DG literature reports error bars across seeds, randomness arising from model selection is often ignored.
This is acceptable if the goal is best-versus-best comparison, but prohibits analyses concerning the model selection process itself.
Instead, we repeat our entire study three times, making every random choice anew: hyperparameters, weight initializations, and dataset splits.
Every number we report is a mean (and its standard error) over these repetitions.
5. R ESULTS
Table 3 summarizes the results of our experiments.
Appendix C contains the full results per dataset and domain.
As anticipated in our introduction, we draw three conclusions from our results.
Claim 1: Carefully tuned ERM outperforms the previously published state-of-the-art
Table(full version in Appendix A.5) shows this result, when we provide ERM with a training-domain validation set for hyperparameter selection.
Such state-of-the-art average performance of our ERM baseline holds even when we select the best competitor available in the literature separately for each benchmark.
One reason for ERM’s strong performance is that we use ResNet-50, whereas some prior work uses smaller ResNet-18 models.
As recently shown in the literature (Hendrycks et al., 2020), this suggests that better in-distribution generalization is a dominant factor behind better out-of-distribution generalization.
Our result does not refute prior work: it is possible that with stronger implementations, some competing methods may improve upon ERM.
Rather, we provide a strong, realistic, and reproducible baseline for future work to build upon.
Claim 2: When evaluated in a consistent setting, no algorithm outperforms ERM in more than one point We observe this result in Table 3, obtained by running from scratch every combination of dataset, algorithm, and model selection criterion in D OMAIN B ED .
Given any model selection criterion, no method improves the average performance of ERM in more than one point.
At the number of trials performed, no improvement over ERM is statistically signiﬁcant according to a t -test at a signiﬁcance level α = 0 .
While new algorithms could improve upon ERM (an exciting premise!), getting substantial DG improvements in a rigorous way proved challenging.
Most of our baselines can achieve ERM-like performance because there have hyperparameter conﬁgurations under which they behave like ERM (e.g. regularization coefﬁcients that can be set to zero).
Our advice to DG practitioners is to use ERM (which is a safe contender) or CORAL (Sun and Saenko, 2016) (which achieved the highest average score).
Claim 3: Model selection methods matter We observe that model selection with a training domain validation set outperforms leave-one-domain-out cross-validation across multiple datasets and algorithms.
This does not mean that using a training domain validation set is the right way to tune hyperparameters.
In fact, the stronger performance of oracle-selection ( +2 .
3 points for ERM) suggests headroom to develop improved DG model selection criteria.
5. A BLATION STUDY ON ERM
To better understand our ERM performance, we perform an ablation study on the neural network architecture and the data augmentation protocol.
Table 5.2 shows that using a ResNet-50 neural network architecture, instead of a smaller ResNet-18, improves DG test accuracy by 3 . 7 points.
Using data augmentation improves DG test accuracy by 0 .
5 points.
However, these ResNet models were pretrained on ImageNet using data augmentation , so the beneﬁts of augmentation are partly absorbed by the model.
In fact, we hypothesize that among models pretrained on ImageNet, domain generalization performance is mainly inﬂuenced by the model’s original test accuracy on ImageNet.
[Table] [Table]
D ISCUSSIONS
We provide several discussions to help the reader interpret our results and motivate future work
[Table]
Our negative claims are fundamentally limited Broad negative claims (e
[Table]
g. “algorithm X does not outperform ERM”) do not specify an exact experimental setting and are therefore impossible to rigorously prove.
In order to be veriﬁable, such claims must be restricted to a speciﬁc setting.
This limitation is fundamental to all negative result claims, and ours (Claim 2) is no exception.
We have shown that many algorithms don’t substantially improve on ERM in our setting
[Table]
but the relevance of that setting is a subjective matter ultimately left for the reader.
In making this judgement, the reader should consider whether they agree with our methodological and implementation choices, which we have explained and motivated throughout the paper.
We also note that our implementation can outperform previous results (Table 1).
Finally, DomainBed is not a black box: our implementation is open-source and actively maintained, and we invite the research community to improve on our results.
Is this as good as it gets?
We question whether DG is possible in some of the considered datasets.
Why do we assume that a neural network should be able to classify cartoons, given only photorealistic training data?
In the case of Rotated MNIST, do truly rotation-invariant features discriminative of the digit class exist?
Are those features expressible by a neural network?
Even in the presence of correct model selection, is the out-of-distribution performance of modern ERM implementations as good as it gets?
Or is it simply as poor as every other alternative?
How far are we from the achievable DG performance?
Is this upper-bound simply the test error in-domain?
Are these the right datasets?
Most datasets considered in the DG literature do not reﬂect realistic situations.
If one wanted to classify cartoons, the easiest option would be to collect a small labeled dataset of cartoons.
Should we consider more realistic, impactful tasks for better research in DG?
Some alternatives are medical imaging in different hospitals and self-driving cars in different cities.
Generalization requires untestable assumptions Every time we use ERM, we assume that training and testing examples follow the same distribution.
This is an untestable assumption in every single instance.
The same applies for DG: each algorithm assumes a different (untestable) type of invariance across domains.
Therefore, the performance of a DG algorithm depends on the problem at hand, and only time can tell if we have made a good choice.
This is akin to the generalization of a scientiﬁc theory such as Newton’s gravitation, which cannot be proven, but rather only resist falsiﬁcation.
C ONCLUSION
Our extensive empirical evaluation of DG algorithms leads to three conclusions.
First, a carefully tuned ERM baseline outperforms the previously published state-of-the-art results in terms of average performance (Claim 1) .
Second, when compared to thirteen popular DG alternatives on the exact same experimental conditions, we ﬁnd out that no competitor is able to outperform ERM by more than one point (Claim 2) .
Third, model selection is non-trivial for DG, and it should be an integral part of any proposed method (Claim 3) .
Going forward, we hope that our results and D OMAIN B ED promote realistic and rigorous evaluation and enable advances in domain generalization.
A A DECADE OF LITERATURE ON DOMAIN GENERALIZATION
In this section, we provide an exhaustive literature review on a decade of domain generalization research.
The following classiﬁes domain generalization algorithms according into four strategies to learn invariant predictors: learning invariant features, sharing parameters, meta-learning, or performing data augmentation.
A. L EARNING INVARIANT FEATURES
Muandet et al.
(2013) use kernel methods to ﬁnd a feature transformation that (i) minimizes the distance between transformed feature distributions across domains, and (ii) does not destroy any of the information between the original features and the targets.
In their pioneering work, Ganin et al.
(2016) propose Domain Adversarial Neural Networks (DANN), a domain adaptation technique which uses generative adversarial networks (GANs, Goodfellow et al.
to learn a feature representation that matches across training domains.
Akuzawa et al.
(2019) extend DANN by considering cases where there exists an statistical dependence between the domain and the class label variables.
Albuquerque et al.
(2019) extend DANN by considering one-versus-all adversaries that try to predict to which training domain does each of the examples belong to.
Li et al.
(2018b) employ
GANs and the maximum mean discrepancy criteria (Gretton et al., 2012) to align feature distributions across domains.
Matsuura and Harada (2019) leverages clustering techniques to learn domaininvariant features even when the separation between training domains is not given.
Li et al.
(2018c;d) learns a feature transformation φ such that the conditional distributions P ( φ ( X d ) | Y d = y ) match for all training domains d and label values y .
Shankar et al.
(2018) use a domain classiﬁer to construct adversarial examples for a label classiﬁer, and use a label classiﬁer to construct adversarial examples for the domain classiﬁer.
This results in a label classiﬁer with better domain generalization.
Li et al.
(2019a) train a robust feature extractor and classiﬁer.
The robustness comes from (i) asking the feature extractor to produce features such that a classiﬁer trained on domain d can classify instances for domain d (cid:48)
(cid:54) =
d , and (ii) asking the classiﬁer to predict labels on domain d using features produced by a feature extractor trained on domain d (cid:48)
(cid:54) =
d .
Li et al.
(2020) adopt a lifelong learning strategy to attack the problem of domain generalization.
Motiian et al.
(2017) learn a feature representation such that (i) examples from different domains but the same class are close, (ii) examples from different domains and classes are far, and (iii) training examples can be correctly classiﬁed.
Ilse et al.
(2019) train a variational autoencoder (Kingma and Welling, 2014) where the bottleneck representation factorizes knowledge about domain, class label, and residual variations in the input space.
Fang et al.
(2013) learn a structural SVM metric such that the neighborhood of each example contains examples from the same category and all training domains.
The algorithms of Sun and Saenko (2016); Sun et al.
; Rahman et al.
(2019a) match the feature covariance (second order statistics) across training domains at some level of representation.
The algorithms of Ghifary et al.
; Hu et al.
(2019) use kernel-based multivariate component analysis to minimize the mismatch between training domains while maximizing class separability.
Although popular, learning domain-invariant features has received some criticism (Zhao et al., 2019;
Johansson et al., 2019).
Some alternatives exist, as we review next.
Peters et al.
; Rojas-Carulla et al.
(2018) considered that one should search for features that lead to the same optimal classiﬁer across training domains.
In their pioneering work, Peters et al.
(2016) linked this type of invariance to the causal structure of data, and provided a basic algorithm to learn invariant linear models, based on feature selection.
Arjovsky et al.
(2019) extend the previous to general gradient-based models, including neural networks, in their Invariant Risk Minimization (IRM) principle.
Teney et al.
(2020) build on IRM to learn a feature transformation that minimizes the relative variance of classiﬁer weights across training datasets.
The authors apply their method to reduce the learning of spurious correlations in Visual Question Answering (VQA) tasks.
Ahuja et al.
(2020) analyze IRM under a game-theoretic perspective to develop an alternative algorithm.
Krueger et al.
(2020) propose an approximation to the IRM problem consisting in reducing the variance of error averages across domains.
Bouvier et al.
(2019) attack the same problem as IRM by re-weighting data samples.
A. S HARING PARAMETERS
Blanchard et al.
(2011) build classiﬁers f ( x d , µ d ) , where µ d is a kernel mean embedding (Muandet et al., 2017) that summarizes the dataset associated to the example x d .
Since the distributional identity of test instances is unknown, these embeddings are estimated using single test examples at test time.
See Blanchard et al.
Deshmukh et al.
(2019) for theoretical results on this family of algorithms (only applicable when using RKHS-based learners).
Zhang et al.
(2020) is an extension of Blanchard et al.
(2011) where a separate CNN computes the domain embedding, appended to the input image as additional channels.
Khosla et al.
(2012) learn one max-margin linear classiﬁer w d = w + ∆ d per domain d , from which they distill their ﬁnal, invariant predictor w .
Ghifary et al.
(2015) use a multitask autoencoder to learn invariances across domains.
To achieve this, the authors assume that each training dataset contains the same examples; for instance, photographs about the same objects under different views.
Mancini et al.
(2018b) train a deep neural network with one set of dedicated batch-normalization layers (Ioffe and Szegedy, 2015) per training dataset.
Then, a softmax domain classiﬁer predicts how to linearly-combine the batch-normalization layers at test time.
Seo et al.
(2020) combines instance normalization with batch-normalization to learn a normalization module per domain, enhancing out-of-distribution generalization.
Similarly, Mancini et al.
(2018a) learn a softmax domain classiﬁer used to linearly-combine domain-speciﬁc predictors at test time.
D’Innocente and Caputo (2018) explore more sophisticated ways of aggregating domain-speciﬁc predictors.
Li et al.
(2017) extends Khosla et al.
(2012) to deep neural networks by extending each of their parameter tensors with one additional dimension, indexed by the training domains, and set to a neutral value to predict domain-agnostic test examples.
Ding and Fu (2017) implement parametertying and low-rank reconstruction losses to learn a predictor that relies on common knowledge across training domains.
Hu et al.
; Sagawa et al.
(2019) weight the importance of the minibatches of the training distributions proportional to their error.
Chattopadhyay et al.
(2020) overlays multiple weight masks over a single network to learn domain-invariant and domain-speciﬁc features.
A. M ETA - LEARNING
Li et al.
(2018a)
employ Model-Agnostic Meta-Learning, or MAML (Finn et al., 2017), to build a predictor that learns how to adapt fast between training domains.
Dou et al.
(2019) use a similar
MAML strategy, together with two regularizers that encourage features from different domains to respect inter-class relationships, and be compactly clustered by class labels.
Li et al.
(2019b) extend the MAML meta-learning strategy to instances of domain generalization where the categories vary from domain to domain.
Balaji et al.
(2018) use MAML to meta-learn a regularizer encouraging the model trained on one domain to perform well on another domain.
A. A UGMENTING DATA
Data augmentation is an effective strategy to address domain generalization (Zhang et al., 2019).
Unfortunately, how to design efﬁcient data augmentation routines depends on the type of data at hand, and demands a signiﬁcant amount of work from human experts.
Xu et al.
Yan et al.
(2020); Wang et al.
(2020b) use mixup (Zhang et al., 2018) to blend examples from the different training distributions.
Carlucci et al.
(2019a) constructs an auxiliary classiﬁcation task aimed at solving jigsaw puzzles of image patches.
The authors show that this self-supervised learning task learns features that improve domain generalization.
Similarly, Wang et al.
(2020a) use metric learning and self-supervised learning to augment the out-of-distribution performance of an image classiﬁer.
Albuquerque et al.
(2020) introduce the self-supervised task of predicting responses to Gabor ﬁlter banks, in order to learn more transferrable features.
Wang et al.
(2019) remove textural information from images to improve domain generalization.
Volpi et al.
(2018) show that training with adversarial data augmentation on a single domain is sufﬁcient to improve domain generalization.
Nam et al.
(2019) promote representations of data that ignore image style and focus on content.
Rahman et al.
(2019b)
; Zhou et al.
; Carlucci et al.
(2019a) are three alternatives that use GANs to augment the data available during training time.
Representation Self-Challenging (Huang et al., 2020) learns robust neural networks by iteratively dropping-out important features.
Hendrycks et al.
(2020) show that, together with larger models and data, data augmentation improves out-of-distribution performance.
A
P REVIOUS STATE - OF - THE - ART NUMBERS
Table 5 compiles the best out-of-distribution test accuracies reported across a decade of domain generalization research.
[Table]
PACS
[Table]
B M ORE ABOUT D OMAIN B ED B. A LGORITHMS
Empirical Risk Minimization ( ERM , Vapnik (1998)) minimizes the errors across domains.
Group Distributionally Robust Optimization ( DRO , Sagawa et al.
(2019)) performs ERM while increasing the importance of domains with larger errors.
Inter-domain Mixup ( Mixup , Xu et al.
Yan et al.
; Wang et al.
(2020b)) performs ERM on linear interpolations of examples from random pairs of domains and their labels.
Meta-Learning for Domain Generalization ( MLDG , Li et al.
(2018a))
leverages MAML (Finn et al., 2017) to meta-learn how to generalize across domains.
Domain-Adversarial Neural Networks ( DANN , Ganin et al. (2016)) employ an adversarial network to match feature distributions across environments.
Class-conditional DANN ( C-DANN , Li et al. (2018d)) is a variant of DANN matching the conditional distributions P ( φ ( X d ) | Y d = y ) across domains, for all labels y .
CORAL (Sun and Saenko, 2016) matches the mean and covariance of feature distributions.
MMD (Li et al., 2018b) matches the MMD (Gretton et al., 2012) of feature distributions.
Invariant Risk Minimization ( IRM Arjovsky et al.
(2019)) learns a feature representation φ ( X d ) such that the optimal linear classiﬁer on top of that representation matches across domains.
Risk Extrapolation ( VREx , Krueger et al.
(2020)) approximates IRM with a variance penalty.
Marginal Transfer Learning ( MTL , Blanchard et al.
(2011; 2017)) estimates a mean embedding per domain, passed as a second argument to the classiﬁer.
Adaptive Risk Min.
( ARM , Zhang et al.
(2020)) extends MTL with a separate embedding CNN.
Style-Agnostic Networks ( SagNets , Nam et al.
(2019)) learns neural networks by keeping image content and randomizing style.
Representation Self-Challenging ( RSC , Huang et al.
(2020)) learns robust neural networks by iteratively discarding (challenging) the most activated features.
B. D ATASETS
D OMAIN B ED includes downloaders and loaders for seven multi-domain image classiﬁcation tasks:
Colored MNIST (Arjovsky et al., 2019) is a variant of the MNIST handwritten digit classiﬁcation dataset (LeCun, 1998).
Domain d ∈ { 0 . 1 , 0 . 3 , 0 . 9 } contains a disjoint set of digits colored either red or blue.
The label is a noisy function of the digit and color, such that color bears correlation d with the label and the digit bears correlation 0.75 with the label.
This dataset contains 70 ,examples of dimension (2 , 28 , 28) and 2 classes.
Rotated MNIST
(Ghifary et al., 2015) is a variant of MNIST where domain d ∈ { 0, 15, 30, 45, 60, 75 } contains digits rotated by d degrees.
Our dataset contains 70 , 000 examples of dimension (1 , 28 , 28) and 10 classes.
PACS (Li et al., 2017) comprises four domains d ∈ { art, cartoons, photos, sketches } .
This dataset contains 9 , 991 examples of dimension (3 , 224 , 224) and 7 classes.
VLCS (Fang et al., 2013) comprises photographic domains d ∈ { Caltech101, LabelMe, SUN09, VOC2007 } .
This dataset contains 10 , 729 examples of dimension (3 , 224 , 224) and 5 classes.
OfﬁceHome (Venkateswara et al., 2017) includes domains d ∈ { art, clipart, product, real } .
This dataset contains 15 , 588 examples of dimension (3 , 224 , 224) and 65 classes.
Terra Incognita (Beery et al., 2018) contains photographs of wild animals taken by camera traps at locations d ∈ { L100 , L38 , L43 , L46 } .
Our version of this dataset contains 24 , 788 examples of dimension (3 , 224 , 224) and 10 classes.
DomainNet (Peng et al., 2019) has six domains d ∈ { clipart, infograph, painting, quickdraw, real, sketch } .
This dataset contains 586 , 575 examples of size (3 , 224 , 224) and 345 classes.
For all datasets, we ﬁrst pool the raw training, validation, and testing images together.
For each random seed, we then instantiate random training, validation, and testing splits.
B.
M
Consider Figure 1, and let T i
= { A i , B i , C i } for i ∈ { 1 , 2 } .
Training-domain validation trains each hyperparameter conﬁguration on T 1 and chooses the conﬁguration with the highest performance in
T 2 .
Leave-one-out validation trains one clone F Z of each hyperparameter conﬁguration on T 1 \ Z , for
Z ∈ T 1 ; then, it chooses the conﬁguration with highest (cid:80) Z ∈ T 1 Performance ( F Z , Z ) .
Test-domain validation trains each hyperparameter conﬁguration on T 1 and chooses the conﬁguration with the highest performance on D 2 , only looking at its ﬁnal epoch.
Finally, result tables show the performance of selected models on D 1 .
[Figure]
B.
A RCHITECTURES AND HYPERPARAMETERS
Neural network architectures used for each dataset:
Dataset
Architecture
[Figure]
Neural network architecture for MNIST experiments:
Layer
Conv2D (in= d , out=64)
ReLU
GroupNorm (groups=8)
Conv2D (in=64, out=128, stride=2)
ReLU
GroupNorm (8 groups)
Conv2D (in=128, out=128)
ReLU
GroupNorm (8 groups)
Conv2D (in=128, out=128)
ReLU
GroupNorm (8 groups)
Global
average
-pooling
For “ResNet-50”, we replace the ﬁnal (softmax) layer of a ResNet50 pretrained on ImageNet and ﬁne-tune the entire network.
Since minibatches from different domains follow different distributions, batch normalization degrades domain generalization algorithms (Seo et al., 2020).
Therefore, we freeze all batch normalization layers before ﬁne-tuning.
We insert a dropout layer before the ﬁnal
ResNet-50 linear layer.
Table 6 lists all algorithm hyperparameters, their default values, and their sweep random search distribution.
We optimize all models using Adam (Kingma and Ba, 2015).
[Table]
B. E XTENDING D OMAIN B ED
Algorithms are classes that implement two methods: .update(minibatches) and .predict(x) .
The update method receives a list of minibatches, one minibatch per training domain, and each minibatch containing one input and one output tensor.
For example, to implement group DRO (Sagawa et al., 2019, Algorithm 1), we simply write the following in algorithms.py :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
def
[Table]
if
[Table]
for
[Table]
ALGORITHMS
[Table]
By inheriting from ERM, the new GroupDRO class has access to a default classiﬁer .network , optimizer .optimizer , and prediction method .predict(x) .
Finally, we tell D OMAIN B ED about the default values and hyperparameter search distributions of the hyperparameters of this new algorithm.
We do so by adding the following to the function hparams in hparams registry.py :
[Equation]
[Equation] [Equation]
To add a new image classiﬁcation dataset to D OMAIN B ED , arrange your image ﬁles as /root/MyDataset/domain/class/image.jpg .
Then, append to datasets.py :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
super
[Equation]
).__init
[Table]
[Equation]
self.dir, test_envs, augment) DATASETS.append(’MyDataset’)
We are now ready to train our new algorithm on our new dataset, using the second domain as test:
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Finally, we can run a fully automated sweep on all datasets, algorithms, test domains, and model selection criteria by simply invoking python sweep.py , after extending the ﬁle command launchers.py to your computing infrastructure.
When the sweep ﬁnishes, the script collect results.py automatically generates all the result tables shown in this manuscript.
Extension to UDA One can use D OMAIN B ED to perform experimentation on unsupervised domain adaptation by extending the .update(minibatches) methods to accept unlabeled examples from the test domain.
RESULTS
M
C.1.
[Table] [Table]
[Table] [Table]
A VERAGES
[Table] [Table]
50.1 ± 0.
10.0 ± 0.
IRM 46.7 ± 2.
51.2 ± 0.
23.1 ± 10.
GroupDRO 50.1 ± 0.
50.0 ± 0.
10.2 ± 0.
Mixup 36.6 ± 10.
53.4 ± 5.
10.2 ± 0.
MLDG 50.1 ± 0.
50.1 ± 0.
10.0 ± 0.
CORAL 49.5 ± 0.
59.5 ± 8.
10.2 ± 0.
MMD 50.3 ± 0.
50.0 ± 0.
9.9 ± 0.
DANN 49.9 ± 0.
62.1 ± 7.
10.0 ± 0.
CDANN 63.2 ± 10.
44.4 ± 4.
9.9 ± 0.
MTL 44.3 ± 4.
50.7 ± 0.
10.1 ± 0.
SagNet
[Table]
MNIST
[Table]
[Table] [Table]
C.2.
A VERAGES
[Table] [Table]
C OLORED MNIST
Algorithm +90%
+80% -90%
Avg
ERM 71.8 ± 0.
72.9 ± 0.
28.7 ± 0.
IRM 72.0 ± 0.
72.5 ± 0.
58.5 ± 3.
GroupDRO 73.5 ± 0.
73.0 ± 0.
[Table] [Table]
[Table] [Table]
[Table] [Table]