B REEDS : B ENCHMARKS FOR S UBPOPULATION S HIFT
Aleksander M ˛adry madry
A BSTRACT
We develop a methodology for assessing the robustness of models to subpopulation shift—speciﬁcally, their ability to generalize to novel data subpopulations that were not observed during training.
Our approach leverages the class structure underlying existing datasets to control the data subpopulations that comprise the training and test distributions.
This enables us to synthesize realistic distribution shifts whose sources can be precisely controlled and characterized, within existing large-scale datasets.
Applying this methodology to the ImageNet dataset, we create a suite of subpopulation shift benchmarks of varying granularity.
We then validate that the corresponding shifts are tractable by obtaining human baselines.
Finally, we utilize these benchmarks to measure the sensitivity of standard model architectures as well as the effectiveness of existing train-time robustness interventions.
I NTRODUCTION
Robustness to distribution shift has been the focus of a long line of work in machine learning (Schlimmer & Granger, 1986; Widmer & Kubat, 1993; Kelly et al., 1999; Shimodaira, 2000; Sugiyama et al., 2007; Quionero-Candela et al., 2009; Moreno-Torres et al., 2012; Sugiyama & Kawanabe, 2012).
At a high-level, the goal is to ensure that models perform well not only on unseen samples from the datasets they are trained on, but also on the diverse set of inputs they are likely to encounter in the real world.
However, building benchmarks for evaluating such robustness is challenging—it requires modeling realistic data variations in a way that is well-deﬁned, controllable, and easy to simulate.
Prior work in this context has focused on building benchmarks that capture distribution shifts caused by natural or adversarial input corruptions (Szegedy et al., 2014; Fawzi & Frossard, 2015; Fawzi et al., 2016; Engstrom et al., 2019b; Ford et al., 2019; Hendrycks & Dietterich, 2019; Kang et al., 2019), differences in data sources (Saenko et al., 2010; Torralba & Efros, 2011; Khosla et al., 2012; Tommasi & Tuytelaars, 2014; Recht et al., 2019), and changes in the frequencies of data subpopulations (Oren et al., 2019; Sagawa et al., 2020).
While each of these approaches captures a different source of real-world distribution shift, we cannot expect any single benchmark to be comprehensive.
Thus, to obtain a holistic understanding of model robustness, we need to keep expanding our testbed to encompass more natural modes of variation.
In this work, we take another step in that direction by studying the following question:
How well do models generalize to data subpopulations they have not seen during training?
The notion of subpopulation shift this question refers to is quite pervasive.
After all, our training datasets will inevitably fail to perfectly capture the diversity of the real word.
Hence, during deployment, our models are bound to encounter unseen subpopulations—for instance, unexpected weather conditions in the self-driving car context or different diagnostic setups in medical applications.
O UR CONTRIBUTIONS
The goal of our work is to create large-scale subpopulation shift benchmarks wherein the data subpopulations present during model training and evaluation differ.
These benchmarks aim to assess how effectively models generalize beyond the limited diversity of their training datasets— e.g., whether models can recognize Dalmatians as “dogs” even when their training data for “dogs” comprises only Poodles and Terriers.
We show how one can simulate such shifts, fairly naturally, within existing datasets, hence eliminating the need for (and the potential biases introduced by) crafting synthetic transformations or collecting additional data.
B REEDS benchmarks.
The crux of our approach is to leverage existing dataset labels and use them to identify superclasses —i.e., groups of semantically similar classes.
This allows us to construct classiﬁcation tasks over such superclasses, and repurpose the original dataset classes to be the subpopulations of interest.
This, in turn, enables us to induce a subpopulation shift by directly making the subpopulations present in the training and test distributions disjoint.
By applying this methodology to the ImageNet dataset (Deng et al., 2009), we create a suite of subpopulation shift benchmarks of varying difﬁculty.
This involves modifying the existing ImageNet class hierarchy—WordNet (Miller, 1995)—to ensure that superclasses comprise visually coherent subpopulations.
We conduct human studies to validate that the resulting benchmarks capture meaningful subpopulation shifts.
Model robustness to subpopulation shift.
In order to demonstrate the utility of our benchmarks, we employ them to evaluate the robustness of standard models to subpopulation shift.
In general, we ﬁnd that model performance drops signiﬁcantly on the shifted distribution—even when this shift does not signiﬁcantly affect humans.
Still, models that are more accurate on the original distribution tend to also be more robust to these subpopulation shifts.
Moreover, adapting models to the shifted domain, by retraining their last layer on this domain, only partially recovers the original model performance.
Impact of robustness interventions.
Finally, we examine whether various train-time interventions, designed to decrease model sensitivity to synthetic data corruptions (e.g., (cid:96) 2 -bounded perturbations) make models more robust to subpopulation shift.
We ﬁnd that many of these methods offer small, yet non-trivial, improvements along this axis—at times, at the expense of performance on the original distribution.
Often, these improvements become more pronounced after retraining the last layer of the model on the shifted distribution.
Nevertheless, the increase in model robustness to subpopulation shifts due to these interventions is much smaller than what is observed for other families of input variations such as data corruptions (Hendrycks & Dietterich, 2019; Ford et al., 2019; Kang et al., 2019; Taori et al., 2020).
This indicates that handling subpopulation shifts, such as those present in the B REEDS benchmarks, might require a different set of robustness tools.
D ESIGNING B ENCHMARKS FOR D ISTRIBUTION S HIFT
When constructing distribution shift benchmarks, the key design choice lies in specifying the target distribution to be used during model evaluation.
This distribution is meant to be a realistic variation of the source distribution , that was used for training.
Typically, studies focus on variations due to:
• Data corruptions : The target distribution is obtained by modifying inputs from the source distribution via a family of transformations that mimic real-world corruptions, as in Fawzi & Frossard (2015); Fawzi et al.
Engstrom et al. (2019b); Hendrycks & Dietterich (2019); Ford et al.
; Kang et al.
; Shankar et al.
• Differences in data sources : Here, the target distribution is an independent dataset for the same task (Saenko et al., 2010; Torralba & Efros, 2011; Tommasi & Tuytelaars, 2014; Recht et al., 2019)—e
g., collected at a different geographic location (Beery et al., 2018), time frame (Kumar et al., 2020) or user population (Caldas et al., 2018).
For instance, this could involve using PASCAL VOC (Everingham et al., 2010) to evaluate Caltech101-trained classiﬁers (Fei-Fei et al., 2006).
The goal is to test whether models are overly reliant on the idiosyncrasies of their training datasets (Ponce et al., 2006; Torralba & Efros, 2011).
• Subpopulation representation : The source and target distributions differ in terms of how well-represented each subpopulation is.
Work in this area typically studies whether models perform equally well across all subpopulations from the perspective of reliability (Meinshausen et al., 2015; Hu et al., 2018; Duchi & Namkoong, 2018;
Caldas et al., 2018; Oren et al., 2019; Sagawa et al., 2020) or algorithmic fairness (Dwork et al., 2012; Kleinberg et al., 2017; Jurgens et al., 2017; Buolamwini & Gebru, 2018; Hashimoto et al., 2018).
.e
variations the data encountered during deployment each the benchmarks real robustness to of these
T HE B REEDS M ETHODOLOGY
In this work, we focus on modeling a pertinent, yet less studied, form of subpopulation shift: one wherein the target distribution (used for testing) contains subpopulations that are entirely absent from the source distribution that the model was trained on.
To simulate such shifts, we need to precisely control the data subpopulations present in the source and target data distributions.
Our procedure for doing this comprises two stages that are outlined below—see Figure 1 for an illustration and
Appendix A.2 for pseudocode.
Devising subpopulation structure.
Typical datasets do not contain annotations for individual subpopulations.
Since collecting such annotations would be challenging, we take an alternative approach: we bootstrap the existing dataset labels to simulate subpopulations.
That is, we group semantically similar classes into broader superclasses which, in turn, allows us to re-purpose existing class labels as the desired subpopulation annotations.
Moreover, we can group classes in a hierarchical manner, obtaining superclasses of different speciﬁcity.
As we will see in Section 4, such class hierarchies are already present in large-scale benchmarks (Deng et al., 2009; Kuznetsova et al., 2018).
Simulating subpopulation shifts.
Given a set of superclasses, we can deﬁne a classiﬁcation task over them: the inputs of each superclass correspond to pooling together the inputs of its subclasses (i.e., the original dataset classes).
Within this setup, we can simulate subpopulation shift in a relatively straightforward manner.
Speciﬁcally, for each superclass, we split its subclasses into two random and disjoint sets, and assign one of them to the source and the other to the target domain.
Then, we can evaluate model robustness under subpopulation shift by simply training on the source domain and testing on the target domain.
Note that the classiﬁcation task remains identical between domains—both domains contain the same (super)classes but the subpopulations that comprise each (super)class differ.
Intuitively, this corresponds to using different dog breeds to represent the class “dog” during training and testing—hence the name of our toolkit.
This methodology is quite general and can be applied to a variety of setting to simulate realistic distribution shifts.
Moreover, it has a number of additional beneﬁts:
• Flexibility: Different semantic groupings of a ﬁxed set of classes lead to B REEDS tasks of varying granularity.
For instance, by only grouping together classes that are quite similar one can reduce the severity of the subpopulation shift.
Alternatively, one can consider broad superclasses, each having multiple subclasses, resulting in a more challenging benchmark.
• Precise characterization: The exact subpopulation shift between the source and target domains is known.
Since both domains are constructed from the same dataset, the impact of any external factors (e.g., differences in data collection pipelines) is minimized.
Note that such external factors can signiﬁcantly impact the difﬁculty of the task (Ponce et al., 2006; Torralba & Efros, 2011; Tsipras et al., 2020).
In fact, minimizing these effects and ensuring that the shift between the source and target domain is caused solely by the intended input variations is one of the major challenges in building distribution shift benchmarks.
For instance, recent work (Engstrom et al., 2020) demonstrates that statistical biases during data collection can signiﬁcantly skew the intended target distribution.
• Symmetry:
Since subpopulations are split into the source and test domains randomly, we expect the resulting tasks to have comparable difﬁculty.
• Reuse of existing datasets: No additional data collection or annotation is required other than choosing the class grouping.
This approach can thus be used to also re-purpose other existing large-scale datasets—even beyond image recognition—with minimal effort.
S IMULATING S UBPOPULATION S HIFTS W ITHIN I MAGE N ET
We now describe how our methodology can be applied to ImageNet (Deng et al., 2009)—speciﬁcally, the ILSVRC2012 subset (Russakovsky et al.
, 2015)—to create a suite of B REEDS benchmarks.
ImageNet contains a large number of classes, making it particularly well-suited for our purpose.
4. U TILIZING THE I MAGE N ET CLASS HIERARCHY
Recall that creating B REEDS tasks requires grouping together similar classes.
For ImageNet, such a semantic grouping already exists—ImageNet classes are a part of the WordNet hierarchy (Miller, 1995).
However, WordNet is not a hierarchy of objects but rather one of word meanings.
Thus, intermediate hierarchy nodes are not always well-suited for object recognition due to:
• Abstract groupings: WordNet nodes often correspond to abstract concepts, e
g., related to the functionality of an object.
Children of such nodes might thus share little visual similarity—e.g., “umbrella” and “roof” are visually different, despite both being “coverings.”
• Non-uniform categorization:
The granularity of object categorization is vastly different across the WordNet hierarchy—e.g., the subtree rooted at “dog” is 25-times larger than the one rooted at “cat.”
Hence, the depth of a node in this hierarchy does not always reﬂect the speciﬁcity of the corresponding object category.
• Lack of tree structure: Nodes in WordNet can have multiple parents and thus the resulting classiﬁcation task would contain overlapping classes, making it inherently ambiguous.
Due to these issues, we cannot directly use WordNet to identify superclasses that correspond to a well-calibrated classiﬁcation task.
To illustrate this, we present some of the superclasses that Huh et al.
(2016) constructed by applying clustering algorithms directly to the WordNet hierarchy in
Appendix Table 7.
Even putting the issue of overlapping classes aside, a B REEDS task based on these superclasses would induce a very skewed subpopulation shift across classes—e.g., varying the types of “bread” is very different that doing the same for different “mammal” species.
To better align the WordNet hierarchy with the task of object recognition in general, and B REEDS benchmarks in particular, we manually modify it according to the following two principles: (i) nodes should be grouped together based on their visual characteristics rather than abstract relationships like functionality, and (ii) nodes of similar speciﬁcity should be at the same distance from the root, irrespective of how detailed their categorization within WordNet is.
Details of this procedure along with the resulting hierarchy are presented in Appendix A.4.
4. C REATING B REEDS TASKS
Once the modiﬁed version of the WordNet hierarchy is in place, B REEDS tasks can be created in an automated manner.
Speciﬁcally, we ﬁrst choose the desired granularity of the task by specifying the distance from the root (“entity”) and retrieving all superclasses at that distance in a top-down manner.
Each resulting superclass corresponds to a subtree of our hierarchy, with ImageNet classes as its leaves.
Note that these superclasses are roughly of the same speciﬁcity, due to our hierarchy restructuring process.
Then, we randomly sample a ﬁxed number of subclasses for each superclass to produce a balanced dataset (omitting superclasses with an insufﬁcient number of subclasses).
Finally, as described in Section 3, we randomly split these subclasses into the source and target domain.
For our analysis, we create four tasks (cf.
Table 2) based on different levels/parts of the hierarchy.
To illustrate what the corresponding subpopulation shifts look like, we present (random) image samples for a subset of the tasks in Figure 3.
Note that while we focus on the tasks in Table 2 in our study, our methodology readily enables us to create other variants of these tasks in an automated manner.
[Table]
[Table]
[Table]
of “subpopulations” (per superclass)
is ﬁxed to create balanced datasets.
We also construct specialized tasks by focusing on subtrees in the hierarchy, e.g., only living (L IVING -17) or non-living (N ON - LIVING -26) objects.
Datasets naming reﬂects the root of the subtree and the number of superclasses they contain.
B REEDS benchmarks beyond ImageNet. It is worth nothing that the methodology we described is not restricted to ImageNet and can be readily applied to other datasets as well.
The only requirement is that we have access to a semantic grouping of the dataset classes, which is the case for many popular vision datasets—e.g., CIFAR-100 (Krizhevsky, 2009),
Pascal-VOC (Everingham et al., 2010),
OpenImages (Kuznetsova et al., 2018), COCO-Stuff (Caesar et al., 2018).
Moreover, even when a class hierarchy is entirely absent, the needed semantic class grouping can be manually constructed with relatively little effort (proportional to the number of classes, not the number of datapoints).
More broadly, the methodology of utilizing existing dataset annotations to construct data subpopulations goes beyond image classiﬁcation tasks.
In particular, by splitting inputs into a source and target domain based on some attribute, we can measure how well models generalize along this axis.
Examples would include grouping by brand in Amazon reviews (McAuley et al., 2015), by location in Berkeley DeepDrive (Yu et al., 2020), and by facial attributes in CelebA (Liu et al., 2015).
4. C ALIBRATING B REEDS BENCHMARKS VIA HUMAN STUDIES
For a distribution shift benchmark to be meaningful, it is essential that the source and target domains capture the same high-level task—otherwise generalizing from one domain to the other would be impossible.
To ensure that this is the case for the B REEDS task, we assess how signiﬁcant the resulting distribution shifts are for human annotators (crowd-sourced via MTurk).
Annotator task.
To obtain meaningful performance estimates, it is crucial that annotators perform the task based only on the visual content of the images , without leveraging prior knowledge.
To achieve this, we design the following annotation task.
First, annotators are shown images from the source domain, grouped by superclass, without being aware of the superclass name (i.e., the grouping it corresponds to).
Then, they are presented with images from the target domain and are asked to assign each of them to one of the groups.
For simplicity, we present two random superclasses at a time, effectively simulating binary classiﬁcation.
Annotator accuracy can be measured directly as the fraction of images that they assign to the superclass to which they belong.
We perform this experiment for each of the B REEDS tasks constructed in Section 4.2.
For comparison, we repeat this experiment without subpopulation shift (test images are sampled from the source domain) and for the superclasses constructed by Huh et al.
(2016) using the WordNet hierarchy directly (cf.
Appendix A.6).
H u m a n : B r e e d s ( S )
H u m a n : B r e e d s ( T )
H u m a n : W o r d N e t ( S )
H u m a n : W o r d N e t ( T )
D e n s e N e t 1 2 1 : B r e e d s ( S )
D e n s e N e t 1 2 1 : B r e e d s ( T )
[Figure]
t o
[Figure]
cc u r [Figure]
c y ( % ) ENTITYH u m a n : B r e e d s ( S ) H u m a n : B r e e d s ( T ) H u m a n : W o r d N e t ( S )
H u m a n : W o r d N e t ( T )
D e n s e N e t 1 2 1 : B r e e d s ( S ) D e n s e N e t 1 2 1 : B r e e d s ( T ) ENTITYH u m a n : B r e e d s ( S ) H u m a n : B r e e d s ( T )
H u m a n : W o r d N e t ( S ) H u m a n : W o r d N e t ( T ) D e n s e N e t 1 2 1 : B r e e d s ( S ) D e n s e N e t 1 2 1 : B r e e d s ( T )
LIVINGH u m a n : B r e e d s ( S ) H u m a n : B r e e d s ( T ) H u m a n : W o r d N e t ( S ) H u m a n : W o r d N e t ( T )
D e n s e N e t 1 2 1 : B r e e d s ( S )
D e n s e N e t 1 2 1 : B r e e d s ( T )
NON-LIVING- indeed better calibrated for object recognition. (We discuss model performance in Section 5.)
Human performance.
We ﬁnd that, across all tasks, annotators perform well on unseen data from the source domain, as expected.
More importantly, annotators also appear to be quite robust to subpopulation shift, experiencing only a small accuracy drop between the source and target domains (cf.
Figure 5).
This indicates that the source and target domains are indeed perceptually similar for humans, making these benchmarks suitable for studying model robustness.
Finally, across all benchmarks, annotators perform better on B REEDS tasks, compared to their WordNet equivalents— even on source domain samples.
This indicates that our modiﬁed class hierarchy is indeed better aligned with the underlying visual recognition task.
E VALUATING M ODEL P ERFORMANCE UNDER S UBPOPULATION S HIFT
We can now use our suite of B REEDS tasks as a testbed for assessing model robustness to subpopulation shift as well as gauging the effectiveness of various train-time robustness interventions.
Speciﬁcs of the evaluation setup and additional experimental results are provided in Appendices A.7 and C.2.
5. S TANDARD TRAINING
We start by evaluating the performance of various model architectures trained in the standard fashion: empirical risk minimization (ERM) on the source distribution (cf.
Appendix A.7.1).
While models perform well on unseen inputs from the domain they are trained on, i [Figure] e., they achieve high source accuracy , their accuracy considerably drops under subpopulation shift—more than 30% in most cases (cf.
Figure 5).
At the same time, models that are more accurate on the source domain also appear to be more robust to subpopulation shift.
Speciﬁcally, the fraction of source accuracy that is preserved in the target domain typically increases with source accuracy.
(If this were not the case, i [Figure] e., the model accuracy dropped by a constant fraction under distribution shift, the target accuracy would match the baseline in Figure [Figure]
This indicates that, improvements in source accuracy do correlate with models generalizing better to variations in testing conditions
[Figure]
would correspond to a constant accuracy drop relative to AlexNet).
Moreover, the drop in model performance can be signiﬁcantly (but not fully) reduced by retraining the ﬁnal model layer with data from the target domain ( green ).
Models vs. Humans.
We compare the best performing model (DenseNet-121 in this case) to our previously obtained human baselines in Figure 4.
To allow for a fair comparison, model accuracy is measured on pairwise superclass classiﬁcation tasks (cf.
Appendix A.7).
We observe that models do exceedingly well on unseen samples from the source domain—signiﬁcantly outperforming annotators under our task setup.
At the same time, models also appear to be more brittle, performing worse than humans on the target domain of these binary B REEDS tasks, despite their higher source accuracy.
Adapting models to the target domain.
Finally, we focus on the intermediate data representations learned by these models, to assess how suitable they are for distinguishing classes in the target domain.
To evaluate this, we retrain the last (fully-connected) layer of models trained on the source domain with data from the target domain.
We ﬁnd that the target accuracy of these models increases signiﬁcantly after retraining, indicating that the learned representations indeed generalize to the target domain.
However, we cannot match the accuracy of models trained directly (end-to-end) on the target domain—see Figure 5—demonstrating that there is signiﬁcant room for improvement.
5. R OBUSTNESS INTERVENTIONS
We now turn our attention to existing methods for decreasing model sensitivity to speciﬁc synthetic perturbations.
Our goal is to assess if these methods enhance model robustness to subpopulation shift too.
Concretely, we consider the following families of interventions (cf.
Appendix A.7.3 for details): (i) adversarial training to enhance robustness to worst-case (cid:96) p -bounded perturbations (in our case (cid:96) 2 ) (Madry et al., 2018), (ii) training on a stylized version of ImageNet to encourage models to rely more on shape rather than texture (Geirhos et al., 2019), and (iii) training with random noise to make models robust
Note that these methods can be viewed as ways of imposing a prior on the features that the model relies on (Heinze-Deml & Meinshausen, 2017; Geirhos et al., 2019; Engstrom et al., 2019a).
That is, by rendering certain features ineffective during training (e.g., texture) they incentivize the model to utilize alternative ones (e.g., shape).
Since different feature families may manifest differently in the target domain, such interventions could signiﬁcantly impact model robustness to subpopulation shift.
[Figure]
To impact of ratio accuracy accuracy
This metric accounts for that train-time interventions can impact model accuracy on the source domain itself
By measuring relative performance, we are able to compare different training methods on an equal footing.
We ﬁnd that robustness interventions do have a small, yet non-trivial, impact on the robustness of a model to subpopulation shift—see Figure 6.
Speciﬁcally, for the case of adversarial training and erase noise, models often retain a larger fraction of their accuracy on the target domain compared to standard training, hence lying on the Pareto frontier of a robustness-accuracy trade-off.
In fact, for some of these interventions, the target accuracy is slightly higher than models obtained via standard training, even without adjusting for their lower source accuracy (raw accuracies are in
Appendix C.2.2).
Nonetheless, it is important to note that none of these methods offer signiﬁcant subpopulation robustness—relative accuracy is not improved by more than a few percentage points.
Adapting models to the target domain.
The impact of these interventions is more pronounced if we consider the accuracy of models after their last layer is retrained on the target domain (cf.
Appendix
Figure 21).
In particular, we ﬁnd that for adversarially robust models, retraining signiﬁcantly boosts accuracy on the target domain—e.g., for L IVING -
it is almost comparable to the initial source accuracy.
This suggests that the feature priors imposed by these interventions incentivize models to learn representations that generalize to other domains—in line with recent results of Utrera et al.
Salman et al.
Moreover, we observe that models trained on stylized inputs perform consistently worse, suggesting that texture might be an important feature for these tasks.
R ELATED W ORK
In Section 2, we surveyed prior work on distribution shift benchmarks.
Here, we discuss further the benchmarks most closely related to ours and defer discussing additional related work to Appendix B.
Our benchmarks can be viewed as an instance of domain generalization .
However, we focus on generalizing between different distributions of real-world images (photographs).
This is in contrast to typical domain generalization benchmarks that focus on generalizing between different stylistic representations, e.g., from cartoons to drawings.
Hence, the only comparable benchmark would be
VLCS (Ghifary et al., 2015), which is however signiﬁcantly smaller in scale and granularity than our benchmarks.
In a similar vein, datasets used in federated learning (Caldas et al., 2018) can be viewed as subpopulation shift benchmarks since the users present during training and testing might differ.
However, to the best of our knowledge, there has been no large-scale vision benchmark in this setting.
Hendrycks & Dietterich (2019), in Appendix G, also (manually) construct a classiﬁcation task over superclasses and use ImageNet classes outside of ILSVRC2012 (ImageNet-1k) to measure “subtype robustness”.
(Unfortunately, these classes are no longer publicly available (Yang et al., 2019).)
Compared to their work, we use a general methodology to create a broader suite of benchmarks.
Also, our analysis of architectures and robustness interventions is signiﬁcantly more extensive.
C ONCLUSION
In this work, we develop a methodology for constructing large-scale subpopulation shift benchmarks.
The motivation behind our B REEDS benchmarks is to test if models can generalize beyond the limited diversity of their training datasets—speciﬁcally, to novel data subpopulations.
A major advantage of our approach is its generality.
It can be applied to any dataset with a meaningful class structure— including tasks beyond classiﬁcation (e.g., object detection) and domains other than computer vision (e.g., natural language processing).
Moreover, the subpopulation shifts are induced in a manner that is both controlled and natural, without altering inputs synthetically or requiring new data.
By applying this approach to the ImageNet dataset, we construct a suite of benchmarks of varying difﬁculty, that we then use to assess model robustness and the efﬁcacy of various train-time interventions.
Further, we obtain human baselines for these tasks to both put model performance in context and validate that the corresponding subpopulation shifts do not signiﬁcantly affect humans.
Overall, our results indicate that existing models still have a long way to go before they can fully tackle B REEDS subpopulation shifts, even using current robustness interventions.
We thus believe that our methodology provides a useful tool for studying and improving model robustness to distribution shift—an increasingly pertinent topic for real-world deployments of machine learning models.
A CKNOWLEDGEMENTS
We thank Andrew Ilyas and Sam Park for helpful discussions.
Work supported in part by the NSF grants CCF-1553428, CNS-1815221, the Google PhD Fellowship, and the Microsoft Corporation.
This material is based upon work supported by the Defense Advanced
Research Projects Agency (DARPA) under Contract No. HR001120C0015.
Research was sponsored by the United States Air Force Research Laboratory and was accomplished under Cooperative Agreement Number FA8750-19-2-1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the ofﬁcial policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. A E XPERIMENTAL S ETUP A. D ATASET
We perform our analysis on the ILSVRC2012 dataset (Russakovsky et al., 2015).
This dataset contains a thousand classes from the ImageNet dataset (Deng et al., 2009) with an independently collected validation set.
The classes are part of the broader hierarchy, WordNet (Miller, 1995), through which words are organized based on their semantic meaning.
We use this hierarchy as a starting point of our investigation but modify it as described in Appendix A.5.
For all the B REEDS superclass classiﬁcation tasks, the train and validation sets are obtained by aggregating the train and validation sets of the descendant ImageNet classes (i.e., subpopulations).
Speciﬁcally, for a given subpopulation, the training and test splits from the original ImageNet dataset are used as is.
A. P IPELINE FORMALIZATION
Recall that our process for evaluating model robustness under subpopulation shift (cf.
Section 3) is as follows.
We present the pseudocode for this process in Algorithm 1.
Choose a level in the hierarchy and use it to deﬁne a set of superclasses by grouping the corresponding dataset classes together.
Note that the original dataset classes form the subpopulations of the superclasses.
For every superclass, select a (random) set of subpopulations (i e., classes in the original dataset) and use them to train the model to distinguish between superclasses (we call this the source domain).
For every superclass, use the remaining unseen subpopulations (i
e., classes in the original dataset) to test how well the model can distinguish between the superclasses (target domain).
if and len( ) ≥ N sub then y ← node.label subclasses ← random choice
) for ( ( subclasses ) do if i then else domain ← for ∈ inputs do ) return ( source, target ) function evaluateMethod( train , H , L , N sub , root ) : source , target ← createDatasets ( H , L , N sub , root ) model ← train ( source ) correct , total ← 0,for ( x, y ) ∈ target do correct += ( model ( x ) =
y ) total += 1 targetAccuracy ← correcttotal return targetAccuracy
A. W ORD N ET ISSUES
As discussed in Section 4, WordNet is a semantic rather than a visual hierarchy.
That is, object classes are arranged based on their meaning rather than their visual appearance.
Thus, using intermediate nodes for a visual object recognition task is not straightforward.
To illustrate this, we examine a sample superclass grouping created by Huh et al.
(2016) via automated bottom-up clustering in
Table 7.
[Table]
beacon, yurt
[Table]
picket fence
[Table]
barbershop
[Table]
fountain
[Table]
library
[Table]
cinema
[Table]
wall
[Table]
planetarium
[Table]
monastery
[Table]
mountain
[Table]
sliding door
[Table]
bakery
[Table]
megalith
[Table]
pedestal covering window shade
[Table]
vestment
[Table]
running shoe
[Table]
diaper
[Table]
sweatshirt
[Table]
breastplate
[Table]
shower curtain
[Table]
shoji
[Table]
miniskirt
[Table]
knee pad
[Table]
apron
[Table]
pajama
[Table]
military uniform
[Table]
theater curtain
[Table]
jersey
[Table]
football helmet
[Table]
book jacket
[Table]
bow tie, suit
[Table]
cloak
[Table]
espresso maker, maillot, iron, bath towel, lab coat, bow tie, washer, jersey, mask, wafﬂe iron, mortarboard, diaper, bolo tie, seat belt, cowboy hat, wig, knee pad, vacuum, microwave, abaya organism thunder snake, stingray, grasshopper, barracouta, Newfoundland,
Mexican hairless, Welsh springer spaniel, bluetick, golden retriever, keeshond, African chameleon, jacamar, water snake, Staffordshire bullterrier, Old English sheepdog, pelican, sea lion, wire-haired fox terrier, ﬂamingo, green mamba produce spaghetti squash, ﬁg, cardoon, mashed potato, pineapple, zucchini, broccoli, cauliﬂower, butternut squash, custard apple, pomegranate, strawberry, Granny Smith, lemon,
[Table]
First, we can notice that these superclasses have vastly different granularities.
For instance, “organism” contains the entire animal kingdom, hence being much broader than “produce”.
Moreover, “covering” is rather abstract class, and hence its subclasses often share little visual similarity (e.g., “window shade”, “pajama”).
Finally, due to the abstract nature of these superclasses, a large number of subclasses overlap—“covering” and “commodity” share 49 ImageNet descendants.
A. M ANUAL CALIBRATION
We manually modify the WordNet hierarchy according to the following two principles so as to make it better aligned for visual object recognition.
Nodes should be grouped together based on their visual characteristics, rather than abstract relationships like functionality—e
g., we eliminate nodes that do not convey visual information such as “covering”.
Nodes of similar speciﬁcity should be at the same distance from the root, irrespective of how detailed their categorization within WordNet is—for instance, we placed “dog” at the same level as “cat” and “ﬂower”, even though the “dog” sub-tree in WordNet is much larger.
Finally, we removed a number of ImageNet classes that did not naturally ﬁt into the hierarchy.
Concretely, we modiﬁed the WordNet hierarchy by applying the following operations:
• Collapse node : Delete a node from the hierarchy and add edges from each parent to each child.
Allows us to remove redundant or overly speciﬁc categorization while preserving the overall structure.
• Insert node above : Add a dummy parent to push a node further down the hierarchy.
Allows us to ensure that nodes of similar granularity are at the same level.
• Delete node : Remove a node and all of its edges.
Used to remove abstract nodes that do not reveal visual characteristics.
• Add edge : Connect a node to a parent.
Used to reassign the children of nodes deleted by the operation above.
We manually examined the hierarchy and implemented these actions in order to produce superclasses that are calibrated for classiﬁcation.
The resulting hierarchy contains nodes of comparable granularity at the same level.
Moreover, as a result of this process, each node ends up having a single parent and thus the resulting hierarchy is a tree.
The full hierarchy can be explored using the notebooks provided with the hierarchy in the Supplementary Material.
A. R ESULTING HIERARCHY
The parameters for constructing the B REEDS benchmarks (hierarchy level, number of subclasses, and tree root) are given in Table 2.
The resulting tasks—obtained by sampling disjoint ImageNet classes (i.e., subpopulations) for the source and target domain—are shown in Tables 8, 9, 10, and 11.
Recall that for each superclass we randomly sample a ﬁxed number of subclasses per superclass to ensure that the dataset is approximately balanced.
[Table]
catamaran, speedboat, ﬁreboat, yawl, airliner, container ship, liner, trimaran, space shuttle, aircraft carrier schooner, gondola, canoe, wreck, warplane, balloon, submarine, pirate, lifeboat, airship equipment volleyball, notebook, basketball, handheld computer, tripod, projector, barbell, monitor, croquet ball, balance beam cassette player, snorkel, horizontal bar, soccer ball, racket, baseball, joystick, microphone, tape player, reﬂex camera furniture wardrobe, toilet seat, ﬁle, mosquito net, four-poster, bassinet, chiffonier, folding chair, ﬁre screen, shoji studio couch, throne, crib, rocking chair, dining table, park bench, chest, window screen, medicine chest, barber chair instrument upright, padlock, lighter, steel drum, parking meter, cleaver, syringe, abacus, scale, corkscrew maraca, saltshaker, magnetic compass, accordion, digital clock, screw, can opener, odometer, organ, screwdriver man-made structure castle, bell cote, fountain, planetarium, trafﬁc light, breakwater, cliff dwelling, monastery, prison, water tower suspension bridge, worm fence, turnstile, tile roof, beacon, street sign, maze, chainlink fence, bakery, drilling platform wheeled vehicle snowplow, trailer truck, racer, shopping cart, unicycle, motor scooter, passenger car, minibus, jeep, recreational vehicle jinrikisha, golfcart, tow truck, ambulance, bullet train, ﬁre engine, horse cart, streetcar, tank, Model T produce broccoli, corn, orange, cucumber, spaghetti squash, butternut squash, acorn squash, cauliﬂower, bell pepper, ﬁg pomegranate, mushroom, strawberry, lemon, head cabbage, Granny Smith, hip, ear, banana, artichoke and target domains.
[Table]
Komodo dragon
[Table]
African
[Table]
wolf
[Table]
spi
[Table]
der
[Table]
dowitcher
[Table]
ﬁddler
[Table]
ﬂat-
[Table]
retriever, otterhound
[Table]
ShihTzu
[Table]
bull
[Table]
lacewing
[Table]
sulphur
[Table]
long-horned beetle, leafhopper
[Table]
dung beetle
[Table]
hog
[Table]
hippopotamus
[Table]
hartebeest
[Table]
warthog
[Table]
howler
[Table]
Madagascar
[Table]
lionﬁsh
[Table]
rock beauty
[Table]
breakwater
[Table]
picket fence
[Table]
turnstile
[Table]
bannister chainlink
[Table]
stone wall
[Table]
worm fence building bookshop, castle
[Table]
mosque
[Table]
butcher shop
[Table]
beacon electronic equipment printer, pay-phone, microphone, computer keyboard modem
[Table]
cassette player
[Table]
dial telephone footwear clog, Loafer, maillot, running shoe sandal, knee pad, cowboy boot, Christmas stocking garment academic gown, apron, miniskirt, fur coat jean, vestment, sarong, swimming trunks headdress pickelhaube, hair slide, shower cap, bonnet bathing cap, cowboy hat, bearskin, crash helmet home appliance washer, microwave, Crock Pot, vacuum toaster, espresso maker, space heater, dishwasher kitchen utensil measuring cup, cleaver, coffeepot, spatula frying pan, cocktail shaker, tray, caldron measuring instrument digital watch, analog clock, parking meter, magnetic compass barometer, wall clock, hourglass, digital clock motor vehicle limousine, school bus, moped, convertible trailer truck, beach wagon, police van, garbage truck musical instrument
French horn, maraca, grand piano, upright acoustic guitar, organ, electric guitar, violin neckwear feather boa, neck brace, bib, Windsor tie necklace, stole, bow tie, bolo tie
[Table]
[Table]
[Table]
A. A NNOTATOR TASK
As described in Section 4.3, the goal of our human studies is to understand whether humans can classify images into superclasses even without knowing the semantic grouping.
Thus, the task involved showing annotators two groups of images, each sampled from the source domain of a random superclass.
Then, annotators were shown a new set of images from the target domain (or the source domain in the case of control) and were asked to assign each of them into one of the two groups.
A screenshot of an (random) instance of our annotator task is shown in Figure 12.
Each task contained 20 images from the source domain of each superclass and 12 images for annotators to classify (the images where rescaled and center-cropped to size 224 × 224 to match the input size use for model predictions).
The two superclasses were randomly permuted at load time.
To ensure good concentration of our accuracy estimates, for every superclass, we performed binary classiﬁcation tasks w.r.t.
3 other (randomly chosen) superclasses.
Further, we used 3 annotators per task and annotators were compensated $0.15 per task.
Comparing with the original hierarchy.
In order to compare our superclasses with those obtained by Huh et al. (2016) via WordNet clustering, 4 we need to deﬁne a correspondence between them.
To do so, for each of our tasks, we selected the clustering (either top-down or bottom-up) that had the closest number of superclasses.
Following the terminology from that work, this mapping is: E NTITY -
→ D OWN U P -36, E NTITY -
→ U P D OWN -127, L IVING -
→ D OWN U P - (restricted to “living” nodes), and N ON - LIVING -
→ D OWN U P -345 (restricted to “non-living” nodes).
A. E VALUATING MODEL PERFORMANCE A.7. M ODEL ARCHITECTURES AND TRAINING
The model architectures used in our analysis are in Table 13 for which we used standard implementations from the PyTorch library ( https://pytorch.org/docs/stable/torchvision/ models.html ).
For training, we use a batch size of 128, weight decay of 10 − 4 , and learning rates listed in Table 13.
Models were trained until convergence.
On E NTITY -13 and E NTITY -30, this required a total of 300 epochs, with 10-fold drops in learning rate every 100 epochs, while on
L IVING -17and N ON - LIVING -26, models a total of 450 epochs, with 10-fold learning rate drops every 150 epochs.
For adapting models, we retrained the last (fully-connected) layer on the train split of the target domain, starting from the parameters of the source-trained model.
We trained that layer using SGD with a batch size of 128 for 40,000 steps and chose the best learning rate out of [0 . 01 , 0 . 1 , 0 . 25 , 0 . 5 , 1 . 0 , 2 . 0 , 3 . 0 , 5 . 0 , 7 . 0 , 8 . 0 , 10 . 0 , 11 . 0 , 12 . 0] , based on test accuracy.
[Table]
A
[Table]
M ODEL PAIRWISE ACCURACY
In order to make a fair comparison between the performance of models and human annotators on the
B REEDS tasks, we evaluate model accuracy on pairs of superclasses.
On images from that pair, we determine the model prediction to be the superclass for which the model’s predicted probability is higher.
A prediction is deemed correct if it matches the superclass label for the image.
Repeating this process over random pairs of superclasses allows us to estimate model accuracy on the average-case binary classiﬁcation task.
A.7. R OBUSTNESS INTERVENTIONS
For model training, we use the hyperparameters provided in Appendix A.7.1.
Additional interventionspeciﬁc hyperparameters are listed in Appendix Table 14.
Due to computational constraints, we trained a restricted set of model architectures with robustness interventions—ResNet-18 and ResNet-50 for adversarial training, and ResNet-18 and ResNet-34 for all others.
Adversarial training was implemented using the robustness library, 5 while random erasing using the PyTorch transforms .
[Table]
B A DDITIONAL R ELATED W ORK
In Section 2, we provide an overview of prior work that is focused on evaluating model robustness to distribution shift.
In Section 6, we discuss existing benchmarks that are most similar to our work.
Here, we discuss other research direction related to model robustness and generalization.
Distributional robustness.
Distribution shifts that are small with respect to some f -divergence have been studied in prior theoretical work (Ben-Tal et al., 2013; Duchi et al., 2016; Esfahani & Kuhn, 2018; Namkoong & Duchi, 2016).
However, this notion of robustness is typically too pessimistic to capture realistic data variations (Hu et al., 2018).
Distributional robustness has also been connected to causality (Meinshausen, 2018): here, the typical approach is to inject spurious correlations into the dataset, and assess to what extent models rely on them for their predictions (Heinze-Deml &
Meinshausen, 2017; Arjovsky et al., 2019; Sagawa et al., 2020).
Domain adaptation and transfer learning.
The goal here is to adapt models to the target domain with relatively few samples from it (Ben-David et al., 2007;
Saenko et al., 2010; Ganin & Lempitsky, 2015; Courty et al., 2016;
Gong et al., 2016; Donahue et al., 2014; Sharif Razavian et al., 2014).
In domain adaptation, the task is the same in both domains, while in transfer learning, the task itself could vary.
In a similar vein, the ﬁeld of domain generalization aims to generalize to samples from a different domain (e.g., from ClipArt to photos) by training on a number of explicitly annotated domains (Muandet et al., 2013; Li et al., 2017; Peng et al., 2019).
Zero-shot learning.
Work in this domain focuses on learning to recognize previously unseen classes (Lampert et al., 2009; Xian et al., 2017), typically described via a semantic embedding (Lampert et al., 2009; Mikolov et al.,
This differs from our setup, where the focus is on generalization to unseen subpopulations for set of
C A DDITIONAL E XPERIMENTAL R ESULTS C. H UMAN B ASELINES FOR B REEDS T ASKS
In Section 4.3, we evaluate human performance on binary versions of our B REEDS tasks.
Appendix
Figures 15a and 15b show the distribution of annotator accuracy over different pairs of superclasses for test data sampled from the source and target domains respectively.
[Figure]
[Figure]
[Figure]
C. M ODEL E VALUATION
In Figures 16- 18, we visualize model performance over B REEDS superclasses for different model architectures.
We observe in general that models perform fairly uniformly over classes when the test data is drawn from the source domain.
This indicates that the tasks are well-calibrated—the various superclasses are of comparable difﬁculty.
At the same time, we see that model robustness to subpopulation shift, i.e., drop in accuracy on the target domain, varies widely over superclasses.
This could be either due to some superclasses being broader by construction or due to models being more sensitive to subpopulation shift for some classes.
[Table]
r e p t il e a r t h r o p o d
m a m m a l a cc e ss
o r y c
r a f t e q u i p m e n t f u r n i t
u r e i n s t r u
m e n t m a n - m a d e s t r u c t u r e w h ee l e d v e h
i c
l e p r o d u c e
[Table]
[Figure]
[Table]
d c r u
[Table]
n [Table] r
[Table]
m
[Table]
u il d
[Table]
n g e l e c t r o n i c
e q u i p m e n t f oo t w e a r
[Table]
a r m e n t h e a dd r e ss h o m e a pp li a n c e k i t c h e n u t e n s il m e a s u r i n g i n s t r u m e n t m o t
o r v e h
i c
l e m u s i c a l i n s t r u m e n t n e c k w e a r s p o r t s e q u i p m e n t t a b l e w a r e t oo l v e ss e l d
i s h v e g e t a b l e f r u i t
A cc u r a c y ( % )
ENTITY-30 | Model=alexnet
Source
Target s a l a m a n d e r t u r t l e li z a r d s n a k e s p i
d
e r g r o u s e p a rr o t c
r a b d o g w o l f f o x d o m e s t
i c c a t b e a r b ee t l e b u tt e r f l y a p e m o n k e y
A cc u r a c y ( % )
LIVING-17 | Model=alexnet
Source
Target b a g b a ll b o a t b o d y a r m o r b
o
tt l e b u s c a r c h a i r
c o a t d i g i t a l c o m p
u t e r d w e lli n g f e n c e h a t k
e y b o a r d i n s t r u m e n t m e r c a n t il e e s t a b li s h m e n t o u t b u il d i n g p e r c u ss i o n i n s t r u m e n t p
o t r oo f s h
i p s k
i r t s t r i n g e d i n s t r u m e n t t i m e p i e c e t r u c k w i n
[Figure]
g a r m e n t b i r d r e p t il e a r t h r o p o d m a m m a l a cc e ss o r y c
r a f t e q u i p m e n t f u r n i t
u r e i n s t r u
m e n t m a n - m a d e s t r u c t u r e w h ee l e d v e h
i c
l e p r o d u c e
[Table]
[Figure]
[Table]
t
[Table]
m
[Table]
n s t r u m
[Table]
n t m o t
[Table]
r v e h
[Table]
l e m
[Table]
l
[Table]
t
[Table]
m
[Table]
n t n e
[Table]
[Figure]
[Table]
[Figure]
Target s a l a m a n d e r t u r t l e li z a r d s n a k e s p i
d
e r g r o u s e p a rr o t c
r a b d o g w o l f f o x d o m e s t
i c c a t b e a r b ee t l e b u tt e r f l y a p e m o n k e y
[Figure]
b a g b a ll b o a t b o d y a r m o r b
o
tt l e b u s c a r c h a i r
c o a t d i g i t a l c o m p
u t e r d w e lli n g f e n c e h a t k
e y b o a r d i n s t r u m e n t m e r c a n t il e e s t a b li s h m e n t o u t b u il d i n g p e r c u ss i o n i n s t r u m e n t p
o t r oo f s h
i p s k
i r t s t r i n g e d i n s t r u m e n t t i m e p i e c e t r u c k w i n d i n s t r u m e n t s q u
a s h
[Figure]
g a r m e n t b i r d r e p t il e a r t h r o p o d m a m m a l a cc e ss o r y c
r a f t e q u i p m e n t f u r n i t
u r e i n s t r u
m e n t m a n - m a d e s t r u c t u r e w h ee l e d v e h
i c
l e p r o d u c e
[Table]
[Figure]
[Table]
[Figure]
[Table]
e
[Table]
u
[Table]
t
[Table]
k
[Table]
n
[Table]
n
[Table]
m e [Table] s u r
[Table]
n
[Table]
i n s t r u m
[Table]
n t m o t o r v e h i c l e m
[Table]
c
[Table]
l i n
[Table]
t r u m e n t n e
[Table]
s p
[Table]
[Figure]
[Table]
[Figure]
s a l a m a n d e r t u r t l e li z a r d s n a k e s p i d e r g r o u s e p a rr o t c
r a b d o g w o l f f o x d o m e s t
i c c a t b e a r b ee t l e b u tt e r f l y a p e m o n k e y
[Figure]
b a g b a ll b o a t b o d y a r m o r b
o
tt l e b u s c a r c h a i r
c o a t d i g i t a l c o m p
u t e r d w e lli n g f e n c e h a t k
e y b o a r d i n s t r u m e n t m e r c a n t il e e s t a b li s h m e n t o u t b u il d i n g p e r c u ss i o n i n s t r u m e n t p
o t r oo f s h
i p s k
i r t s t r i n g e d i n s t r u m e n t t i m e p i e c e t r u c k w i n d i n s t r u m e n t s q u
a s h
[Figure]
C.2. E FFECT OF DIFFERENT SPLITS
As described in Section 3, to create B REEDS tasks, we ﬁrst identify a set of relevant superclasses (at the chosen depth in the hierarchy), and then partition their subpopulations between the source and target domains.
For all the tasks listed in Table 2, the superclasses are balanced—each of them comprise the same number of subpopulations.
To ensure this is the case, the desired number of subpopulations is chosen among all superclass subpopulations at random.
These subpopulations are then randomly split between the source and target domains.
Instead of randomly partitioning subpopultions (of a given superclass) between the two domains, we could instead craft partitions to be more/less adversarial as illustrated in Figure 19.
Speciﬁcally, we could control how similar the subpopulations in the target domain are to those in the source domain.
For instance, a split would be less adversarial ( good ) if subpopulations in the source and target domain share a common parent.
On the other hand, we could make a split more adversarial ( bad ) by ensuring a greater degree of separation (in terms of distance in the hierarchy) between the source and target domain subpopulations.
[Figure]
We now evaluate model performance under such variations in the nature of the splits themselves—see
Figure 20.
As expected, models perform comparably well on test data from the source domain, independent of the how the subpopulations are partitioned into the two domains.
However, model robustness to subpopulation shift varies considerably based on the nature of the split—it is lowest for the most adversarially chosen split.
Finally, we observe that retraining the linear layer on data from the target domain recovers a considerable fraction of the accuracy drop in all cases—indicating that even for the more adversarial splits, models do learn features that transfer well to unknown subpopulations.
C.2. R OBUSTNESS I NTERVENTIONS
In Tables 22 and 23, we present the raw accuracies of models trained using various train-time robustness interventions.
[Figure]
[Figure] [Table]
N ON - LIVING -
[Table] [Table]
evaluated in terms of