Original: M ETA S HIFT : A D ATASET OF D ATASETS FOR E VALUATING C ONTEXTUAL D ISTRIBUTION S HIFTS AND T RAINING C ONFLICTS
Revised: M ETA S HIFT : A D ATASET OF D ATASETS FOR E VALUATING C ONTEXTUAL D ISTRIBUTION S HIFTS AND T RAINING C ONFLICTS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Understanding the performance of machine learning model across diverse data distributions is critically important for reliable applications.
Revised: Understanding the performance of machine learning model across diverse data distributions is critically important for reliable applications.

Original: Motivated by this, there is a growing focus on curating benchmark datasets that capture distribution shifts.
Revised: Motivated by this, there is a growing focus on curating benchmark datasets that capture distribution shifts.

Original: While valuable, the existing benchmarks are limited in that many of them only contain a small number of shifts and they lack systematic annotation about what is different across different shifts.
Revised: While valuable, the existing benchmarks are limited in that many of them only contain a small number of shifts and they lack systematic annotation about what is different across different shifts.

Original: We present MetaShift—a collection of 12,868 sets of natural images across 410 classes—to address this challenge.
Revised: We present MetaShift—a collection of 12,868 sets of natural images across 410 classes—to address this challenge.

Original: We leverage the natural heterogeneity of Visual Genome and its annotations to construct MetaShift.
Revised: We leverage the natural heterogeneity of Visual Genome and its annotations to construct MetaShift.

Original: The key construction idea is to cluster images using its metadata, which provides context for each image (e.g. cats with cars or cats in bathroom ) that represent distinct data distributions.
Revised: The key construction idea is to cluster images using its metadata, which provides context for each image (e.g. cats with cars or cats in bathroom ) that represent distinct data distributions.

Original: MetaShift has two important beneﬁts: ﬁrst it contains orders of magnitude more natural data shifts than previously available.
Revised: MetaShift has two important beneﬁts: ﬁrst it contains orders of magnitude more natural data shifts than previously available.

Original: Second, it provides explicit explanations of what is unique about each of its data sets and a distance score that measures the amount of distribution shift between any two of its data sets.
Revised: Second, it provides explicit explanations of what is unique about each of its data sets and a distance score that measures the amount of distribution shift between any two of its data sets.

Original: We demonstrate the utility of MetaShift in benchmarking several recent proposals for training models to be robust to data shifts.
Revised: We demonstrate the utility of MetaShift in benchmarking several recent proposals for training models to be robust to data shifts.

Original: We ﬁnd that the simple empirical risk minimization performs the best when shifts are moderate and no method had a systematic advantage for large shifts.
Revised: We ﬁnd that the simple empirical risk minimization performs the best when shifts are moderate and no method had a systematic advantage for large shifts.

Original: We also show how MetaShift can help to visualize conﬂicts between data subsets during model training.
Revised: We also show how MetaShift can help to visualize conﬂicts between data subsets during model training.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: A major challenge in machine learning (ML) is that a model can have very different performances and behaviors when it’s applied to different types of natural data.
Revised: A major challenge in machine learning (ML) is that a model can have very different performances and behaviors when it’s applied to different types of natural data.

Original: For example, if the user data have different contexts compared to the model’s training data (e.g. users have outdoor dog photos and the model’s training was mostly on indoor images), then the model’s accuracy can greatly suffer.
Revised: For example, if the user data have different contexts compared to the model’s training data (e.g. users have outdoor dog photos and the model’s training was mostly on indoor images), then the model’s accuracy can greatly suffer.

Original: A model can have disparate performances even within different subsets within its training and evaluation data.
Revised: A model can have disparate performances even within different subsets within its training and evaluation data.

Original: In order to assess the reliability and fairness of a model, we therefore need to evaluate its performance and training behavior across heterogeneous types of data.
Revised: In order to assess the reliability and fairness of a model, we therefore need to evaluate its performance and training behavior across heterogeneous types of data.

Original: However, the lack of well-structured datasets representing diverse data distributions makes systematic evaluation difﬁcult.
Revised: However, the lack of well-structured datasets representing diverse data distributions makes systematic evaluation difﬁcult.

Original: In this paper, we present MetaShift to tackle this challenge.
Revised: In this paper, we present MetaShift to tackle this challenge.

Original: MetaShift is a collection of 12,868 sets of natural images from 410 classes.
Revised: MetaShift is a collection of 12,868 sets of natural images from 410 classes.

Original: Each set corresponds to images in a similar context and represents a coherent real-world data distribution, as shown in Figure 1.
Revised: Each set corresponds to images in a similar context and represents a coherent real-world data distribution, as shown in Figure 1.

Original: The construction of MetaShift is different from and complementary to other efforts to curate benchmarks for data shifts by pulling together data across different experiments or sources.
Revised: The construction of MetaShift is different from and complementary to other efforts to curate benchmarks for data shifts by pulling together data across different experiments or sources.

Original: MetaShift leverages heterogeneity within the large sets of images from the Visual Genome project by clustering the images using metadata that describes the context of each image.
Revised: MetaShift leverages heterogeneity within the large sets of images from the Visual Genome project by clustering the images using metadata that describes the context of each image.

Original: The advantage of this approach is that MetaShift contains many more coherent sets of data compared to other benchmarks.
Revised: The advantage of this approach is that MetaShift contains many more coherent sets of data compared to other benchmarks.

Original: Importantly, we have explicit annotations of what makes each subset unique (e.g. cats with cars or dogs next to a bench ) as well as a score that measures the distance between any two subsets, which is not available in previous benchmarks of natural data.
Revised: Importantly, we have explicit annotations of what makes each subset unique (e.g. cats with cars or dogs next to a bench ) as well as a score that measures the distance between any two subsets, which is not available in previous benchmarks of natural data.

Original: We demonstrate the utility of MetaShift in two applications.
Revised: We demonstrate the utility of MetaShift in two applications.

Original: First, MetaShift supports evaluation on both domain generalization and subpopulation shifts settings.
Revised: First, MetaShift supports evaluation on both domain generalization and subpopulation shifts settings.

Original: Using the score between subsets
Revised: Using the score between subsets

Original: [Figure]
Revised: [Figure]

Original: provided by MetaShift, we study ML models’ behavior under different carefully modulated amounts of distribution shift.
Revised: provided by MetaShift, we study ML models’ behavior under different carefully modulated amounts of distribution shift.

Original: Second, MetaShift can also shed light on the training dynamics of ML models.
Revised: Second, MetaShift can also shed light on the training dynamics of ML models.

Original: Since we have the subset membership information for each training datum, we could attribute the contribution of each gradient step back to the training subsets, and then analyze how different data subsets provide conﬂicting training signals.
Revised: Since we have the subset membership information for each training datum, we could attribute the contribution of each gradient step back to the training subsets, and then analyze how different data subsets provide conﬂicting training signals.

Original: Our contributions: We present MetaShift as an important resource for studying the behavior of
Revised: Our contributions: We present MetaShift as an important resource for studying the behavior of

Original: ML algorithms and training dynamics across data with heterogeneous contexts.
Revised: ML algorithms and training dynamics across data with heterogeneous contexts.

Original: Our methodology for constructing MetaShift can also be applied to other domains where metadata is available.
Revised: Our methodology for constructing MetaShift can also be applied to other domains where metadata is available.

Original: We empirically evaluate the performance of different robust learning algorithms, showing that ERM performs well for modest shifts while no method is the clear winner for larger shifts.
Revised: We empirically evaluate the performance of different robust learning algorithms, showing that ERM performs well for modest shifts while no method is the clear winner for larger shifts.

Original: This ﬁnding suggests that domain generalization is an important and challenging task and that there’s still a lot of room for new methods.
Revised: This ﬁnding suggests that domain generalization is an important and challenging task and that there’s still a lot of room for new methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Existing Benchmarks for Distribution Shift
Revised: Existing Benchmarks for Distribution Shift

Original: Early benchmarks focus on distribution shifts induced by synthetic pixel transformations.
Revised: Early benchmarks focus on distribution shifts induced by synthetic pixel transformations.

Original: Examples include rotated and translated versions of MNIST and CIFAR (Worrall et al., 2017); surface variations such as texture, color, and corruptions like blur in Colored MNIST (Gulrajani & Lopez-Paz, 2020), ImageNet-C (Hendrycks & Dietterich, 2019).
Revised: Examples include rotated and translated versions of MNIST and CIFAR (Worrall et al., 2017); surface variations such as texture, color, and corruptions like blur in Colored MNIST (Gulrajani & Lopez-Paz, 2020), ImageNet-C (Hendrycks & Dietterich, 2019).

Original: Although the synthetic pixel transformations are well-deﬁned, they generally do not represent realistic shifts in real-world images that we capture in MetaShift.
Revised: Although the synthetic pixel transformations are well-deﬁned, they generally do not represent realistic shifts in real-world images that we capture in MetaShift.

Original: Other benchmarks do not rely on transformations but instead pull together data across different experiments or sources.
Revised: Other benchmarks do not rely on transformations but instead pull together data across different experiments or sources.

Original: Ofﬁce-31 (Saenko et al., 2010) and Ofﬁce-home (Venkateswara et al., 2017) contain images collected from different domains like Amazon, clipart.
Revised: Ofﬁce-31 (Saenko et al., 2010) and Ofﬁce-home (Venkateswara et al., 2017) contain images collected from different domains like Amazon, clipart.

Original: These benchmarks typically have only a handful of data distributions.
Revised: These benchmarks typically have only a handful of data distributions.

Original: The benchmarks collected in WILDS (Koh et al., 2020) combine data from different sources (e.g., medical images from different hospitals, animal images from different camera traps).
Revised: The benchmarks collected in WILDS (Koh et al., 2020) combine data from different sources (e.g., medical images from different hospitals, animal images from different camera traps).

Original: Similarly, some meta-learning benchmarks (Triantaﬁllou et al., 2019;
Revised: Similarly, some meta-learning benchmarks (Triantaﬁllou et al., 2019;

Original: Guo et al., 2020) focuses on dataset-level shift by combining different existing datasets like ImageNet,
Revised: Guo et al., 2020) focuses on dataset-level shift by combining different existing datasets like ImageNet,

Original: Omniglot.
Revised: Omniglot.

Original: While valuable, they lack systematic annotation about what is different across different shifts.
Revised: While valuable, they lack systematic annotation about what is different across different shifts.

Original: Santurkar et al.
Revised: Santurkar et al.

Original: Ren et al.
Revised: Ren et al.

Original: (2018) utilize the hierarchical structure of ImageNet to construct training and test sets with disjoint subclasses.
Revised: (2018) utilize the hierarchical structure of ImageNet to construct training and test sets with disjoint subclasses.

Original: For example, the “tableware” class uses “beer glass” and “plate” for training and testing respectively.
Revised: For example, the “tableware” class uses “beer glass” and “plate” for training and testing respectively.

Original: Different from their work, we study the shifts where the core object remains the same while the context changes.
Revised: Different from their work, we study the shifts where the core object remains the same while the context changes.

Original: NICO (He et al., 2020) query different manually-curated phrases on search engines to collect images of objects in different contexts.
Revised: NICO (He et al., 2020) query different manually-curated phrases on search engines to collect images of objects in different contexts.

Original: A key difference is the scale of MetaShift: NICO contains 190 sets of images acrossclasses while MetaShift has 12,868 sets of natural images across 410 classes.
Revised: A key difference is the scale of MetaShift: NICO contains 190 sets of images acrossclasses while MetaShift has 12,868 sets of natural images across 410 classes.

Original: To sum up the advantages of our MetaShift are:
Revised: To sum up the advantages of our MetaShift are:

Original: • Existing benchmark datasets for distribution shifts typically have only a handful of data distributions.
Revised: • Existing benchmark datasets for distribution shifts typically have only a handful of data distributions.

Original: In contrast, our MetaShift has over 12,868 data distributions, thus enabling a much more comprehensive assessment of distribution shifts.
Revised: In contrast, our MetaShift has over 12,868 data distributions, thus enabling a much more comprehensive assessment of distribution shifts.

Original: • Distribution shifts in existing benchmarks are not annotated (i.e. we don’t know what drives the shift) and are not well-controlled (i.e. we can’t easily adjust the magnitude of the shift).
Revised: • Distribution shifts in existing benchmarks are not annotated (i.e. we don’t know what drives the shift) and are not well-controlled (i.e. we can’t easily adjust the magnitude of the shift).

Original: The MetaShift provides explicit annotations of the differences between any two sub-datasets, and it quantiﬁes the distance of the shift.
Revised: The MetaShift provides explicit annotations of the differences between any two sub-datasets, and it quantiﬁes the distance of the shift.

Original: Evaluating conﬂicts on training data Related to our analysis in evaluating training conﬂicts, Data
Revised: Evaluating conﬂicts on training data Related to our analysis in evaluating training conﬂicts, Data

Original: Shapley Ghorbani & Zou (2019) provides a mathematical framework for quantifying the contribution of each training datum.
Revised: Shapley Ghorbani & Zou (2019) provides a mathematical framework for quantifying the contribution of each training datum.

Original: The framework enables users to identify data that are important to collect in the future.
Revised: The framework enables users to identify data that are important to collect in the future.

Original: Data Cartography Swayamdipta et al.
Revised: Data Cartography Swayamdipta et al.

Original: (2020) leverages the changes of a model’s conﬁdence on individual training datum during training to discover hard-to-learn training data points.
Revised: (2020) leverages the changes of a model’s conﬁdence on individual training datum during training to discover hard-to-learn training data points.

Original: Complementary to their work, our analysis sheds light on why a certain portion of the training data are hard-to-learn—because different subsets are providing conﬂicting training signals.
Revised: Complementary to their work, our analysis sheds light on why a certain portion of the training data are hard-to-learn—because different subsets are providing conﬂicting training signals.

Original: classes
Revised: classes

Original: Avg.
Revised: Avg.

Original: 31.4 subsets/class,
Revised: 31.4 subsets/class,

Original: Avg.
Revised: Avg.

Original: 200.4 images/subset 1,853 distinct contexts 1, object presence w/ clock bus w/ bridge
Revised: 200.4 images/subset 1,853 distinct contexts 1, object presence w/ clock bus w/ bridge

Original: w/ sign w/ water w/ carriage w/ policeman w/ fence w/ dog w/ candle w/ toothbrush w/ cat w/ clock horse cup clock, bridge, sign, water, carriage, policeman, ... indoor, outdoor, bedroom, ocean, ... generalcontexts red, plastic, triangular, ... object attributes
Revised: w/ sign w/ water w/ carriage w/ policeman w/ fence w/ dog w/ candle w/ toothbrush w/ cat w/ clock horse cup clock, bridge, sign, water, carriage, policeman, ... indoor, outdoor, bedroom, ocean, ... generalcontexts red, plastic, triangular, ... object attributes

Original: [Figure]
Revised: [Figure]

Original: bus
Revised: bus

Original:  T HE M ETA S HIFT C ONSTRUCTION M ETHODOLOGY
Revised:  T HE M ETA S HIFT C ONSTRUCTION M ETHODOLOGY

Original: What is MetaShift?
Revised: What is MetaShift?

Original: The MetaShift is a collection of subsets of data together with an annotation graph that explains the similarity/distance between two subsets (edge weight) as well as what is unique about each subset (node metadata).
Revised: The MetaShift is a collection of subsets of data together with an annotation graph that explains the similarity/distance between two subsets (edge weight) as well as what is unique about each subset (node metadata).

Original: For each class, say “cat”, we have many subsets of cats, and we can think of each subset as a node in the graph, as shown in Figure 2.
Revised: For each class, say “cat”, we have many subsets of cats, and we can think of each subset as a node in the graph, as shown in Figure 2.

Original: Each subset corresponds to “cat” in a different context: e.g. “cat with sink” or “cat with fence”.
Revised: Each subset corresponds to “cat” in a different context: e.g. “cat with sink” or “cat with fence”.

Original: The context of each subset is the node metadata.
Revised: The context of each subset is the node metadata.

Original: The “cat with sink” subset is more similar to “cat with faucet” subset because there are many images that contain both sink and faucet.
Revised: The “cat with sink” subset is more similar to “cat with faucet” subset because there are many images that contain both sink and faucet.

Original: This similarity is the weight of the edge; a higher weight means the contexts of the two nodes tend to co-occur in the same data.
Revised: This similarity is the weight of the edge; a higher weight means the contexts of the two nodes tend to co-occur in the same data.

Original: How can we use MetaShift?
Revised: How can we use MetaShift?

Original: It is a ﬂexible framework to generate a large number of real-world distribution shifts that are well-annotated and controlled.
Revised: It is a ﬂexible framework to generate a large number of real-world distribution shifts that are well-annotated and controlled.

Original: For each class of interest, say “cats”, we can use the meta-graph of cats to identify a collection of cats nodes for training (e.g. cats with bathroom-related contexts) and a collection of cats nodes for out-of-domain evaluation (e.g. cats in outdoor contexts).
Revised: For each class of interest, say “cats”, we can use the meta-graph of cats to identify a collection of cats nodes for training (e.g. cats with bathroom-related contexts) and a collection of cats nodes for out-of-domain evaluation (e.g. cats in outdoor contexts).

Original: Our meta-graph tells us exactly what is different between the train and test domains (e.g. bathroom vs. outdoor contexts), and it also speciﬁes the similarity between the two contexts via graph distance.
Revised: Our meta-graph tells us exactly what is different between the train and test domains (e.g. bathroom vs. outdoor contexts), and it also speciﬁes the similarity between the two contexts via graph distance.

Original: That makes it easy to carefully modulate the amount of distribution shift.
Revised: That makes it easy to carefully modulate the amount of distribution shift.

Original: For example, if we use cats-in-living-room as the test set, then this is a smaller distribution shift.
Revised: For example, if we use cats-in-living-room as the test set, then this is a smaller distribution shift.

Original: Base Dataset: Visual Genome We leverage the natural heterogeneity of Visual Genome and its annotations to construct MetaShift.
Revised: Base Dataset: Visual Genome We leverage the natural heterogeneity of Visual Genome and its annotations to construct MetaShift.

Original: Visual Genome contains over 100k images across 1,702 object classes.
Revised: Visual Genome contains over 100k images across 1,702 object classes.

Original: For each image, Visual Genome annotates the class labels of all objects that occur in the image.
Revised: For each image, Visual Genome annotates the class labels of all objects that occur in the image.

Original: Formally, for each image x
Revised: Formally, for each image x

Original: ( i ) , we have a list of meta-data tags m ( i ) =
Revised: ( i ) , we have a list of meta-data tags m ( i ) =

Original: { t ( i ) 1 , t ( i ) 2 , . . .
Revised: { t ( i ) 1 , t ( i ) 2 , . . .

Original: , t ( i ) n m } , each indicating the presence of an object in the context.
Revised: , t ( i ) n m } , each indicating the presence of an object in the context.

Original: We denote the vocabulary of the meta-data tags as M = { m 0 , . . .
Revised: We denote the vocabulary of the meta-data tags as M = { m 0 , . . .

Original: , m | M | } .
Revised: , m | M | } .

Original: MetaShift is constructed in a class-by-class basis: For each class, say “cat”, we pull out all cat images and proceeds with the following steps.
Revised: MetaShift is constructed in a class-by-class basis: For each class, say “cat”, we pull out all cat images and proceeds with the following steps.

Original: Step 1: Generate Candidate Subsets
Revised: Step 1: Generate Candidate Subsets

Original: We ﬁrst generate candidate subsets by enumerating all possible meta-data tags.
Revised: We ﬁrst generate candidate subsets by enumerating all possible meta-data tags.

Original: We construct | M | candidate subsets where the i th subset contains all images of the class of interest (i.e., “cat”) that has a meta-tag m i .
Revised: We construct | M | candidate subsets where the i th subset contains all images of the class of interest (i.e., “cat”) that has a meta-tag m i .

Original: We then remove subsets whose sizes are less than a threshold (e.g., 25).
Revised: We then remove subsets whose sizes are less than a threshold (e.g., 25).

Original: Step 2:
Revised: Step 2:

Original: Construct Meta-graphs Since the meta-data are not necessarily disentangled, the candidate subsets might contain signiﬁcant overlaps (e.g., “cat with sink” and “cat with faucet”).
Revised: Construct Meta-graphs Since the meta-data are not necessarily disentangled, the candidate subsets might contain signiﬁcant overlaps (e.g., “cat with sink” and “cat with faucet”).

Original: To capture this phenomenon, we construct a meta-graph to model the relationships among all subsets of each class.
Revised: To capture this phenomenon, we construct a meta-graph to model the relationships among all subsets of each class.

Original: Speciﬁcally, for each class j ∈ Y , we construct meta-graph, a weighted undirected graph
Revised: Speciﬁcally, for each class j ∈ Y , we construct meta-graph, a weighted undirected graph

Original: G =
Revised: G =

Original: ( V , E ) where each node v ∈ V denotes a candidate subset, and the weight of each edge is the overlap coefﬁcient between two subsets:
Revised: ( V , E ) where each node v ∈ V denotes a candidate subset, and the weight of each edge is the overlap coefﬁcient between two subsets:

Original: [Equation]
Revised: [Equation]

Original: We remove the edges whose weights are less than a threshold (e.g., 0.1) to sparsify the graph.
Revised: We remove the edges whose weights are less than a threshold (e.g., 0.1) to sparsify the graph.

Original: As shown in Figure 2, the meta-graph G captures meaningful semantics of the multi-modal data distribution of the class of interest.
Revised: As shown in Figure 2, the meta-graph G captures meaningful semantics of the multi-modal data distribution of the class of interest.

Original: Step 3: Quantify Distances of Distribution Shifts
Revised: Step 3: Quantify Distances of Distribution Shifts

Original: The geometry of meta-graphs provides a natural and systematic way to quantify the distances of shifts across different data distributions:
Revised: The geometry of meta-graphs provides a natural and systematic way to quantify the distances of shifts across different data distributions:

Original: Intuitively, if two subsets are far away from each other in the MetaGraph, then the shift between them tend to be large.
Revised: Intuitively, if two subsets are far away from each other in the MetaGraph, then the shift between them tend to be large.

Original: Following this intuition, we leverage spectral embeddings (Belkin & Niyogi, 2003; Chung & Graham, 1997) to assign an embedding for each node based on the graph geometry.
Revised: Following this intuition, we leverage spectral embeddings (Belkin & Niyogi, 2003; Chung & Graham, 1997) to assign an embedding for each node based on the graph geometry.

Original: Spectral embedding minimizes the expected square distance between nodes that are connected:
Revised: Spectral embedding minimizes the expected square distance between nodes that are connected:

Original: [Equation]
Revised: [Equation]

Original: where X i is the embedding for node i ∈ V and K is the dimension of the embedding.
Revised: where X i is the embedding for node i ∈ V and K is the dimension of the embedding, and A is the adjacency matrix.

Original: We denote by X the matrix of dimension n × K whose i -th
Revised: We denote by X the matrix of dimension n × K whose i -th

Original: row X i corresponds to the embedding of node i .
Revised: row X i corresponds to the embedding of node i .

Original: The constraint X T 1 = 0 forces the embedding to be centered and X T X 
Revised: The constraint X T 1 = 0 forces the embedding to be centered and X T X =

Original: = I K ensures that we do not get trivial solution like all node embeddings located at the origin (i.e., X = 0).
Revised: I K ensures that we do not get trivial solution like all node embeddings located at the origin (i.e., X = 0).

Original: Denoting by L = D − A the Laplacian matrix of the graph, we have:
Revised: Denoting by L = D − A the Laplacian matrix of the graph, we have:

Original: [Equation]
Revised: [Equation]

Original: The minimum is reached for X equal to the matrix of eigenvectors of the Laplacian matrix associated with the eigenvalues λ 2 , ..., λ K +1 .
Revised: The minimum is reached for X equal to the matrix of eigenvectors of the Laplacian matrix associated with the eigenvalues λ 2 , ..., λ K +1 .

Original: After calculating the spectral embeddings, we use the euclidean distance between the embeddings of two nodes as their distance.
Revised: After calculating the spectral embeddings, we use the euclidean distance between the embeddings of two nodes as their distance.

Original: Other off-the-shelf node embedding methods like Node2Vec Grover & Leskovec (2016) can be readily plugged into MetaShift.
Revised: Other off-the-shelf node embedding methods like Node2Vec Grover & Leskovec (2016) can be readily plugged into MetaShift.

Original: We delay exploring them as future work.
Revised: We delay exploring them as future work.

Original: Although the subset overlap (i.e., edge weight) can also be used as a similarity metric, it does not incorporate the structural information from neighboring nodes.
Revised: Although the subset overlap (i.e., edge weight) can also be used as a similarity metric, it does not incorporate the structural information from neighboring nodes.

Original: Our node embedding-based distance captures not only such overlap, but also broader similarities.
Revised: Our node embedding-based distance captures not only such overlap, but also broader similarities.

Original: Step 4: Simulating Distribution Shifts MetaShifts allows users to benchmark both (1) domain generalization and (2) subpopulation shifts in a well-annotated (explicit annotation of what drives the shift) and well-controlled (easy control of the amount of distribution shift) fashion.
Revised: Step 4: Simulating Distribution Shifts MetaShifts allows users to benchmark both (1) domain generalization and (2) subpopulation shifts in a well-annotated (explicit annotation of what drives the shift) and well-controlled (easy control of the amount of distribution shift) fashion.

Original: • In domain generalization , the train and test distributions comprise data from related but distinct domains.
Revised: • In domain generalization , the train and test distributions comprise data from related but distinct domains.

Original: This arises in many real-world scenarios since it is often infeasible to construct a comprehensive training set that spans all domains.
Revised: This arises in many real-world scenarios since it is often infeasible to construct a comprehensive training set that spans all domains.

Original: To simulate this setting, we can sample two distinct collections of subsets as the train domains and the test domains respectively (e
Revised: To simulate this setting, we can sample two distinct collections of subsets as the train domains and the test domains respectively (e

Original: g. bathroom vs. outdoor contexts).
Revised: g. bathroom vs. outdoor contexts).

Original: To adjust the magnitude of the shift, we can ﬁx the test domains and change the train domains with different distances.
Revised: To adjust the magnitude of the shift, we can ﬁx the test domains and change the train domains with different distances.

Original: For example, if we use cats-in-living-room as the test set, then this is a smaller distribution shift.
Revised: For example, if we use cats-in-living-room as the test set, then this is a smaller distribution shift.

Original:  In subpopulation shifts , the train and test distributions are mixtures of the same domains, but the mixture weights change between train and test.
Revised: • In subpopulation shifts , the train and test distributions are mixtures of the same domains, but the mixture weights change between train and test.

Original: It is also an important setting since ML models are often reported to perform poorly on under-represented groups.
Revised: It is also an important setting since ML models are often reported to perform poorly on under-represented groups.

Original: To simulate this setting, we can sample the training set and test set from the same subsets but with different mixture weights.
Revised: To simulate this setting, we can sample the training set and test set from the same subsets but with different mixture weights.

Original: To adjust the magnitude of the shift, we can use different mixture weights for the training set while keeping the test set unchanged.
Revised: To adjust the magnitude of the shift, we can use different mixture weights for the training set while keeping the test set unchanged.

Original: We demonstrate both settings in the experiment section.
Revised: We demonstrate both settings in the experiment section.

Original: It is also worth noting that different subsets may share common images—e.g. a dog image can have both grass and frisbee would occur in both dog with grass and dog with frisbee .
Revised: It is also worth noting that different subsets may share common images—e.g. a dog image can have both grass and frisbee would occur in both dog with grass and dog with frisbee .

Original: Therefore, a post-processing step is needed to remove the training images that also occur in the test set to ensure no data leakage.
Revised: Therefore, a post-processing step is needed to remove the training images that also occur in the test set to ensure no data leakage.

Original: MetaShift Statistics Figure 3 shows the statistics of MetaShift across all tasks.
Revised: MetaShift Statistics Figure 3 shows the statistics of MetaShift across all tasks.

Original: We start from the pre-processed and cleaned version of Visual Genome (Hudson & Manning, 2019), which contains 113,018 distinct images across 1,702 object classes.
Revised: We start from the pre-processed and cleaned version of Visual Genome (Hudson & Manning, 2019), which contains 113,018 distinct images across 1,702 object classes.

Original: After the dataset construction, we have 12, sets of natural images from 410 classes.
Revised: After the dataset construction, we have 12, sets of natural images from 410 classes.

Original: Concretely, each class has 31.4 subsets, and each subset has 200.4 images on average.
Revised: Concretely, each class has 31.4 subsets, and each subset has 200.4 images on average.

Original: The subsets are characterized by a diverse vocabulary of 1,853 distinct contexts.
Revised: The subsets are characterized by a diverse vocabulary of 1,853 distinct contexts.

Original: Beyond 1, contexts deﬁned by object presence, MetaShift also leverages the 37 distinct general contexts and 114 object attributes from Visual Genome.
Revised: Beyond 1, contexts deﬁned by object presence, MetaShift also leverages the 37 distinct general contexts and 114 object attributes from Visual Genome.

Original: The general contexts typically describe locations (i.e., indoor, outdoor), weather (e.g., rainy, cloudless), and places (e.g., bathroom, ocean).
Revised: The general contexts typically describe locations (i.e., indoor, outdoor), weather (e.g., rainy, cloudless), and places (e.g., bathroom, ocean).

Original: The object attributes include color (e.g., white, red), material (e.g., wood, plastic), shape (e.g., round, square), and other object-speciﬁc properties (e.g., empty/full for plates).
Revised: The object attributes include color (e.g., white, red), material (e.g., wood, plastic), shape (e.g., round, square), and other object-speciﬁc properties (e.g., empty/full for plates).

Original: See Appendix A for examples and more information.
Revised: See Appendix A for examples and more information.

Original: Generalizability: Case Study on COCO
Revised: Generalizability: Case Study on COCO

Original: The MetaShift construction methodology is quite simple, and can be extended to any dataset with metadata tags (i.e., multilabel).
Revised: The MetaShift construction methodology is quite simple, and can be extended to any dataset with metadata tags (i.e., multilabel).

Original: To demonstrate this, we apply the construction methodology on MS-COCO dataset (Lin et al., 2014), which provides object detection labels for each image.
Revised: To demonstrate this, we apply the construction methodology on MS-COCO dataset (Lin et al., 2014), which provides object detection labels for each image.

Original: Applying our construction methodology, we are able to construct 1321 subsets, with an average size of 389 and a median size of 124, along with 80 meta-graphs that help to quantify the amount of distribution shift among MS-COCO subsets. 
Revised: Applying our construction methodology, we are able to construct 1321 subsets, with an average size of 389 and a median size of 124, along with 80 meta-graphs that help to quantify the amount of distribution shift among MS-COCO subsets. Experiments on MetaShift from COCO can be found in Appendix D.

Original: Even though the data that we used here (e.g. Visual Genome, COCO) is not new, the idea of turning one dataset into a structured collection of datasets for assessing distribution shift is less explored.
Revised: Even though the data that we used here (e.g. Visual Genome, COCO) is not new, the idea of turning one dataset into a structured collection of datasets for assessing distribution shift is less explored.

Original: COCO (Lin et al., 2014) and many other datasets provide meta-data that could be naturally used for generating candidate subsets.
Revised: COCO (Lin et al., 2014) and many other datasets provide meta-data that could be naturally used for generating candidate subsets.

Original: The main contribution of our methodology is that after generating the candidate subsets, our method provides a meta-graph that could be used to determine the train and test domains, and also speciﬁes the similarity between the two subsets via graph distance.
Revised: The main contribution of our methodology is that after generating the candidate subsets, our method provides a meta-graph that could be used to determine the train and test domains, and also speciﬁes the similarity between the two subsets via graph distance.

Original: MetaShift opens up a new window for systematically evaluating domain shifts.
Revised: MetaShift opens up a new window for systematically evaluating domain shifts.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: We demonstrate the utility of MetaShift in two applications:
Revised: We demonstrate the utility of MetaShift in two applications:

Original: • Evaluating distribution shifts: MetaShift supports evaluating both domain generalization (Section 4.1) and subpopulation shifts (Section 4.2).
Revised: • Evaluating distribution shifts: MetaShift supports evaluating both domain generalization (Section 4.1) and subpopulation shifts (Section 4.2).

Original: We perform controlled studies on ML models’ behavior under different amounts of distribution shift.
Revised: We perform controlled studies on ML models’ behavior under different amounts of distribution shift.

Original: We also evaluate several recent proposals for training models to be robust to data shifts.
Revised: We also evaluate several recent proposals for training models to be robust to data shifts.

Original: Assessing training conﬂicts: The subset information in MetaShift also sheds light on the training dynamics of ML models (Section 4.3).
Revised: Assessing training conﬂicts: The subset information in MetaShift also sheds light on the training dynamics of ML models (Section 4.3).

Original: Since we have the subset membership information for each training datum, we could attribute the contribution of each gradient step back to the training subsets, and then analyze the heterogeneity of the contributions made by different training subsets.
Revised: Since we have the subset membership information for each training datum, we could attribute the contribution of each gradient step back to the training subsets, and then analyze the heterogeneity of the contributions made by different training subsets.

Original: 4. E VALUATING D OMAIN G ENERALIZATION
Revised: 4. E VALUATING D OMAIN G ENERALIZATION

Original: In domain generalization , the train and test distributions comprise data from related but distinct domains.
Revised: In domain generalization , the train and test distributions comprise data from related but distinct domains.

Original: To benchmark domain generalization using MetaShift, we can sample two distinct collections of subsets as the train domains and the test domains respectively.
Revised: To benchmark domain generalization using MetaShift, we can sample two distinct collections of subsets as the train domains and the test domains respectively.

Original: We also showcase adjusting the magnitude of the shift by sampling different training subsets while keeping the test set unchanged.
Revised: We also showcase adjusting the magnitude of the shift by sampling different training subsets while keeping the test set unchanged.

Original: [Table]
Revised: [Table]

Original: Setup As shown in ﬁgure 2, subsets like “cat with sink” and “cat with faucet” are quite similar to each other.
Revised: Setup As shown in ﬁgure 2, subsets like “cat with sink” and “cat with faucet” are quite similar to each other.

Original: To make our evaluation settings more challenging, we merge similar subsets by running
Revised: To make our evaluation settings more challenging, we merge similar subsets by running

Original: Louvain community detection algorithm (Blondel et al., 2008) on each meta-graph.
Revised: Louvain community detection algorithm (Blondel et al., 2008) on each meta-graph.

Original: Node color in Figure 2 indicates the community detection result.
Revised: Node color in Figure 2 indicates the community detection result.

Original: The embeddings of the merged subset are calculated as the average embeddings of subsets being merged, weighted by the subset size.
Revised: The embeddings of the merged subset are calculated as the average embeddings of subsets being merged, weighted by the subset size.

Original: After merging similar subsets, we construct four binary classiﬁcation tasks:
Revised: After merging similar subsets, we construct four binary classiﬁcation tasks:

Original: Cat vs. Dog: Test on dog( shelf ) with 129 images.
Revised: Cat vs. Dog: Test on dog( shelf ) with 129 images.

Original: The cat training data is cat( sofa + bed ) (i e., cat( sofa ) unions cat( bed )).
Revised: The cat training data is cat( sofa + bed ) (i.e., cat( sofa ) unions cat( bed )).

Original: We keep the test set and the cat training data unchanged.
Revised: We keep the test set and the cat training data unchanged.

Original: We keep the total size of training data as 400 images unchanged.
Revised: We keep the total size of training data as 400 images unchanged.

Original: We explore the effect of using 4 different sets of dog training data: (a) dog( cabinet + bed ) as dog training data.
Revised: We explore the effect of using 4 different sets of dog training data: (a) dog( cabinet + bed ) as dog training data.

Original: Its distance to dog( shelf ) is d =0.
Revised: Its distance to dog( shelf ) is d =0.

Original: (b) dog( bag + box ) as dog training data.
Revised: (b) dog( bag + box ) as dog training data.

Original: Its distance to dog( shelf ) is d =0.
Revised: Its distance to dog( shelf ) is d =0.

Original: (c) dog( bench + bike ) as dog training data.
Revised: (c) dog( bench + bike ) as dog training data.

Original: Its distance to dog( shelf ) is d =1.
Revised: Its distance to dog( shelf ) is d =1.

Original: (d) dog( boat + surfboard )
Revised: (d) dog( boat + surfboard )

Original: as dog training data.
Revised: as dog training data.

Original: Its distance to dog( shelf ) is d =1. 2.
Revised: Its distance to dog( shelf ) is d =1. 2.

Original: Bus vs. Truck: Test on truck( airplane ) with 161 images.
Revised: Bus vs. Truck: Test on truck( airplane ) with 161 images.

Original: The bus training data is bus( clock + trafﬁc light ).
Revised: The bus training data is bus( clock + trafﬁc light ).

Original: The 4 different sets of truck training data are truck( cone + fence ), truck( bike + mirror ), truck( ﬂag + tower ), truck( trafﬁc light + dog ).
Revised: The 4 different sets of truck training data are truck( cone + fence ), truck( bike + mirror ), truck( ﬂag + tower ), truck( trafﬁc light + dog ).

Original: Their distances to truck( airplane ) are d =0.81, d =1.20, d =1.42, d =1.52, respectively, as shown in Table 1. 3.
Revised: Their distances to truck( airplane ) are d =0.81, d =1.20, d =1.42, d =1.52, respectively, as shown in Table 1. 3.

Original: Elephant vs. Horse: Test on horse( barn ) with 140 images.
Revised: Elephant vs. Horse: Test on horse( barn ) with 140 images.

Original: The elephant training data is elephant( fence + rock ).
Revised: The elephant training data is elephant( fence + rock ).

Original: The 4 different sets of horse training data are horse( dirt + trees ), horse( fence + helmet ), horse( car + wagon ), horse( statue + cart ), with distances d =0.44/0.63/0.89/1.44.
Revised: The 4 different sets of horse training data are horse( dirt + trees ), horse( fence + helmet ), horse( car + wagon ), horse( statue + cart ), with distances d =0.44/0.63/0.89/1.44.

Original: Bowl vs. Cup: Test on cup( coffee ) with 375 images.
Revised: Bowl vs. Cup: Test on cup( coffee ) with 375 images.

Original: The bowl training data is bowl( fruits + tray ).
Revised: The bowl training data is bowl( fruits + tray ).

Original: The 4 different sets of horse training data are cup( knife + tray ), cup( water + cabinet ), cup( computer + lamp ), cup( toilet + box ), with distances d =0.16/0.47/1.03/1.31 respectively.
Revised: The 4 different sets of horse training data are cup( knife + tray ), cup( water + cabinet ), cup( computer + lamp ), cup( toilet + box ), with distances d =0.16/0.47/1.03/1.31 respectively.

Original: Domain generalization algorithms Following recent benchmarking efforts (Koh et al., 2020;
Revised: Domain generalization algorithms Following recent benchmarking efforts (Koh et al., 2020;

Original: Gulrajani & Lopez-Paz, 2020), we evaluated several recent methods for training models to be robust to data shifts: IRM (Arjovsky et al., 2019), DRO (Sagawa et al., 2020), CORAL (Sun & Saenko, 2016), CDANN (Long et al., 2018).
Revised: Gulrajani & Lopez-Paz, 2020), we evaluated several recent methods for training models to be robust to data shifts: IRM (Arjovsky et al., 2019), GroupDRO (Sagawa et al., 2020), CORAL (Sun & Saenko, 2016), CDANN (Long et al., 2018).

Original: To ensure a fair comparison, we use the implementation and the default hyperparameters from Domainbed (Gulrajani & Lopez-Paz, 2020).
Revised: To ensure a fair comparison, we use the implementation and the default hyperparameters from DomainBed (Gulrajani & Lopez-Paz, 2020).

Revised: We discuss more experiment setup details in Appendix B.1.

Revised: We also extend the binary classiﬁcation experiment here to a 10-class classiﬁcation experiment in Appendix B.2.1.

Original: Results Our experiments show that the standard empirical risk minimization performs the best when shifts are moderate(i.e.
Revised: Results Our experiments show that the standard empirical risk minimization performs the best when shifts are moderate(i.e.

Original: , when d is small).
Revised: , when d is small).

Original: For example, ERM consistently achieves the best test accuracy on the smallest d of all tasks.
Revised: For example, ERM consistently achieves the best test accuracy on the smallest d of all tasks.

Original: This ﬁnding is aligned with previous research, as the domain generalization methods typically impose strong algorithmic regularization in various forms Koh et al.
Revised: This ﬁnding is aligned with previous research, as the domain generalization methods typically impose strong algorithmic regularization in various forms Koh et al.

Original: Santurkar et al.
Revised: Santurkar et al.

Original: However, when the amount of distribution shift increases, the domain generalization algorithms typically outperform the ERM baseline, though no algorithm is a consistent winner compared to other algorithms for large shifts.
Revised: However, when the amount of distribution shift increases, the domain generalization algorithms typically outperform the ERM baseline, though no algorithm is a consistent winner compared to other algorithms for large shifts.

Original: This ﬁnding suggests that domain generalization is an important and challenging task and that there’s still a lot of room for new methods. 
Revised: This ﬁnding suggests that domain generalization is an important and challenging task and that there’s still a lot of room for new methods. [Table]

Original: 4. E VALUATING S UBPOPULATION S HIFTS
Revised: 4. E VALUATING S UBPOPULATION S HIFTS

Original: In subpopulation shifts , the train and test distributions are mixtures of the same domains with different mixture weights.
Revised: In subpopulation shifts [Table] the train and test distributions are mixtures of the same domains with different mixture weights.

Original: This is a more frequently-encountered problem since real-world datasets often have[Table] minority groups, while standard models are often reported to perform poorly on under-represented demographics (Buolamwini & Gebru, 2018; Koenecke et al., 2020).
Revised: This is a more frequently-encountered problem since real-world datasets often have minority groups, while standard models are often reported to perform poorly on under-represented demographics (Buolamwini & Gebru, 2018; Koenecke et al., 2020).

Original: Setup To benchmark subpopulation shifts using MetaShift, we can sample two distinct collections of subsets as the minority groups and majority groups respectively.
Revised: Setup To benchmark subpopulation shifts using MetaShift, we can sample two distinct collections of subsets as the minority groups and majority groups respectively.

Original: We then use different mixture weights to construct the training set and test set.
Revised: We then use different mixture weights to construct the training set and test set.

Original: For “Cat vs. Dog”, we leveraging the general contexts “indoor/outdoor” which has a natural spurious correlation with the class labels.
Revised: For “Cat vs. Dog”, we leveraging the general contexts “indoor/outdoor” which has a natural spurious correlation with the class labels.

Original: Concretely, in the training data, cat( ourdoor ) and dog( indoor ) subsets are the minority groups, while cat( indoor ) and dog( outdoor ) are majority groups.
Revised: Concretely, in the training data, cat( ourdoor ) and dog( indoor ) subsets are the minority groups, while cat( indoor ) and dog( outdoor ) are majority groups.

Original: We keep the total size of training data as 1700 images unchanged and only vary the portion of minority groups.
Revised: We keep the total size of training data as 1700 images unchanged and only vary the portion of minority groups.

Original: We use a balanced test set with 576 images to report both average accuracy and worst group accuracy.
Revised: We use a balanced test set with 576 images to report both average accuracy and worst group accuracy.

Revised: We extend the binary classiﬁcation experiment here to a 10-class classiﬁcation experiment in Appendix B.2.2.

Original: Results Table 2 shows the evaluation results on subpopulation shifts.
Revised: Results Table 2 shows the evaluation results on subpopulation shifts.

Original: In terms of the average accuracy, ERM performs the best when p is large (i.e., the minority group is less underrepresented) as expected.
Revised: In terms of the average accuracy, ERM performs the best when p is large (i.e., the minority group is less underrepresented) as expected.

Original: However, in terms of the worst group accuracy, the algorithms typically outperform
Revised: However, in terms of the worst group accuracy, the algorithms typically outperform

Original: ERM, showcasing their effectiveness in subpopulation shifts.
Revised: ERM, showcasing their effectiveness in subpopulation shifts.

Original: Furthermore, it is worth noting that no algorithm is a consistent winner compared to other algorithms for large shifts.
Revised: Furthermore, it is worth noting that no algorithm is a consistent winner compared to other algorithms for large shifts.

Original: This ﬁnding is aligned with our ﬁnding in the previous section, suggesting that subpopulation shift is an important and challenging problem and there’s still a lot of room for new methods.
Revised: This ﬁnding is aligned with our ﬁnding in the previous section, suggesting that subpopulation shift is an important and challenging problem and there’s still a lot of room for new methods.

Original: 4. A SSESSING T RAINING C ONFLICTS
Revised: 4. A SSESSING T RAINING C ONFLICTS

Original: Next, we demonstrate how MetaShift can shed light on the training dynamics of ML models.
Revised: Next, we demonstrate how MetaShift can shed light on the training dynamics of ML models.

Original: An important feature of MetaShift is that each training datum is not only associated with a class label, but also the annotations of subset membership.
Revised: An important feature of MetaShift is that each training datum is not only associated with a class label, but also the annotations of subset membership.

Original: Such annotations open a window for a systematic evaluation of how training on each subset would affect the evaluation performance on other subsets.
Revised: Such annotations open a window for a systematic evaluation of how training on each subset would affect the evaluation performance on other subsets.

Original: Setup Concretely, we evaluate the ML model after each gradient step .
Revised: Setup Concretely, we evaluate the ML model after each gradient step .

Original: We record the change of the validation loss of each validation subset.
Revised: We record the change of the validation loss of each validation subset.

Original: This gives us a multi-axes description of the effect of the current gradient step.
Revised: This gives us a multi-axes description of the effect of the current gradient step.

Original: Meanwhile, we also record the subset membership information for each of the training data in the current batch.
Revised: Meanwhile, we also record the subset membership information for each of the training data in the current batch.

Original: We then run a linear regression to analyze the contributions of each training subset to the change of the validation loss of each validation subset.
Revised: We then run a linear regression to analyze the contributions of each training subset to the change of the validation loss of each validation subset.

Original: We sampled 48 subsets from cat & dog classes.
Revised: We sampled 48 subsets from cat & dog classes.

Original: We use 22 of them as out-of-domain evaluation sets and the other 26 as training and in-domain evaluation.
Revised: We use 22 of them as out-of-domain evaluation sets and the other 26 as training and in-domain evaluation.

Original: We sample 50 images from each subset to form the training set.
Revised: We sample 50 images from each subset to form the training set.

Original: Figure 4 shows the abridged results for brevity and better visualization.
Revised: Figure 4 shows the abridged results for brevity and better visualization.

Original: Results Region (1) and (2) in Figure 4 highlighted the “conﬂict zones”: indoor cat subsets and indoor dog subsets are having severe training conﬂicts with each other.
Revised: Results Region (1) and (2) in Figure 4 highlighted the “conﬂict zones”: indoor cat subsets and indoor dog subsets are having severe training conﬂicts with each other.

Original: This makes sense since they share the same spurious feature “indoor”, making the model confused.
Revised: This makes sense since they share the same spurious feature “indoor”, making the model confused.

Original: However, such training conﬂicts are important for out-of-domain generalization.
Revised: However, such training con- [Figure] ﬂicts are important for out-of-domain generalization.

Original: As highlighted in Region (6), the dog( animal ) subset (last column, which comprises primarily outdoor images) beneﬁts most from indoor subsets like dog( sheet ), rather than outdoor dog subsets like dog( truck ).
Revised: As highlighted in Region (6), the dog( animal ) subset (last column, which comprises primarily outdoor images) beneﬁts most from indoor subsets like dog( sheet ), rather than outdoor dog subsets like dog( truck ).

Original: These results might indicate that training conﬂicts force the ML model to learn generalizable features.
Revised: These results might indicate that training conﬂicts force the ML model to learn generalizable features.

Original: [Figure] Furthermore, Figure 4 also reveals the subsets that have low training contributions.
Revised: Furthermore, Figure 4 also reveals the subsets that have low training contributions.

Original: As highlighted in Region (3) and (4), training on cat( faucet ) and dog( frisbee ) has little effect on other subsets.
Revised: As highlighted in Region (3) and (4), training on cat( faucet ) and dog( frisbee ) has little effect on other subsets.

Original: We hypothesize that this is because these subsets are too easy and thus provide little insight for the model.
Revised: We hypothesize that this is because these subsets are too easy and thus provide little insight for the model.

Original: To verify this hypothesis, we remove cat( faucet ), and dog( frisbee ) from training and use them for out-of-domain validation.
Revised: To verify this hypothesis, we remove cat( faucet ), and dog( frisbee ) from training and use them for out-of-domain validation.

Original: We found that the model achieves a near perfect accuracy on both subsets: cat( faucet ): 98.1%, dog( frisbee ): 96.6%.
Revised: We found that the model achieves a near perfect accuracy on both subsets: cat( faucet ): 98.1%, dog( frisbee ): 96.6%.

Original: This shows that subsets with low training contributions tend to be the subsets that are too easy.
Revised: This shows that subsets with low training contributions tend to be the subsets that are too easy.

Original: Other observations include: Columns of cat( television ) and dog( television ) in Region (6) look opposite to each other.
Revised: Other observations include: Columns of cat( television ) and dog( television ) in Region (6) look opposite to each other.

Original: In addition, dog( truck ) as highlighted in (5) exhibits a very different behaviour compared to other dog subsets: training on dog(truck) conﬂicts with all other dog subsets.
Revised: In addition, dog( truck ) as highlighted in (5) exhibits a very different behaviour compared to other dog subsets: training on dog(truck) conﬂicts with all other dog subsets.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present MetaShift—a collection of 12,868 sets of natural images from 410 classes—as an important resource for studying the behavior or ML algorithms and training dynamics across data with heterogeneous contexts.
Revised: We present MetaShift—a collection of 12,868 sets of natural images from 410 classes—as an important resource for studying the behavior or ML algorithms and training dynamics across data with heterogeneous contexts.

Original: MetaShift contains orders of magnitude more natural data shifts than previously available.
Revised: MetaShift contains orders of magnitude more natural data shifts than previously available.

Original: It also provides explicit explanations of what is unique about each of its data sets and a distance score that measures the amount of distribution shift between any two of its data sets.
Revised: It also provides explicit explanations of what is unique about each of its data sets and a distance score that measures the amount of distribution shift between any two of its data sets.

Original: We demonstrate the utility of MetaShift in both evaluating distribution shifts, and visualizing conﬂicts between data subsets during model training.
Revised: We demonstrate the utility of MetaShift in both evaluating distribution shifts, and visualizing conﬂicts between data subsets during model training.

Original: Even though the data that we used here (e.g. Visual Genome) is not new, the idea of turning one dataset into a structured collection of datasets for assessing distribution shift is less explored.
Revised: Even though the data that we used here (e.g. Visual Genome) is not new, the idea of turning one dataset into a structured collection of datasets for assessing distribution shift is less explored.

Original: Our methodology for constructing MetaShift is simple and powerful, and can be readily extended to create many new resources for evaluating distribution shifts, which is very much needed.
Revised: Our methodology for constructing MetaShift is simple and powerful, and can be readily extended to create many new resources for evaluating distribution shifts, which is very much needed.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: Our project website https://MetaShift.readthedocs.io/ provides detailed documentation and installation guidelines.
Revised: Our project website https://MetaShift.readthedocs.io/ provides detailed documentation and installation guidelines.

Original: In the experiments of evaluating domain generalization and subpopulation shifts, we use the implementation and the default hyperparameters from Domainbed (Gulrajani & Lopez-Paz, 2020) to ensure a fair comparison.
Revised: In the experiments of evaluating domain generalization and subpopulation shifts, we use the implementation and the default hyperparameters from Domainbed (Gulrajani & Lopez-Paz, 2020) to ensure a fair comparison.

Original: Our MetaShift dataset and code are available in the supplementary material, and would use the Creative Commons Attribution 4.
Revised: Our MetaShift dataset and code are available in the supplementary material, and would use the Creative Commons Attribution 4.

Original: International License.
Revised: International License.

Original: The dataset and code are also available in the anonymous Github repo: https://anonymous.4open.science/r/MetaShift-Supplementary-011E
Revised: The dataset and code are also available in the anonymous Github repo: https://anonymous.4open.science/r/MetaShift-Supplementary-011E

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: One limitation is that our MetaShift might inherit existing biases in Visual Genome, which is the base dataset of our MetaShift.
Revised: One limitation is that our MetaShift might inherit existing biases in Visual Genome, which is the base dataset of our MetaShift.

Original: Potential concerns include minority groups being under-represented in certain classes (e.g., women with snowboard), or annotation bias where people in images are by default labeled as male when gender is unlikely to be identiﬁable.
Revised: Potential concerns include minority groups being under-represented in certain classes (e.g., women with snowboard), or annotation bias where people in images are by default labeled as male when gender is unlikely to be identiﬁable.

Original: Existing work in analyzing, quantifying, and mitigating biases in general computer vision datasets can help with addressing this potential negative societal impact.
Revised: Existing work in analyzing, quantifying, and mitigating biases in general computer vision datasets can help with addressing this potential negative societal impact.

Original: We can also use MetaShift to understand how certain subsets of training data can lead to model biases in other contexts, as done in our experiments in Figure 4.
Revised: We can also use MetaShift to understand how certain subsets of training data can lead to model biases in other contexts, as done in our experiments in Figure 4.

Original: A A DDITIONAL D ATASET I NFORMATION
Revised: A A DDITIONAL D ATASET I NFORMATION

Original: For each image class (e.g. Dogs ), the MetaShift dataset contains different sets of dogs under different contexts to represent diverse data distributions.
Revised: For each image class (e.g. Dogs ), the MetaShift dataset contains different sets of dogs under different contexts to represent diverse data distributions.

Original: The contexts include presence/absence of other objects (e.g. dog with frisbee ).
Revised: The contexts include presence/absence of other objects (e.g. dog with frisbee ).

Original: Contexts can also reﬂect attributes (e.g. black dogs ) and general settings (e.g. dogs in sunny weather ).
Revised: Contexts can also reﬂect attributes (e.g. black dogs ) and general settings (e.g. dogs in sunny weather ).

Original: These concepts thus captures diverse and real-world distribution shifts.
Revised: These concepts thus captures diverse and real-world distribution shifts.

Original: We list the attribute and general location contexts below.
Revised: We list the attribute and general location contexts below.

Original: A. G ENERAL LOCATION AND ATTRIBUTE CONTEXTS A.1. G ENERAL LOCATION CONTEXTS
Revised: A. G ENERAL LOCATION AND ATTRIBUTE CONTEXTS A.1. G ENERAL LOCATION CONTEXTS

Original: [Figure]
Revised: [Figure]

Original: GENERAL_CONTEXT_ONTOLOGY = { 'indoor/outdoor': ['indoors', 'outdoors'], 'weather': ['clear', 'overcast', 'cloudless', 'cloudy', 'sunny', 'foggy', 'rainy'], 'room':
Revised: GENERAL_CONTEXT_ONTOLOGY = { 'indoor/outdoor': ['indoors', 'outdoors'], 'weather': ['clear', 'overcast', 'cloudless', 'cloudy', 'sunny', 'foggy', 'rainy'], 'room':

Original: ['bedroom', 'kitchen', 'bathroom', 'living room'], 'place': ['road', 'sidewalk', 'field', 'beach', 'park', 'grass', 'farm', 'ocean', 'pavement', 'lake', 'street', 'train station', 'hotel room', 'church', 'restaurant', 'forest', 'path', 'display', 'store', 'river', 'yard', 'snow', 'airport', 'parking lot'] }
Revised: ['bedroom', 'kitchen', 'bathroom', 'living room'], 'place': ['road', 'sidewalk', 'field', 'beach', 'park', 'grass', 'farm', 'ocean', 'pavement', 'lake', 'street', 'train station', 'hotel room', 'church', 'restaurant', 'forest', 'path', 'display', 'store', 'river', 'yard', 'snow', 'airport', 'parking lot'] }

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: ', 'staring', 'drinking', 'playing', 'eating', 'cooking', 'resting', 'sleeping', 'posing', 'talking', 'looking down', 'looking up', 'driving', 'reading', 'brushing teeth', 'flying', 'surfing', 'skiing', 'hanging'], 'pose': ['walking', 'standing', 'lying', 'sitting', 'running', 'jumping', 'crouching', 'bending', 'smiling', 'grazing'], 'material': ['wood', 'plastic', 'metal', 'glass', 'leather', 'leather', 'porcelain', 'concrete', 'paper', 'stone', 'brick'], 'color': ['white', 'red', 'black', 'green', 'silver', 'gold', 'khaki', 'gray', 'dark', 'pink', 'dark blue', 'dark brown', 'blue', 'yellow', 'tan', 'brown', 'orange', 'purple', 'beige', 'blond', 'brunette', 'maroon', 'light blue', 'light brown'] }
Revised: ', 'staring', 'drinking', 'playing', 'eating', 'cooking', 'resting', 'sleeping', 'posing', 'talking', 'looking down', 'looking up', 'driving', 'reading', 'brushing teeth', 'flying', 'surfing', 'skiing', 'hanging'], 'pose': ['walking', 'standing', 'lying', 'sitting', 'running', 'jumping', 'crouching', 'bending', 'smiling', 'grazing'], 'material': ['wood', 'plastic', 'metal', 'glass', 'leather', 'leather', 'porcelain', 'concrete', 'paper', 'stone', 'brick'], 'color': ['white', 'red', 'black', 'green', 'silver', 'gold', 'khaki', 'gray', 'dark', 'pink', 'dark blue', 'dark brown', 'blue', 'yellow', 'tan', 'brown', 'orange', 'purple', 'beige', 'blond', 'brunette', 'maroon', 'light blue', 'light brown'] }

Original: A. C OMPUTE REQUIREMENTS
Revised: A. C OMPUTE REQUIREMENTS

Original: All experiments were performed on an Amazon EC2 P3.8 instance with 4 NVIDIA V100 Tensor
Revised: All experiments were performed on an Amazon EC2 P3.8 instance with 4 NVIDIA V100 Tensor

Original: Core GPUs.
Revised: Core GPUs.

Original: Each training trial in evaluating distribution shifts takes less than an hour to ﬁnish.
Revised: Each training trial in evaluating distribution shifts takes less than an hour to ﬁnish.

Original: A. D ATASET L ICENSES
Revised: A. D ATASET L ICENSES

Original: Our MetaShift and the code would use the Creative Commons Attribution 4.0 International License.
Revised: Our MetaShift and the code would use the Creative Commons Attribution 4.0 International License.

Original: Visual Genome Krishna et al.
Revised: Visual Genome Krishna et al.

Original: (2017) is licensed under a Creative Commons Attribution 4.0 International License.
Revised: (2017) is licensed under a Creative Commons Attribution 4.0 International License.

Original: MS-COCO Lin et al.
Revised: MS-COCO Lin et al.

Original: (2014) is licensed under CC-BY 4.0.
Revised: (2014) is licensed under CC-BY 4.0.

Original: The Visual Genome dataset uses 108, 077 images the
Revised: The Visual Genome dataset uses 108, 077 images the

Original: (2016) and MS-COCO.
Revised: (2016) and MS-COCO.

Original: We use the implementation and hyper-parameters from Domainbed Gulrajani & Lopez-Paz (2020) for empirical risk minimization (ERM) and all domain generalization baselines.
Revised: We use the implementation and hyper-parameters from Domainbed Gulrajani & Lopez-Paz (2020) for empirical risk minimization (ERM) and all domain generalization baselines.

Original: Domainbed is licensed under an MIT License.
Revised: Domainbed is licensed under an MIT License.

Original: It is worth noting that the implementation of empirical risk minimization
Revised: It is worth noting that the implementation of empirical risk minimization

Original: (ERM) in Domainbed implicitly up-samples minority domains during training.
Revised: (ERM) in Domainbed implicitly up-samples minority domains during training.

Original: Nevertheless, we stick with their implementation for consistency.
Revised: Nevertheless, we stick with their implementation for consistency.

Original: BA DDITIONAL E XPERIMENT D ETAILS B. A PPLICATION : A SSESSING T RAINING C ONFLICTS Out-of-Domain Evaluation Evaluation Data
Revised: B E XPERIMENT D ETAILS B. E XPERIMENT S ETUP D ETAILS

Revised: In the experiments of evaluating domain generalization and subpopulation shifts, we use the implementation and the default hyperparameters from DomainBed (Gulrajani & Lopez-Paz, 2020) to ensure a fair comparison. More speciﬁcally, we evaluated the following methods for training models to be robust to data shifts: • ERM: the standard Empirical risk minimization baseline.

Revised: • IRM: Invariant risk minimization (Arjovsky et al., 2019), which penalizes feature distributions that have different optimal linear classiﬁers for each domain.

Revised: • GroupDRO: Group Distributionally robust optimization (Sagawa et al., 2020), which uses distributionally robust optimization to explicitly minimize the loss on the worst-case domain during training.

Revised: • CORAL: Correlation Alignment (Sun & Saenko, 2016), which penalizes differences in the means and covariances of the feature distributions for each domain.

Revised: • CDANN: Conditional Adversarial Domain Adaptation (Long et al., 2018), which uses adversarial learning to penalize differences in feature representations from different domains.

Revised: Following DomainBed (Gulrajani & Lopez-Paz, 2020), we use ResNet-18 with the Adam optimizer, cross-entropy loss, learning rate 0 .

Revised: 00005 , batch size 32 .

Revised: The data augmentation steps include the following: crops of random size and aspect ratio, resizing to 224 × 224 pixels, random horizontal ﬂips, random color jitter, grayscaling the image with 10% probability, and normalization using the

Revised: ImageNet channel means and standard deviations.

Revised: We start from ImageNet pre-trained weights, which is consistent with the standard practice of prior work (Gulrajani & Lopez-Paz, 2020; Koh et al., 2020; Sagawa et al., 2020).

Revised: For additional implementation details, we refer readers to Appendix D of

Revised: DomainBed (Gulrajani & Lopez-Paz, 2020).

Revised: B. M ULTI - CLASS C LASSIFICATION E XPERIMENTS

Revised: To demonstrate the generalizability of our ﬁndings, we extend our binary classiﬁcation experiments into a multi-class classiﬁcation setting.

Revised: More speciﬁcally, we augment the “Cat vs. Dog” classiﬁcation task into a 10-class animal classiﬁcation task by incorporating the following classes: “bear”, “bird”, “cow”, “elephant”, “horse”, “sheep”, “giraffe”, “zebra”.

Revised: B.2. M ULTI - CLASS C LASSIFICATION : D OMAIN G ENERALIZATION

Revised: [Table]

Revised: We extend Task 1 in the domain generalization experiment (Section 4.1) into a multi-class classiﬁcation setting.

Revised: Overall, these new results are very consistent with our ﬁndings.

Revised: The multi-class classiﬁcation setting is constructed as follows: we extend the “Cat vs. Dog” task to a 10-class animal classiﬁcation task.

Revised: Similar to Section 4.1, we only change the training data of dog, while keeping the training data of all other classes unchanged, in order to avoid introducing additional confounding effects.

Revised: We explore the effect of using 4 different sets of dog training data:

Revised: dog( cabinet + bed ) as dog training data.

Revised: Its distance to dog( shelf ) is d =0.

Revised: dog( bag + box ) as dog training data.

Revised: Its distance to dog( shelf ) is d =0.

Revised: dog( bench + bike ) as dog training data.

Revised: Its distance to dog( shelf ) is d =1.

Revised: dog( boat + surfboard ) as dog training data.

Revised: Its distance to dog( shelf ) is d =1.

Revised: Results As shown in Table 3, in the multi-class classiﬁcation setting, the standard empirical risk minimization still performs the best when shifts are moderate(i.e.

Revised: , when d is small), which is aligned with previous research, as the domain generalization methods typically impose strong algorithmic regularization in various forms Koh et al.

Revised: Santurkar et al.

Revised: However, when the amount of distribution shift increases, the domain generalization algorithms typically outperform the ERM baseline, though no algorithm is a consistent winner compared to other algorithms for large shifts.

Revised: This ﬁnding suggests that domain generalization is an important and challenging task and that there’s still a lot of room for new methods.

Revised: B.2. M ULTI - CLASS C LASSIFICATION : S UBPOPULATION S HIFT

Original: [Figure]
Revised: We extend the “Cat vs. Dog” subpopulation Shift to a 10-class animal classiﬁcation task. Similarly, to avoid introducing additional confounding effects, we only vary the training data of cat and dog in different experiments, but keep the training data of other classes unchanged across experiments. [Table] Results Table 5 shows the evaluation results on subpopulation shifts in the multi-class classiﬁcation setting.

Revised: In terms of the average accuracy, ERM performs the best when p is large (i.e., the minority group is less underrepresented) as expected.

Revised: However, in terms of the worst group accuracy, the algorithms typically outperform ERM, showcasing their effectiveness in subpopulation shifts.

Revised: Furthermore, it is worth noting that no algorithm is a consistent winner compared to other algorithms for large shifts.

Revised: This ﬁnding is aligned with our ﬁnding in the previous section, suggesting that subpopulation shift is an important and challenging problem and there’s still a lot of room for new methods.

Revised: B. A PPLICATION : A SSESSING T RAINING C ONFLICTS

Original: As data is the fuel powering artiﬁcial intelligence, it is important to understand the training conﬂicts on heterogeneous training data.
Revised: As data is the fuel powering artiﬁcial intelligence, it is important to understand the training conﬂicts on heterogeneous training data.

Original: We next demonstrate the utility of MetaDataset on visualizing how different subsets of data provide conﬂicting training signals.
Revised: We next demonstrate the utility of MetaDataset on visualizing how different subsets of data provide conﬂicting training signals.

Original: The key idea is to analyze the change of validation loss on each validation subset after each gradient step, and then attribute the change to each training subset.
Revised: The key idea is to analyze the change of validation loss on each validation subset after each gradient step, and then attribute the change to each training subset.

Original: As shown in the “in-domain evaluation” region of Figure 9, we randomly sample 26 cat & dog subsets for training and in-domain evaluation: we split the data in each subset into a training set and an in-domain evaluation set.
Revised: As shown in the “in-domain evaluation” region of Figure 9, we randomly sample 26 cat & dog subsets for training and in-domain evaluation: we split the data in each subset into a training set and an in-domain evaluation set.

Original: As shown in the “out-of-domain evaluation” region of Figure 9, we  additionally sample 22 subsets as out-of-domain evaluation sets.
Revised: As shown in the “out-of-domain evaluation” region of Figure 9, we [Figure] additionally sample 22 subsets as out-of-domain evaluation sets.

Original: We downsample the training data to make sure that every training subset has an equal number of data points (i.e., 50).
Revised: We downsample the training data to make sure that every training subset has an equal number of data points (i.e., 50).

Revised: C M ETA -G RAPH AND M ETA - DATA C. V ISUALIZATION OF M ETA - GRAPH S PECTRAL E MBEDDINGS

Revised: To provide a more intuitive understanding of the meta-graph embeddings we visualize the spectral embeddings of all nodes in the “Cat” meta-graph in Figure 10.

Revised: Overall, the spectral embeddings work well as similar nodes are close in the t-SNE visualization.

Revised: Nodes with similar color (i e., assigned to the same cluster by the Louvain community detection algorithm) are also clustered in the t-SNE visualization.

Revised: Finally, we also note that the geometry of the t-SNE visualization (Figure 10) also closely mirrors the geometry of the meta-graph visualization layout in Figure 2.

Original: C L OUVAIN C OMMUNITY D ETECTION A LGORITHM
Revised: C. L OUVAIN C OMMUNITY D ETECTION A LGORITHM

Original: Community detection algorithms detect groups of nodes in a graph that are more densely connected internally than with the rest of the graph.
Revised: Community detection algorithms detect groups of nodes in a graph that are more densely connected internally than with the rest of the graph.

Original: We apply the Louvain community detection algorithm to recover the community structure of meta-graphs.
Revised: We apply the Louvain community detection algorithm to recover the community structure of meta-graphs.

Original: One measure of how well a network is partitioned into communities is Modularity , which is the difference between the actual number of edges in a community and the expected number of such edges.
Revised: One measure of how well a network is partitioned into communities is Modularity , which is the difference between the actual number of edges in a community and the expected number of such edges.

Original: We apply the Louvain community detection algorithm (Blondel et al., 2008), which greedily maximizes modularity since optimising modularity is NP-hard Brandes et al.
Revised: We apply the Louvain community detection algorithm (Blondel et al., 2008), which greedily maximizes modularity since optimizing modularity is NP-hard Brandes et al.

Original: This is a bottom-up algorithm: initially every vertex belongs to a separate community, and vertices are moved between communities iteratively in a way that maximizes the vertices’ local contribution to the overall modularity score.
Revised: This is a bottom-up algorithm: initially, every vertex belongs to a separate community and vertices are moved between communities iteratively in a way that maximizes the vertices’ local contribution to the overall modularity score.

Original: When a consensus is reached (i.e. no single move would increase the modularity score), every community in the original graph is shrank to a single vertex (while keeping the total weight of the adjacent edges) and the process continues on the next level.
Revised: When a consensus is reached (i.e. no single move would increase the modularity score), every community in the original graph is shrank to a single vertex (while keeping the total weight of the adjacent edges) and the process continues on the next level.

Original: The algorithm stops when it is not possible to increase the modularity any more after shrinking the communities to vertices.
Revised: The algorithm stops when it is not possible to increase the modularity anymore after shrinking the communities to vertices.

Revised: C. D ISCUSSIONS ON IMPERFECT OR INCOMPLETE META - DATA

Revised: Even if there is some noise introduced by the imperfect or incomplete metadata, the noise can make the classiﬁcation tasks more challenging and more realistic, as in subpopulation shifts.

Revised: For example, it is rare that someone would collect a dog dataset of 100% indoor images, but it is quite possible that the dataset contains 80% indoor images and 20% outdoor images.

Revised: Therefore, noise from metadata would not invalidate our approach, but make the curated dataset more realistic and challenging.

Revised: D C ONSTRUCTING M ETA S HIFT FROM COCO D ATASET

Revised: MetaShift from COCO To demonstrate the generalizability of the MetaShift construction methodology, we apply our methodology on MS-COCO dataset (Lin et al., 2014).

Revised: We construct 1321 subsets, with an average size of 389 and a median size of 124, along with 80 meta-graphs that help to quantify the amount of distribution shift among MS-COCO subsets.

Revised: Subpopulation Shift Setup Based on the MetaShift from COCO, we constructed a subpopulation shift experiment similar to Section 4.2.

Revised: We construct a “Cat vs. Dog” task, where the “indoor/outdoor” contexts are spuriously correlated with the class labels.

Revised: The “indoor” context is constructed by merging the following super-categories: ’indoor’, ’kitchen’, ’electronic’, ’appliance’, ’furniture’.

Revised: Similarly, the “outdoor” context is constructed by merging the following super-categories: ’outdoor’, ’vehicle’, ’sports’.

Revised: In addition, in the training data, cat( ourdoor ) and dog( indoor ) subsets are the minority groups, while cat( indoor ) and dog( outdoor ) are majority groups.

Revised: We keep the total size of training data as 3000 images unchanged and only vary the portion of minority groups.

Revised: We use a balanced test set with 524 images to report both average accuracy and worst group accuracy.

Revised: [Table]

Revised: Subpopulation Shift Result Overall, we found that the new results are consistent with our previous ﬁndings.

Revised: When p is large (i

Revised: [Table]

Revised: e., the minority group is less underrepresented), ERM outperforms most of the other algorithms in terms of average accuracy.

Revised: It is also worth noting that no algorithm is a consistent winner compared to other algorithms for large shifts.

Revised: These new experiments also show that MetaShift can ﬂexibly accommodate other source datasets including COCO.