E XTENDING THE WILDS B ENCHMARK FOR U NSUPERVISED A DAPTATION
A BSTRACT
Machine learning systems deployed in the wild are often trained on a source distribution but deployed on a different target distribution.
Unlabeled data can be a powerful point of leverage for mitigating these distribution shifts, as it is frequently much more available than labeled data and can often be obtained from distributions beyond the source distribution as well.
However, existing distribution shift benchmarks with unlabeled data do not reﬂect the breadth of scenarios that arise in real-world applications.
In this work, we present the W ILDS 2. update, which extends 8 of the 10 datasets in the W ILDS benchmark of distribution shifts to include curated unlabeled data that would be realistically obtainable in deployment.
These datasets span a wide range of applications (from histology to wildlife conservation), tasks (classiﬁcation, regression, and detection), and modalities (photos, satellite images, microscope slides, text, molecular graphs).
The update maintains consistency with the original W ILDS benchmark by using identical labeled training, validation, and test sets, as well as identical evaluation metrics.
We systematically benchmark state-of-the-art methods that use unlabeled data, including domain-invariant, self-training, and self-supervised methods, and show that their success on W ILDS is limited.
To facilitate method development, we provide an open-source package that automates data loading and contains the model architectures and methods used in this paper. Code and leaderboards are available at https:/ wilds
stanford
edu
I NTRODUCTION
Distribution shifts—when models are trained on a source distribution but deployed on a different target distribution—are frequent problems for machine learning systems in the wild (Qui˜noneroCandela et al., 2009; Geirhos et al., 2020; Koh et al., 2021).
In this paper, we focus on the use of unlabeled data to mitigate these shifts.
Unlabeled data is a powerful point of leverage as it is more readily available than labeled data and can often be obtained from distributions beyond the source distribution.
For example, in the crop detection task in Figure 1, we wish to learn a model that can extrapolate to a set of target domains (farms) (David et al., 2020), and while we only have labeled training examples from some source domains, we have many more unlabeled examples from the source domains, from extra domains, and even directly from the target domains.
Many methods for leveraging unlabeled data have been highly successful on some types of distribution shifts (Berthelot et al., 2021; Zhang et al., 2021).
However, the datasets typically used for evaluating these methods do not reﬂect many of the realistic shifts that might occur in the wild.
These evaluations tend instead to focus on shifts between photos and stylized versions like sketches (Li et al., 2017; Venkateswara et al., 2017; Peng et al., 2019) or synthetic renderings (Peng et al., 2018), or between variants of digits datasets like MNIST (LeCun et al., 1998) and SVHN (Netzer et al., 2011).
Unfortunately, prior work has shown that methods that work well on one type of shift need not generalize to others (Taori et al., 2020;
Djolonga et al., 2020; Xie et al., 2021a; Miller et al., 2021), which raises the question of how well they would work on a wider array of realistic shifts. [Figure]
In this paper, we make two contributions.
First, we present W ILDS 2.0 (Figure 2), an updated version of the recent W ILDS benchmark of in-the-wild distribution shifts (Koh et al., 2021).
W ILDS datasets span a wide range of tasks and modalities, and each dataset reﬂects a domain generalization or subpopulation shift setting with a substantial gap between in-distribution and out-of-distribution performance.
However, W ILDS 1.0 only contained labeled data, which limits the leverage for learning robust models.
In W ILDS 2.0, we extend 8 of the 10 W ILDS datasets 1 with curated unlabeled data acquired from the same source and target domains as the labeled data, as well as from extra domains of the same type: e.g., in the G LOBAL W HEAT - WILDS dataset pictured in Figure 1, we acquired unlabeled photos of wheat ﬁelds from the source and target farms as well as extra farms that were not in the original labeled dataset.
In total, W ILDS 2.0 adds 14.5 million unlabeled examples, expanding the number of examples for each dataset by 3–13 × and allowing us to combine the real-world relevance of W ILDS with the leverage of unlabeled data .
Second, we developed a standardized and consistent protocol for evaluating methods that leverage the unlabeled data in W ILDS 2.0.
We assessed representatives from three popular categories: methods for learning domain-invariant representations (Sun & Saenko, 2016;
Ganin et al., 2016), self-training methods (Lee, 2013; Sohn et al., 2020; Xie et al., 2020), and pre-training methods that rely on self-supervision (Devlin et al., 2019; Caron et al., 2020).
These methods have been successful on some types of shifts, such as going from photos to sketches, or from handwritten digits to street signs (Berthelot et al., 2021; Zhang et al., 2021).
Our results on W ILDS are mixed: many methods did not outperform standard supervised training despite using additional unlabeled data , and the only clear successes were on two image classiﬁcation datasets (C AMELYON 17- WILDS and FM O W- WILDS ).
Successful methods relied heavily on data augmentation (Xie et al., 2020; Caron et al., 2020), which limited their applicability to modalities where augmentations are not as well developed, such as text and molecular graphs.
The same methods were unsuccessful on image regression and detection tasks, which have been relatively understudied: e.g., pseudolabel-based methods do not straightforwardly apply to regression.
For the text datasets, continued language model pre-training did not help, unlike in prior work (Gururangan et al., 2020).
Our results suggest fruitful avenues for future work, such as developing data augmentations for non-image modalities and more effective hyperparameter tuning protocols.
Overall, our results underscore the importance of developing and evaluating methods for unlabeled data on a wider variety of real-world shifts than is typically studied.
To this end, we have updated the open-source Python W ILDS package to include unlabeled data loaders, compatible implementations of all the methods we benchmarked, and scripts to replicate all experiments in this paper (Ap- [Figure] [Figure]
… norm=np. [Figure] p= [Figure]
Popen
[Figure]
)stdout=
p._
[Figure]
[Figure]
[Figure] pendix G). Code and public leaderboards are available at https://wilds.stanford.edu . By allowing developers to easily test algorithms across the variety of datasets in W ILDS 2.0, we hope to accelerate the development of methods that can leverage unlabeled data to improve robustness to real-world distribution shifts.
Finally, we note that W ILDS 2.0 not a separate benchmark from W ILDS 1.0: the labeled data and evaluation metrics are exactly the same in W ILDS 1.0 and W ILDS 2.0, and future results should be reported on the overall W ILDS benchmark, with a note describing what kind of unlabeled data (if any) was used.
In this paper, we discuss the addition of unlabeled data and analyze the performance of methods that use the unlabeled data.
For a more detailed description of the datasets, evaluation metrics, and models used, please refer to the original W ILDS paper (Koh et al., 2021).
C OMPARISON WITH EXISTING UNSUPERVISED ADAPTATION BENCHMARKS
W ILDS 2.0 offers a diverse range of applications and modalities while also providing an extensive amount of unlabeled data that can be used as leverage for training robust models.
In this section, we brieﬂy compare with other existing ML benchmarks for unsupervised adaptation.
Images.
Evaluations of unsupervised adaptation methods for image classiﬁcation have focused on generalizing from natural photos to a range of stylized images, such as sketches and cartoons (PACS (Li et al., 2017), Ofﬁce-Home (Venkateswara et al., 2017), and DomainNet (Peng et al., 2019)), product images (Ofﬁce-31 (Saenko et al., 2010)), and synthetic renderings (VisDA (Peng et al., 2018)), though location-based shifts have also been recently explored (Dubey et al., 2021).
It is also popular to evaluate on shifts between digits datasets, such as MNIST (LeCun et al., 1998), SVHN (Netzer et al., 2011), and USPS (Hull, 1994).
In image detection and segmentation, existing adaptation benchmarks tend to focus on generalizing from synthetic to natural scenes (Ros et al., 2016;
Richter et al., 2016; Cordts et al., 2016; Hoffman et al., 2018), which can be an important tool for realistic problems but is not the focus of this work.
In contrast, W ILDS considers real-world distribution shifts, and it spans diverse modalities (satellite, microscope, agriculture, and camera trap images) and tasks (classiﬁcation, regression, detection).
Text.
Methods for unsupervised adaptation in NLP are typically evaluated on domain shifts between different textual sources, such as news articles, different categories of product reviews, Wikipedia, or social media platforms (Blitzer et al., 2007;
Mansour et al., 2009; Oren et al., 2019; Miller et al., 2020; Kamath et al., 2020; Hendrycks et al., 2020), or even more specialized sources such as legal documents (Chalkidis et al., 2020) or biomedical papers (Lee et al., 2020b; Gu et al., 2020).
Multilingual tasks can also be a setting for unsupervised adaptation (Conneau et al., 2018; Conneau &
Lample, 2019; Hu et al., 2020a; Clark et al., 2020), especially when generalizing to low-resource languages (Nekoto et al., 2020).
The W ILDS text datasets differ in that they focus on subpopulation performance, either to particular demographics in C IVIL C OMMENTS - WILDS or to tail populations in A MAZON - WILDS , rather than on adapting to a completely distinct domain.
Molecules.
While unlabeled molecules have been used for pre-training (Hu et al., 2020c; Rong et al., 2020), no standardized unsupervised adaptation benchmarks have been developed.
P ROBLEM SETTING
As in W ILDS 1.0, we study the domain shift setting where the data is drawn from domains d ∈ D .
Each domain d corresponds to a data distribution P d over ( x, y, d ) , where x is the input, y is the prediction, and all points from P d have domain d . See Koh et al.
(2021) for more details.
The domains come in four types:
[Table]
Unlabeled data
Source domains
Used for training
Can be used for training, if available Extra domains
None
Validation domains
Used for hyperparameter tuning
Target domains
Used
[Table]
evaluation
We consider several variants of the domain shift setting. In some applications, all four types of domains are disjoint (e.g., if we are training on labeled data from some hospitals but seeking to generalize to new hospitals); in others, the target domains are a subset of the source domains (e.g., if we are training on a heterogeneous dataset but seeking to measure model performance on particular demographic subpopulations).
Models are trained on labeled data from the source domains, as well as unlabeled data of one or more types of domains, depending on what is realistic for the application.
D ATASETS
W ILDS 2.0 augments 8 W ILDS datasets with curated unlabeled data.
For consistency, the labeled datasets and evaluation metrics are exactly the same as in W ILDS 1.0, which allows direct evaluations of the utility of unlabeled training data.
The labeled and unlabeled data are disjoint, e.g., the unlabeled target data is different from the labeled target data used for evaluation.
Here, we brieﬂy describe each dataset, why unlabeled data can be realistically obtained for the corresponding task, and how it might help.
In Appendix A, we provide more information on each dataset, including data provenance and details on data processing.
In general, all of the unlabeled datasets in W ILDS 2. were processed in a similar way as their corresponding labeled datasets from W ILDS 1.0.
I W ILD C AM 2020- WILDS :
Species classiﬁcation across different camera traps.
The task is to classify the animal species in a camera trap image (Beery et al., 2020).
We aim to generalize to new camera trap locations despite variations in illumination, background, and label frequencies (Beery et al., 2018).
While hundreds of thousands of camera traps are active worldwide, only a small subset of these traps have had images labeled, and the unlabeled data from the other camera traps capture diverse operating conditions that can be used to learn robust models.
In this work, we add unlabeled images from 3,215 extra camera traps also in the WCS Camera Traps dataset (Beery et al., 2020).
This expands the number of camera traps by 11 × and the number of examples by 5 × .
C AMELYON 17- WILDS : Tumor identiﬁcation across different hospitals.
The task is to classify image patches from lymph node sections as tumor or normal tissue.
We seek to generalize to new hospitals, which can differ in their patient demographics and data acquisition protocols (Veta et al., 2016; AlBadawy et al., 2018; Komura & Ishikawa, 2018; Tellez et al., 2019).
While obtaining labeled data for histopathology applications requires pain-staking annotations from expert pathologists, hospitals typically accumulate unlabeled slide images during normal operation.
These unlabeled images could be used to adapt to differences between hospitals (e.g., different staining protocols might lead to different color distributions).
We provide unlabeled patches from train and test hospitals, which expands the total number of patches by 7 . 5 × .
Both the labeled and unlabeled data are adapted from the Camelyon17 dataset (Bandi et al., 2018).
FM O W- WILDS :
Land use classiﬁcation across different regions and years.
The task is to classify the type of building or land usage in a satellite image.
Given training data from before 2013, we aim to generalize to satellite imagery taken after 2013, while maintaining high accuracy across all geographic regions.
While labeling land use requires combining map data and expert annotations, unlabeled data is available in all locations in the world through constant streams of global satellite imagery.
Prior work has shown that unlabeled satellite data can improve OOD accuracy in landcover and cropland prediction (Xie et al., 2021a) as well as aerial object and scene classiﬁcation (Reed et al., 2021).
We provide unlabeled satellite imagery across all regions from the train and test timeframes deﬁned in W ILDS , expanding the dataset by 3 . 5 × .
Both the labeled and unlabeled data are adapted from the FMoW dataset (Christie et al., 2018).
P OVERTY M AP - WILDS :
Poverty mapping across different countries.
The task is to predict a real-valued asset wealth index of the area in a satellite image.
We consider generalizing across different countries.
Like FM O W- WILDS , unlabeled satellite imagery is available globally, while labeled data is expensive to collect as it requires conducting nationally representative surveys in the ﬁeld.
Prior work on poverty prediction has used unlabeled data for entropy minimization (Jean et al., 2018) and pre-training on auxiliary tasks such as nighttime light prediction (Xie et al., 2016; Jean et al., 2016), but these studies do not study generalization to new countries.
We provide unlabeled satellite imagery from both train and test countries, expanding the dataset by 14 × .
Both the labeled and unlabeled data are adapted from Yeh et al.
G LOBAL W HEAT - WILDS : Wheat head detection across different regions.
The task is to localize wheat heads in overhead ﬁeld images.
We seek to generalize across image acquisition sessions, each of which represents a particular location, time, and sensor; these can differ in wheat genotype, wheat head appearance, growing conditions, background appearance, illumination, and acquisition protocols.
Wheat ﬁeld images contain many densely packed and overlapping instances, making labeling wheat heads in images costly, tedious and sensitive to the individual annotator.
However, hundreds of agricultural research institutes around the world collect terabytes of unlabeled ﬁeld images which could be used for training.
We add unlabeled ﬁeld images from train, test, and extra acquisition sessions, expanding the dataset by 10 × .
The labeled and unlabeled data are adapted from the Global Wheat Head Detection dataset and its underlying sources (David et al., 2020; 2021).
OGB-M OL PCBA: Molecular property prediction across different scaffolds.
The task is to predict the biological activity of small molecules represented as molecular graphs (Wu et al., 2018;
Hu et al., 2020b).
We seek to generalize to molecules with new scaffold structures.
Labels on biological activity are only available for a small portion of molecules, as they require expensive lab experiments to obtain.
However, unlabeled molecule structures are readily available in large-scale chemical databases such as PubChem (Bolton et al., 2008), and have been previously used for pretraining (Hu et al., 2020c) and semi-supervised learning (Sun et al., 2020).
We provide 5 million unlabeled molecules from source and target scaffolds, which expands the number of molecules by 12 . 5 × .
The original labeled data was curated by MoleculeNet (Wu et al., 2018) from PubChem, and we similarly extracted the unlabeled data from PubChem (Bolton et al., 2008).
C IVIL C OMMENTS - WILDS : Toxicity classiﬁcation across demographic identities.
The task is to classify whether a text comment is toxic or not.
We consider the subpopulation shift setting, where the model must classify accurately across groups of comments mentioning different demographic identities.
While labels require large-scale crowdsourcing annotations on both comment toxicity, unlabeled article comments are widely available on the internet.
We provide unannotated comments as unlabeled data, which expands the size of the dataset by 4 . 5 × .
Both the labeled and unlabeled data are adapted from Borkan et al.
A MAZON - WILDS :
Sentiment classiﬁcation across different users.
The task is to classify the star ratings of Amazon reviews.
We seek to perform consistently well across new reviewers.
While the labels (star ratings) are always available for Amazon reviews in practice, unlabeled data is a common source of leverage for sentiment classiﬁcation more generally, with prior work in domain adaptation (Blitzer & Pereira, 2007; Glorot et al., 2011) and semi-supervised learning (Dasgupta & Ng, 2009; Li et al., 2011).
We provide unlabeled reviews from test and extra reviewers, which expands the total number of reviews by 7 . 5 × .
Both the labeled and unlabeled data are adapted from the Amazon review dataset by Ni et al.
A LGORITHMS
For our evaluation, we selected representative methods from the three categories described below.
These methods exemplify current approaches to using unlabeled data to improve robustness, and they have been successful on popular domain adaptation benchmarks like DomainNet (Peng et al., 2019) and semi-supervised settings like improving ImageNet accuracy by leveraging unlabeled images from the internet (Xie et al., 2020; Caron et al., 2020).
For more details, see Appendix B.
Domain-invariant methods.
Domain-invariant methods learn feature representations that are invariant across different domains by penalizing differences between learned source and target representations (Long et al., 2015; Ganin et al., 2016; Sun & Saenko, 2016; Long et al., 2017; 2018;
Saito et al., 2018; Zhang et al., 2018; Xu et al., 2019; Zhang et al., 2019b).
We discuss these methods further in Appendix B.2.
For our experiments, we evaluate two classical methods:
• Domain-Adversarial Neural Networks (DANN) (Ganin et al., 2016) penalize representations on which an auxiliary classiﬁer can easily discriminate between source and target examples.
• Correlation Alignment (CORAL) (Sun et al., 2016; Sun & Saenko, 2016) penalizes differences between the means and covariances of the source and target feature distributions.
Self-training.
Self-training methods “pseudo-label” unlabeled examples with the model’s own predictions and then train on them as if they were labeled examples.
These methods often also use consistency regularization, which encourages the model to make consistent predictions on augmented views of unlabeled examples (Sohn et al., 2020; Xie et al., 2020; Berthelot et al., 2021).
Selftraining methods have recently been successfully applied to unsupervised adaptation (Saito et al., 2017; Berthelot et al., 2021; Zhang et al., 2021).
We include three representative algorithms:
• Pseudo-Label (Lee, 2013) dynamically generates pseudolabels and updates the model each batch.
• FixMatch (Sohn et al., 2020) adds consistency regularization on top of the Pseudo-Label algorithm.
Speciﬁcally, it generates pseudolabels on a weakly augmented view of the unlabeled data, and then minimizes the loss of the model’s prediction on a strongly augmented view.
• Noisy Student
(Xie et al., 2020) leverages weak and strong augmentations like FixMatch, but instead of dynamically generating pseudolabels for each batch, it alternates between a few teacher phases, where it generates pseudolabels, and student phases, where it trains to convergence on the (pseudo)labeled data.
Self-supervision.
Self-supervised methods learn useful representations by training on unlabeled data via auxiliary proxy tasks.
Common approaches include reconstruction tasks (Vincent et al., 2008;
Erhan et al., 2010; Devlin et al., 2019; Gidaris et al., 2018; Lewis et al., 2020), and contrastive learning (He et al., 2020; Chen et al., 2020b; Caron et al., 2020; Radford et al., 2021b), and recent work has shown that self-supervised methods can reduce dependence on spurious correlations and improve performance on domain adaptation tasks (Wang et al., 2021;
Tsai et al., 2021; Mishra et al., 2021).
We use these self-supervision methods for unsupervised adaptation by ﬁrst pre-training models on the unlabeled data, and then ﬁnetuning them on the labeled source data (Shen et al., 2021).
We evaluate popular self-supervised methods for vision and language: • SwAV (Caron et al., 2020) is a contrastive learning algorithm that maps representations to a set of clusters and then enforces similarity between cluster assignments.
• Masked language modeling (MLM) (Devlin et al., 2019) randomly masks some of the tokens from input text and trains the model to predict the missing tokens.
E XPERIMENTS
To evaluate how well existing methods can leverage unlabeled data to be robust to in-the-wild distribution shifts, we benchmarked the methods above on all applicable W ILDS 2.0 datasets.
6. S ETUP
We used the default models, labeled training and test sets, and evaluation metrics from W ILDS .
Unlabeled data.
W ILDS 2.0 contains multiple types of unlabeled data (from source, extra, validation, and/or target domains).
For simplicity, we ran experiments on a single type of unlabeled data for each dataset.
Where possible, we used unlabeled target data to allow methods to directly adapt to the target distribution; for I W ILD C AM 2020- WILDS and C IVIL C OMMENTS - WILDS , which do not have unlabeled target data, we used the extra domains instead.
All methods use exactly the same sets of labeled and unlabeled training data (except ERM, which does not use unlabeled data).
Hyperparameters.
We tuned each method on each dataset separately using random hyperparameter search.
Following W ILDS 1.0, we used the labeled out-of-distribution (OOD) validation set to select hyperparameters and for early stopping (Koh et al., 2021).
This validation set is drawn from a different distribution than both the training and the OOD test set, so tuning on it does not leak information on the test distribution.
We did not use the in-distribution (ID) validation set.
For image classiﬁcation and regression, we used both RandAugment (Cubuk et al., 2020) and Cutout (DeVries & Taylor, 2017) as data augmentation for all methods.
We did not use data augmentation for the remaining datasets.
For some datasets, we also had ground truth labels for the “unlabeled” data, which we used to run fully-labeled ERM experiments.
Overall, we ran 600+ experiments for 7,000 GPU hours on NVIDIA V100s.
See Appendix B for a discussion of which methods were applicable to which experiments
Appendix D further experimental details.
6. R ESULTS
Table 2 shows mixed results on W ILDS : most methods do not improve over standard empirical risk minimization (ERM) despite access to unlabeled data and careful hyperparameter tuning.
In contrast, these methods have been shown to perform well on prior unsupervised adaptation benchmarks; in Appendix E, we verify our implementations by showing that these methods (with the exception of CORAL) outperform ERM on the real → sketch shift in DomainNet, a standard unsupervised adaptation benchmark for object classiﬁcation (Peng et al., 2019).
Image classiﬁcation ( I W ILD C AM 2020- WILDS , C AMELYON 17- WILDS , and FM O W- WILDS ).
Data augmentation improved OOD performance on all three image classiﬁcation datasets.
The gain was the most substantial on C AMELYON 17- WILDS , where vanilla ERM achieved 70.8% accuracy, while ERM with data augmentation achieved 82.0% accuracy.
On C AMELYON 17- WILDS and FM O W- WILDS , where we had access to unlabeled target data, Noisy Student and SwAV pre-training consistently improved OOD performance and reduced variability across replicates.
However, the other methods—CORAL, DANN, Pseudo-Label, and FixMatch— underperformed ERM.
This was especially surprising for FixMatch, which performed very well on DomainNet (Appendix E).
Both FixMatch and Noisy Student use pseudo-labeling and consistency regularization, but FixMatch dynamically computes pseudo-labels in each batch from the start of training, whereas Noisy Student ﬁrst trains a teacher model to convergence on the labeled data and updates pseudolabels at a much slower rate.
As in Xie et al.
, this suggests that dynamically updating pseudo-labels might hurt generalization.
On I W ILD C AM 2020- WILDS , where we had access to 4 × as many unlabeled images from extra domains (distinct camera traps) but not to any images from the target domains, none of the benchmarked methods improved OOD performance compared to ERM.
This was surprising, as many of these methods were originally shown to work in semi-supervised settings.
One difference could be that the labeled and unlabeled examples in I W ILD C AM 2020-
WILDS differ more signiﬁcantly (as [Table] they originate from different camera traps) than in the original FixMatch paper (Sohn et al., 2020), which used i.i.d.
labeled and unlabeled data, or the Noisy Student paper (Xie et al., 2020), which used ImageNet labeled data (Russakovsky et al., 2015) and JFT unlabeled data (Hinton et al., 2015).
Fully-labeled ERM models that used ground truth labels for the “unlabeled” data were available for FM O W- WILDS and I W ILD C AM 2020- WILDS .
They signiﬁcantly outperformed other methods, suggesting room for improvement in how we leverage the unlabeled data.
Image regression (P OVERTY M AP - WILDS ).
Data augmentation had no effect on performance on
P OVERTY M AP - WILDS , which differs from the above image datasets in that it is a regression task and involves multi-spectral satellite images (with 7 channels); both of these aspects are relatively unstudied compared to standard RGB image classiﬁcation.
All applicable methods underperformed standard ERM, despite having access to unlabeled data from the target domains (countries).
Image detection (G LOBAL W HEAT - WILDS ).
We did not apply data augmentation here, as standard augmentation changes the labels (e.g., cropping the image might remove bounding boxes) and would violate the assumption that labels are invariant under augmentations, which contrastive and consistency regularization methods like SwAV, Noisy Student, and FixMatch rely on.
Accordingly, we did not evaluate FixMatch and SwAV, and we modiﬁed Noisy Student to remove data augmentation noise.
All applicable methods underperformed ERM.
Molecule classiﬁcation (OGB-M OL PCBA).
We did not apply data augmentation techniques to OGB-M OL PCBA as they are not well-developed for molecular graphs.
All methods underperformed ERM.
We did not report ID results as this dataset has no separate ID test set.
Text classiﬁcation (C IVIL C OMMENTS - WILDS , A MAZON - WILDS ).
Likewise, we did not apply data augmentation to the text datasets.
On both datasets, other methods performed similarly to ERM (with class-balancing for C IVIL C OMMENTS - WILDS ).
Continued masked LM pre-training on the unlabeled data did not improve target performance, unlike in prior work (Gururangan et al., 2020); this might be because the BERT pre-training corpus (Devlin et al., 2019;
Hendrycks et al., 2020) is more similar to the online comments in C IVIL C OMMENTS - WILDS and product reviews in A MAZON - WILDS than to the biomedical/CS text studied in Gururangan et al.
Also,
C IVIL C OMMENTS - WILDS and A MAZON - WILDS measure subpopulation performance (on minority demographics and on the tail subpopulation, respectively), whereas prior work adapted models to new areas of the input space (e.g., from news to biomedical articles).
Fully-labeled ERM models showed modest gains compared to FM O W- WILDS and I W ILD C AM
2020- WILDS .
As our evaluations on these text datasets focus on subpopulations performance, these results are consistent with prior observations that ERM models can have poor subpopulation performance even with large labeled training sets (Sagawa et al., 2020), necessitating other approaches to subpopulation shifts.
D ISCUSSION
We conclude by discussing several takeaways and promising directions for future work.
The role of data augmentation.
Many unsupervised adaptation methods rely strongly on data augmentation for consistency regularization or contrastive learning.
This reliance on data augmentation techniques—which are largely image-speciﬁc—restricts their generality, as they do not readily generalize to other modalities (or even other types of images besides photos).
Developing data augmentation techniques that can work well in other applications and modalities could be crucial for expanding the applicability of these methods (Verma et al., 2021).
Hyperparameter tuning.
Unsupervised adaptation methods have even more hyperparameters than standard supervised methods, and consistent with prior work, we found that these hyperparameters can signiﬁcantly affect OOD performance (Saito et al., 2021).
Moreover, unlike in standard i.i.d.
settings, we do not have labeled target data that we can use for hyperparameter selection.
Improved methods for hyperparameter tuning could signiﬁcantly improve OOD performance.
Such methods might make use of the unlabeled target data, or even the combination of labeled and unlabeled OOD validation data, which is provided for most datasets in W ILDS 2.0.
Pre-training on broader unlabeled data.
Pre-training on huge amounts of unlabeled data improves robustness to distribution shifts in some settings (Bommasani et al., 2021).
The unlabeled data need not be related to the task: e.g., CLIP was pre-trained on text-image pairs from the internet but tested on tasks including histopathology and satellite image classiﬁcation (Radford et al., 2021a).
Existing techniques for this type of broad pre-training appear insufﬁcient for W ILDS : many of our models were initialized with ImageNet-pretrained weights or derivatives of BERT, but do not generalize well OOD.
While we focused on providing curated unlabeled data that is closely tailored to the task, it could be fruitful to use both broad and curated unlabeled data.
Leveraging domain annotations and task-speciﬁc structure.
OOD robustness is ill-posed in general, as models cannot be robust to arbitrary distribution shifts.
Beyond unlabeled data, W ILDS also has domain annotations and other structured metadata for both labeled and unlabeled data (e.g., in I W ILD C AM 2020-
WILDS , we know which images were taken from which cameras).
Exploiting this type of ﬁne-grained domain structure for unsupervised adaptation—e.g., through multisource/multi-target domain adaptation methods (Zhao et al., 2018; Peng et al.,
2019)—could be a promising avenue for learning models that are more robust to the domain shifts in W ILDS .
E THICS STATEMENT
All W ILDS datasets are curated and adapted from public data sources, with licenses that allow for public release.
The datasets are all anonymized.
The distribution shifts in several of the W ILDS datasets deal with issues of discrimination and bias that arise in real-world applications.
For example, C IVIL C OMMENTS - WILDS studies disparate model performance across online comments that mention different demographic groups, while FM O W- WILDS and P OVERTY M AP - WILDS study countries and regions where labeled satellite data is less readily available.
As our results suggest, standard models trained on these datasets will not perform well on those subpopulations, and their learned representations might also be biased in undesirable ways (Bolukbasi et al., 2016; Caliskan et al., 2017; Garg et al., 2018; Tan & Celis, 2019; Steed & Caliskan, 2021).
We also encourage caution in interpreting positive results on these datasets, as our evaluation metrics might not encompass all relevant facets of discrimination and bias: e.g., the “ground truth” toxicity annotations in C IVIL C OMMENTS - WILDS can themselves be biased, and the particular choice of regions in FM O W- WILDS might obscure lower model performance in sub-regions.
For FM O W- WILDS and P OVERTY M AP - WILDS , surveillance and privacy issues also need to be considered.
In FM O W- WILDS , the image resolution is lower than that of other public satellite data (e.g., from Google Maps), and in P OVERTY M AP - WILDS , the location metadata is noised to protect privacy.
For a deeper discussion of the ethics of remote sensing in the context of humanitarian aid and development, we refer readers to the UNICEF report by Berman et al.
R EPRODUCIBILITY STATEMENT
All W ILDS datasets are publicly available at https://wilds.stanford.edu , together with code and scripts to replicate all of the experiments in this paper.
We also provide all trained model checkpoints and results, together with the exact hyperparameters used. In our appendices, we provide more details on the datasets and experiments:
• In Appendix A, we describe each of the updated datasets in W ILDS 2.0 and their sources of unlabeled data as well as what data processing steps were taken.
• In Appendix B, we describe the implementations of each of our benchmarked methods in detail.
In particular, we discuss any changes we made to their original implementations, either for consistency with other methods or with prior implementations of these methods.
• In Appendix C, we describe details of the data augmentations (if any) that we used across each dataset.
• In Appendix D, we describe our experimental protocol, including the hyperparameter selection procedure and hyperparameter grids for all of the methods and datasets.
• In Appendix E, we describe the details of our experiments on DomainNet. • In Appendix F, we describe the details of our fully-labeled ERM experiments.
• Finally, in Appendix G, we include an illustrative code snippet of how to use the data loaders in the W ILDS library. A UTHOR CONTRIBUTIONS
The project was initiated by Shiori Sagawa, Pang Wei Koh, and Percy Liang. Shiori Sagawa and Pang Wei Koh led the project and coordinated the activities below. Tony Lee developed the experimental infrastructure and ran the experiments.
Tony Lee, Irena Gao, Sang Michael Xie, Kendrick
Shen, Ananya Kumar, and Michihiro Yasunaga designed the evaluation framework and implemented the algorithms.
The unlabeled data loaders and corresponding dataset writeups were added by: • A MAZON - WILDS : Tony Lee • C AMELYON 17- WILDS : Tony Lee
• C IVIL C OMMENTS - WILDS :
Irena Gao • FM O W-
WILDS :
Sang Michael Xie • I W ILD C AM
2020- WILDS :
Henrik Marklund and Sara Beery • OGB-M OL PCBA: Weihua Hu • P OVERTY M AP - WILDS :
Sang Michael Xie • G LOBAL W HEAT - WILDS : Etienne David, Ian Stavness, and Wei Guo.
Tony Lee and Henrik Marklund set up the website and leaderboards.
Jure Leskovec, Kate Saenko,
Tatsunori Hashimoto, Sergey Levine, Chelsea Finn and Percy Liang provided advice on the overall project direction and experimental design and analysis throughout.
Shiori Sagawa, Pang Wei Koh, and Irena Gao drafted the paper; all authors contributed towards writing the ﬁnal paper.
A CKNOWLEDGEMENTS
We would like to thank Ashwin Ramaswami, Berton Earnshaw, Bowen Liu, Hongseok Namkoong,
Junguang Jiang, Ludwig Schmidt, Robbie Jones, Robin Jia, Ruijia Xu, and Yabin Zhang for their helpful advice.
The design of the WILDS benchmark was inspired by the Open Graph Benchmark (Hu et al., 2020b), and we are grateful to the Open Graph Benchmark team for their advice and help in setting up our benchmark.
This project was funded by an Open Philanthropy Project Award and NSF Award Grant
No. 1805310.
Shiori Sagawa was supported by the Herbert Kunzel Stanford Graduate Fellowship and the Apple Scholars in AI/ML PhD fellowship.
Sang Michael Xie was supported by a NDSEG
Graduate Fellowship.
Ananya Kumar was supported by the Rambus Corporation Stanford Graduate
Fellowship.
Weihua Hu was supported by the Funai Overseas Scholarship and the Masason Foundation Fellowship.
Michihiro Yasunaga was supported by the Microsoft Research PhD Fellowship.
Henrik Marklund was supported by the Dr. Tech.
Marcus Wallenberg Foundation for Education in
International Industrial Entrepreneurship, CIFAR, and Google.
Sara Beery was supported by an
NSF Graduate Research Fellowship and is a PIMCO Fellow in Data Science.
Jure Leskovec is a
Chan Zuckerberg Biohub investigator. Chelsea Finn is a CIFAR Fellow in the Learning in Machines and Brains Program. We also gratefully acknowledge the support of DARPA under Nos. N660011924033 (MCS); ARO under Nos.
W911NF-16-1-0342 (MURI), W911NF-16-1-0171 (DURIP); NSF under Nos.
OAC1835598 (CINES), OAC-1934578 (HDR), CCF-1918940 (Expeditions), IIS-2030477 (RAPID);
Stanford Data Science Initiative, Wu Tsai Neurosciences Institute, Chan Zuckerberg Biohub, Amazon, JPMorgan Chase, Docomo, Hitachi, JD.com, KDDI, NVIDIA, Dell, Toshiba, and UnitedHealth
Group.
A PPENDICES A Additional dataset details
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
B Algorithm details
B.1 Empirical risk minimization (ERM)
B.2 Domain-invariant methods . . .
B.3 Self-training methods . .
B.4 Self-supervision methods
C Data augmentation
D Experimental details
D.1 In-
distribution vs. out-of-distribution performance . . . . . . . . . . . . . . . . . .
D.2 Model architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D. Batch sizes and
batch normalization D. Hyperparameter tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.5 Algorithm- speciﬁc hyperparameters
D.6 Compute infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E Experiments on DomainNet
F Fully-labeled ERM experimental details
G Using the W ILDS library with unlabeled data
A A DDITIONAL DATASET DETAILS
In this appendix, we provide additional details on the unlabeled data in W ILDS 2.0.
For more context on the motivation behind each dataset, the choice of evaluation metric, and the labeled data, please refer to the original W ILDS paper (Koh et al., 2021).
A. I W ILD C AM 2020- WILDS
The I W ILD C AM 2020- WILDS dataset was adapted from the iWildCam 2020 competition dataset made up of data provided by the Wildlife Conservation Society (WCS) (Beery et al., 2020) 3 .
Camera trap images are captured by motion-triggered static cameras placed in the wild to study wildlife in a non-invasive manner.
Images are captured at high volumes – a single camera trap can capture 10K images in a month – and annotating these images requires species identiﬁcation expertise and is time-intensive.
However, there are tens of thousands of camera traps worldwide capturing images of wildlife that could be used as unlabeled training data.
For example, Wildlife Insights (Ahumada et al., 2020) now contains almost 20M camera trap images collected across the globe, but a large proportion of that data is still unlabeled.
Ideally we could capture value from those images despite the lack of available labels.
We extend I W ILD C AM 2020- WILDS with unlabeled data from a set of
WCS camera traps entirely disjoint with the labeled dataset, representative of unlabeled data from a newly-deployed sensor network.
Problem setting.
The task is to classify the species of animals in camera trap images.
The input x is an image from a camera trap, and the domain d corresponds to the camera trap that captured the image.
The target y , provided only for the labeled training images, is one of 182 classes of animals.
We seek to learn models that generalize well to new camera trap deployments, so the test data comes from domains unseen during training.
Additionally, we evaluate the in-distribution performance on held-out images from camera traps in the train set.
Data.
The data comes from multiple camera traps around the world, all provided by the Wildlife
Conservation Society (WCS).
The labeled data is the same as in Koh et al.
and the unlabeled data comprise 819,120 images from 3215 WCS camera traps not included in iWildCam 2020:
Source : 243 camera traps.
Validation (OOD): 32 camera traps.
Target (OOD): 48 camera traps.
Extra: 3215 camera traps.
The four sets of camera traps are disjoint.
The distributions of the labeled and unlabeled camera traps are very similar, except that the labeled data does not contain cameras with photos taken before
LandSat 8 data was available.
[Table]
Broader context.
There are large volumes of unlabeled natural world data that have been collected in growing repositories such as iNaturalist (Nugent, 2018), Wildlife Insights (Ahumada et al., 2020), and GBIF (Robertson et al., 2014).
This data includes images or video collected by remote sensors or community scientists, GPS track data from an-animal devices, aerial data from drones or satellites, underwater sonar, bioacoustics, and eDNA.
Methods that can harness the wealth of information in unlabeled ecological data are well-posed to make signiﬁcant breakthroughs in how we think about ecological and conservation-focused research.
Natural-world and ecological benchmarks that provide unlabeled data include NEWT (Van Horn et al., 2021), investigating efﬁcient task learning, and
Semi-Supervised iNat (Su & Maji, 2021), which provides labeled data for only a subset of the taxonomic tree.
Recent work has begun to adapt weakly-supervised and self-supervised approaches for these natural world settings, including probing the generality and efﬁcacy of self-supervision (Cole et al., 2021), incorporating domain-relevant context into self-supervision (Pantazis et al., 2021), or leveraging weak supervision from alternative data modalities (Weinstein et al., 2019) or pre-trained, generic models (Weinstein et al., 2021; Beery et al., 2019).
Active learning also plays a role here in seeking to adapt models efﬁciently to unlabeled data from novel regions with only a few targeted labels (Kellenberger et al., 2019; Norouzzadeh et al., 2021).
A. C AMELYON 17- WILDS
The C AMELYON 17- WILDS dataset (Koh et al., 2021) was adapted from the Camelyon17 dataset (Bandi et al., 2018), which is a collection of whole-slide images (WSIs) of breast cancer metastases in lymph node sections from 5 hospitals in the Netherlands.
The labels were obtained by asking expert pathologists to perform pixel-level annotations of each WSI, which is an expensive and painstaking process.
In practice, unlabeled WSIs (i.e., WSIs without pixel-level annotations) are much easier to obtain.
For example, only a fraction of the WSIs in the original Camelyon17 dataset (Bandi et al., 2018) were labeled; the other WSIs, which are taken from the same 5 hospitals, were provided without labels.
In this work, we augment the C AMELYON
17- WILDS dataset with unlabeled data from these WSIs.
Problem setting.
The task is to classify whether a histological image patch contains any tumor tissue.
We consider generalizing from a set of training hospitals to new hospitals at test time.
The input x corresponds to a 96 × 96 image patch extracted from an WSI of a lymph node section, the label y is a binary indicator of whether the central 32 × 32 patch of the input contains any pixel that was annotated as a tumor in the WSI, and the domain d identiﬁes which hospital the patch came from.
Each patch also includes metadata on which WSI it was extracted from, though we do not use this metadata for training or evaluation.
Models are evaluated by their average accuracy on a class-balanced test dataset.
Data.
All of the labeled and unlabeled data are taken from the Camelyon17 dataset (Bandi et al., 2018), which consists of WSIs from 5 hospitals (domains) in the Netherlands.
We provide unlabeled data from same domains as the labeled C AMELYON
17- WILDS dataset (no extra domains).
The domains are split as follows:
Source: Hospitals 1, 2, and 3. 2.
Validation (OOD): Hospital 4. 3.
Target (OOD): Hospital 5.
C AMELYON
17- WILDS also includes a Validation (ID) set which contains data from the training hospitals.
The C AMELYON 17- WILDS dataset has a total of 455,954 labeled patches across these splits, derived from the 10 WSIs per hospital that have full pixel-level annotations.
We augment the dataset with a total of 2,999,307 unlabeled patches, extracted from an additional 90 unlabeled WSIs per hospital.
There is no overlap between the WSIs used for the labeled versus unlabeled data.
To extract and process each patch, we followed the same data processing steps that were carried out for the labeled data in Koh et al.
Unlike the labeled patches, which were sampled in a class-balanced manner (i.e., half of the patches have positive labels), we sampled the unlabeled patches uniformly at random from the unlabeled
[Table]
WSIs.
We sampled 6,667 patches per unlabeled WSI, with the single exception of one WSI which had only 5,824 valid patches, resulting in a total of 3,000,150 unlabeled patches (Table 4).
While the labeled patches were sampled in a class-balanced manner, the underlying label distribution skews heavily negative (approximately 95% of the patches in a WSI are negative), so we expect the unlabeled patches to be similarly skewed in their label distribution.
Broader context.
We focused on providing unlabeled data from the same hospitals (domains) as in the original labeled C AMELYON
17- WILDS dataset.
This unlabeled data from the training and test hospitals can be used to develop and evaluate methods for semi-supervised learning (Peikari et al., 2018;
Akram et al., 2018;
Lu et al., 2019; Shaw et al., 2020) and domain adaptation (Ren et al., 2018; Zhang et al., 2019a; Koohbanani et al., 2021), respectively.
In practice, there is also a large amount of unlabeled data from different domains that is publicly available: for example,
The
Cancer Genome Atlas (TCGA) hosts tens of thousands of publicly-available slide images across a variety of cancer types and from many different hospitals (Weinstein et al., 2013).
These large and diverse datasets need not even be directly relevant to the task at hand, e.g., one could pretrain a model on images for different types of cancer even if the goal were to develop a model for breast cancer.
Recent work has started to explore the use of these large and diverse datasets for computational pathology applications (Ciga et al., 2020; Dehaene et al., 2020) and in other medical imaging applications (Azizi et al., 2021).
A. FM O W- WILDS
The FM O W- WILDS dataset (Koh et al., 2021) was adapted from the FMoW dataset (Christie et al., 2018), which consists of global satellite images from 2002–2018, labeled with the functional purpose of the buildings or land in the image.
The labels are collected by a process which combines map data with crowdsourced annotations (from a trusted crowd).
In contrast, unlabeled satellite imagery is readily available across the globe.
In this work, we augment the FM O W-
WILDS dataset with unused satellite images that were part of the original FMoW dataset but not in the FM O W- WILDS dataset.
Problem setting.
The task is to classify the building or land-use type of a satellite image.
We consider generalizing from images before 2013 to after 2013, as well as considering the performance on the worst-case geographic region (Africa, the Americas, Oceania, Asia, or Europe).
The input x is an RGB satellite image ( 224 × 224 pixels).
The label y is one of 62 building or land use categories.
The domain d represents both the year and the geographical region of the image.
Each image also includes metadata on the location and time of the image, although we do not use these except for splitting the domains.
Models are evaluated by their average and worst-region accuracies in the
OOD timeframe.
Data.
The labeled and unlabeled data are taken from the FMoW dataset (Christie et al., 2018).
We provide unlabeled data from same domains as the labeled FM O W- WILDS dataset (no extra domains).
The domains are as follows:
Source: Images from 2002–2013. 2.
Validation (OOD): Images from 2013–2016.
[Table]
All of these domains have disjoint locations.
FM O W- WILDS also includes Validation (ID) and
Target (ID) sets which contain data from the training domains of 2002–2013.
The FM O W- WILDS dataset has 141,696 labeled images across these splits.
We augment the dataset with 340,469 unlabeled images.
These images come from two sources:
We use a sequestered split of the dataset, which consists of new locations that are not in the original labeled FM O W- WILDS dataset; these unlabeled data are drawn from the same distribution as the labeled data.
For the unlabeled target and validation splits, we also add unlabeled data in their respective timeframes from the training set locations.
While the unlabeled data from the Validation (OOD) and Target (OOD) domains can come from the same locations as the labeled training data, we note that none of the locations in the labeled Validation (OOD) or Target (OOD) data, which is used for evaluation, is shared with any of the unlabeled or labeled data used for training.
Broader context.
We focus on providing unlabeled data from the years (domains) that were in the original FM O W- WILDS dataset.
Prior works have used unlabeled satellite imagery for pretraining (Xie et al., 2016; Jean et al., 2016;
Xie et al., 2021a;
Reed et al., 2021), self-training (Xie et al., 2021a), and semi-supervised learning (Reed et al., 2021).
Leveraging unlabeled satellite imagery is powerful since it is widely available and can reduce the frequency at which we need to re-collect labeled data.
A. P OVERTY M AP - WILDS
The P OVERTY M AP - WILDS dataset (Koh et al., 2021) was adapted from Yeh et al.
The dataset consists of satellite images from 23 African countries, labeled with a village-level realvalued asset wealth index (measure of wealth).
The labels are collected by conducting a nationally representative survey, which requires sending workers into the ﬁeld to ask each household a number of questions and can be very expensive.
In contrast, unlabeled satellite imagery is readily available across the globe.
In this work, we augment the P OVERTY M AP - WILDS dataset with satellite images from the same LandSat satellite.
Problem setting.
The task is to predict a real-valued asset wealth index from a satellite image.
We consider generalizing across country borders (the dataset contains 5 different cross validation folds, each splitting the countries differently).
The input x is a multispectral LandSat satellite image with 8 channels (resized to 224 × 224 pixels).
The output y is a real-valued asset wealth index.
The domain d represents the country the image was taken in, as well as whether the image was taken at an urban or rural area.
Each image also includes metadata on the location and time, although we do not make use of these except for deﬁning the domains.
Models are evaluated by the average Pearson correlation ( r ) across 5 folds, as well as the lower of the Pearson correlations on the urban or rural
[Table]
subpopulations to test generalization to these subpopulations.
In particular, generalization to rural subpopulations is important as poverty is more common in rural areas.
Data.
We provide unlabeled data from same domains as the labeled P OVERTY M AP - WILDS dataset (no extra domains).
The domains are split as follows:
Source: Images from training countries in the fold. 2.
Validation (OOD): Images from validation countries in the fold. 3.
Target (OOD): Images from test countries in the fold.
All the countries in these splits are disjoint.
Folds also contain a Validation (ID) and Target (ID) set with data from the training countries.
The P OVERTY M AP - WILDS dataset has 19,669 labeled images across these splits.
We augment the dataset with 261,396 unlabeled images from the same 23 countries.
These images are collected using the same process as Yeh et al.
(2020) from the same LandSat satellite.
The image locations are chosen to be roughly near survey locations from the Demographic and Health Surveys (DHS).
Broader context.
We focus on providing unlabeled data from the countries (domains) that were in the original P OVERTY M AP - WILDS dataset.
Prior works on poverty prediction have used pretraining on unlabeled data (to predict an auxiliary task such as nighttime light prediction)
(Xie et al., 2016; Jean et al., 2016) and for semi-supervised learning via entropy minimization (Jean et al., 2018).
However, these works focus on generalization to new locations in the countries in the training set.
Poverty prediction is different from usual tasks in that the output is real-valued.
Most methods for unlabeled data are made for classiﬁcation tasks, and we hope that our dataset will encourage more work on methods for using unlabeled data for improving OOD performance in regression tasks.
A. G LOBAL W HEAT - WILDS
The G LOBAL W HEAT - WILDS dataset was extended from the Global Wheat Head Dataset developed by David et al.
The goal of the dataset is to localize wheat heads from ﬁeld images to assist plant scientists to assess the density, size, and health of wheat heads in a particular wheat ﬁeld.
This imagery is acquired during different periods to cover the development of the vegetation, from the emergence to organ appearance.
Examples in G LOBAL W HEAT - WILDS are labeled by bounding box annotations of each wheat head in the image.
Wheat heads are densely packed and overlapping, making object annotation highly tedious.
Thus, the Global Wheat Head Dataset (GWHD) is relatively small, while in reality more ﬁeld images are available.
We supplement G LOBAL W HEATWILDS with unlabeled examples from the same set of ﬁeld vehicles and sensors but taken in different acquisition sessions, i.e., at different locations or the same location in a different year.
The inclusion of this unlabeled data allows: 1) a much higher spatial coverage of a ﬁeld location when the data comes from an acquisition session which is already included, 2) a much higher temporal resolution when the data comes from a location which is already included, so we have a larger range of wheat
[Table]
growth stages, and 3) slightly more diversity when the session comes from a different location, but with the same image acquisition protocol (i.e., the same ﬁeld vehicle and image sensor).
Problem setting.
The task is to localize wheat heads in high resolution overhead ﬁeld images taken from above the crop canopy.
We consider generalizing across acquisition sessions representing a particular location, time and sensor with which the images were captured.
Variation across sessions includes changes in wheat genotype, wheat head appearance, growing conditions, background appearance, illumination and acquisition protocol.
The input x is an overhead outdoor image of wheat canopy, and the label y is a set of box coordinates bounding the wheat heads (the spike at the top of the wheat plant holding grain), omitting any hair-like awns that may extend from the head.
The domain d designates an acquisition session, which corresponds to a certain location, time, and imaging sensor.
Data.
We provide unlabeled data from same domains as the labeled G LOBAL W HEAT - WILDS dataset.
Additionally, we provide unlabeled data from extra acquisition sessions not in the labeled
G LOBAL W HEAT - WILDS dataset (extra domains).
The domains are split as follows:
Source: 18 acquisition sessions in Europe (France × 13, Norway × 2, Switzerland, United Kingdom, Belgium).
Validation (OOD): 8 acquisition sessions: 7 in Asia (Japan × 4, China × 3) and 1 in Africa (Sudan).
Target (OOD): 21 acquisition sessions: 11 in Australia and 10 in North America (USA × 6, Mexico × 3, Canada).
Extra (OOD): 53 acquisition sessions distributed across the world.
The source, validation, and target sessions are split by continent, while the extra sessions are taken from across the world.
For acquisition sessions with both labeled and unlabeled data, we randomly selected new patches of 1024x1024 pixels from the original underlying data.
The images were preprocessed in the same way as described in David et al.
Broader context.
Utilizing unlabeled data is relatively new in the context of plant phenotyping, due to the lack of a large, unlabeled database of plant images.
However, larger plant image datasets are starting become available, such as from the Terraphenotying Reference Platform (TERRA-Ref,
Burnette et al.
Increasing the sample size and variation within plant datasets is an important goal, because plants from the same species are fairly self-similar within the same ﬁeld and therefore increasing the number of locations, times and image types included in a dataset can be beneﬁcial for making ﬁne-grained visual classiﬁcations for plants.
Further, for plant phenotyping to be used in farming applications, such as for precisely spraying weeds in a ﬁeld with herbicide, models must be highly robust to variations between different ﬁelds.
[Table]
997 Target
Terraref
US
Maricopa 3/20/
Gantry
F  1000 Target
UQ
Australia
Gatton 8/12/
Tractor
PF
Target
UQ
Australia
Gatton 9/8/
Tractor
PF
Target
UQ
Australia
Gatton 9/15/
Tractor
F  0 Target
UQ
Australia
Gatton 10/1/
Tractor
F  0 Target
UQ
Australia
Gatton 10/9/
Tractor
F-R
Target
[Table]
Australia
Gatton 10/14/2015 Tractor
F-R
Target
UQ
[Table]
Australia
Gatton 10/6/
Handheld
R
Target
UQ
[Table]
Australia
McAllister 10/9/
Handheld
R[-
-]
[Table]
UQ
Australia
Brookstead 10/16/2020 Handheld
F-R
Target
UQ
Australia
Gatton 9/22/
Handheld
F-R
Target
[Table]
Australia
Gatton 8/31/
Handheld
PF
Target
[Table]
Canada
Saskatoon 6/6/
Tractor
F-R
[Table]
4/15/2020 Cart
E  1000 Extra CIMMYT 11 Mexico
CiudadObregon 4/22/2020 Cart
E  1000 Extra CIMMYT 12 Mexico
CiudadObregon 4/22/2020 Cart
E  1000 Extra CIMMYT 13 Mexico
CiudadObregon 4/22/2020 Cart
E  1000 Extra CIMMYT 14 Mexico
CiudadObregon 4/22/2020 Cart
PF  1000 Extra CIMMYT 15 Mexico
CiudadObregon 4/28/
Cart
PF  1000 Extra CIMMYT 16 Mexico
CiudadObregon 5/3/
Cart
F-R
Extra ETHZ
Switzerland Eschikon 6/6/
Spidercam F  750 Extra INRAE
France
Clermont-Ferrand 5/29/2019 Handheld
F  1000 Extra KSU
US
Manhattan,KS 5/4/
Tractor
F  1000 Extra KSU
US
Manhattan,KS 4/23/2017 Tractor
P-F  1000 Extra Rres
UK
Rothamsted 7/7/
Gantry
R  1000 Extra Rres
UK
Rothamsted 7/10/2015 Gantry
F  1000 Extra Rres
UK
Rothamsted 7/13/2015 Gantry
F-R  1000 Extra Rres
UK
Rothamsted 7/20/2015 Gantry
F-R  1000 Extra ULi`ege
Belgium
Gembloux 6/11/2020 Cart
PF  1000 Extra ULi`ege
Belgium
Gembloux 6/15/2020 Cart
F  1000 Extra ULi`ege
Belgium
Gembloux 6/16/2020 Cart
F  1000 Extra ULi`ege
Belgium
Gembloux 6/18/2020 Cart
F  1000 Extra ULi`ege
[Table] [Table]
A. OGB-M OL PCBA
The OGB-M OL PCBA dataset was adapted from the Open Graph Benchmark (Hu et al., 2020b) and originally curated by the MoleculeNet (Wu et al., 2018) from the PubChem database (Bolton et al., 2008).
The dataset is a collection of molecules annotated with 128 kinds of binary labels indicating the outcome of different biological assays.
Performing biological assays is expensive, and as a result, the assay labels are only sparsely available over a tiny portion of the molecules curated in the large-scale PubChem database (Bolton et al., 2008).
On the other hand, unlabeled molecule data is abundant and readily available from the database.
Prior work in graph machine learning has leveraged unlabeled molecules to perform pre-training (Hu et al., 2020c) and semi-supervised learning (Sun et al., 2020).
In this work, we augment the OGB-M OL PCBA dataset with unlabeled molecules subsampled from the PubChem database.
Problem setting.
The task is multi-task molecule classiﬁcation, and we consider generalizing to new molecule scaffold structures at test time.
The input x corresponds to a molecular graph (where nodes are atoms and edges are chemical bounds), the label y is a 128-dimensional binary vector, representing the binary outcomes of the biological assay results.
y could contain NaN values, indicating that the corresponding biological assays were not performed on the given molecule.
The domain d indicates the scaffold group a molecule belongs to.
As the binary labels are highly-skewed, the model’s classiﬁcation performance is evaluated using the Average Precision.
Data.
All of the labeled and unlabeled data are taken from the PubChem database (Bolton et al., 2008).
We provide unlabeled data from same domains as the labeled OGB-M OL PCBA dataset (no extra domains).
We curate the unlabeled data by randomly sampling 5 million molecules from the PubChem database.
We then assign these unlabeled molecules to the existing labeled scaffold groups that contain the most similar molecules.
Speciﬁcally, we ﬁrst compute the 1024-dimensional
Morgan ﬁngerprints for all the molecules (Rogers & Hahn, 2010; Landrum et al., 2006).
Then, for each unlabeled molecule, we compute its Jaccard similarity against all the labeled molecules in
OGB-M OL PCBA and obtain a labeled molecule with the highest Jaccard similarity.
Finally, we assign the unlabeled molecule to the scaffold group that the most similar labeled molecule belongs to.
This way, the molecules within the same scaffold groups are structurally similar to each other.
The domains in the OGB-M OL PCBA dataset are as follows:
Source: 44,930 scaffold groups.
Validation (OOD): 31,361 scaffold groups.
Target (OOD): 43,793 scaffold groups.
The largest scaffolds are in the source split and the smallest scaffolds in the target split.
We assign all of the unlabeled molecules to the existing domains, so there are no extra domains added.
While the unlabeled data are similar to the labeled data in that they were all derived from PubChem (Bolton et al., 2008), it is quite possible that there was some selection bias in which molecules in PubChem were chosen to be labeled, which would lead to an undocumented distribution shift between the unlabeled and labeled datasets.
Broader context.
We focused on providing unlabeled data for both training and OOD test domains.
Unlabeled molecules can be used to develop and evaluate methods for domain adaptation, self-training, as well as pre-training (Hu et al., 2020c) and semi-supervised learning (Sun et al., 2020).
In terms of future directions, we think it is fruitful to explore both graph-agnostic methods (e.g., pseudo-label training) and more graph-speciﬁc methods (e.g., self-supervised learning of graph neural networks (Xie et al., 2021b)).
A. C IVIL C OMMENTS - WILDS
The C IVIL C OMMENTS - WILDS dataset (Koh et al., 2021) was adapted from the CivilComments dataset (Borkan et al., 2019), which is a collection of text comments made on online articles.
The data in C IVIL C OMMENTS - WILDS underwent a signiﬁcant labeling and annotation process: each example was labeled toxic or non-toxic and annotated for whether they mentioned certain demographic identities by at least 10 crowdworkers.
Such a substantial labeling and identity annotation process is expensive and time-consuming.
On the other hand, unlabeled, unannotated text comments are readily available.
For example, C IVIL C OMMENTS - WILDS only contains a subset of all data available in the original CivilComments dataset (Borkan et al., 2019), most of which Koh et al.
(2021) excluded because these examples were not annotated for mentioning identities.
In this work, we augment the
C IVIL C OMMENTS - WILDS dataset with these unlabeled, unannotated comments.
Problem setting.
The task is to classify whether a text comment is toxic or not.
The input x is a text comment (at least one sentence long) originally made on an online article, the label y is a binary indicator of whether the comment is rated toxic or not, and the domain d is an 8-dimensional binary vector, where each dimension corresponds to whether the comment mentions each of 8 demographic identities: male, female, LGBTQ, Christian, Muslim, other religions, Black , or White , respectively.
Each comment also includes metadata on which article the comment was made on, although we do not use this metadata for training or evaluation.
We consider the subpopulation shift setting, where the model must perform well across all subpopulations, which are deﬁned based on d .
Koh et al.
(2021) deﬁne 16 subpopulations (groups) based on d .
Models are then evaluated by their worst-group accuracy, i.e., the lowest accuracy over thegroups considered.
In our work, we use the same evaluation setup.
Data.
All of the labeled and unlabeled data are taken from the CivilComments dataset (Borkan et al., 2019).
After preprocessing, Koh et al.
(2021) created the C IVIL C OMMENTS - WILDS dataset using the 448,000 examples that were fully annotated for both toxicity y and the mention of demographic identities d .
In this work, we augment C IVIL C OMMENTS - WILDS with an additional 1,551,515 examples collected by Borkan et al.
We use these examples as unlabeled data.
We follow the same preprocessing steps as was done with the labeled data in Koh et al.
The resulting unlabeled examples have no identity annotations d and no toxicity label y .
We note that
Borkan et al.
(2019) actually do provide toxicity labels for these examples in the original CivilComments dataset, but we ignore these labels and use them neither for training nor evaluation.
Because our unlabeled examples have no identity annotations, we cannot group these examples as
Koh et al.
(2021) group the labeled examples; thus we refer to this data as unlabeled data coming from extra domains (Table 11).
In practice, these comments may actually mention any number of identities.
A substantial amount (1,427,848 or 92%) of the unlabeled comments are drawn from the same articles as the labeled comments.
In particular, 140,082 unlabeled comments are from the same articles as labeled comments in the test split.
C IVIL C OMMENTS - WILDS exhibits class imbalance.
We account for this when benchmarking methods by sampling class-balanced batches of labeled data when applicable (see Appendix B).
Broader context.
In this work, we focused on supplementing C IVIL C OMMENTS - WILDS with extra unannotated data from the original CivilComments dataset (Borkan et al., 2019).
In practice, unannotated text comments are widely available on the internet.
Whether using such unlabeled data, as we do in this work, can help with bias is still an open question.
Previous work suggests that training on large amounts of data alone is not sufﬁcient to avoid unwanted biases, since many
[Table]
papers have pointed out biases in large language models (Abid et et
Gehman et al., 2020).
However, recent work has also suggested that pre-trained models can be trained to be more robust against some types of spurious correlations (Hendrycks et al., 2020; Tu et al., 2020) and that additional domain- and task-speciﬁc pre-training (Gururangan et al., 2020) can also improve performance.
We hope our contributions to the C IVIL C OMMENTS - WILDS dataset can encourage future study on whether unlabeled data can be leveraged to improve generalization across subpopulation shifts.
A. A MAZON - WILDS
The A MAZON - WILDS dataset (Koh et al., 2021) was adapted from the Amazon reviews dataset (Ni et al., 2019), which is a collection of product reviews written by reviewers.
While Amazon reviews are always labeled by the star ratings in practice, unlabeled data is a common source of leverage more generally for sentiment classiﬁcation, with prior work in domain adaptation (Blitzer & Pereira, 2007; Glorot et al., 2011) and semi-supervised learning (Dasgupta & Ng, 2009; Li et al., 2011).
In this work, we augment the A MAZON - WILDS dataset with unlabeled reviews, whose star ratings have been removed.
Problem setting.
The task is sentiment classiﬁcation, and we consider generalizing from a set of reviewers to new reviewers at test time.
The input x corresponds to a review text, the label y is the star rating from 1 to 5, and the domain d identiﬁes which user wrote the review.
For each review, additional metadata (product ID, product category, review time, and summary) are also available.
Because the goal is to train a model that performs well across a wide range of reviewers, models are evaluated by their tail performance, concretely, their accuracy on the user at the 10th percentile.
Data.
All of the labeled and unlabeled data are taken from the Amazon reviews dataset (Ni et al., 2019).
We provide unlabeled data from same domains as the labeled A MAZON - WILDS dataset.
Additionally, we provide unlabeled data from extra reviewers not in the labeled A MAZON - WILDS dataset (extra domains).
The domains are split as follows:
Source: 1,252 reviewers.
Validation (OOD): 1,334 reviewers.
Target (OOD): 1,334 reviewers.
Extra (OOD): 21,694 reviewers.
The reviewers in each split are distinct, and all reviewers have at least 75 reviews.
The distributions of reviewers in each split are identical.
A MAZON - WILDS also includes Validation (ID) and Target (ID) sets which contain data from the source reviewers.
The A MAZON - WILDS dataset has a total of 539,502 labeled reviews across these splits, and we augment the dataset with a total of 3,462,668 unlabeled reviews.
For each split of the unlabeled data, we include all available reviews that are written by the reviewer.
For the Extra (OOD) split, we include all reviewers with at least 75 reviews that are not in Source, Validation (OOD), or Target (OOD) splits.
[Table]
To ﬁlter and process reviews, we followed the same data processing steps as for the labeled data in
A MAZON - WILDS (Koh et al., 2021).
Broader context.
We focused on providing unlabeled data from OOD domains, including both test and extra domains.
Unlabeled data from the test reviewers can be used to develop and evaluate methods for domain adaptation (Ren et al., 2018;
Zhang et al., 2019a; Koohbanani et al., 2021), which has been well-studied in the context of sentiment classiﬁcation (Blitzer & Pereira, 2007;
Glorot et al., 2011).
While there is limited prior work on leveraging unlabeled data from extra domains, some domain adaptation techniques can be readily adapted to leverage such unlabeled data (Ganin et al., 2016).
Finally, we focus on unlabeled data speciﬁc to the task in this work, varying only the domains, and this contrasts with the type of unlabeled data used for pre-training in
NLP, which is much larger and more diverse (Devlin et al., 2019; Brown et al., 2020).
B A LGORITHM DETAILS B. E MPIRICAL RISK MINIMIZATION (ERM)
As a baseline, we consider Empirical Risk Minimization (ERM).
ERM ignores unlabeled data and minimizes the average labeled loss.
We additionally evaluate ERM with strong data augmentation on applicable datasets, i.e., on I W ILD C AM 2020- WILDS , C AMELYON 17- WILDS , P OVERTY M APWILDS , and FM O W- WILDS (see Appendix C).
ERM with strong data augmentation learns a model h that minimizes the labeled training loss
[Equation]
where A strong is a stochastic data augmentation operation, and (cid:96) measures the prediction loss.
We use L L throughout this appendix to refer to the above labeled loss with strong augmentations (on applicable datasets).
For all dataset except C IVIL C OMMENTS - WILDS , we sample labeled batches uniformly at random.
In our experiments, we account for class imbalance in C IVIL C OMMENTS - WILDS by explicitly sampling class-balanced batches of labeled data when computing L L ( h ) .
B. D OMAIN - INVARIANT METHODS
Domain-invariant methods seek to learn feature representations that are invariant across domains.
These methods are motivated by earlier theoretical results showing that the gap between in- and outof-distribution performance depends on some measure of divergence between the source and target distributions (Ben-David et al., 2010).
To minimize this divergence, the methods described below penalize divergence between feature representations across domains, i.e., they encourage the model to produce feature representations that are similar across domains.
Consider a model
h
= g ◦ f , where the featurizer f : X → F maps the inputs to some feature space, and the head g : F → Y maps feature representations to prediction targets.
Domain-invariant methods seek to constrain f to output similar representations for labeled and unlabeled data.
In this work, we adapt all of our domain-invariant methods to use data augmentations on applicable datasets (see Appendix C), and thus the output of f on the labeled batch is
[Equation]
Similarly, the output of f on an unlabeled batch is
[Equation]
Domain-invariant methods seek to minimize some divergence ξ : F × F → R between the labeled data B L and the unlabeled data B U , where the choice of divergence depends on the speciﬁc method.
The divergence is expressed as a penalty term:
[Equation]
The ﬁnal objective is a combination of the labeled loss and penalty loss.
The balance between the two losses is controlled by hyperparameter λ , the penalty weight.
[Equation]
In our experiments, we study two classical domain-invariant methods, Correlation Alignment (CORAL)
(Sun et al., 2016; Sun & Saenko, 2016) and Domain-Adversarial Neural Networks (DANN)
(Ganin et al., 2016).
These methods are well-known and established, but their performance can be lower than that of newer domain-invariant methods that employ different penalties to encourage the source and target representations to be similar (Jiang et al., 2020; Zhang et al., 2021).
Examples of these newer methods are Joint Adaptation Networks (JAN) (Long et al., 2017), Conditional Domain Adversarial Networks (CDAN) (Long et al., 2018), Collaborative and Adversarial
Networks (CAN) (Zhang et al., 2018), and models with Adaptive Feature Norm (AFN) (Xu et al., 2019), as well as methods that minimize the Maximum Classiﬁer Discrepancy (MCD) (Saito et al., 2018) and the Margin Disparity Discrepancy (MDD) (Zhang et al., 2019b).
All of the above methods were developed for the single-source single-target setting, where the source domain is treated as a single distribution, and likewise for the target domain.
As each W ILDS 2. dataset comprises multiple source domains and multiple target domains, it is likely that methods that can leverage this additional structure could perform better.
Examples of these methods include
Multi-source Domain Adversarial Networks (MDAN)
(Zhao et al., 2018) and Moment Matching for Multi-Source Domain Adaptation (M3SDA) (Peng et al., 2019).
The DomainBed (Gulrajani &
Lopez-Paz, 2020) and WILDS (Koh et al., 2021) benchmarks also extended single-source algorithms like CORAL and DANN to take advantage of multiple source domains in the domain generalization setting, and similar extensions in the domain adaptation setting could be promising.
Correlation Alignment (CORAL).
Algorithm 1 describes CORAL, proposed by Sun et al.
(2016); Sun & Saenko (2016).
CORAL measures the divergence ξ between batches of feature representations in terms of the deviation between their ﬁrst and second order statistics.
Given a labeled batch and unlabeled batch of features B L ∈ R n L × m , B U ∈ R n U × m , deﬁne the feature means as [Equation] [Equation] (7) and covariance matrices as
[Equation]
[Equation]
We then compute the CORAL penalty as
[Equation]
We adapted our implementation from DomainBed (Gulrajani & Lopez-Paz, 2020), as done in W ILDS 1.0. We note that these implementations compute the penalty as a sum of deviations in means and covariances, whereas Sun et al.
; Sun & Saenko (2016) penalize deviations in covariances only. (Sun et al.
(2016) considers features that are normalized to zero mean.)
On applicable datasets, we also strongly augmented all labeled and unlabeled examples using A strong , whereas Sun et al.
; Sun & Saenko (2016) do not explicitly require data augmentations.
We add augmentations to allow for a fairer comparison to other methods which use augmentations.
Note that CORAL has also been adapted by Gulrajani & Lopez-Paz (2020);
Koh et al.
(2021) for domain generalization.
In particular, where the original CORAL paper deﬁnes L penalty as the divergence between just two kinds of batches (labeled and unlabeled), these works deﬁne L penalty as the divergence between many kinds of batches, where batches are grouped based on domain annotation d ( i ) .
For simplicity, we followed the original CORAL formulation and differentiate only between labeled and unlabeled batches.
We leave leveraging the domain adaptations d to future work.
A PPLICABLE DATASETS .
We run CORAL on all datasets except G LOBAL W HEAT - WILDS and
C IVIL C OMMENTS - WILDS .
We do not evaluate domain invariant methods on C IVIL C OMMENTSWILDS , since the labeled and unlabeled data are drawn from the same distribution.
We do not evaluate CORAL on G LOBAL W HEAT - WILDS because CORAL does not port straightforwardly to detection settings.
DANN.
Algorithm 2 describes DANN, proposed by Ganin et al.
DANN measures the divergence ξ between batches of feature representations using the performance of a discriminator network h d that aims to discriminate between domains.
Given a batch of features (either B L or B U ), Input: Labeled batch { ( x ( i ) L , y ( i ) L , d ( i ) L ) : i ∈ (1 , · · · , n L ) } , unlabeled batch { ( x ( i ) U , d ( i ) U ) : i ∈ (1 , · · · , n U ) } , strong augmentation function A strong , penalty weight [Equation]
R , dimension of feature representations m 1 Compute feature representations for labeled and unlabeled batches
[Equation]
2 Compute feature mean and covariances for labeled and unlabeled batches
[Equation]
3 Update model h =
g ◦ f on loss
[Equation]
this deep network h d must classify whether examples are from the labeled data or unlabeled data.
h d is optimized using a binary classiﬁcation loss
[Equation]
The loss of h d is exactly related to ξ as
[Equation]
In other words, at the same time that h d is optimized to minimize its loss L ( h d ) , the featurizer f is incentivized to minimize L penalty =
ξ ( B L , B U )
= − L ( h d ) , or maximize L ( h d ) .
See Algorithmfor details.
We adapted our implementation from the Transfer Learning Library (Jiang et al., 2020) and matched all details to the formulation given by Ganin et al.
(2016), except for two changes.
On applicable datasets, we have strongly all labeled and unlabeled examples using A strong , whereas Ganin et al.
(2016) do not explicitly require data augmentations.
We add augmentations to allow for a fairer comparison to other methods which use augmentations.
Second, where Ganin et al.
(2016) optimize f , g , and h d using the same learning rate η , we use three different learning rates η f , η g , η h d , following the implementation of the Transfer Learning Library (Jiang et al., 2020).
A PPLICABLE DATASETS .
We run DANN on all datasets except G LOBAL W HEAT - WILDS and
C IVIL C OMMENTS - WILDS .
We do not evaluate domain invariant methods on C IVIL C OMMENTSWILDS , since the labeled and unlabeled data are drawn from the same distribution.
We do not evaluate DANN on G LOBAL W HEAT - WILDS because DANN does not port straightforwardly to detection settings.
B. S ELF - TRAINING METHODS
Self-training methods leverage unlabeled data by “pseudo-labeling” unlabeled examples with the model’s own predictions and training on them as if they were labeled examples. In certain for- [Table]
f, g, h d using appropriate learning rates η
[Table] , η g , η [Table] mulations, this is equivalent to minimizing the model’s conditional entropy on the unlabeled data (Grandvalet & Bengio, 2005).
Contemporary self-training methods also often make use of consistency regularization, i.e., encouraging the model to make similar predictions on noisy/augmented versions of unlabeled examples.
Self-training methods have recently been shown to be empirically successful at unsupervised domain adaptation (Saito et al., 2017; Berthelot et al., 2021; Zhang et al., 2021).
The self-training methods we study follow this general structure: given an unlabeled example x U , these algorithms generate a pseudolabel ˜
y U
=
ψ ( x U ) , where the pseudolabel-generating function ψ :
X → Y differs between algorithms.
For classiﬁcation problems, we study algorithms that produce hard pseudolabels, which are one-hot class predictions, rather than soft pseudolabels, which are continuous distributions over the classes.
Next, algorithms deﬁne an unlabeled loss L U ( h ) for model h by minimizing the loss between pseudolabels ˜ y U and the model’s predictions.
The algorithms we consider below augment x U during training; i.e., rather than minimizing the loss between ˜ y U and the model’s prediction on x U , the algorithms below minimize the loss of predictions on A ( x U ) , where
A is a stochastic, label-preserving augmentation.
Assuming model h that outputs real-valued logits, the complete unlabeled loss is
[Equation]
This unlabeled loss is jointly optimized with the standard ERM labeled loss.
The balance between the two losses is controlled by hyperparameter λ ( t ) , which is a function of the current step t .
[Equation]
[Equation]
Pseudo-Label.
Algorithm 3 describes Pseudo-Label, proposed by Lee (2013).
In this algorithm, the model dynamically generates pseudolabels and updates each batch.
Formally, the pseudolabelgenerating function ψ is given by
[Equation]
where A strong is the strong augmentation function described in Appendix C. Pseudo-Label then computes the loss between a strongly augmented example and its associated pseudolabel.
In order to more fairly compare Pseudo-Label to FixMatch, we add on conﬁdence thresholding to the Pseudo-Label algorithm, a feature also added in the implementation of Pseudo-Label by Sohn et al.
When conﬁdence thresholding, examples on which the model has low conﬁdence have zero loss, i.e., for some threshold hyperparameter τ , the loss an example x U contributes is
[Table]
Finally, Pseudo-Label increases the balance λ ( t ) between labeled and unlabeled losses over time, initially placing 0 weight on L U ( h ) and then linearly stepping the unlabeled loss weight until it reaches the full value of hyperparameter λ at some threshold step.
We ﬁx the step at which λ ( t ) reaches its maximum value ( λ ) to be 40% of the total number of training steps, matching the implementation of Sohn et al.
This scheduling allows the algorithm to initially prioritize the labeled loss, as generated pseudolabels are mostly incorrect while the model has low accuracy.
Formally, at step t and given total number of steps T ,
[Equation]
We add augmentations to Pseudo-Label in order to allow for a fairer comparison to other methods that use augmentations.
On applicable datasets, we have strongly augmented all labeled and unlabeled examples using A strong , whereas Lee (2013) do not use any data augmentations, i.e., all instances of A strong are replaced with the identity function.
Algorithm 3: Pseudo-Label
Input: Labeled batch { ( x ( i ) L , y ( i ) L , d ( i ) L ) : i ∈ (1 , · · · , n L ) } , unlabeled batch { ( x ( i ) U , d ( i ) U ) : i ∈ (1 , · · · , n U ) } , strong augmentation function A strong , unlabeled loss weight for current step λ ( t ) ∈ R , conﬁdence threshold τ ∈ [0 , 1] 1 Generate pseudolabels ˜ y U = arg max y h ◦
A strong ( x U )
[ y ] for the unlabeled data 2 Update model h on loss n L n L (cid:88)
i = (cid:96) (cid:0)
h ◦ A strong ( x ( i ) L ) , y ( i ) L ) (cid:1) + λ ( t ) n U n U (cid:88) i = [Table]
A PPLICABLE DATASETS .
We evaluate Pseudo-Label on all datasets except P OVERTY M APWILDS , as P OVERTY M AP - WILDS is a regression dataset, and hard pseudolabeling does not port straightforwardly to regression tasks.
FixMatch.
Algorithm 4 describes FixMatch, proposed by Sohn et al.
Like Pseudo-Label, this algorithm dynamically generates pseudolabels and updates each batch.
FixMatch additionally employs consistency regularization on the unlabeled data.
While pseudolabels are generated on a weakly augmented view of the unlabeled examples, the loss is computed with respect to predictions on a strongly augmented view.
This encourages models to make predictions on a strongly augmented example consistent with its prediction on the same example when weakly augmented.
For details about the strong versus weak augmentations we use, see Appendix C.
Formally, the pseudolabel-generating function ψ is given by
[Equation]
Like Pseudo-Label, FixMatch uses conﬁdence thresholding, and unlabeled examples on which the model has low conﬁdence have zero loss.
Following Sohn et al.
(2020), we keep the balance between labeled and unlabeled losses constant at λ ( t )
= λ .
FixMatch’s original authors justify keeping λ ( t ) at a ﬁxed magnitude (as opposed to slowly increasing λ ( t ) as in Pseudo-Label) by noting that most predictions made by FixMatch are initially low conﬁdence, so for sufﬁciently high conﬁdence threshold τ , most unlabeled examples have loss zero, keeping the magnitude of L U ( h ) initially small.
This magnitude grows over time, providing a natural curriculum (Sohn et al., 2020).
We endeavored to match our implementation of FixMatch to the formulation of Sohn et al.
(2020), except in the use of augmentations for labeled data.
Where we have strongly augmented all labeled examples using A strong in Algorithm 4, Sohn et al.
(2020) explicitly choose to use weak instead of strong augmentations on the labeled examples.
However, our results on DomainNet in Appendix E suggest that using strong instead of weak augmentations for the labeled examples improves performance, so we use strong augmentations on the labeled examples in order to allow for a fairer comparison to other methods.
Algorithm [Table]
function A [Table] , unlabeled loss
[Table] R , conﬁdence threshold τ ∈ [0 , 1] 1 Generate pseudolabels ˜ y
U = arg max
y h ◦
A weak ( x U )
[ y ] for the unlabeled data 2 Update model h on loss n
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
n [Equation] [Table] A PPLICABLE
DATASETS .
We evaluate
FixMatch on image classiﬁcation datasets
I W ILD C AM 2020- WILDS , C AMELYON 17- WILDS , P OVERTY M AP - WILDS , and FM O W- WILDS .
We do not evaluate FixMatch on other datasets because FixMatch relies on enforcing consistency across data augmentations, which we only deﬁne for image datasets (see Appendix C).
Noisy Student.
Algorithm 5 describes Noisy Student, proposed by Xie et al.
Unlike
Pseudo-Label and FixMatch, which update the model and re-generate new pseudolabels each batch,
Noisy Student generates pseudolabels, ﬁxes them, and then trains the model until convergence before generating new pseudolabels.
First, an initial teacher model is trained on the labeled data; next, the teacher model pseudolabels the unlabeled data, and a student model is trained on the labeled and pseudolabeled data; ﬁnally, the student model becomes the new teacher, and the cycle repeats (see Algorithm 5).
Each (teacher, student) pair is termed an iteration ; we study the results of two iterations.
We train Noisy Student using hard pseudolabels, which the teacher generates over weakly augmented inputs:
[Equation]
While the teacher generates pseudolabels on a weakly augmented data, the student must make both labeled and unlabeled predictions on noisy (i.e., strongly augmented) data.
Following Xie et al.
(2020), we add a dropout layer ( p = 0 . 5 ) before the student’s last layer, randomly corrupting ﬁnal feature maps.
Students thus are trained to be consistent across both data-based and model-based noise.
We denote the model with inserted dropout as Dropout ◦ f .
Xie et al.
(2020) add even more model-based noise using stochastic depth; for simplicity, we do not use stochastic depth in our implementation.
We follow the original paper and ﬁx the balance between labeled and unlabeled losses as λ ( t ) = 1 .
Noisy Student does not use conﬁdence thresholding.
Note that Xie et al.
(2020) use both dropout and strong data augmentations when training the initial teacher on labeled data.
We reuse models from our ERM + Data Augmentation experiments as initial teacher models; thus we differ from Xie et al.
(2020) in that our initial teachers were trained with strong augmentations, but not dropout (see Algorithm 5).
A PPLICABLE DATASETS .
We evaluate Noisy Student on all datasets except text datasets C IVIL C OMMENTS - WILDS and A MAZON - WILDS . For G LOBAL W HEAT - WILDS and OGBM OL PCBA, we run Noisy Student without noise from data augmentations.
Input: Labeled dataset { ( x L , y L , d L ) } divided into batches of size n L , unlabeled dataset { ( x U , d U ) } divided into batches of size n U , total number of iterations S , weak augmentation function
A weak , strong augmentation function A strong 1 Train an initial teacher model f [0] to convergence on labeled examples using the following batch-wise objective n L n L (cid:88)
i = (cid:96) (cid:0) h ◦
A strong ( x ( i ) L ) , y ( i ) L ) (cid:1) for iteration s ∈ (1 , · · · , S ) do
Generate ﬁxed pseudolabels ˜ y
U = arg max y f
[ s − 1] ◦ A weak ( x U ) for the unlabeled data
Train the next student model f [ s ] to convergence on unlabeled and labeled examples using the following batch-wise objective n L n L (cid:88) i
= (cid:96) (cid:0) Dropout ◦ h ◦
A strong ( x ( i ) L ) , y ( i ) L ) (cid:1) + 1 n
U n U (cid:88) i = (cid:96) (cid:0) Dropout ◦ h ◦ A strong ( x ( i ) U ) , ˜ y ( i ) U ) (cid:1)
B. S ELF - SUPERVISION METHODS
Self-supervised methods learn useful representations by training on unlabeled data via auxiliary “proxy” tasks.
Common approaches include reconstruction tasks (Vincent et al., 2008;
Erhan et al., 2010; Devlin et al., 2019; Gidaris et al., 2018; Lewis et al., 2020), which remove or corrupt a small part of each training example and use it as a prediction goal, and contrastive learning (He et al., 2020;
Chen et al., 2020b; Caron et al., 2020; Radford et al., 2021b), which aims to learn a representation space such that similar example pairs stay close to each other while dissimilar ones are far apart.
The underlying assumption is that feature encoders that solve the proxy tasks will also perform well on the downstream supervised task (Lee et al., 2020a; Wei et al., 2021).
In our work, we consider two self-supervised methods: SwAV Caron et al.
(2020) for images and masked language modeling (Devlin et al., 2019) for text.
We use these methods to pre-train models on the unlabeled data.
In all cases, we start with the same model initialization used for all of the other algorithms on that dataset; do additional pre-training via self-supervision on the unlabeled data; and then initialize a new classiﬁer head and ﬁnetune the model via ERM with data augmentation.
This follows the procedure in Shen et al.
As a concrete example, for FM O W- WILDS , we use the following procedure to run our ERM experiments:
Initialize a DenseNet-121 model (Huang et al., 2017) using ImageNet-pretrained weights.
Finetune the model on labeled data from the source domain.
Evaluate on held-out data from the target domain.
For SwAV, we use the exact same procedure but with the addition of a second step:
Initialize a DenseNet-121 model (Huang et al., 2017) using ImageNet-pretrained weights.
Continue pre-training the model with SwAV on unlabeled data from the target domain.
Finetune the model on labeled data from the source domain. 4.
Evaluate on held-out data from the target domain.
Similarly, for text datasets, we initialized pre-trained BERT models and then continued pre-training them using masked language modeling on the unlabeled data in W ILDS 2.0.
We tuned hyperparameters for ﬁnetuning, following the exact same procedure and hyperparameters as for ERM, but not for pre-training.
SwAV.
We directly use the public SwAV repository available at https://github.com/ facebookresearch/swav .
We keep almost all of the hyperparameters used by the original paper for 400 epoch training with batch size 256.
However, we make the following changes based on issues and tips from the original authors in the SwAV repository:
To stabilize training, we opt not to use a queue; this follows the suggestion in issue #69.
For each dataset, we set the number of prototypes to approximately 10x the number of classes; this follows the suggestion in issue #37.
For P OVERTY M AP - WILDS , which is a regression problem, we use 1000 prototypes, which displayed more stable training thanor 100 prototypes.
We set (cid:15) = 0 .
03 to avoid representation collapse; this follows the suggestion in the Common Issues section of the repository’s readme.
We set the base learning rate via the suggested “linear scaling” rule (issue #37).
In other words, for total batch size (over GPUs) ≥ 512, the learning rate is scaled linearly.
For smaller batch sizes ( < 512), we set the base learning rate at 0.6.
We multiply the base learning rate by 1000 × to obtain the ﬁnal learning rate, since each of the base/ﬁnal pairs that the paper reports differ by that factor.
We set the maximum number of epochs to 400 but stop pre-training early when the loss does not decrease by more than 0. % for consecutive epochs
A PPLICABLE DATASETS .
We evaluate SwAV on I W ILD C AM
2020- WILDS , C AMELYON 17WILDS , P OVERTY M AP - WILDS , and FM O W- WILDS .
We do not evaluate SwAV on other datasets because SwAV relies training with data augmentations, which we only deﬁne for image datasets (see
Appendix C).
Masked language modeling (MLM).
MLM is a popular self-supervised objective for text data and is commonly used to pre-train model representations (Devlin et al., 2019).
Given an unlabeled text corpus X = { X } i
(e.g., a set of comments for CivilComments; a set of reviews for Amazon), a training example ( x, y ) can be generated by randomly masking tokens in each text piece X
(e.g., x = “ The [MASK] is the currency [MASK] the UK ”;
y = (“ pound ” , “ of ”)).
The model is trained to use its representation of the masked input x to predict the original tokens y that should go in each mask.
The MLM objective encourages the model to learn syntactic and semantic knowledge (e.g., to predict “of”) as well as world knowledge
(e.g., to predict “pound”) present in the text corpus (Guu et al., 2020).
For our implementation, we use DistilBERT (Sanh et al., 2019) as our initial model and pre-train it with the MLM objective on the unlabeled data of each task (CivilComments, Amazon).
Following the original BERT implementation (Devlin et al., 2019), we randomly mask 15% of the tokens in each input text piece, of which 80% are replaced with [MASK] , 10% are replaced with a random token (according to the unigram distribution), and 10% are kept unchanged.
A PPLICABLE DATASETS .
We evaluate masked language modeling on the text datasets
C IVIL C OMMENTS - WILDS and A MAZON - WILDS .
C D ATA AUGMENTATION
In this work, several methods we study leverage data augmentations to encourage generalization across domains.
Below, we provide details on our implementations of these augmentations.
Image classiﬁcation ( I W ILD C AM 2020- WILDS , C AMELYON 17- WILDS , and FM O W- WILDS ).
We use a consistent set of data augmentations across image datasets I W ILD C AM 2020- WILDS ,
C AMELYON 17- WILDS , and FM O W- WILDS .
For methods other than SwAV, we deﬁne two strengths of data augmentations: a weak function A weak and a strong function A strong , and we specify both according to Sohn et al.
The weak augmentation function A weak is a random horizontal ﬂip.
The strong augmentation function A strong is a composition of (i) random horizontal ﬂip, (ii)
RandAugment (Cubuk et al., 2020), and (iii) Cutout (DeVries & Taylor, 2017).
For the exact implementation of RandAugment, we directly use the implementation of Zhang et al.
(2021), which is based on the implementation used by Sohn et al.
This implementation speciﬁes a pool of operations and sample magnitudes for each operation uniformly across a pre-speciﬁed range.
The pool of operations includes: autocontrast, brightness, color jitter, contrast, equalize, posterize, rotation, sharpness, horizontal and vertical shearing, solarize, and horizontal and vertical translations.
We apply N = 2 random operations for all experiments (see Appendix D.4).
The labeled loss for all methods, including ﬁnetuning models pre-trained with SwAV, uses this strong augmentation function.
For SwAV pre-training, we use the data augmentation pipeline used in the original paper (Caron et al., 2020), which is almost identical to the strong data augmentation introduced in SimCLR (Chen et al., 2020a) but with different random crop scales to accommodate the several additional lowerresolution crops.
For each image, the pipeline is the following sequence of random transformations: resized crop, horizontal ﬂip, color jitter, grayscale, and Gaussian blur.
P OVERTY M AP - WILDS .
As P OVERTY M AP - WILDS is a dataset of multispectral images, we deﬁne a separate set of data augmentations.
For methods other than SwAV, we deﬁne two strengths of data augmentations: a weak function A weak and a strong function A strong .
The weak augmentation function A weak is a random horizontal ﬂip.
The strong augmentation function A strong is a composition of (i) random horizontal ﬂip, (ii) random afﬁne transformation, (iii) color jitter on the RGB channels, and (iv) Cutout on all channels (DeVries & Taylor, 2017).
We use the same augmentations for SwAV pre-training as above for I W ILD C AM 2020- WILDS ,
C AMELYON 17- WILDS , and FM O W- WILDS , but note that the color jitter module is applied only to the RGB channels.
Other datasets.
We do not deﬁne data augmentations for other datasets, i.e., G LOBAL W HEATWILDS , OGB-M OL PCBA, C IVIL C OMMENTS - WILDS , and A MAZON - WILDS .
Although
G LOBAL W HEAT - WILDS is an image dataset and can be transformed using augmentations deﬁned above, we omit data augmentations for simplicity, because such augmentations would generally require changing y as well as x (e.g., random translations on the input image also require translating the bounding box labels).
For OGB-M OL PCBA, we omit augmentations because data augmentations on graphs are not well developed.
C IVIL C OMMENTS - WILDS and A MAZON - WILDS are text datasets; although data augmentations have been proposed for text datasets, we do not use these augmentations because training with augmentations is not as standard on text datasets as on image datasets.
For these datasets, methods are benchmarked without augmentations, i.e. we substitute all occurrences of A weak , A strong with the identity.
D E XPERIMENTAL DETAILS D. I N - DISTRIBUTION VS . OUT - OF - DISTRIBUTION PERFORMANCE
We report both in-distribution and out-of-distribution performance metrics on all datasets, with the exception of OGB-M OL PCBA, which does not have a separate in-distribution test set.
Using the terminology in W ILDS (Koh et al., 2021), we consider the train-to-train in-distribution comparison on I W ILD C AM 2020- WILDS , C AMELYON 17- WILDS , FM O W- WILDS , and P OVERTY M AP - WILDS , and the average comparison on C IVIL C OMMENTS - WILDS and A MAZON - WILDS .
D. M ODEL ARCHITECTURES
For all experiments, we use the same models for each dataset as in W ILDS 1.0:
• I W ILD C AM
2020- WILDS :
ResNet-50 (He et al., 2016).
• C AMELYON 17- WILDS : DenseNet-121 (Huang et al., 2017).
• FM O W- WILDS :
DenseNet-121 (Huang et al., 2017).
• P OVERTY M AP - WILDS : Multi-spectral ResNet-18 (Yeh et al., 2020).
• G LOBAL W HEAT - WILDS :
Faster-RCNN (Ren et al., 2015).
• OGB-M OL PCBA: Graph Isomorphism Network
(Xu et al., 2018).
• C IVIL C OMMENTS - WILDS : DistilBERT (Sanh et al., 2019).
• A MAZON - WILDS : DistilBERT (Sanh et al., 2019).
The models for I W ILD C AM
- WILDS
, FM O W
WILDS , and G LOBAL W HEAT - WILDS were initialized with weights pre-trained on ImageNet.
Note that models for C AMELYON
17- WILDS were not initialized with ImageNet weights.
The DistilBERT models were also initialized with pre-trained weights from the Transformers library.
D. B ATCH SIZES AND BATCH NORMALIZATION
For each dataset, we set the total batch size (where a batch contains both labeled and unlabeled data) to the maximum that can ﬁt on 12GB of GPU memory (Table 13).
For all the methods that leverage unlabeled data, except the pre-training algorithms, we run with 4 steps of gradient accumulation, resulting in a 4 × larger effective batch size.
For SwAV pre-training, we run with 4 GPUs in parallel, which achieves a similar effect.
For masked LM pre-training, we run with the default setting ofsteps of gradient accumulation.
These larger batch sizes deviate from the defaults used in the W ILDS paper (Koh et al., 2021).
We use these larger batch sizes because methods that leverage unlabeled data tend to use larger batch sizes (Sohn et al., 2020; Xie et al., 2020; Caron et al., 2020). [Table] [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
For models that use batch normalization, the composition of each batch affects the way in which batch normalization is applied.
For CORAL, DANN, and Pseudo-Label, we concatenate the labeled and unlabeled data together in each batch, so the labeled and unlabeled data are jointly normalized.
[Table]
D. H YPERPARAMETER TUNING
We tune each algorithm separately for each dataset by randomly sampling 10 different hyperparameter conﬁgurations within the ranges deﬁned below.
We early stop and select the best hyperparameters based on the OOD validation performance, which is computed on the labeled Validation (OOD) data for each dataset; we do not use the labeled Validation (ID) data in our experiments.
We then run replicates using the best hyperparameters.
For computational reasons, we do not tune hyperparameters for the pre-training algorithms, though we tune the ﬁnetuning of their resulting pre-trained models as usual.
Learning rates.
For all the datasets, except for OGB-M OL PCBA, we multiply the learning rates used in W ILDS by the ratio of the effective batch size to the original batch size used in W ILDS 1.0.
We center the learning rate grid around this modiﬁed learning rate r , and search over r · 10 U ( − 1 , 1) , where U is the uniform distribution.
For OGB-M OL PCBA, we pick r by multiplying the original learning rate by a factor of 10 instead of 4096 / 32 = 128 (for ERM, which does not have grandient accumulation), because we found that the latter led to unstable optimization.
L 2 -regularization.
Across all datasets and methods, we used the same L 2 -regularization strengths used in W ILDS 1.0.
Ratio of unlabeled to labeled data in a batch.
For all the domain-invariant and self-training methods, we search over the ratio of unlabeled to labeled data in a batch, using the values { 3 : 1 , 7 : 1 , 15 : 1 } .
Number of epochs.
We deﬁned an epoch as a complete pass over the labeled data.
This means that the number of batches / gradient steps taken per epoch varies with the ratio of unlabeled to labeled data in a batch, as a higher ratio means that each batch contains fewer labeled examples.
We adjusted the number of epochs accordingly so that the total amount of compute was similar regardless of the ratio of unlabeled to labeled data in a batch.
We allocated roughly twice as much compute (i.e., processing twice as many batches) to methods that used unlabeled data, compared to the purely-supervised ERM baseline.
Overall, we set the number of epochs based on the W ILDS 1. defaults, with some upwards adjustments (due to the different batch sizes and the use of unlabeled data) if we found that the best hyperparameter conﬁguration had not converged on the validation set.
Table 14 shows the total number of epochs used per dataset.
D. A LGORITHM - SPECIFIC HYPERPARAMETERS
We tuned the following algorithm-speciﬁc hyperparameters:
CORAL.
We searched over penalty weights 10 U ( − 1 , 1) .
DANN.
We searched over penalty weights 10 U ( − 1 , 1) and have separate learning rates for the featurizer, classiﬁer and domain discriminator.
We tuned the learning rate for the classiﬁer and domain discriminator, then ﬁxed the learning rate of the featurizer to be a tenth of the learning rate of the classiﬁer.
Pseudo-Label, FixMatch, and Noisy Student.
We ﬁxed the penalty weight to be 1.
For FixMatch and Pseudo-Label, we searched over conﬁdence thresholds U (0 . 7 , 0 . 95) .
Noisy Student does not use a conﬁdence threshold.
SwAV.
We did not tune SwAV hyperparameters.
See Appendix B.4 for a description of the default hyperparameters used.
Masked language modeling.
We did not tune masked LM hyperparameters, opting instead to use default hyperparameters.
For both C IVIL C OMMENTS - WILDS and A MAZON - WILDS , we pre-trained
DistilBERT for 1,000 steps with a learning rate of 10 − 4 and a batch size of 8,192 sequences using gradient accumulation.
Following W ILDS defaults, for CivilComments, we set the max sequence length to be 300 and for Amazon, 512.
We used FP16 training to speed up pretraining.
D. C OMPUTE INFRASTRUCTURE
We ran experiments on a mix of NVIDIA GPUs: V100, K80, GeForce RTX, Titan RTX, Titan Xp, and Titan V. SwAV pre-training took approximately 3 days × 4 V100 GPUs for each dataset, while masked LM pre-training took approximately 3 days for a single GPU for each dataset.
The other algorithms took less than a day on a V100 to run.
The runtime estimates in Section 6 are estimated for V100 GPUs.
We used the Weights and Biases platform (Biewald, 2020) to monitor experiments.
E E XPERIMENTS ON D OMAIN N ET
Prior work has shown that domain-invariant, self-training, and self-supervised methods can perform well on standard benchmarks for unsupervised domain adaptation.
In this section, we describe our experiments on DomainNet (Peng et al., 2019), a standard unsupervised domain adaptation benchmark for object recognition.
Our goal was to verify that our training/tuning protocol and our implementations of the methods we benchmark in Section 6—which differ slightly from prior work in the ways described in Section B—still result in models that can perform well on DomainNet.
Consistent with prior work, the methods we benchmark in Section 6, with the exception of CORAL, all improve over standard ERM training in our DomainNet experiments.
E. S ETUP
DomainNet is an object recognition dataset with approximately 600,000 images across six different domains: sketch , real , quickdraw , painting , infograph and clipart (Peng et al., 2019).
Typically, one of these domains is selected as the source, and another domain as the target for evaluation.
In our experiments, we use the real → sketch setting for two reasons: it is a common choice in prior work on DomainNet, and as our models are pre-trained on ImageNet (following (Zhang et al., 2021)), we wanted to use the real domain as the source to be consistent with the realistic photographs used for ImageNet pretraining.
While it is common to evaluate methods on multiple pairs of source and target domains in DomainNet, in our experiments we only chose one pair, as our goal was only to verify consistency with prior results.
Data.
The DomainNet dataset includes train and test splits for each of the domains, with 70%– 30% split between train and test examples.
The real domain has 172,947 images total: 120, images in the train split and 52,041 images in the test split.
The target domain, sketch , has a total of 69,128 images: 48,212 in the train split and 20,916 images are in the test split.
We used this data in the following way:
For training , we used the source training examples (with labels) and the target training examples (without labels).
For validation , we used the same set of target training examples, but with labels; this overestimates performance in a true domain adaptation setting (where one would not have labeled target data), but it is a common practice in the literature, and we followed it for consistency with Jiang et al.
(2020) and Zhang et al.
For evaluation , we used the source test examples as the in-distribution test set, and the target test examples as the out-of-distribution test set.
Hyperparameters and other details.
Other experimental details followed our main experiments in Section 6.
We used the strong and weak data augmentation described for image classiﬁcation in Appendix C. We set the total batch size to 96, which is the maximum that can ﬁt on 12GB of
GPU memory.
We tuned hyperparameters with the protocol described in Section D.4.
Speciﬁcally, for all methods, we ﬁxed L 2 -regularization at 10 − 4 .
We then randomly sampled learning rates from 10 U ( − 4 , − 2) to train the ERM with data augmentation model.
For all other models, we took the best learning rate that we found for the ERM with data augmentation model and searched over one order of magnitude lower and higher from it.
As in Zhang et al.
(2021), we used a ResNet50 model initialized by pretraining on ImageNet.
For SwAV pre-training, instead of following the early-stopping procedure in Appendix B.4, we trained for the full 400 epochs used in Caron et al.
(2020) since the experiment ﬁnished relatively quickly compared to the larger W ILDS 2.0 datasets.
E. R ESULTS
Table 15 shows the results of our experiments on real → sketch .
The use of (strong) data augmentation improved ERM performance from 34.9% to 35.9%.
All unsupervised adaptation methods except CORAL improved over ERM.
We also tested the use of strong vs. weak augmentation for labeled examples for both Pseudo-Label and FixMatch, and we found that using strong augmentation for the labeled examples improves performance.
[Table]
For DANN, Pseudo-Label, and FixMatch, we compared our results against the results reported in
Zhang et al.
Performance was similar for DANN (ours, 39.4%; theirs, 40.0%).
For FixMatch, our implementation performs better (ours, 50.2%; theirs, 45.3%); this is partially due to our use of strong instead of weak augmentation for the labeled data, which increases performance by 0.9%.
For Pseudo-Label, our implementation performs worse (ours, 36.1%; theirs, 40.6%), and we believe it is due to variation in hyperparameter tuning.
For Noisy Student, Berthelot et al.
(2021) reported signiﬁcantly lower numbers (ours, 39.7%; theirs, 32.6%).
However, this is expected as they trained their models from scratch, whereas we used
ImageNet-pretrained models.
We were unable to ﬁnd comparable results in prior work for CORAL and SwAV pretraining on the real → sketch split.
Prior work has shown that these methods can improve performance on other unsupervised adaptation datasets (Sun & Saenko, 2016;
Shen et al., 2021).
On our DomainNet experiments, we found that SwAV pre-training did improve performance over ERM, though CORAL did not (Table 15).
F F ULLY - LABELED ERM EXPERIMENTAL DETAILS
The self-training methods we evaluate in Section 5 generate a pseudolabel ˜
y U for each unlabeled example x U
and then train on ( x U , ˜ y U ) as if the pseudolabels were true labels.
However, these pseudolabels may not be accurate.
In this section, we describe how we ran fully-labeled ERM experiments using ground truth labels on the “unlabeled” data to establish informal upper bounds on how well we might expect a standard self-training approach to perform with perfect pseudolabel accuracy.
For four of our datasets (A MAZON - WILDS , C IVIL C OMMENTS - WILDS , I W ILD C AM 2020- WILDS , and FM O W- WILDS ), we curated the “unlabeled” data by taking labeled examples and discarding the ground truth labels.
For example, all 268,761 of the unlabeled target reviews in A MAZONWILDS actually have associated star ratings; these are available in our data loaders, but in our main experiments we treat these reviews as unlabeled by not loading the star ratings.
We evaluated models trained via empirical risk minimization (ERM) on the combination of the standard labeled training set and the unlabeled data with these hidden labels revealed.
For example, in A MAZON - WILDS , we pool together the labeled source examples as well as the unlabeled target examples with ground truth labels, and evaluate ERM models trained on all of that data.
As with all of the other experiments in this paper, we evaluate test performance for all datasets on the labeled target splits, so at no point are we training on our actual test examples.
F. H YPERPARAMETERS
Pooling labeled and unlabeled data.
For all datasets, we pooled labeled source examples with examples from the same “unlabeled” split as in our main experiments (Table 2).
We computed gradients for labeled minibatches and unlabeled minibatches separately, which means that for models using batch normalization, the labeled and unlabeled data were normalized separately.
However, we ﬁxed the labeled to “unlabeled” batch size ratios to match the overall labeled to unlabeled dataset size ratio, so other than the batch normalization effects, the training procedure can be viewed as running ERM on the pooled labeled and “unlabeled” data.
Number of epochs.
With the exception of I W ILD C AM 2020- WILDS , detailed below, we followed the procedure in Appendix D.4 to adjust the number of epochs based on the labeled to unlabeled batch size ratios.
This resulted in a similar amount of computation allocated to these fully-labeled
ERM experiments as the other experiments in Table 2.
Other details.
Other experimental details were kept similar to the other experiments in the paper.
Speciﬁcally, we tuned each experiment by randomly sampling 10 different hyperparameters within the ranges deﬁned in Appendix D.4; the only hyperparameter we tuned in these experiments was the learning rate.
We early stopped and selected the best hyperparameters based on the OOD validation performance, and then ran replicates using the best hyperparameters.
We also used data augmentation for I W ILD C AM 2020- WILDS and
FM O W- WILDS
but not for A MAZON - WILDS and
C IVIL C OMMENTS - WILDS .
F. D ATASET - SPECIFIC DETAILS
A MAZON - WILDS .
We matched the experiments in Table 2 by training on the unlabeled target data (268,761 examples).
In addition, we ran a separate experiment where we trained on the unlabeled extra data instead of the unlabeled target data, as the former has 10 × the number of examples (2,927,841 examples).
However, this did not improve performance.
Using the unlabeled target data, we obtained an average accuracy of 73.6 ( ± 0.1) and a 10th percentile accuracy of 56.4 ( ± 0.8), whereas using the unlabeled extra data, we obtained an average accuracy of 73.1 ( ± 0.1) and a 10th percentile accuracy of 54.7 ( ± 0.0).
C IVIL C OMMENTS - WILDS .
We used the unlabeled extra split (1,551,515 examples).
As in our other experiments on C IVIL C OMMENTS - WILDS , we accounted for label imbalance by sampling class-balanced labeled and “unlabeled” batches during training.
I W ILD C AM 2020- WILDS .
We used the unlabeled extra split.
Out of the 819,120 unlabeled extra examples, 108,452 examples have ground truth labels (animal species) that are not present in the labeled training and test sets, so we omitted those examples and trained on the remaining 710, examples.
We found that we required twice as many epochs compared to the other unlabeled methods for the fully-labeled ERM training to converge, so we doubled the amount of compute allocated to the fully-labeled I W ILD C AM 2020- WILDS experiments.
FM O W- WILDS .
We used the unlabeled target split (173,208 examples).
G U SING THE W ILDS LIBRARY WITH UNLABELED DATA
We have extended the existing W ILDS library (Koh et al., 2021) to add data loaders for each of thedatasets with unlabeled data.
These data loaders are compatible with the W ILDS 1.0 APIs, allowing the unlabeled data to be accessed in a similar way to the labeled data:
[Figure]
As in the existing W ILDS library, data downloading is automated.
In addition, we implemented CORAL, DANN, Pseudo-Label, FixMatch, and Noisy Student using the existing W ILDS interfaces.
This allows developers to easily extend these algorithms and evaluate them in a standardized way on all of the W ILDS datasets with unlabeled data.
The W ILDS repository also contains scripts for masked language model pre-training and for SwAV pre-training, which uses a modiﬁed version of the public SwAV repository that can interface with the W ILDS data loaders.