E XTENDING THE WILDS B ENCHMARK FOR U NSUPERVISED A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Machine learning systems deployed in the wild are often trained on a source distribution but deployed on a different target distribution.
Unlabeled data can be a powerful point of leverage for mitigating these distribution shifts, as it is frequently much more available than labeled data.
However, existing distribution shift benchmarks for unlabeled data do not reflect the breadth of scenarios that arise in real-world applications.
In this work, we introduce U-W ILDS , which augments the W ILDS benchmark of in-the-wild distribution shifts with curated unlabeled data that would be realistically obtainable in deployment.
U-W ILDS contains 8 datasets spanning a wide range of applications (from histology to wildlife conservation), tasks (classification, regression, and detection), and modalities (photos, satellite images, microscope slides, text, molecular graphs).
We systematically benchmark contemporary methods that leverage unlabeled data, including domain-invariant, self-training, and self-supervised methods, and show that their success on U-W ILDS is limited.
To facilitate the development of methods that can work reliably on real-world shifts, we provide an open-source package containing all of the relevant data loaders, model architectures, and methods.
I NTRODUCTION
Distribution shifts—when models are trained on a source distribution but deployed on a different target distribution—are frequent problems for machine learning systems deployed in the wild (Geirhos et al., 2020;
Koh et al., 2021).
In this paper, we focus on the use of unlabeled data to mitigate these shifts.
Unlabeled data is a powerful point of leverage as it is frequently much more available than labeled data.
For example, in the crop detection task in Figure 1, we might only have labeled training examples from one set of source domains (farms) available to learn a model that can extrapolate to a different set of target domains (David et al., 2020).
However, we might have many more unlabeled examples from the source domains, from extra domains, or even directly from the target domains.
Many methods for leveraging unlabeled data have been highly successful on some types of distribution shifts (Berthelot et al., 2021; Zhang et al., 2021).
However, the datasets typically used for evaluating these methods do not reflect many of the realistic shifts that might occur in the wild.
These evaluations tend instead to focus on shifts between photos and stylized versions like sketches (Li et al., 2017; Venkateswara et al., 2017; Peng et al., 2019) or synthetic renderings (Peng et al., 2018), or between variants of digits datasets like MNIST (LeCun et al., 1998) and SVHN (Netzer et al., 2011).
Unfortunately, prior work has shown that methods that work well on one type of shift need not generalize to others (Taori et al., 2020;
Djolonga et al., 2020; Xie et al., 2021a; Miller et al., 2021), which raises the question of how well they would work on a wider array of realistic shifts.
In this paper, we make two contributions.
First, we introduce the U-W ILDS benchmark (Figure 2), which builds on the recent W ILDS benchmark of in-the-wild distribution shifts (Koh et al., 2021).
W ILDS datasets span a wide range of tasks and modalities, and each dataset reflects a domain generalization or subpopulation shift setting with a substantial gap between in-distribution and out-ofdistribution performance.
However, W ILDS only contains labeled data, and the leverage points for learning robust models are therefore limited.
In U-W ILDS , we augment 8 of the 10 W ILDS datasets with curated unlabeled data acquired from the same source and target domains in W ILDS , as well as from extra domains of the same type: e.g., in the G LOBAL W HEAT - WILDS dataset pictured in
Figure 1, we acquired unlabeled photos of wheat fields from the source and target farms as well as extra farms that were not in the original labeled dataset.
In total, U-W ILDS adds 14.5 million unlabeled examples, expanding the number of examples by 3–13 × and allowing us to combine the real-world relevance of W ILDS with the leverage of unlabeled data .
[Figure] [Figure]
[Figure]
448,  - - 1,551,  - - - - scaffold
Hu et al.
OGB-MolPCBA bioassays molecular graph 437, 120, 4,052, 44, - - 517, 43, 430, 31,
Overalla solid package that has a good quality of construction for the price.
I *loved* my French press, it’s so perfect and came with all this fun stuff!
user
Ni et al.
Amazon sentiment product review 539, 3, - - 2,927, 21, 268, 1, 266, 1, time, region
Christie et al.
FMoW land use satellite image 141, 16 x11, 11 x- - 173, 2 x155, 3 xlocation, time
David et al.
GlobalWheat wheat head bbox wheat image 6,  5,  42,  8,  2, 11 Validation domains camera
Beery et al.iWildCam animal species camera trap photo 203,  - - 819, 3, - - - -
Yeh et al.
PovertyMap asset wealth satellite image 19, 23 x181, 13 x- - 55, 5 x24, 5 xcountry, ru/ur source, extra, validation, or target domains) depends on what is realistic and available for the application.
Second, to measure the effectiveness of contemporary methods that leverage unlabeled data, we developed a standardized and consistent protocol for benchmarking using U-W ILDS .
We evaluated representatives from three popular categories: methods for learning domain-invariant representations (Sun & Saenko, 2016;
Ganin et al., 2016), self-training methods (Lee, 2013; Sohn et al., 2020;
Xie et al., 2020), and pre-training methods that rely on self-supervision (Devlin et al., 2019; Caron et al., 2020).
These methods have been successful on some types of shifts, such as going from photos to sketches, or from handwritten digits to street signs (Berthelot et al., 2021; Zhang et al., 2021).
Our results on U-W ILDS are mixed: many methods performed similarly to or worse than purelysupervised training, and the only clear successes were on the two image classification datasets with unlabeled target data (C AMELYON 17- WILDS and FM O W- WILDS ).
On those, successful methods relied heavily on data augmentation (Xie et al., 2020; Caron et al., 2020), which limited their applicability to other modalities where augmentation techniques are not as well developed, such as text and molecular graphs.
The same methods were unsuccessful on the image regression and detection tasks, which have been relatively understudied: e.g., the notion of hard pseudolabels, which is common to many of the benchmarked methods, does not apply to regression.
For the text datasets, continued language model pre-training did not help; this could be because the online comments in
C IVIL C OMMENTS - WILDS and product reviews in A MAZON - WILDS are already more similar to the
BERT pre-training corpus (Devlin et al., 2019; Hendrycks et al., 2020) compared to other types of text studied in prior work, such as biomedical papers (Gururangan et al., 2020).
These results suggest potentially fruitful avenues for future work, such as developing data augmentation techniques for non-image modalities and more realistic hyperparameter tuning protocols.
They also underscore the importance of developing and evaluating methods for unlabeled data on a wider variety of real-world shifts than is typically studied.
To this end, we developed an open-source
Python package containing data loaders, compatible implementations of the methods we benchmarked, and scripts to replicate all experiments in this paper (Appendix G).
In addition, we will release a public leaderboard.
By allowing developers to easily test algorithms across the variety of datasets in U-W ILDS , we hope to accelerate the development of methods that can leverage unlabeled data to improve robustness to real-world distribution shifts.
C OMPARISON WITH EXISTING UNSUPERVISED ADAPTATION BENCHMARKS
The advantage of U-W ILDS is that it builds on the diverse range of applications and modalities covered in W ILDS (Koh et al., 2021), while providing an extensive amount of unlabeled data that can be used as leverage for training robust models.
In this section, we briefly compare with other existing ML benchmarks for unsupervised adaptation.
Images.
Evaluations of unsupervised adaptation methods for image classification have focused on generalizing from natural photos to a range of stylized images, such as sketches and cartoons (PACS (Li et al., 2017), Office-Home (Venkateswara et al., 2017), and DomainNet (Peng et al., 2019)), product images (Office-31 (Saenko et al., 2010)), and synthetic renderings (VisDA (Peng et al., 2018)), though location-based shifts have also been recently explored (Dubey et al., 2021).
It is also popular to evaluate on shifts between digits datasets, such as MNIST (LeCun et al., 1998), SVHN (Netzer et al., 2011), and USPS (Hull, 1994).
In contrast, U-W ILDS contains satellite, microscope, agriculture, and camera trap images, and it includes image regression and detection tasks; these modalities and settings are comparatively understudied.
Existing adaptation benchmarks for image segmentation, which is closely related to detection, focus on generalizing from natural to synthetic scenes (Ros et al., 2016;
Richter et al., 2016; Cordts et al., 2016; Hoffman et al., 2018).
Text.
Methods for unsupervised adaptation in NLP are typically evaluated on domain shifts between different textual sources, such as news articles, different categories of product reviews, Wikipedia, or social media platforms (Blitzer et al., 2007;
Mansour et al., 2009; Oren et al., 2019; Miller et al., 2020; Kamath et al., 2020; Hendrycks et al., 2020), or even more specialized sources such as legal documents (Chalkidis et al., 2020) or biomedical papers (Lee et al., 2020b; Gu et al., 2020).
Multilingual tasks can also be a setting for unsupervised adaptation (Conneau et al., 2018; Conneau &
Lample, 2019; Hu et al., 2020a; Clark et al., 2020), especially when generalizing to low-resource languages (Nekoto et al., 2020).
The U-W ILDS text datasets differ in that they focus on subpopulation performance, either to particular demographics in C IVIL C OMMENTS - WILDS or to tail populations in A MAZON - WILDS , rather than on adapting to a completely distinct domain.
Molecules.
While unlabeled molecules have been used for pre-training (Hu et al., 2020c; Rong et al., 2020), no standardized unsupervised adaptation benchmarks have been developed.
In U-W ILDS , we extend OGB-M OL PCBA to include unlabeled data from source, validation, and target domains.
P ROBLEM SETTING
Following W ILDS , we study the domain shift setting where the data is drawn from domains d ∈ D .
Each domain d corresponds to a data distribution P d over ( x, y, d ) , where x is the input, y is the prediction, and all points from P d have domain d .
The domains come in four types:
[Table]
Used for training
Can be used for training, if available
[Table]
domains
None
Validation domains
Used for hyperparameter tuning
Target domains
Used for
[Table]
evaluation
We consider domain adaptation, where all four types of domains are disjoint; subpopulation shift, where the target domains are a subset of the source domains; and hybrids of these two problem settings.
Models are trained on labeled data from the source domains, as well as on unlabeled data from one or more types of domains, depending on what is realistic for the application.
D ATASETS
The U-W ILDS datasets augment the W ILDS datasets with curated unlabeled data.
For consistency, the labeled datasets are exactly the same as in W ILDS , which allows direct evaluations of the utility of additional unlabeled training data.
The labeled and unlabeled data are disjoint; for instance, the unlabeled data from the target domains is different from the labeled target data used for evaluation.
Here, we briefly describe each dataset, why unlabeled data is realistically obtainable for the corresponding task, and how it might help.
We provide details for each dataset in Appendix A.
I W ILD C AM 2020- WILDS :
Species classification across different camera traps.
The task is to classify the animal species captured in a camera trap image (Beery et al., 2020).
We aim to generalize to new camera trap locations despite variations in illumination, background, and label frequencies (Beery et al., 2018).
While hundreds of thousands of camera traps are active worldwide, only a small subset of these traps have had images labeled.
On the other hand, unlabeled data from additional sensors operating across diverse conditions is plentiful.
In this work, we add unlabeled images from 3,215 extra camera traps also in the WCS Camera Traps dataset (Beery et al., 2020).
This expands the number of camera traps by a factor of 11 and the number of examples by a factor of 5.
C AMELYON 17- WILDS : Tumor identification across different hospitals.
The task is to classify image patches from lymph node sections as tumor or normal tissue.
We seek to generalize to new hospitals, which can differ in their patient demographics and data acquisition protocols (Veta et al., 2016; AlBadawy et al., 2018; Komura & Ishikawa, 2018; Tellez et al., 2019).
While obtaining labeled data for histopathology applications requires expensive and pain-staking annotations from expert pathologists, hospitals accumulate plentiful unlabeled slide images during normal operation.
Unlabeled images could be used to adapt to differences in the test hospitals (e.g., different staining protocols might lead to different color distributions).
We provide unlabeled patches from train and test hospitals, which expands the total number of patches by a factor of 7.5.
Both the labeled and unlabeled data are adapted from the Camelyon17 dataset (Bandi et al., 2018).
FM O W- WILDS :
Land use classification across different regions and years.
The task is to classify the type of building or land usage in a satellite image.
Given training data from before 2013, we aim to generalize to satellite imagery taken after 2013, while maintaining high accuracy across all geographic regions.
While labeling land use requires combining map data and expert annotations, unlabeled data is available in all locations in the world through constant streams of global satellite imagery.
Prior work has shown that unlabeled satellite data can improve OOD accuracy in landcover and cropland prediction (Xie et al., 2021a) as well as aerial object and scene classification (Reed et al., 2021).
We provide unlabeled satellite imagery from source and target timeframes across all regions, expanding the dataset by a factor of 3.5.
Both the labeled and unlabeled data are adapted from the FMoW dataset (Christie et al., 2018).
P OVERTY M AP - WILDS :
Poverty mapping across different countries.
The task is to predict a real-valued asset wealth index of the area in a satellite image.
We consider generalizing across different countries.
Like FM O W- WILDS , unlabeled satellite imagery is available globally, while labeled data is expensive to collect as it requires conducting nationally representative surveys in the field.
Prior work on poverty prediction has used unlabeled data for entropy minimization (Jean et al., 2018) and pre-training on auxiliary tasks such as nighttime light prediction (Xie et al., 2016; Jean et al., 2016), but these studies do not study generalization to new countries.
We provide unlabeled satellite imagery from both train and test countries, expanding the dataset by a factor of 14.
Both the labeled and unlabeled data are adapted from Yeh et al.
G LOBAL W HEAT - WILDS : Wheat head detection across different regions.
The task is to localize wheat heads in overhead field images.
We seek to generalize across image acquisition sessions, which each represent a particular location, time, and sensor.
Wheat field images contain many densely packed and overlapping instances, making labeling wheat heads in images costly, tedious and sensitive to the individual annotator.
However, hundreds of agricultural research institutes around the world collect terabytes of unlabeled field images which could be used for training.
We add unlabeled field images from train, test, and extra acquisition sessions, expanding the dataset by a factor of 10.
The labeled and unlabeled data are adapted from the Global Wheat Head Detection dataset and its underlying sources (David et al., 2020; 2021).
OGB-M OL PCBA: Molecular property prediction across different scaffolds.
The task is to predict the biological activity of small molecules represented as molecular graphs (Wu et al., 2018;
Hu et al., 2020b).
We seek to generalize to molecules with new scaffold structures.
Labels on biological activity are only available for a small portion of molecules, as they require expensive lab experiments to obtain.
However, unlabeled molecule structures are readily available in large-scale chemical databases such as PubChem (Bolton et al., 2008), and have been previously used for pretraining (Hu et al., 2020c) and semi-supervised learning (Sun et al., 2020).
We provide 5 million unlabeled molecules from source and target scaffolds, which expands the number of molecules by a factor of 12.5.
The original labeled data was curated by MoleculeNet (Wu et al., 2018) from
PubChem, and we similarly extracted the unlabeled data from PubChem (Bolton et al., 2008).
C IVIL C OMMENTS - WILDS :
Toxicity classification across demographic identities.
The task is to classify whether a text comment is toxic or not.
We consider the subpopulation shift setting, where the model must classify accurately across groups of comments mentioning different demographic identities.
While labels require large-scale crowdsourcing annotations on both comment toxicity, unlabeled article comments are widely available on the internet.
We provide unannotated comments as unlabeled data, which expands the size of the dataset by a factor of 4.5.
Both the labeled and unlabeled data are adapted from Borkan et al.
A MAZON - WILDS :
Sentiment classification across different users.
The task is to classify the star ratings of Amazon reviews.
We seek to perform consistently well across new reviewers.
While the labels (star ratings) are always available for Amazon reviews in practice, unlabeled data is a common source of leverage for sentiment classification more generally, with prior work in domain adaptation (Blitzer & Pereira, 2007; Glorot et al., 2011) and semi-supervised learning (Dasgupta & Ng, 2009; Li et al., 2011).
We provide unlabeled reviews from test and extra reviewers, which expands the total number of reviews by a factor of 7.5.
Both the labeled and unlabeled data are adapted from the Amazon review dataset by Ni et al.
A LGORITHMS
For our evaluation, we selected representative methods from the three categories described below.
These methods exemplify current approaches to using unlabeled data to improve robustness, and they have been successful on popular domain adaptation benchmarks like DomainNet (Peng et al., 2019) and semi-supervised settings like improving ImageNet accuracy by leveraging unlabeled images from the internet (Xie et al., 2020; Caron et al., 2020).
For more details, see Appendix B.
Domain-invariant methods.
Domain-invariant methods learn feature representations that are invariant across different domains by penalizing differences between learned source and target representations (Long et al., 2015; Ganin et al., 2016; Sun & Saenko, 2016; Long et al., 2017; 2018;
Saito et al., 2018; Zhang et al., 2018; Xu et al., 2019; Zhang et al., 2019b).
We discuss these methods further in Appendix B.2.
For our experiments, we evaluate two classical methods:
• Domain-Adversarial Neural Networks (DANN) (Ganin
et al.
2016) penalize representations on which an auxiliary classifier can easily discriminate between source and target examples.
• Correlation Alignment (CORAL) (Sun et al., 2016; Sun & Saenko, 2016) penalizes differences between the means and covariances of the source and target feature distributions.
Self-training.
Self-training methods “pseudo-label” unlabeled examples with the model’s own predictions and then train on them as if they were labeled examples.
These methods often also use consistency regularization, which encourages the model to make consistent predictions on augmented views of unlabeled examples (Sohn et al., 2020; Xie et al., 2020; Berthelot et al., 2021).
Self-training methods have recently been successfully applied to unsupervised adaptation (Saito et al., 2017; Berthelot et al., 2021; Zhang et al., 2021).
We include three representative algorithms:
• Pseudo-Label (Lee, 2013) dynamically generates pseudolabels and updates the model each batch.
• FixMatch (Sohn et al., 2020) adds consistency regularization on top of the Pseudo-Label algorithm.
Specifically, it generates pseudolabels on a weakly augmented view of the unlabeled data, and then minimizes the loss of the model’s prediction on a strongly augmented view.
• Noisy Student
(Xie et al., 2020) is similar to FixMatch, but instead of dynamically generating pseudolabels for each batch, it alternates between a few teacher phases, where it generates pseudolabels, and student phases, where it trains to convergence on the (pseudo)labeled data.
Self-supervision.
Self-supervised methods
learn useful representations by training on unlabeled data via auxiliary proxy tasks.
Common approaches include reconstruction tasks (Vincent et al., 2008;
Erhan et al., 2010; Devlin et al., 2019; Gidaris et al., 2018; Lewis et al., 2020), and contrastive learning (He et al., 2020; Chen et al., 2020b; Caron et al., 2020; Radford et al., 2021b), and recent work has shown that self-supervised methods can reduce dependence on spurious correlations and improve performance on domain adaptation tasks (Wang et al., 2021;
Tsai et al., 2021; Mishra et al.,
We carry out unsupervised adaptation by using self-supervision to pre-train models on the unlabeled data before fine-tuning them on the labeled data.
We evaluate popular self-supervised methods for vision and language: • SwAV (Caron et al., 2020) is a contrastive learning algorithm that maps representations to a set of clusters and then enforces similarity between cluster assignments.
• Masked language modeling (MLM) (Devlin et al., 2019) randomly masks some of the tokens from input text and trains the model to predict the missing tokens.
E XPERIMENTS
To evaluate how well existing methods can leverage unlabeled data to be robust to in-the-wild distribution shifts, we benchmarked the methods described above on all applicable U-W ILDS datasets.
6. S ETUP
We used the default models, labeled training and test sets,
Unlabeled data.
U-W ILDS contains multiple types of unlabeled data (from source, extra, validation, and/or target domains).
For simplicity, we ran experiments on a single type of unlabeled data for each dataset.
Where possible, we used unlabeled target data to allow methods to directly adapt to the target distribution; for I W ILD C AM 2020- WILDS and C IVIL C OMMENTS - WILDS , which do not have unlabeled target data, we used the extra domains instead.
All methods use exactly the same sets of labeled and unlabeled training data (except ERM, which does not use unlabeled data).
Hyperparameters.
We tuned each method on each dataset separately using random hyperparameter search.
Following W ILDS , we used the labeled out-of-distribution (OOD) validation set to select hyperparameters and for early stopping (Koh et al., 2021).
This validation set drawn from a different distribution than both the training and the OOD test set, so tuning on it does not leak information on the test distribution.
We did not use the in-distribution (ID) validation set.
For image classification and regression, we used both RandAugment (Cubuk et al., 2020) and Cutout (DeVries & Taylor, 2017) as data augmentation for all methods.
We did not use data augmentation for the remaining datasets.
For some datasets, we also had ground truth labels for the “unlabeled” data, which we used to run fully-labeled ERM experiments.
Overall, we ran 600+ experiments for 7,000 GPU hours on NVIDIA V100s.
See Appendix B for a discussion of which methods were applicable to which datasets; Appendix C for augmentation details; Appendix F for the fully-labeled experiments;
Appendix D for further experimental details.
6. R ESULTS
Table 2 shows that our results on U-W ILDS are mixed: while some methods improve performance on the target distribution, many methods perform similarly to or even underperform purely-supervised empirical risk minimization (ERM) on many datasets, despite access to unlabeled data from the target distribution and careful hyperparameter tuning.
In contrast, these methods have been shown to perform well on prior unsupervised adaptation benchmarks; in Appendix E, we verify our implementations by showing that all methods outperform ERM on the real → sketch shift in DomainNet, a standard unsupervised adaptation benchmark for object classification (Peng et al., 2019).
Image classification ( I W ILD C AM 2020- WILDS , C AMELYON 17- WILDS , and FM O W- WILDS ).
Data augmentation improved OOD performance on all three image classification datasets.
The gain was the most substantial on C AMELYON 17- WILDS , where vanilla ERM achieved 70.8% accuracy, while ERM with data augmentation achieved 82.0% accuracy.
On C AMELYON 17- WILDS and FM O W- WILDS , where we had access to unlabeled target data, several methods—Noisy Student, SwAV pre-training, and to a lesser extent CORAL—consistently improved OOD performance and reduced variability across replicates.
However, the other methods—
DANN, Pseudo-Label, and FixMatch—underperformed ERM.
This was especially surprising for
FixMatch, which performed very well on DomainNet (Appendix E).
Both FixMatch and Noisy
Student use pseudo-labeling and consistency regularization, but FixMatch dynamically computes pseudo-labels in each batch from the start of training, whereas Noisy Student first trains a teacher model to convergence on the labeled data and updates pseudolabels at a much slower rate.
As in Xie et al.
, this suggests that dynamically updating pseudo-labels might hurt generalization.
On I W ILD C AM 2020- WILDS , where we had access to 4 × as many unlabeled images from extra domains (distinct camera traps) but not to any images from the target domains, none of the benchmarked methods improved OOD performance compared to ERM.
This was surprising, as many of these methods were originally shown to work in semi-supervised settings.
One difference could be that the labeled and unlabeled examples in I W ILD C AM 2020-
WILDS differ more significantly (as they originate from different camera traps) than in the original FixMatch paper (Sohn et al., 2020), which used i.i.d.
labeled and unlabeled data, or the Noisy Student paper (Xie et al., 2020), which used ImageNet labeled data (Russakovsky et al., 2015) and JFT unlabeled data (Hinton et al., 2015).
Fully-labeled ERM models that used ground truth labels for the “unlabeled” data were available for FM O W- WILDS and I W ILD C AM 2020- WILDS .
They significantly outperformed other methods, suggesting room for improvement in how we leverage the unlabeled data.
[Table]
Image regression (P OVERTY M AP - WILDS ).
Data augmentation had no effect on performance on
P OVERTY M AP - WILDS , which differs from the above image datasets in that it is a regression task and involves multi-spectral satellite images (with 7 channels); both of these aspects are relatively unstudied compared to standard RGB image classification.
All applicable methods underperformed standard ERM, despite having access to unlabeled data from the target domains (countries).
Notably, even SwAV pre-training—which uses an independent auxiliary task, and should therefore be unaffected by how the final task is regression instead of classification—underperformed ERM.
Image detection (G LOBAL W HEAT - WILDS ).
We did not apply data augmentation to
G LOBAL W HEAT - WILDS , as standard augmentation changes the labels (e.g., cropping the image might remove bounding boxes) and would violate the assumption that labels are invariant under augmentations, which contrastive and consistency regularization methods like SwAV, Noisy Student, and FixMatch rely on.
Accordingly, we did not evaluate FixMatch and SwAV, and we modified
Noisy Student to remove data augmentation noise.
All applicable methods underperformed ERM.
Molecule classification (OGB-M OL PCBA).
We also did not apply data augmentation to OGBM OL PCBA, as such techniques are not well-developed for molecular graphs.
All applicable methods underperformed ERM.
We do not report ID results as this dataset has no separate ID test set.
Text classification (C IVIL C OMMENTS - WILDS , A MAZON - WILDS ).
Similarly, we did not apply data augmentation to the text datasets.
On both datasets, the benchmarked methods performed similarly to ERM (with class-balancing for C IVIL C OMMENTS - WILDS ).
Continued masked LM pretraining on the target distribution failed to improve target performance, unlike in prior work (Gururangan et al., 2020).
This difference might be because the online comments in C IVIL C OMMENTSWILDS and product reviews in A MAZON - WILDS are more similar to the BERT pre-training corpus (Devlin et al., 2019; Hendrycks et al., 2020) than the other types of text (e.g., biomedical and CS papers) studied in Gururangan et al.
(2020) are.
Also, C IVIL C OMMENTS - WILDS and A MAZONWILDS both measure subpopulation performance (on minority demographics and on the tail subpopulation, respectively), whereas prior work adapted models to new areas of the input space (e.g., from news to biomedical articles).
Fully-labeled ERM models were available but showed modest gains compared to FM O W- WILDS and I W ILD C AM 2020- WILDS .
As the text datasets focus on subpopulation performance, these results are consistent with prior observations that ERM models can have poor subpopulation performance even on large labeled training sets (Sagawa et al., 2020), necessitating other approaches to subpopulation shifts.
D ISCUSSION
We conclude by discussing several takeaways and promising directions for future work.
The role of data augmentation.
Many unsupervised adaptation methods rely strongly on data augmentation for consistency regularization or contrastive learning.
This reliance on data augmentation techniques—which are largely image-specific—restricts their generality, as they do not readily generalize to other modalities (or even other types of images besides photos).
Developing data augmentation techniques that can work well in other applications and modalities could be crucial for expanding the applicability of these methods (Verma et al., 2021).
Hyperparameter tuning.
Unsupervised adaptation methods have even more hyperparameters than standard supervised methods, and consistent with prior work, we found that these hyperparameters can significantly affect OOD performance (Saito et al., 2021).
Moreover, unlike in standard i.i.d.
settings, we do not have labeled target data that we can use for hyperparameter selection.
Improved methods for hyperparameter tuning could significantly improve OOD performance.
Such methods might make use of the unlabeled target data, or even the combination of unlabeled and labeled OOD validation data, which is provided for most datasets in U-W ILDS .
Pre-training on broader unlabeled data.
Pre-training on huge amounts of unlabeled data improves robustness to distribution shifts in some settings (Bommasani et al., 2021).
This unlabeled data need not be very related to the task: e.g., CLIP was pre-trained on text-image pairs from the internet but tested on tasks including histopathology and satellite image classification (Radford et al., 2021a).
This type of broad pre-training appears insufficient for U-W ILDS : many of our models were initialized with ImageNet-pretrained weights or derivatives of BERT but do not generalize well OOD.
However, it might still be helpful in conjunction with other techniques.
While we focused on providing curated unlabeled data from the same types of domains, it could be fruitful to explore methods that use both broad unlabeled data and unlabeled data that is more closely tailored to the task.
Leveraging domain annotations and task-specific structure.
OOD robustness is ill-posed in general, as one cannot expect to learn models that are robust to arbitrary distribution shifts.
Unlabeled data is one means of obtaining leverage on this problem.
Another leverage point is domain annotations and other structued metadata, which are provided in U-W ILDS for both labeled and unlabeled data (e.g., in I W ILD C AM
2020- WILDS , we know which images were taken from which cameras).
Exploiting this type of fine-grained domain structure for unsupervised adaptation—e.g., through multi-source/multi-target domain adaptation methods (Zhao et al., 2018; Peng et al.
, 2019)—could be a promising avenue for learning models that are more robust to the domain shifts in U-W ILDS .
E THICS STATEMENT
All of the U-W ILDS datasets are curated and adapted from public data sources, with licenses that allow for public release.
The datasets are all anonymized.
The distribution shifts in several of the U-W ILDS datasets deal with issues of discrimination and bias that arise in real-world applications.
For example, the C IVIL C OMMENTS - WILDS text dataset studies disparate model performance across comments that mention different demographic groups, while the FM O W- WILDS and P OVERTY M AP - WILDS satellite datasets study model performance on countries and regions where labeled data is less readily available.
As our results suggest, standard models trained on these datasets will not perform well on those subpopulations, and their learned representations might also be biased in undesirable ways (Bolukbasi et al., 2016; Caliskan et al., 2017; Garg et al., 2018; Tan & Celis, 2019; Steed & Caliskan, 2021).
We also encourage caution in interpreting positive results on these datasets, as our evaluation metrics might not encompass all relevant facets of discrimination and bias: for example, the “ground truth” toxicity annotations in C IVIL C OMMENTS - WILDS can themselves be biased, and the particular groupings of regions in
FM O W- WILDS might hide lower model performance in sub-regions.
For FM O W- WILDS and P OVERTY M AP - WILDS , surveillance and privacy issues also need to be considered.
In FM O W- WILDS , the image resolution is lower than that of other public satellite data (e.g., from Google Maps), and in P OVERTY M AP - WILDS , the location metadata is noised to protect privacy.
For a deeper discussion of the ethics of remote sensing in the context of humanitarian aid and development, we refer readers to the UNICEF report by Berman et al.
R EPRODUCIBILITY STATEMENT
All of the U-W ILDS datasets will be publicly available, together with code and scripts to replicate all of the experiments in this paper.
Following author guidelines, we will post a private link to an anonymized GitHub repository containing the code and scripts, including code used for dataset processing, as well as code for all of the methods benchmarked in this paper.
We have also stored all trained model checkpoints and results, together with the exact hyperparameters used, and will post this in a subsequent public version.
In Appendix A, we describe each U-W ILDS dataset and their sources of unlabeled data as well as what data processing steps were taken.
In Appendix B, we describe the implementations of each of our benchmarked methods in detail.
In particular, we discuss any changes we made to their original implementations, either for consistency with other methods or with prior implementations of these methods.
In Appendix C, we describe details of the data augmentations (if any) that we used across each dataset.
In Appendix D, we describe our experimental protocol, including the hyperparameter selection procedure and hyperparameter grids for all of the methods and datasets.
In Appendix F, we describe the details of our fully-labeled ERM experiments.
In Appendix E, we describe the details of our experiments on DomainNet.
Finally, in Appendix G, we include an illustrative code snippet of how to use the data loaders in the
U-W ILDS library.
A PPENDICES A Additional dataset details
A.
[Table]
AM 2020-
[Table]
A.
[Table] [Table]
A.
[Table]
FM
[Table]
W
[Table]
A.
[Table]
M
[Table]
A.
[Table]
W
[Table]
A.
[Table]
OGB-M
[Table]
PCBA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.
[Table]
IVIL C OMMENTS - WILDS . .
A.8 A MAZON -
[Table]
B
Algorithm details
B.1 Empirical risk minimization (ERM)
B.2 Domain-invariant methods
B.3 Self-training methods .
B.4 Self-supervision methods . . . . .
C Data augmentation
D Experimental details
D.1 In-distribution vs. out-of-distribution performance . . . . . . . . . . . . . . . . . .
D.2 Batch sizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.3 Hyperparameter tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.4 Algorithm-specific hyperparmeters . . . . . . . . . . . . . . . . . . . . . . . . . .
D.5 Compute infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
E Experiments on DomainNet
E.1 Setup . .
E.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F Fully-labeled ERM experimental details
F.
Hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
F.
Dataset-specific details . .
G Using the U-W ILDS library
A A DDITIONAL DATASET DETAILS
In this appendix, we provide additional details on the unlabeled data in U-W ILDS .
For more context on the motivation behind each dataset, the choice of evaluation metric, and the labeled data, please refer to the original W ILDS paper (Koh et al., 2021).
A. I W ILD C AM 2020- WILDS
The I W ILD C AM 2020- WILDS dataset was adapted from the iWildCam 2020 competition dataset made up of data provided by the Wildlife Conservation Society (WCS) (Beery et al., 2020) 3 .
Camera trap images are captured by motion-triggered static cameras placed in the wild to study wildlife in a non-invasive manner.
Images are captured at high volumes – a single camera trap can capture 10K images in a month – and annotating these images requires species identification expertise and is time-intensive.
However, there are tens of thousands of camera traps worldwide capturing images of wildlife that could be used as unlabeled training data.
For example, Wildlife Insights (Ahumada et al., 2020) now contains almost 20M camera trap images collected across the globe, but a large proportion of that data is still unlabeled.
Ideally we could capture value from those images despite the lack of available labels.
We extend I W ILD C AM 2020- WILDS with unlabeled data from a set of
WCS camera traps entirely disjoint with the labeled dataset, representative of unlabeled data from a newly-deployed sensor network.
Problem setting.
The task is to classify the species of animals in camera trap images.
The input x is an image from a camera trap, and the domain d corresponds to the camera trap that captured the image.
The target y , provided only for the labeled training images, is one of 182 classes of animals.
We seek to learn models that generalize well to new camera trap deployments, so the test data comes from domains unseen during training.
Additionally, we evaluate the in-distribution performance on held-out images from camera traps in the train set.
Data.
The data comes from multiple camera traps around the world, all provided by the Wildlife
Conservation Society (WCS).
The labeled data is the same as in Koh et al.
and the unlabeled data comprise 819,120 images from 3215 WCS camera traps not included in iWildCam 2020:
Source : 243 camera traps.
Validation (OOD): 32 camera traps.
Target (OOD): 48 camera traps.
Extra: 3215 camera traps.
The four sets of camera traps are disjoint.
The distributions of the labeled and unlabeled camera traps are very similar, except that the labeled data does not contains cameras with photos taken before LandSat 8 data was available.
Split # Domains (camera traps)
# Labeled examples #
[Table]
examples
Source
[Table]
Validation
[Table]
ID) 7,
Target
[Table]
ID)
[Table]
Validation
[Table] [Table]
Target
[Table] [Table]
Extra
[Table] [Table]
Broader context.
There are large volumes of unlabeled natural world data that have been collected in growing repositories such as iNaturalist (Nugent, 2018), Wildlife Insights (Ahumada et al., 2020), and GBIF (Robertson et al., 2014).
This data includes images or video collected by remote sensors or community scientists, GPS track data from an-animal devices, aerial data from drones or satellites, underwater sonar, bioacoustics, and eDNA.
Methods that can harness the wealth of information in unlabeled ecological data are well-posed to make significant breakthroughs in how we think about ecological and conservation-focused research.
Natural-world and ecological benchmarks that provide unlabeled data include NEWT (Van Horn et al., 2021), investigating efficient task learning, and
Semi-Supervised iNat (Su & Maji, 2021), which provides labeled data for only a subset of the taxonomic tree.
Recent work has begun to adapt weakly-supervised and self-supervised approaches for these natural world settings, including probing the generality and efficacy of self-supervision (Cole et al., 2021), incorporating domain-relevant context into self-supervision (Pantazis et al., 2021), or leveraging weak supervision from alternative data modalities (Weinstein et al., 2019) or pre-trained, generic models (Weinstein et al., 2021; Beery et al., 2019).
Active learning also plays a role here, seeking to adapt models efficiently to unlabeled data from novel regions with only a few targeted labels (Kellenberger et al., 2019; Norouzzadeh et al., 2021).
A. C AMELYON 17- WILDS
The C AMELYON 17- WILDS dataset (Koh et al., 2021) was adapted from the Camelyon17 dataset (Bandi et al., 2018), which is a collection of whole-slide images (WSIs) of breast cancer metastases in lymph node sections from 5 hospitals in the Netherlands.
The labels were obtained by asking expert pathologists to perform pixel-level annotations of each WSI, which is an expensive and painstaking process.
In practice, unlabeled WSIs (i.e., WSIs without pixel-level annotations) are much easier to obtain.
For example, only a fraction of the WSIs in the original Camelyon17 dataset (Bandi et al., 2018) were labeled; the other WSIs, which are taken from the same 5 hospitals, were provided without labels.
In this work, we augment the C AMELYON
17- WILDS dataset with unlabeled data from these WSIs.
Problem setting.
The task is to classify if a histological image patch contains any tumor tissue.
We consider generalizing from a set of training hospitals to new hospitals at test time.
The input x corresponds to a 96 × 96 image patch extracted from an WSI of a lymph node section, the label y is a binary indicator of whether the central 32 × 32 patch of the input contains any pixel that was annotated as a tumor in the WSI, and the domain d identifies which hospital the patch came from.
Each patch also includes metadata on which WSI it was extracted from, though we do not use this metadata for training or evaluation.
Models are evaluated by their average accuracy on a class-balanced test dataset.
Data.
All of the labeled and unlabeled data are taken from the Camelyon17 dataset (Bandi et al., 2018), which consists of WSIs from 5 hospitals (domains) in the Netherlands.
We provide unlabeled data from same domains as the labeled C AMELYON
17- WILDS dataset (no extra domains).
The domains are split as follows:
Source: Hospitals 1, 2, and 3. 2.
Validation (OOD): Hospital 4. 3.
Target (OOD): Hospital 5.
C AMELYON
WILDS also includes a Validation (ID) set which contains data from the training hospitals.
The C AMELYON 17- WILDS dataset has a total of 455,954 labeled patches across these splits, derived from the 10 WSIs per hospital that have full pixel-level annotations.
We augment the dataset with a total of 2,999,307 unlabeled patches, extracted from an additional 90 unlabeled WSIs per hospital.
There is no overlap between the WSIs used for the labeled versus unlabeled data.
To extract and process each patch, we followed the same data processing steps that were carried out for the labeled data in Koh et al.
Unlike the labeled patches, which were sampled in a class-balanced manner (i.e., half of the patches have positive labels), we sampled the unlabeled patches sampled uniformly at random from the un-
[Table]
labeled WSIs.
We sampled 6,667 patches per unlabeled WSI, with the single exception of one WSI which had only 5,824 valid patches, resulting in a total of 3,000,150 unlabeled patches (Table 4).
While the labeled patches were sampled in a class-balanced manner, the underlying label distribution skews heavily negative (approximately 95% of the patches in a WSI are negative), so we expect the unlabeled patches to be similarly skewed in their label distribution.
Broader context.
We focused on providing unlabeled data from the same hospitals (domains) as in the original labeled C AMELYON
17- WILDS dataset.
This unlabeled data from the training and test hospitals can be used to develop and evaluate methods for semi-supervised learning (Peikari et al., 2018;
Akram et al., 2018;
Lu et al., 2019; Shaw et al., 2020) and domain adaptation (Ren et al., 2018; Zhang et al., 2019a; Koohbanani et al., 2021), respectively.
In practice, there is also a large amount of unlabeled data from different domains that is publicly available: for example,
The
Cancer Genome Atlas (TCGA) hosts tens of thousands of publicly-available slide images across a variety of cancer types and from many different hospitals (Weinstein et al., 2013).
These large and diverse datasets need not even be directly relevant to the task at hand, e.g., one could pretrain a model on images for different types of cancer even if the goal were to develop a model for breast cancer.
Recent work has started to explore the use of these large and diverse datasets for computational pathology applications (Ciga et al., 2020; Dehaene et al., 2020) and in other medical imaging applications (Azizi et al., 2021).
A. FM O W- WILDS
The FM O W- WILDS dataset (Koh et al., 2021) was adapted from the FMoW dataset (Christie et al., 2018), which consists of global satellite images from 2002–2018, labeled with the functional purpose of the buildings or land in the image.
The labels are collected by a process which combines map data with crowdsourced annotations (from a trusted crowd).
In contrast, unlabeled satellite imagery is readily available across the globe.
In this work, we augment the FM O W-
WILDS dataset with unused satellite images that were part of the original FMoW dataset but not in the FM O W- WILDS dataset.
Problem setting.
The task is to classify the building or land-use type of a satellite image.
We consider generalizing from images before 2013 to after 2013, as well as considering the performance on the worst-case geographic region (Africa, the Americas, Oceania, Asia, or Europe).
The input x is an RGB satellite image ( 224 × 224 pixels).
The label y is one of 62 building or land use categories.
The domain d represents both the year and the geographical region of the image.
Each image also includes metadata on the location and time of the image, although we do not use these except for splitting the domains.
Models are evaluated by their average and worst-region accuracies in the
OOD timeframe.
Data.
The labeled and unlabeled data are taken from the FMoW dataset (Christie et al., 2018).
We provide unlabeled data from same domains as the labeled FM O W- WILDS dataset (no extra domains).
The domains are as follows:
Source: Images from 2002–2013. 2.
Validation (OOD): Images from 2013–2016.
[Table]
Target (OOD): Images from 2016–2018.
All of these domains have disjoint locations.
FM O W- WILDS also includes Validation (ID) and
Target (ID) sets which contain data from the training domains of 2002–2013.
The FM O W- WILDS dataset has 141,696 labeled images across these splits.
We augment the dataset with 340,469 unlabeled images.
These images come from two sources:
We use a sequestered split of the dataset, which consists of new locations that are not in the original labeled FM O W- WILDS dataset; these unlabeled data are drawn from the same distribution as the labeled data.
For the unlabeled target and validation splits, we also add unlabeled data in their respective timeframes from the training set locations.
While the unlabeled data from the Validation (OOD) and Target (OOD) domains can come from the same locations as the labeled training data, we note that none of the locations in the labeled Validation (OOD) or Target (OOD) data, which is used for evaluation, is shared with any of the unlabeled or labeled data used for training.
Broader context.
We focus on providing unlabeled data from the years (domains) that were in the original FM O W- WILDS dataset.
Prior works have used unlabeled satellite imagery for pretraining (Xie et al., 2016; Jean et al., 2016;
Xie et al., 2021a;
Reed et al., 2021), self-training (Xie et al., 2021a), and semi-supervised learning (Reed et al., 2021).
Leveraging unlabeled satellite imagery is powerful since it is widely available and can reduce the frequency at which we need to re-collect labeled data.
A. P OVERTY M AP - WILDS
The P OVERTY M AP - WILDS dataset (Koh et al., 2021) was adapted from Yeh et al.
The dataset consists of satellite images from 23 African countries, labeled with a village-level realvalued asset wealth index (measure of wealth).
The labels are collected by conducting a nationally representative survey, which requires sending workers into the field to ask each household a number of questions and can be very expensive.
In contrast, unlabeled satellite imagery is readily available across the globe.
In this work, we augment the P OVERTY M AP - WILDS dataset with satellite images from the same LandSat satellite.
Problem setting.
The task is to predict a real-valued asset wealth index from a satellite image.
We consider generalizing across country borders (the dataset contains 5 different cross validation folds, each splitting the countries differently).
The input x is a multispectral LandSat satellite image with 8 channels (resized to 224 × 224 pixels).
The output y is a real-valued asset wealth index.
The domain d represents the country the image was taken in, as well as whether the image was taken at an urban or rural area.
Each image also includes metadata on the location and time, although we do not make use of these except for defining the domains.
Models are evaluated by the average Pearson correlation ( r ) across 5 folds, as well as the lower of the Pearson correlations on the urban or rural
[Table]
subpopulations to test generalization to these subpopulations.
In particular, generalization to rural subpopulations is important as poverty is more common in rural areas.
Data.
We provide unlabeled data from same domains as the labeled P OVERTY M AP - WILDS dataset (no extra domains).
The domains are split as follows:
Source: Images from training countries in the fold. 2.
Validation (OOD): Images from validation countries in the fold. 3.
Target (OOD): Images from test countries in the fold.
All the countries in these splits are disjoint.
Folds also contain a Validation (ID) and Target (ID) set with data from the training countries.
The P OVERTY M AP - WILDS dataset has 19,669 labeled images across these splits.
We augment the dataset with 261,396 unlabeled images from the same 23 countries.
These images are collected using the same process as Yeh et al.
(2020) from the same LandSat satellite.
The image locations are chosen to be roughly near survey locations from the Demographic and Health Surveys (DHS).
Broader context.
We focus on providing unlabeled data from the countries (domains) that were in the original P OVERTY M AP - WILDS dataset.
Prior works on poverty prediction have used pretraining on unlabeled data (to predict an auxiliary task such as nighttime light prediction)
(Xie et al., 2016; Jean et al., 2016) and for semi-supervised learning via entropy minimization (Jean et al., 2018).
However, these works focus on generalization to new locations in the countries in the training set.
Poverty prediction is different from usual tasks in that the output is real-valued.
Most methods for unlabeled data are made for classification tasks, and we hope that our dataset will encourage more work on methods for using unlabeled data for improving OOD performance in regression tasks.
A. G LOBAL W HEAT - WILDS
The G LOBAL W HEAT - WILDS dataset was extended from the Global Wheat Head Dataset developed by David et al.
The goal of the dataset is to localize wheat heads from field images to assist plant scientists to assess the density, size, and health of wheat heads in a particular wheat field.
This imagery is acquired during different periods to cover the development of the vegetation, from the emergence to organ appearance.
Examples in G LOBAL W HEAT - WILDS are labeled by bounding box annotations of each wheat head in the image.
Wheat heads are densely packed and overlapping, making object annotation highly tedious.
Thus, the Global Wheat Head Dataset (GWHD) is relatively small, while in reality more field images are available.
We supplement G LOBAL W HEATWILDS with unlabeled examples from the same set of field vehicles and sensors but taken in different acquisition sessions, i.e., at different locations or the same location in a different year.
The inclusion of this unlabeled data allows: 1) a much higher spatial coverage of a field location when the data comes from an acquisition session which is already included, 2) a much higher temporal resolution when the data comes from a location which is already included, so we have a larger range of wheat
[Table]
growth stages, and 3) slightly more diversity when the session comes from a different location, but with the same image acquisition protocol (i.e., the same field vehicle and image sensor).
Problem setting.
The task is to localize wheat heads in high resolution overhead field images taken from above the crop canopy.
We consider generalizing across acquisition sessions representing a particular location, time and sensor with which the images were captured.
Variation across sessions includes changes in wheat genotype, wheat head appearance, growing conditions, background appearance, illumination and acquisition protocol.
The input x is an overhead outdoor image of wheat canopy, and the label y is a set of box coordinates bounding the wheat heads (the spike at the top of the wheat plant holding grain), omitting any hair-like awns that may extend from the head.
The domain d designates an acquisition session, which corresponds to a certain location, time, and imaging sensor.
Data.
We provide unlabeled data from same domains as the labeled G LOBAL W HEAT - WILDS dataset.
Additionally, we provide unlabeled data from extra acquisition sessions not in the labeled
G LOBAL W HEAT - WILDS dataset (extra domains).
The domains are split as follows:
Source: 18 acquisition sessions in Europe (France × 13, Norway × 2, Switzerland, United Kingdom, Belgium).
Validation (OOD): 8 acquisition sessions: 7 in Asia (Japan × 4, China × 3) and 1 in Africa (Sudan).
Target (OOD): 21 acquisition sessions: 11 in Australia and 10 in North America (USA × 6, Mexico × 3, Canada).
Extra (OOD): 53 acquisition sessions distributed across the world
The source, validation, and target sessions are split by continent, while the extra sessions are taken from across the world.
For acquisition sessions with both labeled and unlabeled data, we randomly selected new patches of 1024x1024 pixels from the original underlying data.
The images were preprocessed in the same way as described in David et al.
Broader context.
Utilizing unlabeled data is relatively new in the context of plant phenotyping, due to the lack of a large, unlabeled database of plant images.
However, larger plant image datasets are starting become available, such as from the Terraphenotying Reference Platform (TERRA-Ref,
Burnette et al.
Increasing the sample size and variation within plant datasets is an important goal, because plants from the same species are fairly self-similar within the same field and therefore increasing the number of locations, times and image types included in a dataset can be beneficial for making fine-grained visual classifications for plants.
Further, for plant phenotyping to be used in farming applications, such as for precisely spraying weeds in a field with herbicide, models must be highly robust to variations between different fields.
[Table]
997 Target
Terraref
US
Maricopa 3/20/
Gantry
F  1000 Target
UQ
Australia
Gatton 8/12/
Tractor
PF
Target
UQ
Australia
Gatton 9/8/
Tractor
PF
Target
UQ
Australia
Gatton 9/15/
Tractor
F  0 Target
UQ
Australia
Gatton 10/1/
Tractor
F  0 Target
UQ
Australia
Gatton 10/9/
Tractor
F-R
Target
UQ
Australia
Gatton 10/14/2015 Tractor
F-R
Target
UQ
Australia
Gatton 10/6/
Handheld
R
Target
UQ
Australia
McAllister 10/9/
Handheld
R
Target
UQ
Australia
Brookstead 10/16/2020 Handheld
F-R
Target
UQ
Australia
Gatton 9/22/
Handheld
F-R
Target
UQ
Australia
Gatton 8/31/
Handheld
PF
Target
Usask
Canada
Saskatoon 6/6/
Tractor
F-R
[Table]
4/15/2020 Cart
E  1000 Extra CIMMYT 11 Mexico
CiudadObregon 4/22/2020 Cart
E  1000 Extra CIMMYT 12 Mexico
CiudadObregon 4/22/2020 Cart
E  1000 Extra CIMMYT 13 Mexico
CiudadObregon 4/22/2020 Cart
E  1000 Extra CIMMYT 14 Mexico
CiudadObregon 4/22/2020 Cart
PF  1000 Extra CIMMYT 15 Mexico
CiudadObregon 4/28/
Cart
PF  1000 Extra CIMMYT 16 Mexico
CiudadObregon 5/3/
Cart
F-R
Extra ETHZ
Switzerland Eschikon 6/6/
Spidercam F  750 Extra INRAE
France
Clermont-Ferrand 5/29/2019 Handheld
F  1000 Extra KSU
US
Manhattan,KS 5/4/
Tractor
F  1000 Extra KSU
US
Manhattan,KS 4/23/2017 Tractor
P-F  1000 Extra Rres
UK
Rothamsted 7/7/
Gantry
R  1000 Extra Rres
UK
Rothamsted 7/10/2015 Gantry
F  1000 Extra Rres
UK
Rothamsted 7/13/2015 Gantry
F-R  1000 Extra Rres
UK
Rothamsted 7/20/2015 Gantry
F-R  1000 Extra ULi`ege
Belgium
Gembloux 6/11/2020 Cart
PF  1000 Extra ULi`ege
Belgium
Gembloux 6/15/2020 Cart
F  1000 Extra ULi`ege
Belgium
Gembloux 6/16/2020 Cart
F  1000 Extra ULi`ege
Belgium
Gembloux 6/18/2020 Cart
F  1000 Extra ULi`ege
[Table] [Table]
A. OGB-M OL PCBA
The OGB-M OL PCBA dataset was adapted from the Open Graph Benchmark (Hu et al., 2020b) and originally curated by the MoleculeNet (Wu et al., 2018) from the PubChem database (Bolton et al., 2008).
The dataset is a collection of molecules annotated with 128 kinds of binary labels indicating the outcome of different biological assays.
Performing biological assays is expensive, and as a result, the assay labels are only sparsely available over a tiny portion of the molecules curated in the large-scale PubChem database (Bolton et al., 2008).
On the other hand, unlabeled molecule data is abundant and readily available from the database.
Prior work in graph machine learning has leveraged unlabeled molecules to perform pre-training (Hu et al., 2020c) and semi-supervised learning (Sun et al., 2020).
In this work, we augment the OGB-M OL PCBA dataset with unlabeled molecules subsampled from the PubChem database.
Problem setting.
The task is multi-task molecule classification, and we consider generalizing to new molecule scaffold structures at test time.
The input x corresponds to a molecular graph (where nodes are atoms and edges are chemical bounds), the label y is a 128-dimensional binary vector, representing the binary outcomes of the biological assay results.
y could contain NaN values, indicating that the corresponding biological assays were not performed on the given molecule.
The domain d indicates the scaffold group a molecule belongs to.
As the binary labels are highly-skewed, the model’s classification performance is evaluated using the Average Precision.
Data.
All of the labeled and unlabeled data are taken from the PubChem database (Bolton et al., 2008).
We provide unlabeled data from same domains as the labeled OGB-M OL PCBA dataset (no extra domains).
We curate the unlabeled data by randomly sampling 5 million molecules from the PubChem database.
We then assign these unlabeled molecules to the existing labeled scaffold groups that contain the most similar molecules.
Specifically, we first compute the 1024-dimensional
Morgan fingerprints for all the molecules (Rogers & Hahn, 2010; Landrum et al., 2006).
Then, for each unlabeled molecule, we compute its Jaccard similarity against all the labeled molecules in
OGB-M OL PCBA and obtain a labeled molecule with the highest Jaccard similarity.
Finally, we assign the unlabeled molecule to the scaffold group that the most similar labeled molecule belongs to.
This way, the molecules within the same scaffold groups are structurally similar to each other.
The domains in the OGB-M OL PCBA dataset are as follows:
Source: 44,930 scaffold groups.
Validation (OOD): 31,361 scaffold groups.
Target (OOD): 43,793 scaffold groups.
The largest scaffolds are in the source split and the smallest scaffolds in the target split.
We assign all of the unlabeled molecules to the existing domains, so there are no
extra
domains added.
While the unlabeled data are similar to the labeled data in that they were all derived from PubChem (Bolton et al., 2008), it is quite possible that there was some selection bias in which molecules in PubChem were chosen to be labeled, which would lead to an undocumented distribution shift between the unlabeled nad labeled datasets.
Broader context.
We focused on providing unlabeled data for both training and OOD test domains.
Unlabeled molecules can be used to develop and evaluate methods for domain adaptation, self-training, as well as pre-training (Hu et al., 2020c) and semi-supervised learning (Sun et al., 2020).
In terms of future directions, we think it is fruitful to explore both graph-agnostic methods (e.g., pseudo-label training) and more graph-specific methods (e.g., self-supervised learning of graph neural networks (Xie et al., 2021b)).
A. C IVIL C OMMENTS - WILDS
The C IVIL C OMMENTS - WILDS dataset (Koh et al., 2021) was adapted from the CivilComments dataset (Borkan et al., 2019), which is a collection of text comments made on online articles.
The data in C IVIL C OMMENTS - WILDS underwent a significant labeling and annotation process: each example was labeled toxic or non-toxic and annotated for whether they mentioned certain demographic identities by at least 10 crowdworkers.
Such a substantial labeling and identity annotation process is expensive and time-consuming.
On the other hand, unlabeled, unannotated text comments are readily available.
For example, C IVIL C OMMENTS - WILDS only contains a subset of all data available in the original CivilComments dataset (Borkan et al., 2019), most of which Koh et al.
(2021) excluded because these examples were not annotated for mentioning identities.
In this work, we augment the
C IVIL C OMMENTS - WILDS dataset with these unlabeled, unannotated comments.
Problem setting.
The task is to classify whether a text comment is toxic or not.
The input x is a text comment (at least one sentence long) originally made on an online article, the label y is a binary indicator of whether the comment is rated toxic or not, and the domain d is an 8-dimensional binary vector, where each dimension corresponds to whether the comment mentions each of 8 demographic identities: male, female, LGBTQ, Christian, Muslim, other religions, Black , or White , respectively.
Each comment also includes metadata on which article the comment was made on, although we do not use this metadata for training or evaluation.
We consider the subpopulation shift setting, where the model must perform well across all subpopulations, which are defined based on d .
Koh et al.
(2021) define 16 subpopulations (groups) based on d .
Models are then evaluated by their worst-group accuracy, i.e., the lowest accuracy over thegroups considered.
In our work, we use the same evaluation setup.
Data.
All of the labeled and unlabeled data are taken from the CivilComments dataset (Borkan et al., 2019).
After preprocessing, Koh et al.
(2021) created the C IVIL C OMMENTS - WILDS dataset using the 448,000 examples that were fully annotated for both toxicity y and the mention of demographic identities d .
In this work, we augment C IVIL C OMMENTS - WILDS with an additional 1,551,515 examples collected by Borkan et al.
We use these examples as unlabeled data.
We follow the same preprocessing steps as was done with the labeled data in Koh et al.
The resulting unlabeled examples have no identity annotations d and no toxicity label y .
We note that
Borkan et al.
(2019) actually do provide toxicity labels for these examples in the original CivilComments dataset, but we ignore these labels and use them neither for training nor evaluation.
Because our unlabeled examples have no identity annotations, we cannot group these examples as
Koh et al.
(2021) group the labeled examples; thus we refer to this data as unlabeled data coming from extra domains (Table 11).
In practice, these comments may actually mention any number of identities.
A substantial amount (1,427,848 or 92%) of the unlabeled comments are drawn from the same articles as the labeled comments.
In particular, 140,082 unlabeled comments are from the same articles as labeled comments in the test split.
C
IVIL C OMMENTS - WILDS exhibits class imbalance.
We account for this when benchmarking methods by sampling class-balanced batches of labeled data when applicable (see Appendix B).
Broader context.
In this work, we focused on supplementing C IVIL C OMMENTS - WILDS with extra unannotated data from the original CivilComments dataset (Borkan et al., 2019).
In practice, unannotated text comments are widely available on the internet.
Whether using such unlabeled data, as we do in this work, can help with bias is still an open question.
Previous work suggests that training on large amounts of data alone is not sufficient to avoid unwanted biases, since many
[Table]
papers have pointed out biases in large language models ( et
Nadeem et al
Gehman et al., 2020).
However, recent work has also suggested that pre-trained models can be trained to be more robust against some types of spurious correlations (Hendrycks et al., 2020; Tu et al., 2020) and that additional domain- and task-specific pre-training (Gururangan et al., 2020) can also improve performance.
We hope our contributions to the C IVIL C OMMENTS - WILDS dataset can encourage future study on whether unlabeled data can be leveraged to improve generalization across subpopulation shifts.
A. A MAZON - WILDS
The A MAZON - WILDS dataset (Koh et al., 2021) was adapted from the Amazon reviews dataset (Ni et al., 2019), which is a collection of product reviews written by reviewers.
While Amazon reviews are always labeled by the star ratings in practice, unlabeled data is a common source of leverage more generally for sentiment classification, with prior work in domain adaptation (Blitzer & Pereira, 2007; Glorot et al., 2011) and semi-supervised learning (Dasgupta & Ng, 2009; Li et al., 2011).
In this work, we augment the A MAZON - WILDS dataset with unlabeled reviews, whose star ratings have been removed.
Problem setting.
The task is sentiment classification, and we consider generalizing from a set of reviewers to new reviewers at test time.
The input x corresponds to a review text, the label y is the star rating from 1 to 5, and the domain d identifies which user wrote the review.
For each review, additional metadata (product ID, product category, review time, and summary) are also available.
Because the goal is to train a model that performs well across a wide range of reviewers, models are evaluated by their tail performance, concretely, their accuracy on the user at the 10th percentile.
Data.
All of the labeled and unlabeled data are taken from the Amazon reviews dataset (Ni et al., 2019).
We provide unlabeled data from same domains as the labeled A MAZON - WILDS dataset.
Additionally, we provide unlabeled data from extra reviewers not in the labeled A MAZON - WILDS dataset (extra domains).
The domains are split as follows:
Source: 1,252 reviewers.
Validation (OOD): 1,334 reviewers.
Target (OOD): 1,334 reviewers.
Extra (OOD): 21,694 reviewers.
The reviewers in each split are distinct, and all reviewers have at least 75 reviews.
The distributions of reviewers in each split are identical.
A MAZON - WILDS also includes Validation (ID) and Target (ID) sets which contain data from the source reviewers.
The A MAZON - WILDS dataset has a total of 539,502 labeled reviews across these splits, and we augment the dataset with a total of 3,462,668 unlabeled reviews.
For each split of the unlabeled data, we include all available reviews that are written by the reviewer.
For the Extra (OOD) split, we include all reviewers with at least 75 reviews that are not in Source, Validation (OOD), or Target (OOD) splits.
[Table]
To filter and process reviews, we followed the same data processing steps as for the labeled data in
A MAZON - WILDS (Koh et al., 2021).
Broader context.
We focused on providing unlabeled data from OOD domains, including both test and extra domains.
Unlabeled data from the test reviewers can be used to develop and evaluate methods for domain adaptation (Ren et al., 2018;
Zhang et al., 2019a; Koohbanani et al., 2021), which has been well-studied in the context of sentiment classification (Blitzer & Pereira, 2007;
Glorot et al., 2011).
While there is limited prior work on leveraging unlabeled data from extra domains, some domain adaptation techniques can be readily adapted to leverage such unlabeled data (Ganin et al., 2016).
Finally, we focus on unlabeled data specific to the task in this work, varying only the domains, and this contrasts with the type of unlabeled data used for pre-training in
NLP, which is much larger and more diverse (Devlin et al., 2019; Brown et al., 2020).
B A LGORITHM DETAILS B. E MPIRICAL RISK MINIMIZATION (ERM)
As a baseline, we consider Empirical Risk Minimization (ERM).
ERM ignores unlabeled data and minimizes the average labeled loss.
We additionally evaluate ERM with strong data augmentation on applicable datasets, i.e., on I W ILD C AM 2020- WILDS , C AMELYON 17- WILDS , P OVERTY M APWILDS , and FM O W- WILDS (see Appendix C).
ERM with strong data augmentation learns a model h that minimizes the labeled training loss
[Equation]
where A strong is a stochastic data augmentation operation, and ℓ measures the prediction loss.
We use L L throughout this appendix to refer to the above labeled loss with strong augmentations (on applicable datasets).
For all dataset except C IVIL C OMMENTS - WILDS , we sample labeled batches uniformly at random.
In our experiments, we account for class imbalance in C IVIL C OMMENTS - WILDS by explicitly sampling class-balanced batches of labeled data when computing L L ( h ) .
B. D OMAIN - INVARIANT METHODS
Domain-invariant methods seek to learn feature representations that are invariant across domains.
These methods are motivated by earlier theoretical results showing that the gap between in- and outof-distribution performance depends on some measure of divergence between the source and target distributions (Ben-David et al., 2010).
To minimize this divergence, the methods described below penalize divergence between feature representations across domains, i.e., they encourage the model to produce feature representations that are similar across domains.
Consider a model
h
= g ◦ f , where the featurizer f : X → F maps the inputs to some feature space, and the head g : F → Y maps feature representations to prediction targets.
Domain-invariant methods seek to constrain f to output similar representations for labeled and unlabeled data.
In this work, we adapt all of our domain-invariant methods to use data augmentations on applicable datasets (see Appendix C), and thus the output of f on the labeled batch is
[Equation]
Similarly, the output of f on an unlabeled batch is
[Equation]
Domain-invariant methods seek to minimize some divergence ξ : F × F → R between the labeled data B L and the unlabeled data B U , where the choice of divergence depends on the specific method.
The divergence is expressed as a penalty term:
[Equation]
The final objective is a combination of the labeled loss and penalty loss.
The balance between the two losses is controlled by hyperparameter λ , the penalty weight.
[Equation]
In our experiments, we study two classical domain-invariant methods, Correlation Alignment (CORAL)
(Sun et al., 2016; Sun & Saenko, 2016) and Domain-Adversarial Neural Networks (DANN)
(Ganin et al., 2016).
These methods are well-known and established, but their performance can be lower than that of newer domain-invariant methods that employ different penalities to encourage the source and target representations to be similar (Jiang et al., 2020; Zhang et al., 2021).
Examples of these newer methods are Joint Adaptation Networks (JAN) (Long et al., 2017), Conditional Domain Adversarial Networks (CDAN) (Long et al., 2018), Collaborative and Adversarial
Networks (CAN) (Zhang et al., 2018), and models with Adaptive Feature Norm (AFN) (Xu et al., 2019), as well as methods that minimize the Maximum Classifier Discrepancy (MCD) (Saito et al., 2018) and the Margin Disparity Discrepancy (MDD) (Zhang et al., 2019b).
All of the above methods were developed for the single-source single-target setting, where the source domain is treated as a single distribution, and likewise for the target domain.
As each U-W ILDS dataset comprises multiple source domains and multiple target domains, it is likely that methods that can leverage this additional structure could perform better.
Examples of these methods include
Multi-source Domain Adversarial Networks (MDAN)
(Zhao et al., 2018) and Moment Matching for Multi-Source Domain Adaptation (M3SDA) (Peng et al., 2019).
The DomainBed (Gulrajani &
Lopez-Paz, 2020) and WILDS (Koh et al., 2021) benchmarks also extended single-source algorithms like CORAL and DANN to take advantage of multiple source domains in the domain generalization setting, and similar extensions in the domain adaptation setting could be promising.
Correlation Alignment (CORAL).
Algorithm 1 describes CORAL, proposed by Sun et al.
(2016); Sun & Saenko (2016).
CORAL measures the divergence ξ between batches of feature representations in terms of the deviation between their first and second order statistics.
Given a labeled batch and unlabeled batch of features B L ∈ R n L × m , B U ∈ R n U × m , define the feature covariance matrices as
[Equation]
CORAL then defines a penalty function
[Equation]
where m is the dimension of the feature representations.
We adapted our implementation from the Transfer Learning Library (Jiang et al., 2020) and matched all details to the formulation given by Sun et al.
; Sun & Saenko (2016),
except in the addition of augmentations.
On applicable datasets, we strongly augmented all labeled and unlabeled examples using A strong , whereas Sun et al.
; Sun & Saenko (2016) do not explicitly require data augmentations.
We add augmentations to allow for a fairer comparison to other methods which use augmentations.
Note that CORAL has also been adapted by Gulrajani & Lopez-Paz (2020);
Koh et al.
(2021) for domain generalization.
In particular, where the original CORAL paper defines L penalty as the divergence between just two kinds of batches (labeled and unlabeled), these works define L penalty as the divergence between many kinds of batches, where batches are grouped based on domain annotation d ( i ) .
For simplicity, we followed the original CORAL formulation and differentiate only between labeled and unlabeled batches.
We leave leveraging the domain adaptations d to future work.
Algorithm 1: CORAL
Input: Labeled batch { ( x ( i ) L , y
( i ) L , d ( i ) L ) : i ∈ (1 , · · · , n L ) } , unlabeled batch { ( x ( i ) U , d ( i ) U ) : i ∈ (1 , · · · , n U ) } , strong augmentation function A strong , penalty weight λ ∈ R , dimension of feature representations m 1 Compute feature representations for labeled and unlabeled batches
[Equation]
2 Compute feature covariance matrices for labeled and unlabeled batches
[Equation]
[Table]
[Table] [Table]
[Equation]
A PPLICABLE DATASETS .
We run CORAL on all datasets except G LOBAL W HEAT - WILDS and
C IVIL C OMMENTS - WILDS .
We do not evaluate domain invariant methods on C IVIL C OMMENTSWILDS , since the labeled and unlabeled data are drawn from the same distribution.
We do not evaluate CORAL on G LOBAL W HEAT - WILDS because CORAL does not port straightforwardly to detection settings.
DANN.
Algorithm 2 describes DANN, proposed by Ganin et al.
DANN measures the divergence ξ between batches of feature representations using the performance of a discriminator network h d that aims to discriminate between domains.
Given a batch of features (either B L or B U ), this deep network h d must classify whether examples are from the labeled data or unlabeled data.
h d is optimized using a binary classification loss
[Equation]
The loss of h d is exactly related to ξ as
[Equation]
[Equation]
In other words, at the same time that h d is optimized to minimize its loss L ( h d ) , the featurizer f is incentivized to minimize L penalty =
ξ ( B L , B U )
= − L ( h d ) , or maximize L ( h d ) .
See Algorithmfor details.
We adapted our implementation from the Transfer Learning Library (Jiang et al., 2020) and matched all details to the formulation given by Ganin et al.
(2016), except for two changes.
On applicable datasets, we have strongly all labeled and unlabeled examples using A strong , whereas Ganin et al.
(2016) do not explicitly require data augmentations.
We add augmentations to allow for a fairer comparison to other methods which use augmentations.
Second, where Ganin et al.
(2016) optimize f , g , and h d using the same learning rate η , we use three different learning rates η f , η g , η h d , following the implementation of the Transfer Learning Library (Jiang et al., 2020).
g
[Table] [Table]
[Equation]
[Table] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Table]
[Table] [Table]
DATASETS .
We run DANN on all datasets except G LOBAL W HEAT - WILDS and
C IVIL C OMMENTS - WILDS .
We do not evaluate domain invariant methods on C IVIL C OMMENTSWILDS , since the labeled and unlabeled data are drawn from the same distribution.
We do not evaluate DANN on G LOBAL W HEAT - WILDS because DANN does not port straightforwardly to detection settings.
B. S ELF - TRAINING METHODS
Self-training methods leverage unlabeled data by “pseudo-labeling” unlabeled examples with the model’s own predictions and training on them as if they were labeled examples.
In certain formulations, this is equivalent to minimizing the model’s conditional entropy on the unlabeled data (Grandvalet & Bengio, 2005).
Contemporary self-training methods also often make use of consistency regularization, i.e., encouraging the model to make similar predictions on noisy/augmented versions of unlabeled examples.
Self-training methods have recently been shown to be empirically successful at unsupervised domain adaptation (Saito et al., 2017; Berthelot et al., 2021; Zhang et al., 2021).
The self-training methods we study follow this general structure: given an unlabeled example x U , these algorithms generate a pseudolabel ˜
y U
=
ψ ( x U ) , where the pseudolabel-generating function ψ :
X → Y differs between algorithms.
For classification problems, we study algorithms that produce hard pseudolabels, which are one-hot class predictions, rather than soft pseudolabels, which are continuous distributions over the classes.
Next, algorithms define an unlabeled loss L U ( h ) for model h by minimizing the loss between pseudolabels ˜ y U and the model’s predictions.
The algorithms we consider below augment x U during training; i.e., rather than minimizing the loss between ˜ y U and the model’s prediction on x U , the algorithms below minimize the loss of predictions on A ( x U ) , where
A is a stochastic, label-preserving augmentation.
Assuming model h that outputs real-valued logits, the complete unlabeled loss is
[Equation]
This unlabeled loss is jointly optimized with the standard ERM labeled loss.
The balance between the two losses is controlled by hyperparameter λ ( t ) , which is a function of the current step t .
[Equation]
Pseudo-Label.
Algorithm 3 describes Pseudo-Label, proposed by Lee (2013).
In this algorithm, the model dynamically generates pseudolabels and updates each batch.
Formally, the pseudolabelgenerating function ψ is given by
[Equation]
where A strong is the strong augmentation function described in Appendix C. Pseudo-Label then computes the loss between a strongly augmented example and its associated pseudolabel.
In order to more fairly compare Pseudo-Label to FixMatch, we add on confidence thresholding to the Pseudo-Label algorithm, a feature also added in the implementation of Pseudo-Label by Sohn et al.
When confidence thresholding, examples on which the model has low confidence have zero loss, i.e., for some threshold hyperparameter τ , the loss an example x U contributes is
[Equation]
[Equation]
Finally, Pseudo-Label increases the balance λ ( t ) between labeled and unlabeled losses over time, initially placing 0 weight on L U ( h ) and then linearly stepping the unlabeled loss weight until it reaches the full value of hyperparameter λ at some threshold step.
We fix the step at which λ ( t ) reaches its maximum value ( λ ) to be 40% of the total number of training steps, matching the implementation of Sohn et al.
This scheduling allows the algorithm to initially prioritize the labeled loss, as generated pseudolabels are mostly incorrect while the model has low accuracy.
Formally, at step t and given total number of steps T ,
[Equation]
We endeavored to match our implementation of Pseudo-Label to the formulation given by Lee (2013), except in the use of augmentations.
On applicable datasets, we have strongly augmented all labeled and unlabeled examples using A strong , whereas Lee (2013) do not use any data augmentations, i.e., all instances of A strong are replaced with the identity function.
We add augmentations to
Pseudo-Label in order to allow for a fairer comparison to other methods that use augmentations.
Algorithm
Input: Labeled batch { ( x i ) L , y ( i ) L , d ( i ) L ) : i ∈ (1 , · · · , n L ) } , unlabeled batch { ( x i ) U , d ( i ) U ) : i ∈ (1 , · · · , n U ) } , strong augmentation function A strong , unlabeled loss weight for current step λ ( t ) ∈ R , confidence threshold τ ∈ [0 , 1]
[Equation]
A PPLICABLE DATASETS .
We evaluate Pseudo-Label on all datasets except P OVERTY M APWILDS , as P OVERTY M AP - WILDS is a regression dataset, and hard pseudolabeling does not port straightforwardly to regression tasks.
FixMatch.
Algorithm 4 describes FixMatch, proposed by Sohn et al.
Like Pseudo-Label, this algorithm dynamically generates pseudolabels and updates each batch.
FixMatch additionally employs consistency regularization on the unlabeled data.
While pseudolabels are generated on a weakly augmented view of the unlabeled examples, the loss is computed with respect to predictions on a strongly augmented view.
This encourages models to make predictions on a strongly augmented example consistent with its prediction on the same example when weakly augmented.
For details about the strong versus weak augmentations we use, see Appendix C.
Formally, the pseudolabel-generating function ψ is given by
[Equation]
Like Pseudo-Label, FixMatch uses confidence thresholding, and unlabeled examples on which the model has low confidence have zero loss.
Following Sohn et al.
(2020), we keep the balance between labeled and unlabeled losses constant at λ ( t )
= λ .
FixMatch’s original authors justify keeping λ ( t ) at a fixed magnitude (as opposed to slowly increasing λ ( t ) as in Pseudo-Label) by noting that most predictions made by FixMatch are initially low confidence, so for sufficiently high confidence threshold τ , most unlabeled examples have loss zero, keeping the magnitude of L U ( h ) initially small.
This magnitude grows over time, providing a natural curriculum (Sohn et al., 2020).
We endeavored to match our implementation of FixMatch to the formulation of Sohn et al.
(2020), except in the use of augmentations for labeled data.
Where we have strongly augmented all labeled examples using A strong in Algorithm 4, Sohn et al.
(2020) explicitly choose to use weak instead of strong augmentations on the labeled examples.
However, our results on DomainNet in Appendix E suggest that using strong instead of weak augmentations for the labeled examples improves performance, so we use strong augmentations on the labeled examples in order to allow for a fairer comparison to other methods.
Algorithm 4: FixMatch
Input: Labeled batch { ( x (
i ) L , y ( i ) L , d ( i ) L ) : i ∈ (1 , · · · , n L ) } , unlabeled batch { ( x ( i ) U , d ( i ) U ) : i ∈ (1 , · · · , n U ) } , weak augmentation function A weak , strong augmentation function A strong , unlabeled loss weight λ ∈ R , confidence threshold τ ∈ [0 , 1] 1 Generate pseudolabels ˜ y
U = arg max
y h ◦
A weak ( x U )
[ y ] for the unlabeled data 2 Update model h on loss
[Equation]
A PPLICABLE
DATASETS .
We evaluate
FixMatch on image classification datasets
I W ILD C AM 2020- WILDS , C AMELYON 17- WILDS , P OVERTY M AP - WILDS , and FM O W- WILDS .
We do not evaluate FixMatch on other datasets because FixMatch relies on enforcing consistency across data augmentations, which we only define for image datasets (see Appendix C).
Noisy Student.
Algorithm 5 describes Noisy Student, proposed by Xie et al.
Unlike
Pseudo-Label and FixMatch, which update the model and re-generate new pseudolabels each batch,
Noisy Student generates pseudolabels, fixes them, and then trains the model until convergence before generating new pseudolabels.
First, an initial teacher model is trained on the labeled data; next, the teacher model pseudolabels the unlabeled data, and a student model is trained on the labeled and pseudolabeled data; finally, the student model becomes the new teacher, and the cycle repeats (see Algorithm 5).
Each (teacher, student) pair is termed an iteration ; we study the results of two iterations.
We train Noisy Student using hard pseudolabels, which the teacher generates over weakly augmented inputs:
[Equation]
While the teacher generates pseudolabels on a weakly augmented data, the student must make both labeled and unlabeled predictions on noisy (i.e., strongly augmented) data.
Following Xie et al.
(2020), we add a dropout layer ( p = 0 . 5 ) before the student’s last layer, randomly corrupting final feature maps.
Students thus are trained to be consistent across both data-based and model-based noise.
We denote the model with inserted dropout as Dropout ◦ f .
Xie et al.
(2020) add even more model-based noise using stochastic depth; for simplicity, we do not use stochastic depth in our implementation.
We follow the original paper and fix the balance between labeled and unlabeled losses as λ ( t ) = 1 .
Noisy Student does not use confidence thresholding.
Note that Xie et al.
(2020) use both dropout and strong data augmentations when training the initial teacher on labeled data.
We reuse models from our ERM + Data Augmentation experiments as initial teacher models; thus we differ from Xie et al.
(2020) in that our initial teachers were trained with strong augmentations, but not dropout (see Algorithm 5).
Algorithm 5: Noisy Student
1 Train an initial teacher model f [0] to convergence on labeled examples using the following batch-wise objective
[Equation]
for iteration s ∈ (1 , · · · , S ) do
Generate fixed pseudolabels ˜
y U = arg max y f
[ s − 1] ◦ A weak ( x U ) for the unlabeled data
Train the next student model f [ s ] to convergence on unlabeled and labeled examples using the following batch-wise objective
[Equation]
A PPLICABLE DATASETS .
We evaluate Noisy Student on all datasets except text datasets
C IVIL C OMMENTS - WILDS and A MAZON - WILDS .
For G LOBAL W HEAT - WILDS and OGBM OL PCBA, we run Noisy Student without noise from data augmentations.
B. S ELF - SUPERVISION METHODS
Self-supervised methods learn useful representations by training on unlabeled data via auxiliary “proxy” tasks.
Common approaches include reconstruction tasks (Vincent et al., 2008;
Erhan et al., 2010; Devlin et al., 2019; Gidaris et al., 2018; Lewis et al., 2020), which remove or corrupt a small part of each training example and use it as a prediction goal, and contrastive learning (He et al., 2020;
Chen et al., 2020b; Caron et al., 2020; Radford et al., 2021b).
which aims to learn a representation space such that similar example pairs stay close to each other while dissimilar ones are far apart.
The underlying assumption is that feature encoders that solve the proxy tasks will also perform well on the downstream supervised task (Lee et al., 2020a; Wei et al., 2021).
In our work, we consider two self-supervised methods: SwAV Caron et al.
(2020) for images and masked language modeling (Devlin et al., 2019) for text. We use these methods to pre-train models on the unlabeled data. In all cases, we start with the same model initialization used for all of the other algorithms on that dataset; do additional pre-training via self-supervision on the unlabeled data; and then initialize a new classifier head and finetune the model via ERM with data augmentation. As a concrete example, for FM O W- WILDS , we use the following procedure to run our ERM experiments:
Initialize a DenseNet-121 model (Huang et al., 2017) using ImageNet-pretrained weights. Finetune the model on labeled data from the source domain. Evaluate on held-out data from the target domain. For SwAV, we use the exact same procedure but with the addition of a second step:
Initialize a DenseNet-121 model (Huang et al., 2017) using ImageNet-pretrained weights. Continue pre-training the model with SwAV on unlabeled data from the target domain. Finetune the model on labeled data from the source domain. 4. Evaluate on held-out data from the target domain.
Similarly, for text datasets, we initialized pre-trained BERT models and then continued pre-training them using masked language modeling on the unlabeled data in U-W ILDS .
We tuned hyperparameters for finetuning, following the exact same procedure and hyperparameters as for ERM, but not for pre-training.
SwAV.
We directly use the public SwAV repository made available by Caron et al. (2020) at https://github.com/facebookresearch/swav .
We keep almost all of the hyperparameters used by the original paper for 400 epoch training with batch size 256.
However, we make the following changes based on issues and tips from the original authors in the SwAV repository:
To stabilize training, we opt not to use a queue; this follows the suggestion in issue #69.
For each dataset, we set the number of prototypes to approximately 10x the number of classes; this follows the suggestion in issue #37.
For P OVERTY M AP - WILDS , which is a regression problem, we use 1000 prototypes, which displayed more stable training thanor 100 prototypes.
We set ϵ = 0 .
03 to avoid representation collapse; this follows the suggestion in the Common Issues section of the repository’s readme.
We set the base learning rate via the suggested “linear scaling” rule (issue #37).
In other words, for total batch size (over GPUs) ≥ 512, the learning rate is scaled linearly.
For smaller batch sizes ( < 512), we set the base learning rate at 0.6.
We multiply the base learning rate by 1000 × to obtain the final learning rate, since each of the base/final pairs that the paper reports differ by that factor.
We set the maximum number of epochs to 400 but stop pre-training early when the loss does not decrease by more than 0.3% for 5 consecutive epochs.
A PPLICABLE DATASETS .
We evaluate SwAV on I W ILD C AM
2020- WILDS , C AMELYON 17WILDS , P OVERTY M AP - WILDS , and FM O W- WILDS .
We do not evaluate SwAV on other datasets because SwAV relies training with data augmentations, which we only define for image datasets (see
Appendix C).
Masked language modeling (MLM).
MLM is a popular self-supervised objective for text data and is commonly used to pre-train model representations (Devlin et al., 2019).
Given an unlabeled text corpus X = { X } i
(e.g., a set of comments for CivilComments; a set of reviews for Amazon), a training example ( x, y ) can be generated by randomly masking tokens in each text piece X
(e.g., x = “ The [MASK] is the currency [MASK] the UK ”;
y = (“ pound ” , “ of ”)).
The model is trained to use its representation of the masked input x to predict the original tokens y that should go in each mask.
The MLM objective encourages the model to learn syntactic and semantic knowledge (e.g., to predict “of”) as well as world knowledge
(e.g., to predict “pound”) present in the text corpus (Guu et al., 2020).
For our implementation, we use DistilBERT (Sanh et al., 2019) as our initial model and pre-train it with the MLM objective on the unlabeled data of each task (CivilComments, Amazon).
Following the original BERT implementation (Devlin et al., 2019), we randomly mask 15% of the tokens in each input text piece, of which 80% are replaced with [MASK] , 10% are replaced with a random token (according to the unigram distribution), and 10% are kept unchanged.
A PPLICABLE DATASETS .
We evaluate masked language modeling on the text datasets
C IVIL C OMMENTS - WILDS and A MAZON - WILDS .
C D ATA AUGMENTATION
In this work, several methods we study leverage data augmentations to encourage generalization across domains.
Below, we provide details on our implementations of these augmentations.
Image classification ( I W ILD C AM 2020- WILDS , C AMELYON 17- WILDS , and FM O W- WILDS ).
We use a consistent set of data augmentations across image datasets I W ILD C AM 2020- WILDS ,
C AMELYON 17- WILDS , and FM O W- WILDS .
For methods other than SwAV, we define two strengths of data augmentations: a weak function A weak and a strong function A strong , and we specify both according to Sohn et al.
The weak augmentation function A weak is a random horizontal flip.
The strong augmentation function A strong is a composition of (i) random horizontal flip, (ii)
RandAugment (Cubuk et al., 2020), and (iii) Cutout (DeVries & Taylor, 2017).
For the exact implementation of RandAugment, we directly use the implementation of Zhang et al.
(2021), which is based on the implementation used by Sohn et al.
This implementation specifies a pool of operations and sample magnitudes for each operation uniformly across a pre-specified range.
The pool of operations includes: autocontrast, brightness, color jitter, contrast, equalize, posterize, rotation, sharpness, horizontal and vertical shearing, solarize, and horizontal and vertical translations.
We apply N = 2 random operations for all experiments (see Appendix D.3).
The labeled loss for all methods, including finetuning models pre-trained with SwAV, uses this strong augmentation function.
For SwAV pre-training, we use the data augmentation pipeline used in the original paper (Caron et al., 2020), which is almost identical to the strong data augmentation introduced in SimCLR (Chen et al., 2020a) but with different random crop scales to accommodate the several additional lowerresolution crops.
For each image, the pipeline is the following sequence of random transformations: resized crop, horizontal flip, color jitter, grayscale, and Gaussian blur.
P OVERTY M AP - WILDS .
As P OVERTY M AP - WILDS is a dataset of multispectral images, we define a separate set of data augmentations.
For methods other than SwAV, we define two strengths of data augmentations: a weak function A weak and a strong function A strong .
The weak augmentation function A weak is a random horizontal flip.
The strong augmentation function A strong is a composition of (i) random horizontal flip, (ii) random affine transformation, (iii) color jitter on the RGB channels, and (iv) Cutout on all channels (DeVries & Taylor, 2017).
We use the same augmentations for SwAV pre-training as above for I W ILD C AM 2020- WILDS ,
C AMELYON 17- WILDS , and FM O W- WILDS , but note that the color jitter module is applied only to the RGB channels.
Other datasets.
We do not define data augmentations for other datasets, i.e., G LOBAL W HEATWILDS , OGB-M OL PCBA, C IVIL C OMMENTS - WILDS , and A MAZON - WILDS .
Although
G LOBAL W HEAT - WILDS is an image dataset and can be transformed using augmentations defined above, we omit data augmentations for simplicity, because such augmentations would generally require changing y as well as x (e.g., random translations on the input image also require translating the bounding box labels).
For OGB-M OL PCBA, we omit augmentations because data augmentations on graphs are not well developed.
C IVIL C OMMENTS - WILDS and A MAZON - WILDS are text datasets; although data augmentations have been proposed for text datasets, we do not use these augmentations because training with augmentations is not as standard on text datasets as on image datasets.
For these datasets, methods are benchmarked without augmentations, i.e. we substitute all occurrences of A weak , A strong with the identity.
D E XPERIMENTAL DETAILS D. I N - DISTRIBUTION VS . OUT - OF - DISTRIBUTION PERFORMANCE
We report both in-distribution and out-of-distribution performance metrics on all datasets, with the exception of OGB-M OL PCBA, which does not have a separate in-distribution test set.
Using the terminology in W ILDS (Koh et al., 2021), we consider the train-to-train in-distribution comparison on I W ILD C AM 2020- WILDS , C AMELYON 17- WILDS , FM O W- WILDS , and P OVERTY M AP - WILDS , and the average comparison on C IVIL C OMMENTS - WILDS and A MAZON - WILDS .
D. B ATCH SIZES
For each dataset, we set the total batch size (where a batch contains both labeled and unlabeled data) to the maximum that can fit on 12GB of GPU memory (Table 13).
For all the methods that leverage unlabeled data, except the pre-training algorithms, we run with 4 steps of gradient accumulation, resulting in a 4 × larger effective batch size.
For SwAV pre-training, we run with 4 GPUs in parallel, which achieves a similar effect.
For masked LM pre-training, we run with the default setting ofsteps of gradient accumulation.
These larger batch sizes deviate from the defaults used in the W ILDS paper (Koh et al., 2021).
We use these larger batch sizes because methods that leverage unlabeled data tend to use larger batch sizes (Sohn et al., 2020; Xie et al., 2020; Caron et al., 2020).
[Table]
D. H YPERPARAMETER TUNING
We tune each algorithm separately for each dataset by randomly sampling 10 different hyperparameter configurations within the ranges defined below.
We early stop and select the best hyperparameters based on the OOD validation performance, which is computed on the labeled Validation (OOD) data for each dataset; we do not use the labeled Validation (ID) data in our experiments.
We then run replicates using the best hyperparameters.
For computational reasons, we do not tune hyperparameters for the pre-training algorithms, though we tune the finetuning of their resulting pre-trained models as usual.
Learning rates.
For all the datasets, except for OGB-M OL PCBA, we multiply the learning rates used in W ILDS by the ratio of the effective batch size to the original batch size used in W ILDS .
We center the learning rate grid around this modified learning rate r , and search over 10 U (0 .
1 r, 10 r ) , where U is the uniform distribution.
For OGB-M OL PCBA, we pick r by multiplying the original learning rate by a factor of 10 instead of 4096 / 32 = 128 , because we found that the latter led to unstable optimization.
L 2 -regularization.
Across all datasets and methods, we used the same L 2 -regularization strengths used in W ILDS .
Ratio of unlabeled to labeled data in a batch.
For all the domain-invariant and self-training methods, we search over the ratio of unlabeled to labeled data in a batch, using the values { 3 : 1 , 7 : 1 , 15 : 1 } .
Number of epochs.
We defined an epoch as a complete pass over the labeled data.
This means that the number of batches / gradient steps taken per epoch varies with the ratio of unlabeled to [Table] labeled data in a batch, as a higher ratio means that each batch contains fewer labeled examples.
We adjusted the number of epochs accordingly so that the total amount of compute was similar regardless of the ratio of unlabeled to labeled data in a batch.
We allocated roughly twice as much compute (i.e., processing twice as many batches) to methods that used unlabeled data, compared to the purely-supervised ERM baseline.
Overall, we set the number of epochs based on the W ILDS defaults, with some upwards adjustments (due to the different batch sizes and the use of unlabeled data) if we found that the best hyperparameter configuration had not converged on the validation set.
Table 14 shows the total number of epochs used per dataset.
D. A LGORITHM - SPECIFIC HYPERPARMETERS
We tuned the following algorithm-specific hyperparameters:
CORAL.
We searched over penalty weights 10 U ( −
DANN.
We searched over penalty weights 10 U ( − 1 , 1) and have separate learning rates for the featurizer, classifier and domain discriminator.
We tuned the learning rate for the classifier and domain discriminator, then fixed the learning rate of the featurizer to be a tenth of the learning rate of the classifier.
Pseudo-Label, FixMatch, and NoisyStudent.
We fixed the penalty weight to be 1.
For FixMatch and Pseudo-Label, we searched over confidence thresholds U (0 . 7 , 0 . 95) .
Noisy Student does not use a confidence threshold.
SwAV.
We did not tune SwAV hyperparameters.
See Appendix B.4 for a description of the default hyperparameters used.
Masked language modeling.
We did not tune masked LM hyperparameters, opting instead to use default hyperparameters.
For both C IVIL C OMMENTS - WILDS and A MAZON - WILDS , we pre-trained
DistilBERT for 1,000 steps with a learning rate of 10 − 4 and a batch size of 8,192 sequences using gradient accumulation.
Following W ILDS defaults, for CivilComments, we set the max sequence length to be 300 and for Amazon, 512.
We used FP16 training to speed up pretraining.
D. C OMPUTE INFRASTRUCTURE
We ran experiments on a mix of NVIDIA GPUs: V100, K80, GeForce RTX, Titan RTX, Titan Xp, and Titan V. SwAV pre-training took approximately 3 days × 4 V100 GPUs for each dataset, while masked LM pre-training took approximately 3 days for a single GPU for each dataset.
The other algorithms took less than a day on a V100 to run.
The runtime estimates in Section 6 are estimated for V100 GPUs.
We used the Weights and Biases platform (Biewald, 2020) to monitor experiments.
E E XPERIMENTS ON D OMAIN N ET
Prior work has shown that domain-invariant, self-training, and self-supervised methods can perform well on standard benchmarks for unsupervised domain adaptation.
In this section, we describe our experiments on DomainNet (Peng et al., 2019), a standard unsupervised domain adaptation benchmark for object recognition.
Our goal was to verify that our training/tuning protocol and our implementations of the methods we benchmark in Section 6—which differ slightly from prior work in the ways described in Section B—still result in models that can perform well on DomainNet.
Consistent with prior work, the methods we benchmark in Section 6 all improve over standard ERM training in our DomainNet experiments.
E. S ETUP
DomainNet is an object recognition dataset with approximately 600,000 images across six different domains: sketch , real , quickdraw , painting , infograph and clipart (Peng et al., 2019).
Typically, one of these domains is selected as the source, and another domain as the target for evaluation.
In our experiments, we use the real → sketch setting for two reasons: it is a common choice in prior work on DomainNet, and as our models are pre-trained on ImageNet (following (Zhang et al., 2021)), we wanted to use the real domain as the source to be consistent with the realistic photographs used for ImageNet pretraining.
While it is common to evaluate methods on multiple pairs of source and target domains in DomainNet, in our experiments we only chose one pair, as our goal was only to verify consistency with prior results.
Data.
The DomainNet dataset includes train and test splits for each of the domains, with 70%–30% split between train and test examples.
The real domain has 172,947 images total: 120,906 images in the train split and 52,041 images in the test split.
The target domain, sketch , has a total of 69, images: 48,212 in the train split and 20,916 images are in the test split.
We used this data in the following way:
For training , we used the source training examples (with labels) and the target training examples (without labels).
For validation , we used the same set of target training examples, but with labels; this overestimates performance in a true domain adaptation setting (where one would not have labeled target data), but it is a common practice in the literature, and we followed it for consistency with Jiang et al.
(2020) and Zhang et al.
For evaluation , we used the source test examples as the in-distribution test set, and the target test examples as the out-of-distribution test set.
Hyperparameters and other details.
Other experimental details followed our main experiments in Section 6.
We used the strong and weak data augmentation described for image classification in Appendix C. We set the total batch size to 96, which is the maximum that can fit on 12GB of
GPU memory.
We tuned hyperparameters with the protocol described in Section D.3.
Specifically, for all methods, we fixed L 2 -regularization at 10 − 4 .
We then randomly sampled learning rates from 10 U ( − 4 , − 2) to train the ERM with data augmentation model.
For all other models, we took the best learning rate that we found for the ERM with data augmentation model and searched over one order of magnitude lower and higher from it.
As in Zhang et al.
(2021), we used a ResNet50 model initialized by pretraining on ImageNet.
For SwAV pre-training, instead of following the early-stopping procedure in Appendix B.4, we trained for the full 400 epochs used in Caron et al.
(2020) since the experiment finished relatively quickly compared to the larger U-W ILDS datasets.
E. R ESULTS
Table 15 shows the results of our experiments on real → sketch .
The use of (strong) data augmentation improved ERM performance from 34.9% to 35.9%.
All unsupervised adaptation methods improved over ERM with data augmentation.
We also tested the use of strong vs. weak augmentation for labeled examples for both Pseudo-Label and FixMatch, and we found that using strong augmentation for the labeled examples improves performance.
For DANN, Pseudo-Label, and FixMatch, we compared our results against the results reported in
Zhang et al.
Performance was similar for DANN (ours, 39.4%; theirs, 40.0%).
For FixMatch, our implementation performs better (ours, 50.2%; theirs, 45.3%); this is partially due to our use of strong instead of weak augmentation for the labeled data, which increases performance by
For PseudoLabel, our implementation performs worse (ours, 36.1%; theirs, 40.6%), and we believe it is due to variation in hyperparameter tuning.
For Noisy Student, Berthelot et al.
(2021) reported significantly lower numbers (ours, 39.7%; theirs, 32.6%).
However, this is expected as they trained their models from scratch, whereas we used
ImageNet-pretrained models.
We were unable to find comparable results in prior work for CORAL and SwAV pretraining on the real → sketch split.
However, prior work has shown that these methods can improve performance on other unsupervised adaptation datasets [Table] and Table 15 shows that they both improve performance on real → sketch . [Table]
F F ULLY - LABELED ERM EXPERIMENTAL DETAILS
The self-training methods we evaluate in Section 5 generate a pseudolabel ˜
y U for each unlabeled example x U
and then train on ( x U , ˜ y U ) as if the pseudolabels were true labels.
However, these pseudolabels may not be accurate.
In this section, we describe how we ran fully-labeled ERM experiments using ground truth labels on the “unlabeled” data to establish informal upper bounds on how well we might expect a standard self-training approach to perform with perfect pseudolabel accuracy.
For four of our datasets (A MAZON - WILDS , C IVIL C OMMENTS - WILDS , I W ILD C AM 2020- WILDS , and FM O W- WILDS ), we curated the “unlabeled” data by taking labeled examples and discarding the ground truth labels.
For example, all 268,761 of the unlabeled target reviews in A MAZONWILDS actually have associated star ratings; these are available in our data loaders, but in our main experiments we treat these reviews as unlabeled by not loading the star ratings.
We evaluated models trained via empirical risk minimization (ERM) on the combination of the standard labeled training set and the unlabeled data with these hidden labels revealed.
For example, in A MAZON - WILDS , we pool together the labeled source examples as well as the unlabeled target examples with ground truth labels, and evaluate ERM models trained on all of that data.
As with all of the other experiments in this paper, we evaluate test performance for all datasets on the labeled target splits, so at no point are we training on our actual test examples.
F. H YPERPARAMETERS
Pooling labeled and unlabeled data.
For all datasets, we pooled labeled source examples with examples from the same “unlabeled” split as in our main experiments (Table 2).
We computed gradients for labeled minibatches and unlabeled minibatches separately, which means that for models using batch normalization, the labeled and unlabeled data were normalized separately.
However, we fixed the labeled to “unlabeled” batch size ratios to match the overall labeled to unlabeled dataset size ratio, so other than the batch normalization effects, the training procedure can be viewed as running ERM on the pooled labeled and “unlabeled” data.
Number of epochs.
With the exception of I W ILD C AM 2020- WILDS , detailed below, we follwed the procedure in Appendix D.3 to adjust the number of epochs based on the labeled to unlabeled batch size ratios.
This resulted in a similar amount of computation allocated to these fully-labeled
ERM experiments as the other experiments in Table 2.
Other details.
Other experimental details were kept similar to the other experiments in the paper.
Specifically, we tuned each experiment by randomly sampling 10 different hyperparameters within the ranges defined in Appendix D.3; the only hyperparameter we tuned in these experiments was the learning rate.
We early stopped and selected the best hyperparameters based on the OOD validation performance, and then ran replicates using the best hyperparameters.
We also used data augmentation for I W ILD C AM 2020- WILDS and
FM O W- WILDS
but not for A MAZON - WILDS and
C IVIL C OMMENTS - WILDS .
F. D ATASET - SPECIFIC DETAILS
A MAZON - WILDS .
We matched the experiments in Table 2 by training on the unlabeled target data (268,761 examples).
In addition, we ran a separate experiment where we trained on the unlabeled extra data instead of the unlabeled target data, as the former has 10 × the number of examples (2,927,841 examples).
However, this did not improve performance.
Using the unlabeled target data, we obtained an average accuracy of 73.6 ( ± 0.1) and a 10th percentile accuracy of 56.4 ( ± 0.8), whereas using the unlabeled extra data, we obtained an average accuracy of 73.1 ( ± 0.1) and a 10th percentile accuracy of 54.7 ( ± 0.0).
C IVIL C OMMENTS - WILDS .
We used the unlabeled extra split (1,551,515 examples).
As in our other experiments on C IVIL C OMMENTS - WILDS , we accounted for label imbalance by sampling class-balanced labeled and “unlabeled” batches during training.
[Table] - WILDS .
We used the unlabeled extra split.
Out of the 819,120 unlabeled extra examples, 108,452 examples have ground truth labels (animal species) that are not present in the labeled training and test sets, so we omitted those examples and trained on the remaining 710, examples.
We found that we required twice as many epochs compared to the other unlabeled methods for the fully-labeled ERM training to converge, so we doubled the amount of compute allocated to the fully-labeled I W ILD C AM 2020- WILDS experiments.
FM O W- WILDS .
We used the unlabeled target split (173,208 examples).
G U SING THE U-W ILDS LIBRARY
We provide an open-source PyTorch-based library that builds on top of the existing W ILDS library (Koh et al., 2021).
The U-W ILDS library adds data loaders for each of the 8 datasets with unlabeled data.
These data loaders are compatible with the W ILDS APIs, allowing the unlabeled data to be accessed in a similar way to the labeled data:
[Equation] [Figure]
In addition, we implemented CORAL, DANN, Pseudo-Label, FixMatch, and Noisy Student using the existing W ILDS interfaces.
This allows developers to easily extend these algorithms and evaluate them in a standardized way on all of the U-W ILDS datasets.
The U-W ILDS repository also contains scripts for SwAV pre-training, which uses a modified version of the public SwAV repository that can interface with the W ILDS data loaders, and for masked language model pre-training.