I F YOUR DATA DISTRIBUTION SHIFTS , USE SELF - LEARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We demonstrate that self-learning techniques like entropy minimization and pseudo-labeling are simple and effective at improving performance of a deployed computer vision model under systematic domain shifts.
We show consistent improvements irrespective of the model architecture, the pre-training technique or the type of distribution shift.
At the same time, self-learning is simple to use in practice because it does not require knowledge or access to the original training data or scheme, is robust to hyperparameter choices, is straight-forward to implement and requires only a few adaptation epochs.
This makes selflearning techniques highly attractive for any practitioner who applies machine learning algorithms in the real world.
We present state-of-the art adaptation results on CIFAR10-C (8.5% error), ImageNet-C (22.0% mCE), ImageNet-R (17.4% error) and ImageNet-A (14.8% error), theoretically study the dynamics of self-supervised adaptation methods and propose a new classiﬁcation dataset (ImageNet-D) which is challenging even with adaptation.
I NTRODUCTION
Deep Neural Networks (DNNs) can reach human-level performance in complex cognitive tasks (Brown et al., 2020;
He et al., 2016a; Berner et al., 2019) if the distribution of the test data is sufﬁciently similar to the training data.
However, DNNs are known to struggle if the distribution of the test data is shifted relatively to the training data (Geirhos et al., 2018; Dodge & Karam, 2017).
Two largely distinct communities aim to increase the performance of models under test-time distribution shifts:
The robustness community generally considers ImageNet-scale datasets and evaluates models in an ad-hoc scenario.
Models are trained on a clean source dataset like ImageNet, using heavy data augmentation (Hendrycks et al., 2020a;
Rusak et al., 2020; Geirhos et al., 2019) and/or large-scale pre-training
(Xie et al., 2020a; Mahajan et al., 2018).
The trained models are not adapted in any way to test-time distribution shifts.
This evaluation scenario is relevant for applications in which very different distribution shifts are encountered in an unpredictable order, and hence misses out on the gains of adaptation to unlabeled samples of the target distribution.
The unsupervised domain adaptation (UDA) community often considers smaller-scale datasets and assumes that both the source and the (unlabeled) target dataset are known.
Models are trained on both datasets (e.g., with an adversarial domain objective, Ganin et al., 2016) before evaluation on the target domain data.
This evaluation scenario provides optimal conditions for adaptation, but the reliance on the source dataset makes UDA more computationally expensive, more impractical and prevents the use of pre-trained models for which the source dataset is unknown or simply too large.
In this work, we consider the source-free domain adaptation setting , a middle ground between the classical ad-hoc robustness setting and UDA in which models can adapt to the target distribution but without using the source dataset (Kundu et al., 2020;
Kim et al., 2021; Li et al., 2020; Liang et al., 2020).
This evaluation scenario is interesting for many practitioners and applications as an extension of the ad-hoc robustness scenario.
It evaluates the possible performance of a deployed model on a systematic, unseen distribution shift at inference time: an embedded computer vision system in an autonomous car should adapt to changes without being trained on all available training data; an image-based quality control software may not necessarily open-source the images it has been trained on, but still has to be adapted to the lighting conditions at the operation location; a computer vision system in a hospital should perform robustly when tested on a scanner different from the training images—importantly, it might not be known at development/training time which scanner it will be tested on, and it might be prohibited to share images from many hospitals to run an UDA technique.
Can self-learning methods like pseudo-labeling and entropy-minimization also be used in this source-free domain adaptation setting?
To answer this question, we perform an extensive study of several self-learning variants, and ﬁnd consistent and substantial gains in test-time performance across several robustness and out-of-domain benchmarks and a wide range of models and pretraining methods, including models trained with UDA methods that do not use self-learning.
We also ﬁnd that self-learning outperforms state-of-the-art source-free domain adaptation methods, namely Test-Time Training which is based on a self-supervised auxiliary objective and continual training (Sun et al., 2019b), test-time entropy minimization (Wang et al., 2020) and (gradient-free)
BatchNorm adaptation (Schneider et al., 2020; Nado et al., 2020).
We perform a large number of ablations to study important design choices for self-learning methods in source-free domain adaptation.
Furthermore, we show that a variant of pseudo-labeling with a robust loss function consistently outperforms entropy minimization on ImageNet-scale datasets.
We theoretically analyze and empirically verify the inﬂuence of the temperature parameter in self-learning and provide guidelines how this single parameter should be chosen.
Our approach is visualized in Figure 1.
We do not consider test-time adaptation in an online setting like is studied e.g., by Zhang et al. (2021), where the model is adapted to one example at a time, and reset after each example. Related Work. Variants of self-learning have been used for UDA (Berthelot et al., 2021), for example using auxiliary information (Xie et al., 2020b), consistency (Wei et al., 2020; Cai et al., 2021; Prabhu et al., 2021) or conﬁdence (Zou et al., 2019) regularization.
The main difference from these works to ours is that they 1) utilize both source and target data for self-learning whereas we only require access to unlabeled target data, 2) train their models from scratch whereas we merely ﬁne-tune pretrained checkpoints on the unlabeled target data, and 3) are generally more complicated than our approach due to using more than one term in the objective function.
Our work is conceptually most similar to virtual adversarial domain adaptation in the ﬁne-tuning phase of DIRT-T (Shu et al., 2018)) and Test-time entropy minimization (TENT; Wang et al., 2020).
In contrast to DIRT-T, our objective is simpler and we scale the approach to considerably larger datasets on ImageNet scale.
TENT, on the other hand, only evaluated a single method (entropy minimization) on a single vanilla model (ResNet-50) on IN-C.
We substantially expand this analysis to show that self-learning almost universally increases test-time performance under distribution shifts, regardless of the type of distribution shift, the model architecture or the pre-training method.
Self-learning has also been applied to UDA for semantic segmentation (Zou et al., 2018), for gradual domain adaptation (Kumar et al., 2020), for semi-supervised learning (Rizve et al., 2021; Mukherjee & Awadallah, 2020), for learning in biased datasets (Chen et al., 2020) and for automated data annotation (De Sousa Ribeiro et al., 2020).
Zoph et al.
(2020) show that self-learning outperforms pretraining when stronger data augmentation is used and more labeled data is present.
A more detailed discussion of related work alongside with the main differences to our work can be found in
Appendix F. Our main contribution beyond these works is to show the effectiveness of self-learning on top of both robust, large scale, and domain adapted models, at scale.
S ELF - LEARNING FOR T EST -T IME A DAPTATION
Different variants of self-learning have been used in both unsupervised domain adaptation (French et al., 2018; Shu et al., 2018), self-supervised representation learning (Caron et al., 2021), and in semi-supervised learning (Xie et al., 2020a).
In a typical self-learning setting a teacher network f t trained on the source domain predicts labels on the target domain.
Then, a student model f s is ﬁne-tuned on the predicted labels.
In the following, let f t ( x ) denote the logits for sample x and let p t ( j | x ) ≡ σ j ( f t ( x )) denote the probability for class j obtained from a softmax function σ j ( · ) .
Similarly, f s ( x ) and p s ( j | x ) denote the logits and probabilities for the student model f s .
For all techniques, one can optionally only admit samples where the probability max j p t ( j | x ) exceeds some threshold.
We consider three popular variants of self-learning: Pseudo-labeling with hard or soft labels, as well as entropy minimization.
Hard Pseudo-Labeling (Lee, 2013; Galstyan & Cohen, 2007).
We generate labels using the teacher and train the student on pseudo-labels i using the standard cross-entropy loss,
[Equation]
Usually, only samples with a conﬁdence above a certain threshold are considered for training the student.
We test several thresholds but note that thresholding means discarding a potentially large portion of the data which leads to a performance decrease in itself.
The teacher is updated after each epoch.
Soft Pseudo-Labeling (Lee, 2013; Galstyan & Cohen, 2007).
In contrast to the hard pseudolabeling variant, we here train the student on class probabilities predicted by the teacher,
[Equation]
Soft pseudo-labeling is typically not used in conjunction with thresholding, since it already incorporates the certainty of the model.
The teacher is updated after each epoch.
Entropy Minimization ( ENT ; Grandvalet & Bengio, 2004).
This variant is similar to soft pseudolabeling, but we no longer differentiate between a teacher and student network.
It corresponds to an “instantaneous” update of the teacher.
The training objective becomes
[Equation]
Intuitively, self-training with entropy minimization leads to a sharpening of the output distribution for each sample, making the model more conﬁdent in its predictions.
Robust Pseudo-Labeling ( RPL ). Virtually all introduced self-training variants use the standard cross-entropy classiﬁcation objective.
However, the standard cross-entropy loss has been shown to be sensitive to label noise (Zhang & Sabuncu, 2018; Zhang et al., 2017).
In the setting of domain adaptation, inaccuracies in the teacher predictions and, thus, the labels for the student, are inescapable, with severe repercussions for training stability and hyperparameter sensitivity as we show in the results.
As a straight-forward solution to this problem, we propose to replace the cross-entropy loss by a robust classiﬁcation loss designed to withstand certain amounts of label noise (Ghosh et al., 2017;
Song et al., 2020; Shu et al., 2020; Zhang & Sabuncu, 2018).
A popular candidate is the Generalized
Cross Entropy (GCE) loss which combines the noise-tolerant Mean Absolute Error (MAE) loss (Ghosh et al., 2017) with the CE loss.
We only consider the hard labels and use the robust GCE loss as the training loss for the student,
[Equation] [Equation]
with q ∈ (0 , 1] .
For the limit case q → 0 , the GCE loss approaches the CE loss and for q = 1 , the
GCE loss is the MAE loss (Zhang & Sabuncu, 2018).
We test updating the teacher both after every update step of the student ( RPL ) and once per epoch ( RPL ep ).
E XPERIMENT DESIGN
Datasets.
IN-C (Hendrycks & Dietterich, 2019) contains corrupted versions of the 50 000 images in the IN validation set.
There are ﬁfteen test and four hold-out corruptions, and there are ﬁve severity levels for each corruption.
The established metric to report model performance on IN-C is the mean
Corruption Error (mCE) where the error is normalized by the AlexNet error, and averaged over all corruptions and severity levels, see Eq.
20, Appendix C.1. IN-R (Hendrycks et al., 2020a) contains 30 000 images with artistic renditions of 200 classes of the IN dataset.
IN-A (an, 2019) is composed of 7500 unmodiﬁed real-world images on which standard IN-trained ResNet50 (He et al., 2016b) models yield chance level performance.
CIFAR10 (Krizhevsky et al., 2009) and STL10 (Coates et al., 2011) are small-scale image recognition datasets with 10 classes each, and training sets of 50 000 / 5000 images and test sets of 10 000 / 8000 images, respectively.
The digit datasets MNIST (Deng, 2012) and MNIST-M (Ganin et al., 2016) both have 60 000 training and 10 000 test images.
Hyperparameters. The different self-learning variants have a range of hyperparameters such as the learning rate or the stopping criterion.
Our goal is to give a realistic estimation on the performance to be expected in practice..
To this end, we optimize hyperparameters for each variant of pseudolabeling on a hold-out set of IN-C that contains four types of image corruptions (“speckle noise”, “Gaussian blur”, “saturate” and “spatter”) with ﬁve different strengths each, following the procedure suggested in Hendrycks & Dietterich (2019).
We refer to the hold-out set of IN-C as our dev set.
Models for ImageNet-scale datasets. We consider four popular model architectures:
ResNet (He et al., 2016b), DenseNet161 (Huang et al., 2017), ResNeXt101 (Xie et al., 2017) and
EfﬁcientNet-L2 (Tan & Le, 2019)
(see Appendix B.1 for details on the used models).
For
ResNet50, DenseNet and ResNeXt101, we include a simple vanilla version trained on IN only.
For
ResNet50 and ResNeXt101, we additionally include a state-of-the-art robust version trained with
DeepAugment and Augmix (DAug+AM, Hendrycks et al., 2020a) 1 .
For the ResNeXt model, we also include a version that was trained on 3.5 billion weakly labeled images (IG-3.5B, Mahajan et al., 2018).
Finally, for EfﬁcientNet-L2 we select the current state of the art on IN-C which was trained on 300 million images from JFT-300M (Chollet, 2017; Hinton et al., 2014) using a noisy studentteacher protocol (Xie et al., 2020a).
We validate the IN and IN-C performance of all considered models and match the originally reported scores (Schneider et al., 2020).
For EfﬁcientNet-L2, we match IN top-1 accuracy up to 0.1% points, and IN-C up to 0.6% mCE.
Models for CIFAR10/MNIST-scale datasets. For CIFAR10-C experiments, we use two
WideResNets (WRN, Zagoruyko & Komodakis, 2016): the ﬁrst one is trained on CIFAR10 and has a depth of 28 and a width of 10 and the second one is trained with AugMix (Hendrycks et al., 2020b) and has a depth of 40 and a width of 2.
The remaining small-scale models are trained with unsupervised domain adaptation (UDA) methods.
We propose to regard any UDA method which requires joint training with source and target data as a pre-training step, similar to regular pretraining on IN, and use self-learning on top of the ﬁnal checkpoint.
We consider two popular UDA methods: self-supervised domain adaptation (UDA-SS; Sun et al., 2019a) and Domain-Adversarial
Training of Neural Networks (DANN; Ganin et al., 2016).
In UDA-SS, the authors seek to align the representations of both domains by performing an auxiliary self-supervised task on both domains simultaneously.
In all UDA-SS experiments, we use a WideResNet with a depth of 26 and a width of 16.
In DANN, the authors learn a domain-invariant embedding by optimizing a minimax objective.
For all DANN experiments except for MNIST → MNIST-M, we use the same WRN architecture as above.
For the MNIST → MNIST-M experiment, the training with the larger model diverged and we used a smaller WideResNet version with a width of 2.
We note that DANN training involves optimizing a minimax objective and is generally harder to tune.
Refer to Appendix B.1 for all details.
R ESULTS : S ELF - LEARNING UNIVERSALLY IMPROVES MODELS
Self-learning is a powerful learning scheme, and in the following section we show that it allows to perform test-time adaptation on robustiﬁed models, models obtained with large-scale pre-training, as well as domain adapted models across a wide range of datasets and distribution shifts.
Our main results on large-scale and small-scale datasets are shown in Tables 1 and 2, respectively.
These summary tables show ﬁnal results, and all experiments use the hyperparameters we determined separately on the dev set. architectures on IN-C, IN-A and IN-R.
We adapt the vanilla ResNet50, ResNeXt101 and
DenseNet161 models to IN-C and decrease the mCE by over 19 percent points in all models.
Further, self-learning works for models irrespective of their size: Self-learning substantially improves the performance of the ResNet50 and the ResNext101 trained with DAug+AM, on IN-C by 11.9 and 9.7 percent points, respectively.
Finally, we further improve the current state-of-the-art model on IN-C—the EfﬁcientNet-L2 Noisy Student model—and report a new state-of-the-art result of 22% mCE (which corresponds to a error of 1%) on this benchmark with
[Table]
Self-learning is not limited to the distribution shifts in IN-C like compression artefacts or blur.
On IN-R, a dataset with renditions, self-learning improves both the vanilla ResNet50 and the EfﬁcientNet-L2 model, the latter of which improves from 23.5% to a new state-of-the art of 17.4%
top-1 error. For a vanilla ResNet50, we improve the top-1 error from 63.8% (Hendrycks et al., 2020a) to 54.1%.
On IN-A, adapting the EfﬁcientNet-L2 model using self-learning decreases the top-1 error from 16.5% (Xie et al., 2020a) to 14.8% top-1 error, again constituting a new state of the art with test-time adaptation on this dataset. et al., 2019a), and show that self-learning is effective at further tuning such models to the target domain.
We suggest to view unsupervised source/target domain adaptation as a step comparable to pre-training under corruptions, rather than an adaptation technique speciﬁcally tuned to the target set—indeed, we can achieve error rates using, e.g., DANN + target adaptation previously only possible with source/target based pseudo-labeling, across different common domain adaptation benchmarks. Self-learning also decreases the error on CIFAR10-C of the Wide ResNet model trained with AugMix (AM, Hendrycks et al., 2020b) and reaches a new state of the art on CIFAR10C of 8.5% top1 error with test-time adaptation. † denotes preliminary results on CIFAR-C dev only, due to instabilities in training the adversarial network in DANN.
[Table] [Table]
[Table] [Table] [Table]
been
[Table]
The recently published DINO method (Caron et al., 2021) is another variant of selfsupervised learning that has proven to be effective for unsupervised representation learning. At the core, the method uses soft pseudo-labeling. Here, we test whether a model trained with DINO on the source dataset can be test-time adapted on IN-C using DINO to further improve out-of-distribution performance. Since the used model is a vision transformer model, we test different choices of adaptation parameters and ﬁnd considerable performance improvements in all cases, yielding an mCE of 43.5%mCE at a parameter count comparable to a ResNet50 model.
For adapting the afﬁne layers, we follow Houlsby et al.
[Table]
U NDERSTANDING TEST - TIME ADAPTATION WITH SELF - LEARNING
In the following section, we show ablations and interesting insights of using self-learning for testtime adaptation.
If not speciﬁed otherwise, all ablations are run on the holdout corruptions of IN-C (our dev set) with a vanilla ResNet50. large while the reverse is true on small-scale datasets.
We ﬁnd that robust pseudo-labeling consistently improves over entropy minimization on IN-C, while entropy minimization performs better on smaller scale data (CIFAR10, STL10, MNIST).
The ﬁnding highlights the importance of testing both algorithms on new datasets. The improvement is typically on the order of one percent point:
[Table] [Table]
Having established that
RPL generally performs better than soft- and hard-labeling, we vary the update interval for the teacher.
We ﬁnd that instant updates are most effective.
In entropy minimization, the update interval is instant per default
[Table] [Table]
Note that for Vision Transformers, full model adaptation works better than afﬁne adaptation (see
Table 5).
We also noticed that on convolutional models with a smaller parameter count like
ResNet18, full model adaptation is possible.
Hyperparameters obtained on corruption datasets transfer well to real world datasets.
When evaluating models [Table] we select the hyperparameters discussed above (the learning rate and the epoch used for early stopping are the most critical ones) on the holdout set of IN-C.
We note that this technique transfers well to IN-R, -A and -D, highlighting the practical value of corruption robustness datasets for adapting models on real distribution shifts.
On IN-D, we performed a control experiment where we selected hyperparameters with leave-oneout cross validation—this selection scheme actually performed worse than IN-C parameter selection (see Appendix D.1).
A DAPTING MODELS ON A WIDER RANGE OF DISTRIBUTION SHIFTS REVEALS LIMITATIONS OF ROBUSTIFICATION AND ADAPTATION METHODS
Robustness datasets on ImageNet-scale have so far been limited to a few selected domains (image corruptions in IN-C, image renditions in IN-R, difﬁcult images for ResNet50 classiﬁers in IN-A). In order to test our approach on a wider range of complex distribution shifts, we re-purpose the dataset from the Visual Domain Adaptation Challenge 2019 (DomainNet, Saenko et al., 2019) as an additional robustness benchmark. This dataset comes with six image styles: Clipart, Real, Infograph, Painting, Quickdraw and Sketch.
It has 345 classes in total, of which 164 overlap with IN.
To benchmark robustness of IN trained models out of the box, we ﬁlter out the classes that cannot be mapped to IN and refer to the smaller version of DomainNet as ImageNet-D (IN-D).
We mapclasses in IN to these 164 IN-D classes, e.g., for an image from the “bird” class in IN-D, we accept all 39 bird classes in IN as valid predictions.
We show example images from IN-D in Table 10. The detailed evaluation protocol along with justiﬁcations for our design choices and additional analysis are outlined in Appendix D. The beneﬁt of IN-D over DomainNet is the re-mapping to ImageNet classes which allows robustness researchers to easily benchmark on this dataset, without the need of re-training a model (as common in UDA). To test whether self-learning is helpful for more complex distribution shifts, we adapt a vanilla ResNet50, several robust IN-C models and the EfﬁcientNet-L2 Noisy Student model on IND.
We use the same hyperparameters we obtained on IN-C dev for all our IN-D experiments.
We show our main results in Table 10.
More robust models perform better on IN-D. Comparing the performance of the vanilla ResNet model to its robust DAug+AM variant, we ﬁnd that the DAug+AM model performs better on all domains, with the most signiﬁcant gains on the “Clipart”, “Painting” and “Sketch” domains.
We show detailed results for all domains and all tested models in Appendix D.2, along with results on IN-C and IN-R for comparison.
We ﬁnd that the best performing models on IN-D are also the [Table] strongest ones on IN-C and IN-R which indicates good generalization capabilities of the techniques combined for these models, given the large differences between the three considered datasets.
However, even the best models perform 20 to 30 percentage points worse on IN-D compared to their performance on IN-C or IN-R, indicating that IN-D might be a more challenging benchmark.
All models struggle with some domains of IN-D. The EfﬁcientNet-L2 Noisy Student model obtains the best results on most domains.
However, we note that the overall error rates are surprisingly high compared to the model’s strong performance on the other considered datasets (IN-A: 14.8% top-1 error, IN-R: 17.4% top-1 error, IN-C: 22.0% mCE).
Even on the “Real” domain closest to clean IN where the EfﬁcientNet-L2 model has a top-1 error of 11.6%, the model only reaches a top-1 error of 29.2%.
Self-learning decreases the top1 error on all domains except for “Infograph” and “Quickdraw”.
We note that both domains have very high error rates from the beginning and thus hypothesize that the produced pseudo-labels are of low quality.
Error analysis on IN-D.
We investigate the errors a ResNet50 model makes on IN-D by analyzing the most frequently predicted classes for different domains to reveal systematic errors indicative of the encountered distribution shifts.
We ﬁnd most errors interpretable: the classiﬁer assigns the label “comic book” to images from the “Clipart” or “Painting” domains, “website” to images from the “Infograph” domain, and “envelope” to images from the “Sketch” domain.
Thus, the classiﬁer predicts the domain rather than the class.
We ﬁnd
no systematic errors on the “Real” domain which is expected since this domain should be similar to IN.
Detailed results on the top-3 most frequently predicted classes for different domains can be found in Fig.
6, Appendix D.4.
IN-D should be used as an additional robustness benchmark.
While the error rates on IN-C, -R and -A are at a well-acceptable level for our largest EfﬁcientNet-L2 model after adaptation,
IN-D performance is consistently worse for all models.
We propose to move from isolated benchmark settings like IN-R (single domain) to benchmarks more common in domain adaptation (like DomainNet) and make IN-D publicly available as an easy to use dataset for this purpose. Additional experiments and limitations.
We discuss additional proof-of-concept implementations on the WILDS benchmark, non-batchnorm trained BigTransfer (BiT; ( ? )) models and on selflearning based UDA models in Appendix E.
A SIMPLE MODEL OF STABILITY IN SELF - LEARNING
We observed that different self-learning schemes are optimal for small-scale vs. large-scale datasets and varying amount of classes.
We reconsider the used loss functions, and unify them into
[Equation]
We introduced student and teacher temperature τ s and τ t as parameters in the softmax function and the stop gradient operation sg .
Caron et al.
ﬁxed τ s and varied τ t during training, and empirically found an upper bound for τ t above which the training was no longer stable.
To better understand such behavior, we study the learning dynamics of the loss function in equation 5 theoretically in a simple two-datapoints, two-classes model with linear student and teacher networks f s ( x ) =
x (cid:62) w s and f t ( x ) = x (cid:62) w t deﬁned in Appendix A.1.
Gradient descent with stop gradient corresponds to hard pseudo-labeling in the limit τ t → 0 and to soft pseudo-labeling when τ s = τ t = 1 .
Gradient descent without stop gradient, i e., setting w s = w t =
w corresponds to entropy minimization.
We obtain the following result:
[Figure]
Proposition 1 (Collapse in the two-point model) .
The student and teacher networks w s and w t trained with stop gradient does not collapse to the trivial representation ∀ x : x (cid:62) w s = 0 , x (cid:62) w t[-= 
-]
=if τ s > τ t .
The network w trained without stop gradient does not collapse if τ s > τ t / 2 .
Proof.
see § A.2.
We validate both propositions on a simulated toy model, as well as on the CIFAR-C real dataset and outline the results in Figure 2.
We observe that the size and location of the unstable region depends on the initial classiﬁer and the learning rate.
In practice, the result suggests that student temperatures should exceed the teacher temperatures for pseudo-labeling , and student temperatures should exceed half the teacher temperature for entropy minimization .
Entropy minimization with standard temperatures ( τ s = τ t = 1 ) and hard pseudo-labeling ( τ t → 0 ) are hence stable.
The two-point learning dynamics vanish for soft pseudo-labeling with τ s = τ t , suggesting that one would have to analyze a more complex model with more data points.
While this does not directly imply that the learning is unstable at this point, we empirically observe that both entropy minimization and hard labeling outperform soft-labeling in practice.
C ONCLUSION
We evaluated and analysed how self-learning, an essential component in many unsupervised domain adaptation and self-supervised pre-training techniques, can be applied for adaptation to both small and large-scale image recognition problems common in robustness research.
We demonstrated new state-of-the-art adaptation results with the EfﬁcientNet-L2 model on the benchmarks ImageNet-C, -R, and -A, and introduced a new benchmark dataset (ImageNet-D) which remains challenging even after adaptation.
Our theoretical analysis shows the inﬂuence of the temperature parameter in the self-learning loss function on the training stability and provides guidelines how to choose a suitable value.
We showed that self-learning universally improves test-time performance under diverse distribution shifts irrespective of the architecture or pre-training method.
We hope that our work encourages both researchers and practitioners to use self-learning if their data distribution shifts.
Reproducibility Statement We attempted to make our work as reproducible as possible: We mostly used pre-trained models which are publicly available and we denoted the URL addresses of all used checkpoints; for the checkpoints that were necessary to retrain, we report the Github directories with the source code.
We report all used hyperparameters in the Appendix and will release our code upon acceptance of the paper.
A A TWO - POINT MODEL OF SELF - LEARNING A. D EFINITION OF THE TWO - POINT MODEL
To understand the learning dynamics and properties of different loss functions and their hyperparameters, we propose a simple model of self-learning, both for entropy minimization and pseudo-labeling.
A student network w s ∈ R d and a teacher network w t ∈ R
d are trained on N data points { x i } Ni = with the cross-entropy loss function L deﬁned as
[Equation]
where σ t ( z
[Equation]
=
[Equation]
and σ
[Equation]
Here τ s and τ t denote the student and teacher temperature parameters.
With stop gradient, student and teacher evolve in time according to
[Equation]
where α is the learning rate of the teacher.
Without stop gradient, student and teacher are set equal to each other, and they evolve as
[Equation]
We restrict the theoretical analysis to the time evolution of the components of w s,t in direction of two data points x k and x l , y s,tk
[Equation]
≡ x
[Equation]
l w s,t .
All other components y s,ti with i (cid:54) =
k, l are neglected to reduce the dimensionality of the equation system.
It turns out that the resulting model captures the neural network dynamics quite well despite the drastic simpliﬁcation of taking only two data points into account (see Figure 2).
with stop gradient: ˙ y sk =
[Equation]
without stop gradient:
[Equation]
A. P ROOF OF P ROPOSITION
Learning dynamics with stop gradient.
Computing the stop gradient evolution deﬁned in equation 7 explicitly yields
[Equation]
The second equality uses the well-known derivative of the sigmoid function, ∂ z σ ( z )
= σ ( z ) σ ( − z ) .
The equation system of 2 d nonlinear, coupled ODEs for w s ∈ R d and w t ∈ R d in equation 10 is analytically difﬁcult to analyze.
Instead of studying the ODEs directly, we act on them with the data points x (cid:62) k , k = 1 , . . .
, N , and investigate the dynamics of the components x (cid:62) k w s,t ≡ y s,tk :
[Equation]
The learning rate of each mode y sk is scaled by ( x (cid:62) k x i ) which is much larger for i =
k than for i (cid:54)
=
k in high-dimensional spaces.
In the two-point approximation, we consider only the two (in absolute value) largest terms i , l in
Any that and y l ( t ) might induce in other modes y s,ti ( t ) are neglected, and so we are left with only four ODEs:
[Equation]
The ﬁxed points of equation 12 satisfy
[Equation]
For α > 0 , requiring ˙
y tk = ˙
y tl = 0 implies that y sk =
y tk
and y sl = y tl .
For τ s = τ t , the two remaining equations ˙ y sk
= ˙
y sl = 0 vanish automatically so that there are no non-trivial two-point learning dynamics.
For τ s (cid:54) =
τ t , there is a ﬁxed point at y s,tk = y s,tl = 0 since at this point, each bracket in equation 12 vanishes individually:
[Equation]
At the ﬁxed point y s,tk = y s,tl = 0 , w s and w t are orthogonal to both x k and x l and hence classiﬁcation fails.
If this ﬁxed point is stable, w s and w t will stay at the ﬁxed point once they have reached it, i.e. the model collapses.
The ﬁxed point is stable when all eigenvalues of the Jacobian J of the ODE system equation 12 evaluated at y s,tk = y s,tl = 0 are negative.
This is the case whenever τ s < τ t :
[Equation]
eigenvalues: λ
1 = λ 2 = − α < 0 , λ 3 , 4 = 1(cid:18)τ t −
[Equation]
To sum up
[Equation]
training with stop gradient and τ s > τ t avoids a collapse of the two-point model to the trivial representation y s,tk = y s,tl = 0 since the ﬁxed point is not stable in this parameter regime.
Learning dynamics without stop gradient Without stop gradient
[Equation] we set w t = w s ≡ w which leads to an additional term in the gradient:
[Equation]
As before, we focus on the evolution of the two components y k
= w (cid:62) x k and y l = w (cid:62) x l .
[Equation]
There is a ﬁxed point at y k =
y l = 0 where each bracket in equation 17 vanishes individually,
[Equation]
The Jacobian of the ODE system in equation 17 and its eigenvalues evaluated at the ﬁxed point are given by
[Equation]
Hence the ﬁxed point is unstable when τ s > τ t / 2 and thus the model without stop gradient does not collapse onto y k
= y l = 0 in this regime [Equation]
A. S IMULATION OF THE TWO - POINT MODEL
For visualization purposes [Equation] we set w s = w t =
5] (cid:62) and train the model using instant gradient updates on the dataset with points x 1 = [1 , 0] and x 2 = [0 , − 1] using SGD with learning rate 0 . 1 and momentum 0 . We varied student and teacher temperatures on a log-scale withpoints from 10 − 2 to 10 .
Qualitatively similar results can be obtained without momentum training, at higher learning rates (most likely due to the implicit learning rate scaling introduced by the momentum term).
Note that the temperature scales for observing the collapse effect depend on the learning rate—lower learning rates can empirically prevent the model from collapsing and shift the convergence region.
The result in Figure 2 will hence depend on the exact choice of learning rate (which is currently not considered in our continuous time evolution theory), while the stability region is robust to the learning rate.
B A DDITIONAL INFORMATION ON USED MODELS B. D ETAILS ON ALL HYPERPARAMETERS WE TESTED FOR DIFFERENT MODELS
For all models except EfﬁcientNet-L2, we adapt the batch norm statistics to the test domains following (Schneider et al., 2020). We do not expect signiﬁcant gains for combining EfﬁcientNet-L with batch norm adaptation: as demonstrated in (Schneider et al., 2020), models trained with large amounts of weakly labeled data do not seem to beneﬁt from batch norm adaptation.
ResNet50 models We use a vanilla ResNet50 model and compare soft- and hard-labeling against entropy minimization and robust pseudo-labeling.
To ﬁnd optimal hyperparameters for all methods, we perform an extensive evaluation and test (i) three different adaptation mechanisms (ii) several learning rates 1 .
× 10 − 4 , 1 .
× 10 − 3 , 1 .
× 10 − 2 and 5 .
× 10 − 2 , (iii) the number of training epochs and (iv) updating the teacher after each epoch or each iteration.
For all experiments, we use a batch size of 128.
The hyperparameter search is performed on IN-C dev.
We then use the optimal hyperparameters to evaluate the methods on the IN-C test set.
ResNeXt101 models The ResNeXt101 model is considerably larger than the ResNet50 model and we therefore limit the number of ablation studies we perform for this architecture.
Besides a baseline, we include a state-of-the-art robust version trained with DeepAugment+Augmix (DAug+AM, Hendrycks et al., 2020a) and a version that was trained on 3.5 billion weakly labeled images (IG-3.5B, Mahajan et al., 2018).
We only test the two leading methods on the ResNeXt models ( ENT and RPL ).
We vary the learning rate in same interval as for the ResNet50 model but scale it down linearly to account for the smaller batch size of 32.
We only train the afﬁne batch normalization parameters because adapting only these parameters leads to the best results on ResNet50 and is much more resource efﬁcient than adapting all model parameters.
Again, the hyperparameter search is performed only on the development corruptions of IN-C.
We then use the optimal hyperparameters to evaluate the methods on the IN-C test set.
EfﬁcientNet-L2 models
The current state of the art on IN, IN-C, IN-R and IN-A is an
EfﬁcientNet-L2 trained on 300 million images from JFT-300M (Chollet, 2017; Hinton et al., 2014) using a noisy student-teacher protocol (Xie et al., 2020a).
We adapt this model for only one epoch due to resource constraints.
During the hyperparameter search, we only evaluate three corruptions on the IN-C development set 2 and test the learning rates 4 .
6 × 10 − 2 , 4 . 6 × 10 − 3 , 4 . 6 × 10 − 4 and 4 .
6 × 10 − 5 .
We use the optimal hyperparameters to evaluate ENT and RPL on the full IN-C test set (with all severity levels).
UDA-SS models We trained the models using the scripts from the ofﬁcial code base at github.
com/yueatsprograms/uda release.
We used the provided scripts for the cases: (a) source: CIFAR10, target:
STL10 and (b) source: MNIST, target: MNIST-M. For the case (c) source: CIFAR10, target:
CIFAR10-C, we used the hyperparameters from case (a) since this case seemed to be the closest match to the new setting.
We think that the baseline performance of the UDA-SS models can be further improved with hyperparameter tuning.
DANN models To train models with the DANN-method, we used the PyTorch implementation of this paper at https://github.com/fungtion/DANN py3.
The code base only provides scripts and hyperparameters for the case (b) source: MNIST, target: MNIST-M.
For the cases (a) and (c), we used the same optimizer and trained the model for 100 epochs.
We think that the baseline performance of the DANN models can be further improved with hyperparameter tuning.
Preprocessing For IN, IN-R, IN-A and IN-D, we resize all images to 256 × 256 px and take the center 224 × 224 px crop.
The IN-C images are already rescaled and cropped.
We center and re-scale the color values with µ RGB =
456 , 0 . 406] and σ RGB =
For the EfﬁcientNet-L2, we follow the procedure in Xie et al. (2020a)
and rescale all inputs to a resolution of 507 × 507 px and then center-crop them to 475 × 475 px.
B. F ULL LIST OF USED MODELS
ImageNet scale models ImageNet trained models (ResNet50, DenseNet161, ResNeXt) are taken directly from torchvision (Marcel & Rodriguez, 2010).
The model variants trained with
DeepAugment and AugMix augmentations (Hendrycks et al.
, 2020b;a) are taken from https: //github.com/hendrycks/imagenet-r.
The weakly-supervised ResNeXt101 model is taken from the
PyTorch Hub.
For EfﬁcientNet (Tan & Le, 2019), we use the PyTorch re-implementation available at https://github.com/rwightman/gen-efﬁcientnet-pytorch.
This is a veriﬁed re-implementation of the original work by Xie et al.
(2020a). We verify the performance on ImageNet, yielding a 88.23% top-1 accuracy and 98.546% top-5 accuracy which is within 0.2% points of the originally reported result (Xie et al., 2020a).
On ImageNet-C, our reproduced baseline achieves 28.9% mCE vs. 28.3% mCE originally reported by Xie et al. (2020a).
As noted in the re-implementation, this offset is possible due to minor differences in the pre-processing.
It is possible that our adaptation results would improve further when applied on the original codebase by Xie et al. .
Small scale models We train the UDA-SS models using the original code base at github.com/ yueatsprograms/uda release, with the hyperparameters given in the provided bash scripts.
For our
DANN experiments, we use the PyTorch implementation at github.com/fungtion/DANN py3.
We use the hyperparameters in the provided bash scripts.
The following Table 11 contains all models we evaluated on various datasets with references and links to the corresponding source code.
[Table]
The established performance metric on IN-C is the mean Corruption Error (mCE), which is obtained by normalizing the model’s top-1 errors with the top-1 errors of AlexNet across the C=15 test corruptions and S=5 severities:
[Equation]
The AlexNet errors used for normalization are shown in Table
[Table]
C. D ETAILED RESULTS FOR TUNING EPOCHS AND LEARNING RATES
We tune the learning rate for all models and the number of training epochs for all models except the EfﬁcientNet-L2.
In this section, we present detailed results for tuning these hyperparameters for all considered models.
The best hyperparameters that we found in this analysis, are summarized in
Table 17. for
ENT and RPL for different numbers of training epochs when adapting the afﬁne batch norm parameters of a ResNet50 model.
[Table]
[Table]
[Table]
[Table] [Table] C. D ETAILED RESULTS FOR ALL IN-C CORRUPTIONS
We outline detailed results for all corruptions and models in Table 18.
Performance across the severities in the dataset is depicted in Figure 3.
All detailed results presented here are obtained by following the model selection protocol outlined in the main text [Figure] [Table]
C. D ETAILED RESULTS FOR THE CIFAR10-C AND UDA ADAPTATION
[Table] [Table] C. A BLATION OVER THE HYPERPARAMETER q FOR RPL
For RPL [Table] we must choose the hyperparameter q .
We performed an ablation study over q and show results in Table [Table] demonstrating that RPL is robust to the choice of q , with slight preference to higher values.
Note:
In the initial parameter sweep for this paper, we only compared q = 0 .
7 and q = 0 . 8 .
Given the result in Table 21, it could be interesting to re-run the models in Table 1 of the main paper with q = 0 . 9 , which could yield another (small) improvement in mCE.
[Table]
C. S ELF - TRAINING OUTPERFORMS CONTRASTIVE T EST -T IME T RAINING (S UN ET AL ., 2019 B )
Sun et al.
(2019b) use a ResNet18 for their experiments on ImageNet and only evaluate their method on severity 5 of IN-C.
To enable a fair comparison, we trained a ResNet18 with both hard labeling and RPL and compare the efﬁcacy of both methods to Test-Time Training in Table 22.
For both hard labeling and RPL , we use the hyperparameters we found for the vanilla ResNet50 model and thus, we expect even better results for hyperparameters tuned on the vanilla ResNet18 model and following our general hyperparameter search protocol.
While all methods (self-learning and TTT) improve the performance over a simple vanilla ResNet [Table] we note that even the very simple baseline using hard labeling already outperfoms Test-Time
Training; further gains are possible with RPL.
The result highlights the importance of simple baselines (like self-learning) when proposing new domain adaptation schemes.
It is likely that many established DA techniques more complex than the basic self-learning techniques considered in this work will even further improve over TTT and other adaptation approaches developed exclusively in robustness settings [Table] [Equation] [Table]
C. E FFECT OF BATCH SIZE AND LINEAR LEARNING RATE SCALING
How is self-learning performance affected by batch size constraints?
We compare the effect of different batch sizes and linear learning rate scaling.
In general, we found that afﬁne adaptation experiments on ResNet50 scale can be run with batch size 128 on a Nvidia V100 GPU (16GB), while only batch size 96 experiments are possible on RTX 2080 GPUs
[Table]
The results in Table 23 show that for a ResNet50 model, higher batch size yields a generally better performance.
[Table]
To limit the amount of compute
[Table] we ran RPL and ENT for our vanilla ResNet50 model three times with the optimal hyperparameters.
The averaged results, displayed as “mean (unbiased std)” are: [Table] [Table] [Table]
C. S ELF - LEARNING AS CONTINUOUS TEST - TEST ADAPTATION
We test our method on continuous test-time adaptation where the model adapts to a continuous stream of data from the same domain. In Fig. 4, we display the error of the Noisy Student L2 model while it is being adapted to ImageNet-C and ImageNet -R.
The model performance improves as the model sees more data from the new domain.
We differentiate continuous test-time adaptation from the online test-time adaptation setting (Zhang et al., 2021) where the model is adapted to each test sample individually, and reset after each test sample.
[Figure]
D D ETAILED AND ADDITIONAL R ESULTS ON IN-D D. E VALUATION PROTOCOL ON IN-D
The domains in IN-D differ in terms of their difﬁculty for the studied models.
Therefore, to calculate an aggregate score, we propose normalizing the error rates by the error achieved by AlexNet on the respective domains to calculate the mean error, following the approach in Hendrycks & Dietterich (2019) for IN-C.
This way, we obtain the aggregate score mean Domain Error (mDE) by calculating the mean over different domains,
[Equation]
where E fd is the top-1 error of a classiﬁer f on domain d .
Leave-one-out-cross-validation For all IN-D results we report in this paper, we chose the hyperparameters on the IN-C dev set.
We tried a different model selection scheme on IN-D as a control experiment with “Leave one out cross-validation” (L1outCV): with a round-robin procedure, we choose the hyperparameters for the test domain on all other domains.
We select the same hyperparameters as when tuning on the “dev” set: For the ResNet50 model, we select over the number of training epochs (with a maximum of 7 training epochs) and search for the optimal learning rate in the set [ 0 . 01 , 0 .
For the EfﬁcientNet-L2 model, we train only for one epoch as before and select the optimal learning rate in the set [ 4 . 6 × 10 − 3 , 4 .
6 × 10 − 4 , 4 . 6 × 10 − 5 , 4 . 6 × 10 − 6 ].
This model selection leads to worse results both for the ResNet50 and the EfﬁcientNetL2 models, highlighting the robustness of our model selection process, see Table 25. [Table]
D. D ETAILED RESULTS FOR ROBUST R ES N ET 50 MODELS ON IN-D
We show detailed results for all models on IN-D for vanilla evaluation (Table 26) BN adaptation (Table 27), RPL q =0 .
8 (Table 28) and ENT (Table 29).
For RPL q =0 . 8 and ENT , we use the same hyperparameters that we chose on our IN-C ‘dev’ set.
This means we train the models for 5 epochs with RPL q =0 .
8 and for one epoch with ENT .
We evaluate the pre-trained and public checkpoints of SIN (Geirhos et al., 2019), ANT (Rusak et al., 2020), ANT+SIN (Rusak et al., 2020), AugMix (Hendrycks et al., 2020b), DeepAugment (Hendrycks et al., 2020a) and DeepAug+Augmix (Hendrycks et al., 2020a) in the following tables.
[Table]
[Table]
[Table] [Table] [Table] [Table]
AugMix 70.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
DeepAugment
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
DeepAug
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
The summary results for all models are shown in Table 30.
We show the top-1 error for the different IN-D domains versus training epochs for a vanilla ResNet in Fig.
We indicate the epochs 1 and 5 at which we extract the errors with dashed black lines.
[Figure]
D. D ETAILED RESULTS FOR THE E FFICIENT N ET -L2 N OISY S TUDENT MODEL ON IN-D
We show the detailed results for the EfﬁcientNet-L2 Noisy Student model on IN-D in Table
[Table]
[Table]
D ETAILED RESULTS ON THE ERROR ANALYSIS ON IN-D
Frequently predicted classes We analyze the most frequently predicted classes on IN-D by a vanilla ResNet50 and show the results in Fig.
We make several interesting observations:
First, we ﬁnd most errors interpretable: it makes sense that a ResNet50 assigns the label “comic book” to images from the “clipart” or “painting” domains, or “website” to images from the “infograph” domain, or “envelope” to images from the “sketch” domain.
Second, on the hard domain “quickdraw”, the ResNet50 mostly predicts non-sensical classes that are not in IN-D, mirroring its almost chance performance on this domain.
Third, we ﬁnd no systematic errors on the “real” domain which is expected since this domain should be similar to IN.
Filtering predictions on IN-D that cannot be mapped to ImageNet-D
We perform a second analysis: We ﬁlter the predicted labels according to whether they can be mapped to IN-D and report the ﬁltered top-1 errors as well as the percentage of ﬁltered out inputs in Table 32.
We note that for the domains “infograph” and “quickdraw”, the ResNet50 predicts labels that cannot be mapped to
IN-D in over 70% of all cases, highlighting the hardness of these two domains.
[Table] Filtering labels and predictions on IN that cannot be mapped to ImageNet-D To test for possible class-bias effects, we test the performance of a ResNet50 model on IN classes that can be mapped to IN-D and report the results in Table [Table]
First, we map IN labels to IN-D to make the setting as similar as possible to our experiments on
IN-D and report the top-1 error (12.1%).
This error is signiﬁcantly lower compared to the toperror a ResNet50 obtains following the standard evaluation protocol (23.9%).
This can be explained by the simpliﬁcation of the task: While in IN there are 39 bird classes, these are all mapped to the same hierarchical class in IN-D.
Therefore, the classes in IN-D are more dissimilar from each other than in IN.
Additionally, there are only 164 IN-D classes compared to the 1000 IN classes, raising the chance level prediction.
If we further only accept predictions that can be mapped to IN-D [Table] the top-1 error is slightly increased to 13.4%.
In total, about 52.7% of all images in the IN validation set cannot be mapped to IN-D. [Figure] f p r e d i c t i o [Figure]
r
[Figure]
f
[Figure] r e [Figure] i c
[Figure]
i
[Figure] r [Figure] i c t
[Figure]
D. T OP -1 ERROR ON IN-D FOR A LEX N ET
We report the top-1 error numbers on different IN-D as achieved by AlexNet in Table 33.
We used these numbers for normalization when calculating mDE.
[Table]
E A DDITIONAL EXPERIMENTS E. B EYOND I MAGE N ET CLASSES : S ELF - LEARNING ON WILDS
The WILDS benchmark (Koh et al., 2021) is comprised of ten tasks to test domain generalization, subpopulation shift, and combinations thereof.
In contrast to the setting considered here, many of the datasets in WILDS mix several 10s or 100s domains during test time.
The Camelyon17 dataset in WILDS contains histopathological images, with the labels being binary indicators of whether the central 32 × 32 region contains any tumor tissue; the domain identiﬁes the hospital that the patch was taken from.
Camelyon17 contains three different test splits with different domains and varying difﬁculty levels.
For evaluation, we took the pretrained checkpoint from worksheets.codalab.org/worksheets/0x00d14c55993548a1823a710642f6d608 (camelyon erm densenet121 seed0) for a DenseNet121 model (Huang et al., 2017) and veriﬁed the reported baseline performance numbers.
We adapt the models using ENT or RPL for a maximum ofepochs using learning rates { 3 × 10 − 5 , 3 × 10 − 4 , . . .
3 × 10 − 1 } .
The best hyperparameter is selected according to OOD Validation accuracy.
The RxRx1 dataset in WILDS contains RGB images of cells obtained by ﬂuorescent microscopy, with the labels indicating which of the 1,139 genetic treatments (including no treatment) the cells received; the domain identiﬁes the batch in which the imaging experiment was run.
The RxRx dataset contains three test splits, however, unlike Camelyon17, in all of the splits the domains are mixed.
For evaluation, we took the pretrained checkpoint from worksheets.codalab.org/bundles/ 0x7d33860545b64acca5047396d42c0ea0 for a ResNet50 model and veriﬁed the reported baseline performance numbers.
{+We adapt the models using ENT or RPL for a maximum of 10 epochs using learning rates { 6 . 
+}
× 10 − 6 , 6 . 25 × 10 − 5 , . . .
25 × 10 − 2 } .
The best hyperparameter is selected according to OOD Validation accuracy.
[Table]
The FMoW dataset in WILDS contains RGB satellite images, with the labels being one ofbuilding or land use categories; the domain speciﬁes the year in which the image was taken and its geographical region (Africa, the Americas, Oceania, Asia, or Europe).
The FMoW dataset contains four test splits for different time periods, for which all regions are mixed together.
For evaluation, we took the pretrained checkpoint from //worksheets.codalab.org/ bundles/0x20182ee424504e4a916fe88c91afd5a2 for a DenseNet121 model and veriﬁed the reported baseline performance numbers.
{+We adapt the models using ENT or RPL for a maximum of 10 epochs using learning rates { 5 . 
+}
{+× 10 − 6 , 5 . 
+}
× 10 − 5 , . . .
× 10 − 2 } .
The best hyperparameter is selected according to OOD Validation accuracy.
While we see improvements on Camelyon17, neither BN adaptation nor self-learning can improve performance on RxRx1 or FMoW. Initial experiments on PovertyMap and iWildsCam also do not show improvements with self-learning.
We hypothesize that the reason lies in the mixing of the domains: Both BN adaptation and our self-learning methods work best on systematic domain shifts.
These results support our claim that self-learning is effective, while showing the important limitation when applied to more diverse shifts.
E. S MALL IMPROVEMENTS ON B IG T RANSFER MODELS WITH G ROUP NORMALIZATION LAYERS
We evaluated BigTransfer models (Kolesnikov et al., 2020) provided by the timm library (Wightman, 2019).
A difference to the ResNet50, ResNeXt101 and EfﬁcientNet models is the use of group normalization layers, which might inﬂuence the optimal method for adaptation.
For afﬁne adaptation, a distilled BigTransfer ResNet50 model improves from 49.6 % to 48.6 % mCE on the ImageNet-C development set, and from 55.0 % to 54.4 % mCE on the ImageNet-C test set when using RPL ( q = 0 . 8 ) for adaptation, at learning rate 7 . 5 × 10 − 4 at batch size 64 after a single adaptation epoch.
E. C AN S ELF -L EARNING IMPROVE OVER S ELF -L EARNING BASED UDA?
An interesting question is whether test-time adaptation with self-learning can improve upon selflearning based UDA methods.
To investigate this question, we build upon French et al.
(2018) and their released code base at github.com/Britefury/self-ensemble-visual-domain-adapt.
We trained the Baseline models from scratch using the provided shell scripts with the default hyperparameters and veriﬁed the reported performance.
For adaptation, we tested BN adaptation, ENT , RPL , as well as continuing to train in exactly the setup of French et al.
(2018), but without the supervised loss.
For the different losses, we adapt the models for a maximum of 10 epochs using learning rates { 1 × 10 − 5 , 1 × 10 − 4 , . . .
, 1 × 10 − 1 } .
Note that for this experiment, in contrast to any other result in this paper, we purposefully do not perform proper hyperparameter selection based on a validation dataset —instead we report the best accuracy across all tested epochs and learning rates to give an upper bound on the achievable performance for test-time adaptation.
As highlighted in Table 35, none of the four tested variants is able to meaningfully improve over the baseline, corroborating our initial hypothesis that self-learning within a full UDA setting is the optimal strategy, if dataset size and compute permits. On the other hand, results like the teacher reﬁnement step in DIRT-T (Shu et al., 2018) show that with additional modiﬁcations in the loss function, it might be possible to improve over standard UDA with additional adaptation at test time. [Table] F D ETAILED D ISCUSSION OF R ELATED W ORK
Self-learning for domain adaptation Xie et al.
(2020b) introduce “In-N-Out” which uses auxiliary information to boost both in- and out-of-distribution performance.
AdaMatch (Berthelot et al., 2021) builds upon FixMatch (Sohn et al., 2020) and can be used for the tasks of unsupervised domain adaptation, semi-supervised learning and semi-supervised domain adaptation as a generalpurpose algorithm.
Prabhu et al.
(2021) propose SENTRY, an algorithm based on judging the predictive consistency of samples from the target domain under different image transformations.
Zou et al.
(2019) show that different types of conﬁdence regularization can improve the performance of self-learning.
A theoretically motivated framework for self-learning in domain adaptation based on consistency regularization has been proposed by Wei et al.
and then extended by Cai et al.
Self-learning has also been used for semantic segmentation (Zou et al., 2018).
The main difference from these works to ours is that they 1) utilize both source and target data during training (i.e., the classical UDA setup) whereas we only require access to unlabeled target data (source-free setup), and 2) train their models from scratch whereas we adapt pretrained checkpoints to the unlabeled target data, 3) are oftentimes more complicated (also in terms of the number of hyperparameters) than our approach due to using more than one term in the objective function.
We would like to highlight that utilizing source data should always result in better performance compared to not using source data.
Our contribution is to show that self-learning can still be very beneﬁcial with a small compute budget and no access to source data.
Our setup targets “deployed systems”, e.g., a self-driving car or a detection algorithm in a production line which adapts to the distribution shift “on-the-ﬂy” and cannot (or should not) be retrained from scratch for every new domain shift.
Kumar et al.
(2020) study the setting of self-learning for gradual domain adaptation.
They ﬁnd that self-learning works better if the data distribution changes slowly.
The gradual domain adaptation setting differs from ours; instead of a gradual shift over time, we focus on a ﬁxed, systematic shift at test time dataset.
Kumar et al.
(2020) tested their method on a synthetic Gaussian dataset, MNIST and the Portraits datasets; building and evaluating ImageNet-scale datasets for a gradual domain adaptation perspective is a very interesting extension of our work, but left for future work, and would not only require changes/adaptations to the self-learning method, but also to the evaluation datasets.
Chen et al.
(2020) prove that under certain conditions, self-learning can improve performance in biased datasets where spurious features correlate with the label in the source domain but are independent of the label in the target domain.
While Chen et al.
(2020) also consider the setting of source-free domain adaptation (like we do), they limit their experiments to small scale models on MNIST and Celeb-A, while we conduct a large scale empirical study on all common robustness datasets with large scale models – some of the observed and studied effects in our paper (effectiveness of different loss functions at different problem scales, adaptation mechanisms, etc.) can be attributed to this large scale evaluation setting, and extending our insights over small scale experiments.
Similar to us, Chen et al.
ﬁnd that a strong source classiﬁer is necessary for self-learning to work; however, in their case, a teacher accuracy of 72% (on CMNIST10) is already too low and leads to worse student accuracy.
In contrast, in our experiments, self-learning still works for an mCE as high as 80% (cf. appendix Figure 3, severity 5) and teacher accuracies as low as 10.4% (on ImageNet-D “Infograph”), and breaks down at accuracies around 1-2% (on ImageNet-D “Quickdraw”).
This discrepancy might be due to the spurious correlations that Chen et al.
(2020) introduced in their dataset leading to systematic biases, which are not present in the datasets we studied.
Self-learning in semi-supervised learning
(SSL)
In a different line of work which is not related to domain adaptation directly, self-learning has been used in a semi-supervised setting.
Zoph et al.
(2020) show that self-learning outperforms pretraining when stronger data augmentation is used and more labeled data is present.
They use human labels on the target task (e.g., object detection on
COCO) and pseudo-labels on an unlabeled dataset (e.g. ImageNet), and optimize the loss on both datasets, with the aim to improve performance on the task where ground truth labels are known.
The work of Zoph et al.
(2020) is orthogonal to ours, in the sense that we could adapt their ﬁnal checkpoint to a new domain with our method, similar to how we adapted the Noisy Student model which was also trained using self-learning.
Rizve et al.
(2021) propose an uncertainty-aware pseudo-label selection (UPS) framework which outperforms other SSL methods in a few-label regime.
UPS is helpful to reduce the impact of noisy pseudo-labels; in our case, we use the generalized cross-entropy loss for this purpose.
Testing the
UPS framework (and other means for improving the quality of pseudo-labels, or robustness against label noise) on robustness datasets would be an interesting direction for future work.
De Sousa Ribeiro et al.
(2020) propose Deep Bayesian Self-Training (DBST) for automatic data annotation.
Mukherjee & Awadallah (2020) suggest using self-learning in a semi-supervised setting for text classiﬁcation with few labels.
G S OFTWARE STACK
We use different open source software packages for our experiments, most notably Docker (Merkel, 2014), scipy and numpy (Virtanen et al., 2020), GNU parallel (Tange, 2011), Tensorﬂow (Abadi et al., 2016), PyTorch (Paszke et al., 2017), timm (Wightman, 2019), Self-ensembling for visual domain adaptation (French et al., 2018), the WILDS benchmark (Koh et al., 2021), and torchvision (Marcel & Rodriguez, 2010).