A DA M ATCH : A U NIFIED A PPROACH TO S EMIS UPERVISED L EARNING AND D OMAIN A DAPTATION
A BSTRACT
We extend semi-supervised learning to the problem of domain adaptation to learn signiﬁcantly higher-accuracy models that train on one data distribution and test on a different one.
With the goal of generality, we introduce AdaMatch, a uniﬁed solution for unsupervised domain adaptation (UDA), semi-supervised learning (SSL), and semi-supervised domain adaptation (SSDA).
In an extensive experimental study, we compare its behavior with respective state-of-the-art techniques from SSL, SSDA, and UDA and ﬁnd that AdaMatch either matches or signiﬁcantly exceeds the state-of-the-art in each case using the same hyper-parameters regardless of the dataset or task.
For example, AdaMatch nearly doubles the accuracy compared to that of the prior state-of-the-art on the UDA task for DomainNet and even exceeds the accuracy of the prior state-of-the-art obtained with pre-training by 6.4% when AdaMatch is trained completely from scratch.
Furthermore, by providing AdaMatch with just one labeled example per class from the target domain (i.e., the SSDA setting), we increase the target accuracy by an additional 6.1%, and with 5 labeled examples, by 13.6%.
I NTRODUCTION
Since the inception of domain adaptation and knowledge transfer, researchers have been well aware of various conﬁgurations of labeled or unlabeled data and assumptions on domain shift (Csurka, 2017).
Unsupervised domain adaptation (UDA), semi-supervised learning (SSL), and semi-supervised domain adaptation (SSDA) all use different conﬁgurations of labeled and unlabeled data, with the major distinction being that, unlike SSL, UDA and SSDA assume a domain shift between the labeled and unlabeled data (see Table 1).
However, currently the ﬁelds of SSL and UDA/SSDA are fragmented: different techniques are developed in isolation for each setting, and there are only a handful of algorithms that are evaluated on both (French et al., 2018).
Techniques that leverage unlabeled data are of utmost importance in practical applications of machine learning because labeling data is expensive.
It is also the case that in practice the available unlabeled data will have a distribution shift.
Addressing this distribution shift is necessary because neural networks are not robust (Recht et al., 2019a; Biggio & Roli, 2018; Szegedy et al., 2013; Hendrycks & Dietterich, 2019; Azulay & Weiss, 2018; Shankar et al., 2019;
Gu et al., 2019; Taori et al., 2020) to even slight differences between the training distribution and test distribution.
Although there are techniques to improve out-of-distribution robustness assuming no access to unlabeled data from the target domain (Hendrycks et al., 2020; Zhang, 2019; Engstrom et al., 2019; Geirhos et al., 2018; Yang et al., 2019; Zhang et al., 2019), it is common in practice to have access to unlabeled data in a shifted domain (i.e. the UDA or SSDA setting), and leveraging this unlabeled data allows for much higher accuracy.
Moreover, while SSDA (Donahue et al., 2013; Yao et al., 2015; Ao et al., 2017; Saito et al., 2019) has received less attention than both SSL and UDA, we believe it describes a realistic scenario in practice and should be equally considered.
In this work, we introduce AdaMatch, a uniﬁed solution designed to solve the tasks of UDA, SSL, and
SSDA using the same set of hyperparameters regardless of the dataset or task .
AdaMatch extends
FixMatch (Sohn et al., 2020) by (1) addressing the distribution shift between source and target domains present in the batch norm statistics, (2) adjusting the pseudo-label conﬁdence threshold on-the-ﬂy, and (3) using a modiﬁed version of distribution alignment from Berthelot et al.
[Table]
AdaMatch sets a new state-of-the-art accuracy of 28.7% for UDA without pre-training and 33.4% with pre-training on DomainNet, an increase of 11.1% when compared on the same code base.
With just one label per class on the target dataset, AdaMatch is more data efﬁcient than other method, achieving a gain of 6.1% over UDA and 13.6% with 5 labels.
We additionally promote democratic research by reporting results on a smaller 64 × 64 DomainNet.
This results in a minimal drop in accuracy compared to the full resolution, and compared to the practice of sub-selecting dataset pairs, does not bias the results towards easier or harder datasets.
Finally, we perform an extensive ablation analysis to understand the importance of each improvement and modiﬁcation that distinguishes
AdaMatch from prior semi-supervised learning methods.
R ELATED W ORK
Unsupervised Domain Adaptation (UDA).
UDA studies the performance of models trained on a labeled source domain and an unlabeled target domain with the goal of obtaining high accuracy on the target domain.
Inspired by the theoretical analysis of domain adaptation (Ben-David et al., 2010), a major focus has been reducing the discrepancy of representations between domains, so that a classiﬁer that is learned on the source features works well on the target features.
UDA methods can be categorized by the technique they use to measure this discrepancy.
For example, (Long et al., 2013;
Tzeng et al., 2014; 2015) use the maximum mean discrepancy (Gretton et al., 2012), (Sun & Saenko, 2016) use correlation alignment across domains, and domain adversarial neural networks (Ajakan et al., 2014; Ganin et al., 2016;
Bousmalis et al., 2017;
Saito et al., 2018) measure the domain discrepancy using a discriminator network.
Maximum classiﬁer discrepancy (Saito et al., 2018) (MCD) measures the domain discrepancy via multiple task classiﬁers, achieving SOTA performance.
Semi-Supervised Learning (SSL) .
In SSL, a portion of the training dataset is labeled and the remaining portion is unlabeled.
SSL has seen great progress in recent years, including temporal ensemble (Laine & Aila, 2017), mean teacher (Tarvainen & Valpola, 2017), MixMatch (Berthelot et al., 2019), ReMixMatch (Berthelot et al., 2020), FixMatch (Sohn et al., 2020), and unsupervised data augmentation (Xie et al., 2019).
While there is no technical barrier to applying SSL to UDA, only a few SSL methods have been applied to solve UDA problems; for example, on top of the discrepancy reduction techniques of UDA, several works (French et al., 2018;
Long et al., 2018;
Saito et al., 2019;
Tran et al., 2019) propose to combine SSL techniques such as entropy minimization (Grandvalet &
Bengio, 2005) or pseudo-labeling (Lee, 2013).
While NoisyStudent (Xie et al., 2020) uses labeled data from ImageNet and unlabeled data from JFT in training, they do not leverage this distribution shift during training, and they only evaluate on the source domain (i.e., ImageNet).
Semi-Supervised Domain Adaptation (SSDA) has been studied in several settings, including vision (Donahue et al., 2013; Yao et al., 2015; Ao et al., 2017; Saito et al., 2019) and natural language processing (Jiang & Zhai, 2007; Daumé III et al., 2010; Guo & Xiao, 2012).
Since SSDA assumes access to labeled data from multiple domains, early works have used separate models for each domain and regularized them with constraints (Donahue et al., 2013; Yao et al., 2015; Ao et al., 2017; Daumé III et al., 2010; Guo & Xiao, 2012).
However, such methods are difﬁcult to adapt to the UDA setting where labeled data is only available in a single domain.
One recent exception is minimax entropy (MME) regularization (Saito et al., 2019), which can work in both the UDA and
SSDA setting.
However, unlike AdaMatch, MME requires a pre-trained network to work well.
Transfer learning is used to boost accuracy on small datasets by initializing model parameters with pre-trained weights ﬁrst learned on a separate, larger dataset, which can compensate for the limited amount of labeled source data and boost the overall performance (Recht et al., 2019b; Kolesnikov et al., 2019).
For example, standard experimental protocols on several UDA benchmarks, including
Ofﬁce-31 (Saenko et al., 2010), PACS (Li et al., 2017), and DomainNet (Peng et al., 2019), use
ImageNet-pretrained models to initialize model parameters.
Though useful for some cases in practice, it may not be the most general protocol to evaluate the advancement of UDA algorithms, especially in situations where no datasets exist for pre-training (for example, images with arbitrary number of
[Figure]
channels), or for domains other than vision, where no pre-training datasets exist.
Thus, in this work, we mainly focus on evaluating methods under a non-transfer learning setting, which we consider to be more general.
Although we also achieve state of the art results with transfer learning, we only present them for historical reasons and to illustrate that our method can use that setting too.
A DA M ATCH
We now introduce AdaMatch, a new algorithm inspired by modern semi-supervised learning techniques, aimed at solving UDA, SSL, and SSDA.
As typical in SSL, AdaMatch takes both an unlabeled and labeled dataset as input.
We assume that the labeled data is drawn from a source domain while the unlabeled data is drawn from a target domain (for the SSL task, these domains are the same).
Notation.
We use capital letters X , Y , Z to denote minibatches of examples, labels and logits.
Speciﬁcally, X SL ⊂ R n SL × d and Y SL ⊂ { 0 , 1 } n SL × k denote the minibatch of source images and labels, respectively.
Similarly, the minibatch of unlabeled target images is X TU ⊂ R n TU × d .
Here, k is the number of classes and d is the input dimension (for images
d = h · w · c , where h is height, w is width, and c is the number of channels).
The minibatch size for the labeled data is n SL and the minibatch size of the unlabeled images is n TU .
Additionally, we use Y ( i ) to refer to its i -th row, and
Y ( i,j ) to refer to the i, j − th element of Y .
The model f : R d → R k takes images as input and outputs logits for each of k classes.
Importantly, the source and target domain are the same classiﬁcation task, so the number of classes k and the image dimension d is the same for both domains.
3. M ETHOD DESCRIPTION
AdaMatch introduces three new techniques to account for differences between the source and target distributions – random logit interpolation, a relative conﬁdence threshold, and a modiﬁed distribution alignment from ReMixMatch (Berthelot et al., 2020) – but builds upon the algorithmic backbone from FixMatch (Sohn et al., 2020).
We ﬁrst provide a high-level overview of the algorithm and then discuss the implementation details of the various components.
For a brief summary of how each component helps with distribution shift and accompanying motivational examples, see Appendix A.
Overview.
A high-level depiction of AdaMatch is in Figure 1.
Two augmentations are made for each image: a weak and a strong one with the intent to make the class prediction harder on the strongly augmented image 2 .
Next, we obtain logits by running two batches through the model: a batch of the source images and batch composed of both the source and target images.
Each of the resulting batches of logits are inﬂuenced by their respective batch norm statistics, i.e. the source batch is only inﬂuenced by the source data batch norm statistics while the batch that combines source and target is inﬂuenced by both domains batch norm statistics.
Two loss terms are then computed:
• The source loss term is responsible for predicting correct source labels and for aligning source and target logit domains.
We ﬁrst combine logits for the source images using random logit interpolation ,
which encourages the
model
to
produce the same label for the hyperspace connecting source logits obtained from 1) only source examples and 2) a combination of source and target examples.
In practice, this creates an implicit constraint to align the source and target domains in logit space.
The newly obtained source logits are then used to compute the cross-entropy loss for the source data.
• The target loss term is responsible for predicting the correct target labels and for aligning the target predictions to a desired class distribution.
Since we don’t assume access to labels for the target images, we create a pseudo-label for these images as follows.
First, we rectify the class distribution obtained from weakly augmented target images to a desired class distribution using distribution alignment .
If the target class distribution is known, it can be used directly.
In the general case where it is not known, we use the source class distribution instead.
We then select entries of the batch for which the rectiﬁed probabilities of the weakly augmented target image predictions are above a user-deﬁned conﬁdence threshold .
A pseudo label is then made for these outputs by selecting the most conﬁdent class, and these pseudo-labels are used with a standard cross-entropy loss applied to the logits of the strongly augmented images.
Augmentation.
For a dataset D ∈ { SL, TU } , we augment each image batch X D fed into AdaMatch twice, once with a weak augmentation and once with a strong augmentation, using the same types of weak and strong augmentations as (Berthelot et al., 2020).
This forms a pair of batches X D,w and X D,s respectively, which we denote together as X augD = { X D,w , X D,s } .
From these pairs of batches, we then compute logits Z (cid:48) SL , Z (cid:48)(cid:48) SL and Z TU as follows:
[Equation]
That is, we compute logits by calling the model twice for both the strong and weakly augmented images.
The ﬁrst time we pass both source and target inputs together in the same batch so the batch normalization statistics are shared, and the second time we only pass the source label data.
We only update batch normalization statistics when computing { Z (cid:48) SL , Z TU } =
f ( { X augSL , X augTU } ; θ ) to avoid double counting source labeled data.
Note that without batch normalization, we would have
Z (cid:48)
SL ≡ Z (cid:48)(cid:48) SL .
But batch normalization may make them slightly different.
Random logit interpolation.
The role of random logit interpolation is to randomly combine the joint batch statistics from the source and target domains with the batch statistics from the source domain, which has the effect of producing batch statistics that are more representative of both domains.
More precisely, during training, we obtain logits Z SL by randomly interpolating the logits Z (cid:48) SL and Z (cid:48)(cid:48) SL computed with different batch statistics:
[Equation]
where we sample λ ∼ U n SL · k (0 , 1) .
Note that each individual logit gets its own random factor.
Our underlying goal here is to minimize the loss for every point between Z (cid:48) SL and Z (cid:48)(cid:48) SL , which can be accomplished by either 1) having Z (cid:48) SL and Z (cid:48)(cid:48) SL be equal to each other, or 2) having the whole line between Z (cid:48) SL and Z (cid:48)(cid:48) SL be a minima.
Rather than picking one of the two ways, we formulate the problem as minimizing the loss for all connecting points, which gives the model the freedom to ﬁnd the best possible solution.
The key point is that we randomly choose the interpolation value , and then minimize the loss on this randomly interpolated value.
Because we randomly choose the point on the line each time, over the course of training we can ensure that the entire line segment between Z (cid:48) SL and Z (cid:48)(cid:48) SL reaches low loss.
Another way to achieve the same result would be to divide the interval into N (a large number) of different segments, and then minimize the loss on all N points.
However this would be computationally expensive and so by randomly choosing a new each time we achieve the same result without increasing the computation cost by a factor of N .
Distribution alignment.
Distribution alignment (Berthelot et al., 2020) can be seen as an additional form of model regularization that helps constrain the distribution of the class predictions to be more aligned with the true distribution.
Without it, the classiﬁer could just predict the most prevalent class or exhibit other failure modes.
Ideally, if the target label distribution is known, we would use it directly.
However, when the target label distribution is unknown, we approximate it using the only available distribution – the source label distribution.
A limitation of this approach is that the more the source label distribution differs from the target distribution, the more incorrect the approximation will be, which may cause the model performance to degrade.
However, in practice, we ﬁnd that aligning the target pseudo-labels to match the source label distribution helps signiﬁcantly.
Unlike ReMixMatch (Berthelot et al., 2020), we estimate the source label distribution from the output of the model rather than using the true labels.
We make this change since the model may not be capable of matching the ground truth source label distribution (particularly when source accuracy is low) but matching the source output distribution is a more attainable goal.
To implement this, we ﬁrst extract the logits for weakly and strongly augmented samples from the batch (by indexing):
[Equation]
Then, we compute pseudo-labels for labeled sources and unlabeled targets:
[Equation]
Using distribution alignment, we rectify the target unlabeled pseudo-labels by multiplying them by the ratio of the expected value of the weakly augmented source labels E
[ ˆ Y SL,w ] ∈ R k to the expected value of the target labels E
[ ˆ Y TU,w ] ∈ R k , obtaining the ﬁnal pseudo-labels ˜
Y
[Equation]
normalize ensures that the distribution still sums to 1.
As could be seen E [ ˜ Y TU,w ] =
E [ ˆ Y SL,w ] , which conﬁrms that distribution alignment makes the target pseudo-labels follow the source label distribution.
If the target label distribution is known, one can simply replace the term E
[ ˆ Y SL,w ] with it in the formula above.
Relative conﬁdence threshold.
A conﬁdence threshold is typically used to select which predicted labels are conﬁdent enough to be used as pseudo-labels (Lee, 2013).
However, since machine learning models are poorly calibrated (Guo et al., 2017), especially on out-of-distribution data Ovadia et al.
(2019), the conﬁdence varies from dataset to dataset depending on the ability of the model to learn its task.
To address this issue, we introduce a relative conﬁdence threshold which adjusts a user-provided conﬁdence threshold relative to the conﬁdence level of the classiﬁer on the weakly augmented source data.
Speciﬁcally, we deﬁne the relative conﬁdence threshold c τ as the mean conﬁdence of the topprediction on the weakly augmented source data multiplied by a user provided threshold τ :
[Equation]
We then compute a binary mask ∈ { 0 , 1 } n TU by thresholding the weakly augmented target images with the relative conﬁdence threshold c τ :
[Equation]
Loss function .
The loss L ( θ ) sums L source ( θ ) for the source and L target ( θ ) for the target.
[Equation]
where H ( p, q ) = − (cid:80) p ( x ) log q ( x ) is the cross-entropy loss and stop_gradient is a function that prevents gradient from back-propagating on its argument.
Prevention of gradient back-propagation on guessed labels is a standard practice in SSL works that favors convergence.
µ ( t ) is a warmup function that controls the unlabeled loss weight at every step of the training.
The purpose is to shorten the convergence time for the model, and it should not signiﬁcantly effect the model’s ﬁnal accuracy.
In practice we use µ ( t ) = 1 / 2 − cos (cid:0) min( π, 2 πt/T ) (cid:1) / 2 where T is the total training steps.
This particular function smoothly raises from 0 to 1 for the ﬁrst half of the training and remains at 1 for the second half.
L source ( θ ) is the typical cross-entropy loss with the nuance that we call it twice: once on weakly augmented samples and once on strongly augmented ones.
Similarly, L target ( θ ) is the masked crossentropy loss, where entries for which the conﬁdence is less than c τ are zeroed.
This loss term is exactly the same as in FixMatch (Sohn et al., 2020).
3. H YPER - PARAMETERS
AdaMatch only requires the following two hyper-parameters: (1) Conﬁdence threshold τ (set to 0 .for
all experiments).
(2) Unlabeled target batch size ratio uratio (set to 3 for all experiments) which deﬁnes how much larger is the unlabeled batch, e.g. n tu = n sl · uratio .
E XTENSION TO SSL AND SSDA
SSL takes as input two types of data: labeled samples and their labels X L and unlabeled samples
X U .
AdaMatch can be used without change for the SSL task by feeding X L in place of X SL and
X U in place of X TU .
The same trick is provided for other methods in the experimental study.
SSDA differs from UDA by the presence of targeted labeled data X TL .
AdaMatch can be used without change by feeding the concatenated batch X STL
=
{ X SL , X TL } in place of X SL .
As in
SSL, the same trick works for other methods.
Note there’s a subtle effect on the uratio which ends up being implicitly halved since the labeled batch is twice bigger in SSDA than in standard UDA.
E XPERIMENTAL SETUP
We evaluate AdaMatch on the SSL, UDA, and SSDA tasks using the standard DigitFive (Ganin et al., 2016) and DomainNet (Peng et al., 2019) visual domain adaptation benchmarks and we compare to various existing methods.
DigitFive experiments and ablation studies were run on a single V
GPU, other experiments were run on a single TPU.
Datasets.
Digit-Five is composed of 5 domains, USPS (Hull, 1994), MNIST (LeCun et al., 1998),
MNIST-M (Ganin et al., 2016), SVHN (Netzer et al., 2011), and synthetic numbers (Ganin et al., 2016).
We resize all images into 32 × 32 3 and convert them to RGB.
DomainNet (Peng et al., 2019) is composed of six domains from 345 object categories.
Unlike prior work (Ganin et al., 2016; Saito et al., 2018; Peng et al., 2019), unless otherwise stated, we train models from scratch to focus on evaluating the efﬁcacy of the algorithms themselves.
Democratic research.
For DomainNet, we experiment with two image resolutions, 64 × 64 and 224 × 224 .
Though Peng et al.
(2019) only evaluate with resolution 224 × 224 , we include 64 ×to make future experiments using DomainNet more accessible to researchers with limited compute budget.
Moreover, we ﬁnd that AdaMatch can outperform the previous SOTA for DomainNet at 224 × 224 resolution even when trained with 64 × 64 resolution images.
Network and training hyperparameters.
We use ResNetV2-101 (He et al., 2016) for resolution 224 × 224 , WRN-34-2 (Zagoruyko & Komodakis, 2016) for 64 × 64, and WRN-28-2 for 32 × 32.
We tried three conﬁdence thresholds ( 0 . 8 , 0 . 9 , 0 . 95 ) on 64 × 64 UDA clipart → infograph and picked the best.
Additionally, we picked the largest uratio allowed on 8 GPUs for a batch size ofusing 224 × 224 images.
We coarsely tuned (using 3 candidate values) the learning rate, learning rate decay, and weight decay using only the 64 × 64 supervised source data.
For weight decay, we used a empirical rule: halving it for 32 × 32 neural networks and doubling it for the 224 × 224 to coarsely adapt the regularization to the capacity of the network (we used a weight decay of 0 .for WRN-28-2, 0 .
001 for WRN-34-2 and 0 .
002 for ResNetV2-101).
We set learning rate to 0 .
03 and learning rate cosine decay to 0 .
We trained DigitFive for 32M images, and DomainNet for 8M.
Baselines.
We compare against popular SOTA methods from UDA and SSL, including maximum classiﬁer discrepancy (MCD) (Saito et al., 2018), FixMatch with distribution alignment (FixMatch+), (Sohn et al., 2020), and NoisyStudent (Xie et al., 2019).
For UDA, we include MCD since it is the current single-source SOTA on DomainNet (Peng et al., 2019).
For SSL, we include FixMatch+ since it achieves SOTA performance on standard SSL benchmarks and our algorithm borrows many components from it.
We also include NoisyStudent since it is a popular approach to SSL that achieves competitive results on large-scale datasets like ImageNet (Deng et al., 2009).
Finally, we include our own baseline, BaselineBN, which uses fully supervised learning on the source domain but additionally feeds unlabeled target data through the network to update batch norm statistics (see Appendix B).
Evaluation.
For each source and target dataset pair, we calculate the ﬁnal accuracy as the median accuracy reported over the last ten checkpoints (we save a checkpoint every 2 16 samples).
In summary tables, we report the average ﬁnal accuracy over all target datasets.
Unlike SSL, most prior work in UDA evaluates using ImageNet pre-training.
As explained in Section 2, we choose to report the majority of our results without pre-training.
However, for easier comparison with prior work, we also evaluate ImageNet pre-trained AdaMatch, FixMatch+, and MCD for UDA in Section 5.1.1.
R ESULTS
We now summarize our results for the UDA, SSL, and SSDA tasks.
We ran all ﬁve algorithms on all source-target pairs for every dataset (20 for DigitFive and 30 for DomainNet64/224), varying the number for labels in SSL/SSDA tasks, for a comprehensive total of 2 , 420 experiments.
This represents a major experimental undertaking and we hope that the results (especially the ﬁner grained results on individual dataset pairs in Appendix E and the democratized setting) will serve future work.
5. U NSUPERVISED D OMAIN A DAPTATION (UDA)
[Table]
Table 2 shows the average target accuracy achieved by each algorithm for the UDA task on the
DigitFive, DomainNet64, and DomainNet224 benchmarks.
For the UDA task, we additionally compare to a fully-supervised learning setup that uses an oracle model that has access to all labels from both the source and target datasets (Oracle).
We ﬁnd that AdaMatch outperforms all other algorithms we compare against, and the improvement is highest for the larger dataset size, DomainNet224, where AdaMatch achieves an average target accuracy of 28.7% compared to 23.9% for NoisyStudent.
Additionally, on all three benchmark datasets, AdaMatch, FixMatch+, and NoisyStudent all signiﬁcantly outperform both MCD and
BaselineBN.
This success illustrates that SSL methods can be applied out-of-the-box on UDA problems, but we can also signiﬁcantly improve upon them if, as in AdaMatch, the distributional differences between the source and target data are accounted for.
For DomainNet64, we also observe that both AdaMatch and FixMatch+ without pre-training on
DomainNet64 are able to out-perform the previous state-of-the-art accuracy of 21.9% set by MCD with pre-training on DomainNet224 (Peng et al., 2019).
Overall, the accuracy gap between AdaMatch on
DomainNet64 and DomainNet224 is relatively low – only 2% – and, for the purposes of democratizing research, we encourage future UDA research to evaluate on DomainNet64 since it is signiﬁcantly less computationally intensive to run experiments on the smaller image size.
5.1. UDA WITH PRE - TRAINING
[Table]
In order to compare with prior UDA work which uses ImageNet pre-training, in Table 3 we evaluate AdaMatch versus MCD on the DomainNet224 benchmark when initializing from pre-trained
ImageNet ResNet101 weights.
As is typical with pre-training, we also found in our experiments that early stopping is necessary to get the best results for AdaMatch.
Thus, for the pre-training results in Table 3, we report the ﬁnal target accuracy achieved after training for 8 million images (our standard training protocol), the ﬁnal accuracy achieved after early stopping at 2 million images, and the maximum median accuracy (over a window of 10 checkpoints) achieved across the entire run.
Using the standard training protocol of 8 million images, AdaMatch with pre-training is able to outperform MCD with pre-training, a difference of 28.2% versus 22.3%, respectively (we also conﬁrm that the accuracy we are able to achieve on MCD closely matches the reported accuracy of 21.9% from (Peng et al., 2019)).
However, we also observe that pre-trained AdaMatch peaks early on in training, and a signiﬁcantly higher accuracy of 35.6% is achievable if an Oracle were to tell when to stop training.
To account for this, we recommend early-stopping AdaMatch in the pre-trained setting.
[Figure]
In the SSL setting, we only train on a single dataset (since there is no notion of source nor target) which we randomly divide into two groups: labeled and unlabeled.
Figure 2 shows the average target accuracy for each algorithm on the SSL task as we vary the number of target labels (we also include the corresponding numerical results in Appendix D).
We ﬁnd that AdaMatch achieves state-of-the-art performance for DomainNet224, and competitive performance to FixMatch+ on the DigitFive and
DomainNet64 benchmarks.
Additionally, we observe that increasing the number of target labels generally results in better accuracy for all methods, and on the DomainNet224 dataset in particular, the gap in performance between AdaMatch and FixMatch+ widens as we increase the number labels.
Overall, AdaMatch’s improvement over FixMatch+ is smaller on the SSL task compared to the SSDA or DA tasks, which is expected since AdaMatch by design is an extension of FixMatch+ to handle distribution shifts.
This also suggests that the additional components of AdaMatch, i.e. random logit interpolation and relative conﬁdence thresholding, improve accuracy more in settings where the unlabeled and labeled data are not drawn from the same distribution.
5. S EMI -S UPERVISED D OMAIN A DAPTATION (SSDA)
In the SSDA setting, we randomly sample a subset of the target dataset and treat it as labeled.
Figure 3 shows the performance of AdaMatch compared to all other algorithms on the SSDA task as we vary the number of target labels (corresponding numerical results are in Appendix D).
AdaMatch
[Table]
We evaluate algorithms by excluding the components one at a time.
outperforms all other algorithms on both DomainNet64 and DomainNet224.
As expected, increasing the number of target labels improves the accuracy for all methods.
In very low label regime, AdaMatch further improves its lead over other methods.
A BLATION S TUDY
In this section, we perform an ablation analysis on each component of AdaMatch to better understand their importance and we provide a sensitivity analysis of hyperparameters, such as uratio, weight decay, conﬁdence threshold, or augmentation strategies.
We conduct our study on DomainNet using 64 × 64 as input and report the average accuracy across 6 domain adaptation protocols.
Exclude-One-Out Analysis.
AdaMatch improves upon FixMatch with several innovative techniques: random logit interpolation, adaptive conﬁdence thresholding and distribution alignment (initially introduced by ReMixMatch and discussed in FixMatch as an extension but not per-se part of it).
To better understand the role of each component, we conduct experiments by excluding one component at a time from AdaMatch.
As we see in Table 4 all components contribute to the success of AdaMatch.
Sensitivity Analysis.
AdaMatch only has two hyper-parameters to tune: uratio and conﬁdence threshold.
While it is not speciﬁc to AdaMatch, we also measure the sensitivity to weight decay as it was shown to be important to achieve a good performance in (Sohn et al., 2020).
Results are in Figure 4.
Similarly to the ﬁndings from FixMatch (Sohn et al., 2020), AdaMatch requires high conﬁdence threshold.
Also, we observe improved accuracy with higher uratio at the expense of more compute.
(As a reminder, uratio deﬁnes the ratio of unlabeled to labeled data within a mini-batch, and not the ratio between the total number of unlabeled and labeled examples seen over the course of training.)
For L2 weight decay, we ﬁnd fairly reliable performance between 0 . 0001 and 0 . 001 .
[Figure]
C ONCLUSION
Machine learning models still suffer a drop in accuracy on out-of-distribution data.
However, if we have access to unlabeled data from a shifted domain (the UDA setting) or even a small amount of labeled data from the shifted domain (the SSDA setting), we can greatly improve accuracy.
In this work, we present AdaMatch, a general method designed to boost accuracy on domain shifts when given access to unlabeled data from the new domain.
AdaMatch uniﬁes the domains of UDA, SSL, and SSDA, demonstrating that one method can perform well at all three.
Overall, our work shows that it is possible to apply SSL algorithms out-of-the-box to the domain adaptation problem.
By taking into account the distribution shift, AdaMatch can signiﬁcantly improve upon SSL methods.
This suggests that a promising direction for future work is to take new advances from SSL and translate or modify them for the domain adaptation setting.
While AdaMatch outperforms prior work by a large margin, there is still an even larger margin left for improvement on out-of-distribution shifts.
This leaves open an important question: which is more important for future progress, making use of unlabeled data more efﬁciently, or attempting to better model domain shifts?
A CKNOWLEDGEMENTS
We would like to thank Ekin Dogus Cubuk and Colin Raffel for helpful feedback on this work.
E THICS S TATEMENT
AdaMatch is designed to make ML models robust to a domain shift between train and test time using a limited amount of labeled data and a large amount of unlabeled data.
The development of data efﬁcient ML algorithms are of utmost importance in democratization of ML methods.
However, the conﬁrmation bias of self-training (Arazo et al., 2019) could be a concern for building a fair ML model across major and minor classes.
AdaMatch partially resolves the issue with the distribution alignment, but more in-depth investigation on the fairness and robustness of data-efﬁcient ML algorithms should be done in the future.
R EPRODUCIBILITY
We wrote our experimental codes using the open-source Objax library (Objax Developers, 2020) and used the same hyperparameters for most of our experiments, which we specify in Section 4.
Moreover, we released the open source code publicly at https://github.com/google-research/ adamatch .
A M OTIVATION AND ILLUSTRATING EXAMPLES
We ﬁrst brieﬂy summarize the motivation for each of the components we introduce and explain how they help with distribution shift:
Random logit has the effect of producing batch statistics that are more representative of both domains (Section 3.1: Random Logit Interpolation) and creates an implicit constraint to align the source and target domains in logit space (Section 3.1: Overview).
Distribution alignment helps constrain the distribution of the class predictions to be more aligned with the true distribution (Section 3.1: Distribution Alignment).
If we left it out, then the model would have no incentive to match the target distribution to the source distribution.
Relative conﬁdence thresholding addresses the issue that models are poorly calibrated on out-of-distribution data.
(Section 3.1: Relative Conﬁdence Threshold)
By including relative conﬁdence thresholding, we can ensure that the target data is used as pseudo labels as often as it should be.
We next provide some illustrating examples to clarify the concepts of distribution alignment and relative conﬁdence thresholding.
Distribution alignment example : Assume we have a source dataset that has two classes that should follow a frequency distribution of { 0 . 4 , 0 . 6 } , e.g. 40% of samples are from class 1 and 60% from class 2.
Let’s say the model for the weakly augmented source data currently predicts an empirical frequency distribution of { 0 . 3 , 0 . 7 } , e.g. 30% for class 1 and 70% for class 2.
However, for the weakly augmented target data the model predicts on average { 0 . 6 , 0 . 4 } .
We rectify a new prediction P TU,w by multiplying it elementwise by { 0 . 3 , 0 . 7 } { 0 . 6 , 0 . 4 } , which has the effect of making the ﬁrst class of P TU,w half as likely and the second class (roughly) twice as likely.
In other words, it changes the class distribution on the weakly augmented unlabeled target domain from { 0 . 6 , 0 . 4 } to { 0 . 3 , 0 . 7 } .
Relative conﬁdence thresholding example : Conﬁdence thresholding determines what is a conﬁdent pseudo-label for unlabeled data.
However, since machine learning models are poorly calibrated (Guo et al., 2017), especially on out-of-distribution data, the conﬁdence varies from dataset to dataset depending on the ability of the model to learn its task.
Suppose that we have a dataset X where a given model’s average top-1 conﬁdence is 0 .
7 on labeled data.
Using a default conﬁdence threshold of τ = 0 .
9 for pseudo-labels will exclude almost all unlabeled data since they are unlikely to exceed the maximum labeled data conﬁdence.
In this scenario, relative conﬁdence thresholding is particularly useful.
By multiplying the default conﬁdence threshold τ by the average top-1 labeled conﬁdence, we can obtain a relative conﬁdence ratio of c τ = 0 .
9 × 0 .
7 = 0 .
63 which is more likely to capture a meaningful fraction of the unlabeled data.
In the case of CIFAR-10, typically the softmax of most if not all labeled training examples will reach 1 . 0 .
When the average top-1 conﬁdence on the labeled data is 1 . 0 , the relative conﬁdence threshold c τ and the default conﬁdence threshold τ are the same.
Therefore, one can see relative conﬁdence thresholding as a generalization of the conﬁdence thresholding concept.
B B ASELINE BN
For BaselineBN, the same model and hyper-parameters are used, but the loss is computed differently.
Both source and target are concatenated as a batch, and the loss is only computed on the (source) labeled logits for both weak and strong augmentations.
[Equation]
C UDA M ODEL E VAL C URVES W ITH AND W ITHOUT P RE -T RAINED W EIGHTS
We randomly select a dataset pair from DomainNet224 (Sketch → Clipart) and plot the target accuracy versus the number of training images for both pre-trained and randomly initialized AdaMatch and
MCD in Figure 5.
For all pre-trained runs, we set weight decay to 0 and initialize model weights using an ImageNet pre-trained ResNet101 architecture.
Figure 5 shows that AdaMatch beneﬁts signiﬁcantly from early stopping.
Although not plotted, we observe a similar pattern in other dataset pairs.
Using a consistent early stopping rule of 2 million images, we found that, on average across all dataset pairs, AdaMatch with pre-training and early stopping outperforms randomly initialized AdaMatch by 4 .
7% (see Table 3).
D S UMMARY RESULTS
In this section
[Table]
we include the corresponding tables for the SSL and SSDA summary ﬁgures (Figure 2 and 3, respectively).
D. SSL
[Table]
D
[Table]
See Figure 3 for plot of number
E I NDIVIDUAL DATASET RESULTS
In this section
[Table]
we evaluate all methods on the individual dataset pairs from the DigitFive, DomainNet64, and DomainNet224 benchmarks for the UDA, SSL
[Table]
and SSDA tasks.
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table]
infograph
[Table] [Table]
[Table]
[Table] [Table]
[Table] [Table] [Table] [Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
for DomainNet224 in the UDA setting using
NoisyStudent.
[Table]
clipart infograph
[Table]
clipart -
[Table] [Table]
[Table] [Table]
[Table]
infograph 22.
painting 39.
[Table] [Table]
[Table]
quickdraw
[Table] [Table]
real
[Table] [Table]
clipart
[Table]
infograph
[Table] [Table]
painting 27.
[Table] [Table]
[Table] [Table]
[Table]
quickdraw 10.
real 39.
sketch 40.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
for DomainNet64 in the UDA setting using MCD.
[Table]
clipart infograph painting quickdraw real sketch
[Table]
clipart10.
infograph 21.
painting 36.
quickdraw 19.
real 44.
sketch
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]