Surprisingly Simple Semi-Supervised Domain Adaptation with Pretraining and Consistency
Abstract
Most modern unsupervised domain adaptation (UDA) approaches are rooted in domain alignment, i.e ., learning to align source and target features to learn a target domain classiﬁer using source labels.
In semi-supervised domain adaptation (SSDA), when the learner can access few target domain labels, prior approaches have followed UDA theory to use domain alignment for learning.
We show that the case of SSDA is different and a good target classiﬁer can be learned without needing explicit alignment.
We use self-supervised pretraining and consistency regularization to achieve well separated target clusters, aiding in learning a low error target classiﬁer, allowing our method to outperform recent state of the art approaches on large, challenging benchmarks like DomainNet and VisDA-17. Code for our experiments can be found at https://github.com/venkatesh-saligrama/PAC.
Introduction
The problem of visual domain adaptation arises when a learner must leverage labeled source domain data to classify instances in the target domain, where it has limited access to ground-truth annotated labels.
An example of this is the problem of learning to classify real-world images based on handsketched depictions.
The problem is challenging because discriminative features that are learnt while training to classify source domain instances may not be meaningful or sufﬁciently discriminative in the target domain.
As described in prior works, this situation can be viewed as arising from a “domain-shift” where the joint distribution of features and labels in the source domain does not follow the same law in the target domain.
We propose a novel method for semi-supervised domain adaptation (SSDA) where the learner, in addition to unlabeled target examples is granted access to a few labeled target domain examples for training.
Our method is based on enhancing clusterability of target domain features independent of the source domain.
Prior SSDA methods [28, 15, 16] draw upon approaches developed within the context of visual domain adaptation using labelled source images and unlabelled target images [1, 21, 30, 34, 38].
These works in turn are based on adversarial domain alignment, and draw upon
Ben-David et al .
[2]’s elegant theory of domain adaptation.
Generalization bounds in Ben-David et al .
[2] suggest that, good adaptation is possible if the divergence between induced source and target domain feature distributions is small.
While domain alignment is a meaningful goal in the absence of labels, we believe that the situation is dramatically different in the presence of a few target domain labels.
For our work, we draw inspiration from Castelli and Cover [5], who showed that in binary classiﬁcation problems, a learner with no knowledge of underlying feature distributions, can beneﬁt exponentially from labelled examples.
Speciﬁcally, with u unlabelled examples and (cid:96) labelled ones, they showed the learners probability of error approaches Bayes error as
[Equation]
Consistency
Rotation Prediction
[Figure]
Final Task
Training
[Figure]
(x
[Figure]
where D ( q 1 , q 2 ) or the Bhattacharya distance is a measure of separation between the class distributions q 1 and q 2 .
Notice that the error probability decays exponentially with the product of inter-class distance and number of labels.
As such, our key insight is that we can forego domain alignment if we learn representations that result in compact, well-separated clusters in the target domain.
The identity of these clusters can then be deduced from few labels.
Our contributions in this paper are two-fold: (1) We propose a novel semi-supervised domain adaptation method
PAC (pretraining and consistency), based on label consistency and rotation prediction for pretraining, which performs comparably or better than state of the art on SSDA across multiple datasets.
In contrast to prior works, we forego domain alignment, and pose objectives that improve target clusterability.
(2) We perform ablative analysis on individual components of our method, illustrating their behavior, and their impact on performance.
Our analysis provides an understanding of these components and shows how they can be combined with other techniques.
Pretraining and Consistency (PAC)
Notating some variables: Available to the model are two sets of labelled images :
D s = { ( x si , y si ) } n s i =1 , the labelled source images and D t = { ( x ti , y ti ) } n t i =1 , the few labelled target images, and additionally the set of unlabelled target images D u = { x ui } n u
i =1 .
The goal is to predict labels for these images in D u .
The ﬁnal classiﬁcation model consists of two components : the feature extractor F and the classiﬁer C .
F generates features F ( x ) for an input image x which the classiﬁer uses produce output class scores C ( F ( x ))
∈ R K , where K is the number of categories that the images in the dataset could belong to.
In our experiments, F is a convolutional network and produces features with unit (cid:96) 2 -norm, i.e . ||
F ( x ) ||
2 = 1 (following [28]).
C consists of one or two fully connected layers.
An overview of PAC is shown in Fig 1.
Our ﬁnal model is trained in two stages:
Pretraining with Rotation Prediction.
We ﬁrst train our feature extractor F with the self-supervised task of predicting image rotations (Fig 1 (left)) on both the source and target datasets, i.e ., all images in D s , D t and D u .
Without using image category labels, we train a 4-way classiﬁer to predict one out of 4 possible angles ( 0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ ) that an input image has been rotated by.
This backbone is then used as the initialization for training the ﬁnal classiﬁer in the next stage.
Consistency Regularized Classiﬁer Training.
Consistency regularization promotes the ﬁnal model
C ◦ F to produce the same output for both an input image x and a perturbed version x + δ .
We introduce these perturbations using image level augmentations — RandAugment [8] along with additional color jittering.
Given an unlabelled image x ∈ D u , we ﬁrst compute the model’s predicted class distributions
[Equation]
( F ( x + δ )))
p x is then conﬁdence thresholded using a threshold τ and the following is used as the consistency regularization loss.
[Equation]
where 1 is an indicator function and H ( p x , q x ) = (cid:80)
k ∈ K − p x ( k ) log( q x ( k )) is cross-entropy.
Note that p x ( · ) has been used to index into the K elements of p x .
Intuitively, an unperturbed version of image x is used to compute pseudo-targets for the perturbed version x + δ , which is only used when the pseudo-target has high conﬁdence ( max k p x ( k ) ≥ τ ).
We also note here that the target p x is treated as a constant for gradient computation with respect to the network parameters.
For the labelled examples from D s and D t , we use the same perturbations but with ground truth labels as targets.
The model is optimized using minibatch-SGD, with minibatches M s , M t and M u sampled from
D s , D t and D u respectively.
The ﬁnal optimization criterion used is
[Equation]
where ¯ y ∈ R K is the one-hot representation of y ∈
[ K ] or ¯ y ( i ) =
[ i = y ] and H (¯ y, x ) has been overloaded to mean H (¯ y, C ( F ( x ))) .
Experiments
Datasets.
From DomainNet[25], we used 4 domains (Clipart, Paintings, Real and Sketch) following [28], each with 126 classes resulting in an avg 36500 images per domain.
In VisDA-17 [26] the source domain (the VisDA "train" split) consists of 152,398 synthetic images from 12 categories, and the target domain (the VisDA "validation" split) consists of 55,388 real images.
We used 1 and 3-shot settings for evaluation, which contained 1 and 3 labelled examples in the target domain.
Additionally, for VisDA-17, we evaluated with settings where 1 and 5% of the target examples were labelled.
Implementation Details.
All our experiments were implemented in PyTorch [24] using W&B [4] for tracking experiments.
For evaluations on the DomainNet dataset, we used an Alexnet and a
Resnet-34 [14] backbone, while on VisDA-17 we evaluated our method with a ResNet-34 backbone. 1 or 2 fully connected layers we used for the classiﬁer C .
Optimization was done via SGD with a momentum 0.9.
Complete details of all experiments are in the supplementary.
3.1 Results
Comparison to other approaches.
We compare PAC with different recent SSDA approaches :
MME [28], BiAT [15], Meta-MME [19], APE [16], LIRR [18], and CDAC [20] using results reported by these papers.
Besides this, we also include in the tables, baseline approaches using adversarial domain alignment—DANN [11], ADR [29] and CDAN [21], that were evaluated by Saito et al .
The baseline “S+T” is a method that simply uses all labelled data available to it to train the network using cross-entropy loss.
In Table 1, we compare the accuracy of PAC with different recent approaches on DomainNet.
Remarkably our simple approach outperforms other SSDA approaches by 3-5% on this benchmark with different backbones (with the exception of CDAC, with which it is competitive on most scenarios).
In Table 2, besides our method, we report results of S+T and MME on VisDA-17, that we replicated from the implementation of [28].
We see that PAC shows strong performance, with close to 10% improvement in accuracy over MME in the 3-shot scenario, and 4% improvement over LIRR.
Ablative analysis.
In Table 3, we see what rotation prediction pretraining and consistency regularization do for ﬁnal target classiﬁcation performance separately.
The two components provide boosts to the ﬁnal performance individually, with the combination of both performing best.
We see that in most cases consistency regularization helps performance signiﬁcantly, especially in the 3-shot scenarios.
For more analyses including a feature space comparison of methods, effects of different pretraining and perturbation methods and PAC’s sensitivity to different number of target examples, see the supplementary.
[Table] [Table]
[Table]
on the real to sketch scenario of Domainnet using both Alexnet and Resnet-34 backbones.
Conclusion
We showed that consistency regularization and pretraining using rotation prediction are powerful techniques in SSDA.
Our method, using simply a combination of these without requiring any domain alignment, could outperform recent state of the art on this task, most of which use adversarial alignment.
With our approach we demonstrated that domain alignment is not a necessity for SSDA, and that achieving well-separated target clusters allows for low classiﬁer error with a few labelled examples.
We presented a thorough analysis of both the aforementioned techniques showing how they can improve target clustering and why they are better than other options for similar approaches.
We hope our analysis can help inform future SSDA work.
Acknowledgements.
This work was supported by the Hariri Institute at Boston University.
Supplementary A Feature Space Comparison of Methods.
In Fig 2 we plot the 2-D TSNE [22] embeddings for features generated by 5 differently trained
Alexnet backbones.
The embeddings are plotted for all points from 5 randomly picked classes.
The source domain points which are light colored circles, come from real images of Ofﬁce-Home and the target domain points which are dark colored markers come from clipart images.
The labelled target examples are marked with X’s.
The two plots on the left compare differently pretrained backbones and the three on the right use backbones at the end of different SSDA training processes.
In the plots we can see that pretraining the backbone for rotation prediction starts to align and cluster points according to their classes a little better than what a backbone pretrained just on Imagenet can do.
Out of the ﬁnal classiﬁers on the right, we see that both PAC and MME create well separated classes in feature space allowing for the classiﬁer to have decision boundaries in low-density regions.
MME explicitly minimizes conditional entropy which may draw samples even further apart from the classiﬁer boundaries, as compared to our method which simply tries to ensure that the classiﬁer does not separate an example and its perturbed version.
[Figure]
In Table 4, we quantitatively analyze the features using different metrics :
The A -distance is a distance metric between the two domains in feature space computed using an SVM trained to classify domains as done in [3].
The higher the error of the SVM domain classiﬁer, the lower is the A -distance.
The other two metrics are accuracies of distance based nearest neighbor (NN) classiﬁers in feature space.
The ﬁrst one, “NN Acc.
(Target)” is the accuracy of a classiﬁer that assigns any unlabelled target example, the class label of the target labelled examples closest to it on average in the feature space.
“NN Acc.
(Source)” similarly uses only the source examples, all of which are labelled, to compute the class label for an unlabelled target example.
Finally BD is an empirical inter-class
Bhattacharyya distance estimate (details in supp.).
Comparing the pretrained backbones, we see that rotation pretraining improves the feature space both by bringing closer the features across the two domains (as indicated by the low A -distance) and aligning them so that features within a class are closer (indicated by the higher NN accuracies).
When it comes to ﬁnal feature spaces of the SSDA methods, we see that MME, being a domain alignment method, reduces A -distance more than PAC.
However, PAC is able to better maintain the class-deﬁned neighborhood of features, as indicated by the higher accuracies.
Also, PAC has higher inter-class divergence (BD), which leads to lower error according to Eq.
(1) in the main paper.
This shows that in SSDA, for learning good target classiﬁcation, low divergence between domains is not a necessity.
B PAC performance with different target shots.
In Figure 3, we plot the target accuracy of 4 methods on the real to clipart adaptation scenario of
Ofﬁce-Home, for different number of labelled target examples.
The method “CR” represents the consistency regularization part of PAC, meaning it starts with an Imagenet pretrained backbone, same
[Table]
as S+T and MME [28].
We see that with its domain alignment approach MME performs well atshots.
However, along with pretraining using rotation prediction which has some alignment effect
PAC does not lag far behind.
As the number of labelled examples increase we see all methods enjoy a signiﬁcant boost in performance, where the error has an exponential relation to the number of labelled examples as indicated by Eq.
1 (main paper).
Since PAC and CR have better feature space clustering, i e ., they have a higher inter-class divergence D , they see a bigger reduction in error
[Figure] [Figure]
[Figure]
C More questions
How does pretraining with rotation prediction compare to a constrastive method?
Contrastive pretraining methods [13, 6, 7] have been shown to attain remarkable performance in learning features from unlabelled images that are useful for tasks like image recognition and object detection.
We evaluate how momentum contrast (MoCo) [13] and SimSiam [7] perform for pretraining our feature extractor on both source and target images, compared with rotation prediction.
Table 5 compares most of the same metrics as Table 4 with the addition of ﬁnal model (training with labels and consistency regularization) performance on target classiﬁcation.
We see that, both MoCo and SimSiam improve the imagenet pretrained features to some extent helping the ﬁnal classiﬁcation performance.
However, this improvement is not as high as in the case of rotation prediction pretraining.
We see that MoCo has marginally better class-deﬁned structure across domains, but a poorer structure in the target domain indicated by the accuracies of the distance based NN classiﬁers.
Interestingly, we see that nearest neighbors classiﬁers perform much poorer in the case of SimSiam pretraining as compared to
Backbonepretraining
[Table]
other cases, while the pretraining still beneﬁts ﬁnal SSDA accuracy.
This might indicate that this pretraining helps with optimization in some manner while not clustering the initial feature space as much.
[Figure]
Which perturbation technique is best for consistency?
We compared three different image augmentation approaches : RandAugment [8] involves a list of 14 different augmentation schemes like translations, rotations, shears, color/brightness enhancements etc., 2 out of which are chosen randomly anytime an image is augmented.
We also evaluated color jittering, since common objects in our datasets are largely invariant to small changes in color.
Finally we tried a combination of both, and found that this performed best for our method.
Fig 4 shows the comparison of the ﬁnal target accuracies achieved using an Alexnet backbone on the real to clipart adaptation scenario of
Ofﬁce-Home.
Besides perturbations based on augmentation, we also evaluated adversarial image perturbation via virtual adversarial training (VAT) [23].
When using VAT, we found improvements over the simple “S+T” method (48.3% using VAT vs 44.6% without), but as seen from Fig 4, we found this was much lower than image augmentation approaches.
This is quite likely because image augmentation imposes a more meaningful neighborhood on images where class labels do not change, while adversarial perturbation does not have this guarantee.
Can pretraining and consistency help other methods?
An indication towards the afﬁrmative is seen when we train MME with pretraining and consistency on the 3-shot real to sketch scenario of
DomainNet using a Resnet-34 backbone.
The results are shown in Table 6, where we can see that pretraining and consistency both individually help MME’s performance, and their combination helps it the most.
[Table]
It was explained how pretraining improves initial feature space, but prior work has also used “pretext” tasks like rotation prediction alongside classiﬁcation for training [37, 33].
How does pretraining compare to that?
A comparison of this can be found in table 7, which reports the 3-shot
SSDA target accuracies of the two methods on the DomainNet dataset.
As can be seen, pretraining using rotation prediction provides more of a performance beneﬁt as compared to using rotation prediction as an auxiliary task like [37, 33].
The latter can help regularize ﬁnal target classiﬁer training, but likely does not have the beneﬁts that pretraining provides the method via a better initial feature space for training.
[Table]
What if pretraining uses rotation prediction only on target?
We train the backbone only on target domain data for pretraining with rotation prediction, and then train it like PAC using consistency regularization.
On the 3-shot real to clipart SSDA scenario of Ofﬁce-Home using an Alexnet backbone, this achieves a ﬁnal target accuracy of 57 .
5 % compared to 58 .
9 % of PAC.
This is indicative of target-only rotation prediction helping the initial feature extractor, but not as much as in the case when source domain data is used along with it. Rot
[Equation]
CR
Accuracy (with source)
Accuracy (only target) (cid:88)
[Table]
How big is the role of source domain data in ﬁnal target performance?
To see this, we train our method with no access to source domain data.
This is similar to the semi-supervised learning problem.
Target accuracy with only 3 labelled target examples and access to all other unlabelled examples, on the clipart domain of Ofﬁce-Home using an Alexnet backbone, are in the last column of Table 8.
For reference, the accuracies of our method with source domain data from the real domain ( i.e . R2C adaptation scenario) are provided in the 3 rd column.
D PAC sensitivity to conﬁdence threshold
Our consistency regularization approach uses soft targets based on outputs of the classiﬁer only in cases where the conﬁdence of labelling is high.
In Fig 5, we compare the sensitivity of our method to this threshold.
We see that higher conﬁdence thresholds up to 0.9 help ﬁnal target classiﬁcation performance.
[Figure] [Figure]
E Results on Ofﬁce and Ofﬁce-Home
Ofﬁce-Home [35] is a dataset with 65 categories of objects found in typical ofﬁce and home environments.
It has 4 different visual domains (Art, Clipart, Product, and Real), and we evaluate our methods on all 12 different adaptation scenarios.
The 4 domains have close to 3800 images on average.
Ofﬁce [27] dataset has objects of 31 different categories in 3 different domains—amazon, webcam and dslr, with approx.
2800, 800 and 500 images respectively.
Following [28] we evaluated only on the 2 cases with amazon as the target domain, since the other two domains have a lot fewer images.
Table 10 shows the results of PAC on the different scenarios of Ofﬁce-Home, the average accuracy over all these scenarios was also reported in Table 3 in the main paper.
Table 9 shows the accuracy of
PAC on two scenarios of Ofﬁce.
We see that PAC performs comparably to state of the art.
It lags behind a little in the 1-shot scenarios as compared to 3-shot ones.
F Experiment details
All our experiments were implemented in PyTorch [24] using W&B [4] for managing experiments.
F.1 PAC experiments
We used three different backbones for evaluation in different experiments—Alexnet [17], VGG- [32] and Resnet-34 [14].
Our backbones before being trained using the rotation prediction task, are pretrained on the Imagenet [9] dataset, same as other methods used for comparison.
While using an Alexnet or VGG-16 feature extractor, we use 1 fully connected layer as the classiﬁer, and while using the Resnet-34 backbone, we use a 2-layer MLP with 512 intermediate nodes.
The classiﬁer C uses a temperature parameter set to 0 .
05 to sharpen the distribution it outputs using a softmax.
For consistency regularization, the conﬁdence threshold τ was set to 0.9 across all experiments, having validated on the real to sketch scenario of DomainNet.
Same as [28], we train the models using minibatch-SGD, with s source examples, s labelled target examples and 2 s unlabelled target examples that the learner “sees” at each training step.
s = 24 for the VGG and Resnet backbones, while s = 32 for Alexnet.
The SGD optimizer used a momentum parameter 0 .
9 and a weight decay (coefﬁcient of (cid:96) 2 regularizer on parameter norm) of 0 . 0005 .
For all experiments, the parameters of the backbone are updated with a learning rate of 0 .
001 , while the parameters of the classiﬁer are updated with a learning rate 0 .
Both of these are decayed as training progresses using a decay schedule similar to [10].
Learning rate at step i ( η i ) is set as below:
[Table] [Table]
{+η i = η(1 + 0 . 
+}
× i ) 0 .
For experiments on the Ofﬁce and Ofﬁce-Home dataset, we trained PAC using both an Alexnet and a VGG-16 backbone, and the models were trained for 10000 steps with the stopping point chosen using best validation accuracy.
For the experiments on DomainNet, we use both Alexnet and Resnet-34 backbones, while for VisDA17, we use only Resnet-34.
All models in these experiments were trained for 50000 steps, using validation accuracy for determining the best stopping point.
F.2 Pretraining
As mentioned above, we pretrain our models for rotation prediction starting from Imagenet pretrained weights.
A comparison of PAC with a backbone trained with rotation prediction starting from imagenet pretraining (ﬁnal target accuracy = 9 %) vs one that does not use any imagenet pretraining (ﬁnal target accuracy = 43 .
7 %), revealed that there is important feature space information in imagenet pretrained weights that rotation prediction could not capture on its own.
This comparison was done using an Alexnet on the real to clipart adaptation scenario of Ofﬁce-Home.
Following Gidaris et al .
[12], we trained the model on all 4 rotations of a single image in each minibatch.
Each minibatch contained s images each from source and target domains, which translates to 4 s images considering all rotations.
The Alexnet backbones are trained using a learning rate of 0 .
01 and s = 128 .
The Resnet-34 and VGG backbones are both trained using s = 16 and a learning rate of 0 .
We found that beyond a certain point early on in training, the number of steps of training for rotation prediction did not make a big difference to the ﬁnal task accuracy, and ﬁnally the chosen number of training steps was 4000 for Alexnet, 2000 for VGG-16 and 5000 for Resnetbackbones.
F.3 Other Experiments
MoCo pretraining.
Using the Alexnet backbone, we trained momentum contrast [13] fortraining steps, where in each step the model saw 32 images each from the real and the clipart domains of Ofﬁce-Home.
The queue length used for MoCo was 4096 and the momentum parameter was 0 . 999 .
SimSiam pretraining.
Using the Alexnet backbone, we trained SimSiam [7] for 200 epochs (or 6800 training steps) on a mix of the source ( real ) and target ( clipart ) sets of Ofﬁce-Home, with a batch size of 256.
Virtual Adversarial Training.
For adding a VAT criterion to our model, we closely followed the
VAT criterion in VADA [31].
We used a radius of 3 . 5 for adversarial perturbations and a coefﬁcient of 0 .
01 for the VAT criterion, which is the KL divergence between the outputs of the perturbed and the unperturbed input from the target domain.
Empirical Bhattacharyya Distance Estimate.
We use this estimate to compare target domain inter-class separation in Table 4.
For computing an approximation, we made the assumption that features for each class in the target domain are distributed as gaussians with identity covariance and used the closed form Bhattacharyya Distance (BD) between two multivariate gaussians [36].
The estimate then reduces to:
[Equation]
where µ i is the mean of class i features of images in the target domain ( D t ∪ D u ).