Con 2 DA: Simplifying Semi-supervised Domain Adaptation by Learning Consistent and Contrastive Feature Representations
Abstract
In this work, we present Con 2 DA, a simple framework that extends recent advances in semi-supervised learning to the semi-supervised domain adaptation (SSDA) problem.
Our framework generates pairs of associated samples by performing stochastic data transformations to a given input.
Associated data pairs are mapped to a feature representation space using a feature extractor.
We use different loss functions to enforce consistency between the feature representations of associated data pairs of samples.
We show that these learned representations are useful to deal with differences in data distributions in the domain adaptation problem.
We performed experiments to study the main components of our model and we show that (i) learning of the consistent and contrastive feature representations is crucial to extract good discriminative features across different domains, and ii) our model beneﬁts from the use of strong augmentation policies.
With these ﬁndings, our method achieves state-of-the-art performances in three benchmark datasets for
SSDA.
Introduction
Even though deep neural networks have become the state of the art for image classiﬁcation [17, 30, 12] they often require many labeled data to achieve good results.
Furthermore, these models usually perform poorly when test data are drawn from a different distribution or feature space.
This difference between data distributions is usually called domain shift [27, 32].
To alleviate such labeling efforts, domain adaptation (DA;[1]) aims at reducing the domain shift when a model trained on a source domain is tested on a target set sampled from a different distribution, by ﬁnding a common representation between them [23].
Semi-supervised domain adaptation (SSDA) methods address the problem by considering a labeled source and both (a few) labeled and unlabeled target data during training [28, 13, 15].
For instance,
Minimax Entropy (MME) [28] uses a (cid:96) 2 normalization on the output of a feature extractor and minimizes the cosine similarity between the normalized feature representation of the labeled data and a linear classiﬁer.
Thus, the weights of the linear classiﬁer can be seen as representative points of each class (i.e. prototypes).
In this work we adopted the normalized feature representation, which has proven to be effective to learn feature representations under labels restrictions [10, 4, 3].
Advances in semi-supervised learning such as consistency regularization methods [18, 35, 2, 3, 31] have demonstrated to improve the generalization of the model by generating the same model predictions for different randomly perturbed inputs.
Also, recent works suggest that the use of heavily distorted samples (e.g., Cutout [7], or RandAugment [6]) are effective to improve the model’s performance and generalization [31].
On the other hand, recent contrastive learning approaches have shown state-of-the-art results by learning discriminative feature representations from unlabeled data.
The general idea is to align a pair of positive samples and push negative samples far apart [8, 34, 3, 14].
Despite the promising results of the aforementioned approaches to learning in presence of domain shift, to the best of our knowledge, they have not been extended yet to a SSDA scenario.
In this work we adopted the normalized feature representation proposed by [28], and we extent recent advances on Con sistency regularization and Con trastive learning for the SS DA problem.
We present
Con 2 DA , a simple framework that takes input images and generates pairs of associated augmented versions of the input.
These pairs are mapped to the unit hypersphere using a feature extractor.
When labels are available, we move the associated normalized representations of the pairs that come from the same class towards their correspondent class prototype.
When labels are not available, we treat each pair of associated versions of the input as positive samples and pull them together, while the remaining dissimilar data pairs that come from different input images are pushed apart.
An overview of our model can be seen in Figure 1.
Using this simple yet effective method, we achieved state-of-the-art performances in the commonly used benchmark datasets for SSDA DomainNet [25], Ofﬁce-Home [33], and Ofﬁce-31 [27], improving up to 1 .
3% , 1 . 3% , and 1 . 4% in average for each dataset respectively.
Method
In a semi-supervised domain adaptation (SSDA) problem, we are given a source dataset composed by the images and their respective labels D s =
{ x si , y si } n s i =1 , an unlabeled target dataset D u = { ( x ui ) } n u i =1 , and labeled target data D t
= { ( x ti , y ti ) } n t i =1 .
In SSDA, n t is assumed to be small (usually 1 or 3 labeled samples per class).
For both domains we have the same K classes, i.e. y si ∈ { 1 , ..., K } , y ti ∈ { 1 , ..., K } .
Our goal is to reduce the domain shift between D s and {D u , D t } , to improve the performance of the model when evaluating on D u .
In this work we propose a method that learns Con sistent and Con trastive feature representations for SS DA .
An overview of our method, named Con 2 DA can be seen in Figure 1.
Our framework is composed of three major components:
• Stochastic data augmentation:
Given an input x ∈ {D s , D u , D t } , two random transformations are applied, generating the two associated versions of the same sample ˜ x weak ∼ T weak ( x ) and ˜ x strong ∼ T strong ( x ) .
For T weak ( · ) , we used simple transformations such as random cropping , random horizontal ﬂipping , and Gaussian blur .
For T strong ( · ) , we used the same augmentations, plus RandAugment [6].
We show that by adopting an augmentation strategy that strongly perturbs input images, our algorithm improves in performance compared with the method using only weak augmentations (see Appendix D).
• Feature extractor:
We used a feature extractor f ( · ) that maps transformations ˜ x ∈ { ˜ x weak , ˜ x strong } into a representation vector h
= f (˜ x ) ∈ R d , where d is the representation dimension (we set d = 256 in all our experiments).
This vector is normalized to the unit hypersphere using the (cid:96) 2 normalization, that takes the form z ( · ) =
f ( · ) (cid:107) f
( · ) (cid:107) .
Depending on whether ˜ x is a labeled or unlabeled sample, different loss functions are optimized.
• Classiﬁer: We pass the normalized feature representations through a linear classiﬁer W ∈ R d × K and we scale the output by a temperature hyperparameter T .
Using a softmax activation function we obtained the output probability distribution p ( y | ˜ x ) =
Softmax (cid:0) 1 T W T z (˜ x ) (cid:1) .
It is worth noticing that for accurate classiﬁcation, both the weight vector and the normalized features of the same class should be pointing to the same direction into the hypersphere.
This way, each weight vector w i ∈ W , i ∈ { 1 , ..., K } can be seen as a representative prototype of each class [28].
Supervised Loss Function:
For the labeled objects, we align associated data pairs towards their respective class prototype, regardless of whether the objects come from the source or target domain.
Speciﬁcally, we sample a mini-batch of data { ( x i , y i ) } Ni =1 ∈ {D s , D t } , composed of N/ 2 number of source labeled objects and N/ 2 number of target labeled objects.
For each sample x i , two stochastic augmentations are applied, obtaining a weak augmented version ˜ x iweak , and a strong augmented version ˜ x istrong .
Both images are passed through the feature extractor and classiﬁer, obtaining the probability vectors p weaki =
p ( y i | ˜ x weaki ) and p strongi = p ( y i | ˜ x strongi ) .
Then, the loss function for each mini-batch can be computed using Equation 1:
[Equation]
where L ce is a standard cross-entropy loss.
Intuitively, by minimizing this objective with respect to the feature extractor and the linear classiﬁer, we force the labeled feature representations and prototypes that represent the same class to point in the same direction into the hypersphere.
Unsupervised Loss Functions:
In the unsupervised case, we expect the distance of normalized feature representations of associated data pairs to be low, while we expect the distance with feature representations of objects that were generated from different inputs to be high.
For this purpose, we sample a mini-batch { x i } Ni =1 ∈ D u of N unlabeled target objects
and we use the normalized temperature-scaled cross-entropy loss function (NT-Xent; [3]), that takes two associated normalized feature representations z weaki =
z (˜ x weaki ) and z strongi
= z (˜ x strongi ) , and computes the cosine similarity between them sim ( z weaki , z strongi ) =
z weaki · z strongi .
This similarity is forced to be high, while the similarities sim ( z i , z a ) between each feature representation z i ∈ { z weaki , z strongi } and the rest of the dissimilar augmented samples z a ∈ A ( i ) =
{ z weak, ..., z weaki − 1 , z weaki +1 , ..., z weakN } ∪ { z strong, ..., z strongi −, z strongi + , ..., z strongN } are forced to be low.
The mathematical formulation for the NT-Xent loss can be written as follows:
[Equation]
where T denotes a temperature hyperparameter.
Notice that this loss function is computed for both z i
=
z weaki and z i =
z strongi .
Our model also computes an artiﬁcial label for each pair of strong and weak augmented unlabeled samples (i.e. pseudo-labeling [22, 19, 31]).
This label can be obtained by computing the arg max function over the averaged model predictions for the k classes, for a given pair of associated samples ¯ y i =
arg max 1 ≤ k ≤ K (( p weakik + p strongik ) / 2) .
Similarly, an averaged model conﬁdence can be computed as ¯ p i = max 1 ≤ k ≤ K (( p weakik + p strongik ) / 2) .
Then, for each mini-batch our method minimizes a self-supervised loss function with respect to the feature extractor:
[Equation]
where 1 ¯ p i ≥ τ ∈ { 0 , 1 } is the indicator function that takes value 1 iff ¯ p i is greater than τ , a probability threshold hyperparameter, and
0 otherwise.
For simplicity, we assume that ¯ p i are valid one-hot probability distributions for the cross-entropy loss.
The overall optimization procedure can be seen in Appendix E.
Experiments
As in the standard SSDA scenario, we randomly chose one or three labeled training samples per target class for training (one and three-shot respectively).
We also randomly selected three target labeled samples as the validation set.
We used all the unlabeled data for training, and we reveal their labels to evaluate and report the ﬁnal model performance.
Datasets.
We performed experiments using three benchmark datasets: DomainNet [25] a large-scale domain adaptation dataset that contains six domains and 345 classes in each domain.
Following [28], four domains (R: Real, P: Painting, S: Sketch, C: Clipart), 126 classes, and 7 different adaptation scenarios are used for evaluation.
Ofﬁce-Home [33] contains four domains (R: Real, A: Art, C:
Clipart, P: Product) and 65 classes.
We performed experiments using 12 adaptation scenarios.
Ofﬁce31 [27] contains three domains (W: Webcam, D: DSLR, A: Amazon), and 31 classes.
We used two adaptation scenarios for evaluation (D to A, and W to A).
Baselines.
We compared our method against state-of-the-art SSDA and UDA approaches, as well as using no domain adaptation at all.
The baselines for SSDA consist of the state-of-the-art methods
MME [29], APE [15], and BiAT [13].
For UDA we compared with DANN [9], ADR [29], and
CDAN [20].
We trained UDA methods treating target labeled samples as if they were source labeled samples.
Non adaptation methods consisted of models trained on all labeled samples using crossentropy, leveraging only labeled data (S+T; [5]), and considering unlabeled samples by minimizing the conditional entropy (ENT; [11]).
Results A summary of our results is shown in Table 1 (see Appendix C for complete result tables).
For the DomainNet dataset, our model achieved competitive results compared to the rest of state-of-the-art methods for AlexNet and ResNet34 in both one and three shot cases.
As can be seen for Ofﬁce-Home our method outperforms the averaged results reported by previous state-of-the-art results by 1 . 3% of accuracy in average.
Finally, for Ofﬁce31 our model outperforms previous state-of-the-art methods by a margin of 1 . 0% and 1 .
4% of accuracy in one-shot and three-shot scenarios respectively.
[Table]
Conclusions
Despite the huge labeling efforts that deep neural network models need to achieve their best performances
[Table]
SSDA methods can be applied to greatly reduce labeling costs.
In this work
[Table]
we present
Con 2 DA [Table] a simple framework that learns from artiﬁcially augmented images using consistent and contrastive visual representation to improve generalization on target domain.
We studied the main components of our model and we showed the effect of different combinations of augmentation and training strategies.
Using these ﬁndings, we compared our methods with different SSDA and UDA methods and we obtained state-of-the-art performances in three commonly used domain adaptation benchmarks DomainNet, Ofﬁce-Home, and Ofﬁce31 by margins of 1 . 3% , 1 . 3% , and 1 . 4% respectively.
Acknowledgements
We gratefully acknowledge support from ANID through the FONDECYT Initiation grant Nº and from the Data Science Unit at University of Concepción.
G
C
V. acknowledges support from the
ANID –Millennium Science Initiative Program–ICN12_009 awarded to the Millennium Institute of
Astrophysics (MAS).
Appendix Appendix A: Implementation Details :
We implemented our model using PyTorch [24].
For fair comparison with previous methods, we used AlexNet [17] or ResNet34 [12] pretrained on Imagenet [26] as the feature extractor.
If the number of labeled objects class in the target domain is greater than 256, we set the batch size to 256.
If not, we set the batch size to the number of target labeled objects.
We sample two mini-batches, where the ﬁrst mini-batch contains the same number of labeled objects per domain, and the second mini-batch is composed of only unlabeled target samples.
We optimized our model using Adam [16] with hyperparameters β 1 = 0.9, β 2 = 0 .
999 , and a learning rate of 0.00008 for all experiments.
We decayed the learning rate with the cosine decay schedule without restarts as in [21].
In all scenarios, the augmentation policy was set to H = 1 and the distortion magnitude to M = 10 (details in
Appendix A).
The rest of the hyperparameters were set to ( T = 0 .
05 , τ = 0 .
for DomainNet, ( T = 0 . 3 , τ = 0 .
95 ) for Ofﬁce-Home, and ( T = 0 .
5 , τ = 0 .
95 ) for Ofﬁce31.
We did a forward pass using the two mini-batches (labeled source and target, and unlabeled target), calculated their respective losses, and did a backward pass.
We called each of these forward-backward passes an iteration.
We trained each model over 5000 iterations, setting a patience of 50 iterations for early stopping.
Appendix B: Hyperparameter Optimization
We randomly sample three target labeled objects per class as the validation set and we use this set to set the hyperparameters temperature T and threshold τ .
We selected the hyperparameters using different domains and adaptation scenarios.
Speciﬁcally, for Ofﬁce31 we used the W
→ A scenario, and for Ofﬁce-Home and DomainNet we used the R → C scenario.
We vary the hyperparameters from
T ∈ { 0 . 01 , 0 . 05 , 0 . 07 , 0 . 1 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } and τ ∈ { 0 . 8 , 0 . 9 , 0 .
Figure 6 shows the results in terms of accuracy each case.
As can be seen, the best results on validation set are achieved using the hyperparameters ( T = 0 .
05 , τ = 0 .
for DomainNet (AlexNet),
( T = 0 .
05 , τ = 0 . 8 ) for DomainNet (ResNet34),
( T = 0 .
3 , τ = 0 .
95 ) for Ofﬁce-Home, and ( T = 0 .
5 , τ = 0 .
95 ) for Ofﬁce31.
Therefore, we setted the hyperparameters to these values for the remaining adaptation scenarios in each case for the results.
Appendix C: Results per adaptaton setting
Results of our method on the DomainNet dataset are shown in Table 2.
In both the one and three-shot cases, our model achieved competitive results compared to the rest of state-of-the-art methods for
AlexNet and ResNet34.
As can be seen in Table 3, for Ofﬁce-Home our method outperforms the averaged results reported by previous state-of-the-art results by margins between 0 .
1% up to 1 .
3% of accuracy.
Finally, as can be seen in Table 4, for Ofﬁce31 our model outperforms previous stateof-the-art methods by a margin of 1 . 0% and 1 .
4% of accuracy in one-shot and three-shot scenario respectively.
[Table] [Table]
[Table] [Table]
[Table] [Table]
Appendix D: Ablation Study
[Table]
[Figure]
[Table]
[Figure]
[Table]
MME
[Table] [Table]
[Table] [Table]
In order to understand the key components of our method, we performed an ablation study to compare gains in performance using variations of our algorithm.
We ﬁrst investigated the performance of different strong augmentation policies.
We started by ﬁrst measuring the performance on a baseline where no strong augmentations are applied (i.e both ˜ x weak and ˜ x strong are samples from T weak ).
In a three-shot scenario, the accuracy on validation set is 72 . 0% for Ofﬁce31 in the W
→ A setting, and 50 .
5% for DomainNet in the R → C setting.
We added strong augmentation policies to ˜ x strong such as color jitter, random grayscale, Cutout [7], and RandAugment [6].
The performance between combinations of these augmentation policies is shown in Figure 3.
As can be seen, our model consistently improved the baseline and the rest of the augmentation strategies in both settings by using RandAugment as a strong augmentation policy.
Using RandAugment, we also investigate the
[Table]
model performance removing components from the loss function.
Firstly, we removed the contrastive loss function deﬁned in Equation 2 (w/o L cont ), the self-supervised loss function deﬁned in Equation 3, w/o L self .
Then, we forced the prototypes to be unit vectors by normalizing the linear classiﬁer
W , and we computed the cosine similarity between the normalized linear classiﬁer and feature vector (w cosine).
Results of these variations are shown in Table 5.
As can be seen, our algorithm is beneﬁted from the use of contrastive and self-supervised loss.
Finally, the linear classiﬁer without normalization helped our model to obtain signiﬁcant performance improvements.
validation set.
[Table]
Appendix E: Algorithm Algorithm 1 summarizes the Con 2 DA training procedure.
Input: Mini batches
Transformations
1: while not converged do : 2: for i ∈ { 1 , ..., N } do 3: # Augmentations for labeled data 4:
˜ x weaki ∼ T weak ( x i ) 5:
˜ x strongi ∼ T strong ( x i ) 6: # Probability vectors for labeled samples.
7: p weaki =
p ( y i | ˜ x weaki ) 8: p strongi = p ( y i | ˜ x strongi ) 9: # Augmentations for unlabeled data 10: ˜ x u,weaki ∼ T weak ( x ui ) 11: ˜ x u,strongi ∼ T strong ( x ui ) 12: # Normalized feature representations.
13: z u,weaki =
f (˜ x u,weaki )) / (cid:107) f
(˜ x u,weaki ))
(cid:107) 14: z u,strongi = f
(˜ x u,strongi )) / (cid:107) f
(˜ x u,strongi ))
(cid:107) 15: # Probability vectors for unlabeled samples.
16: p u,weaki =
p ( y i | ˜ x u,weaki ) 17: p u,strongi = p ( y i | ˜ x u,strongi ) 18: # Averaged prediction and pseudo-label.
[Equation]
20: ¯
[Equation]
21: end for 22: for i ∈ { 1 , ..., N } do 23: sim
[Equation]
z
[Equation]
weaki · z u,strongi 24:
Deﬁne A ( i ) 25: for z a ∈ A ( i ) do 26: sim ia =
z ui · z a 27: end for 28: end for 29:
Compute Equations 1, 2, and30:
Update parameters of f and W to minimize Eq.
Update parameters of f to minimize Eq.
2 and 3.
32: end while
Output: Trained feature extractor f and classiﬁer W