N EIGHBOR C LASS C ONSISTENCY ON U NSUPERVISED D OMAIN A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Unsupervised domain adaptation (UDA) is to make a prediction for unlabeled data in a target domain with labeled data from the source domain available.
Recent advances exploit entropy minimization and self-training to align the feature of two domains.
However, as decision boundary is largely biased towards source data, class-wise pseudo labels generated by target predictions are usually very noisy, and trusting those noisy supervisions might potentially deteriorate the intrinsic target discriminative feature.
Motivated by agglomerative clustering which assumes that features in the near neighborhood should be clustered together, we observe that target features from source pre-trained model are highly intrinsic discriminative and have a high probability of sharing the same label with their neighbors.
Based on those observations, we propose a simple but effective method to impose Neighbor Class Consistency on target features to preserve and further strengthen the intrinsic discriminative nature of target data while regularizing the uniﬁed classiﬁer less biased towards source data.
We also introduce an entropybased weighting scheme to help our framework more robust to the potential noisy neighbor supervision.
We conduct ablation studies and extensive experiments on three UDA image classiﬁcation benchmarks.
Our method outperforms all existing
UDA state-of-the-art.
I NTRODUCTION
Recent advances in deep neural network have dominated many computer vision tasks, such as image recognition
He et al.
(2016), object detectionGirshick (2015), and semantic segmentationLong et al.
However, collection and manual annotation need no trivial human effort, especially for vision tasks like semantic segmentation where dense annotations are required.
Thanks to the growth of computer graphics ﬁeld, it is now possible to leverage CNN to synthetic images with computergenerated annotations (Richter et al.
; Ros et al.
, so unlimited amount of data with free annotation is available for training network in scale.
However, directly applying the model trained on synthetic source data to unlabeled target data leads to performance degradation and Unsupervised
Domain Adaptation (UDA) aims to tackle this domain shift problem.
A widespread of UDA methods were proposed to align the domain-invariant representations by simultaneously minimizing the source error and discrepancy(e.g.
H -divergence Ben-David et al.
(2010); Hoffman et al.
H(cid:52)H -divergenceBen-David et al.
between two domain such as the maximum mean discrepancy Tzeng et al.
(2014), correlation distanceSun et al.
(2016) and etc.
Further, adversarial learning-based UDA Ganin & Lempitsky (2015); Tzeng et al.
; Radford et al.
; Hoffman et al.
; Tsai et al.
; Sankaranarayanan et al.
; Luo et al.
(2019) methods aim to reduce this discrepancy between two domain by minimizing the adversarial loss.
However, the major limitation of adversarial learning is that it only aligns the global feature distribution of two domains without considering the class labels.
As the result, a small H(cid:52)H distance does not guarantee the small error on ideal joint hypothesis on the features of two domains
Liu et al.
To alleviate this issue, Entropy minimization (Grandvalet & Bengio (2005); Vu et al.
and
Self-Training (Lee (2013); Zou et al.
(2018)) are the two dominant methods to enforce the cluster assumption such that network can learn a discriminative feature space by pushing the decision boundary away from densely-distributed area.
However, as decision boundary is largely biased to-
[Figure]
wards source data, trusting biased network predictions will push target features towards their nearest source class prototypes while deteriorating the intrinsic discriminative target structure as shown in Fig.
1(2.a).
Motivated by agglomerative clustering methods (Sarfraz et al.
(2019)) which assume that features in the nearby region should be clustered together
, we investigate target features from source pretrained model and observe that they are intrinsically discriminative and have a very high possibility of sharing the same label with their neighbors as shown in Fig. 1(2.b).
To utilize this high-quality pairwise neighbor supervision, we propose a simple and effective approach to impose Neighbor Class Consistency between target samples and their neighbors.
To alleviate propagated errors from false neighbor supervision, we introduce an Entropy-based weighting scheme to emphasize more on the reliable pairwise neighbor supervision.
Additionally, we categorize Self Class Consistency as a special case of our method where the nearest neighbor of a sample is its self-augmentation.
Further, we explore feature representation learning based on the ranking relationship between selfaugmentation and the ﬁrst neighbor given an anchor.
We enforce the features of anchors to be closer to their self-augmentation than their ﬁrst neighbors.
In summary, our main contributions are shown as follows: (1) We revisit the source pre-trained model and observe the intrinsic discriminative nature of target features from source model.
Based on this observation, we propose Neighbor Class Consistency (NC) to utilize the high-quality pairwise neighbor pseudo supervision over noisy class-wise pseudo supervision from Self-Training methods.
(3) We introduce an Entropy-based weighting scheme to help our framework be more robust to unreliable neighbor supervision.
(4) We categorize Self Class Consistency as a special case of our framework and explore the ﬁrst neighbor for feature representation learning.
(5) We conduct extensive experiments on three UDA benchmarks datasets.
NC outperforms all existing methods and achieves a new UDA state-of-the-art performance.
Notably, we achieve 86.2% on challenging VisDA17 dataset.
R ELATED W ORK
Discrepancy based domain adaptation Following the theoretical upper bound proposed in BenDavid et al.
(2007), existing methods have explored to align the feature representations of the source and target images by minimizing the distribution discrepancy.
For example, Maximum Mean Discrepancy (MMD) Tzeng et al.
(2014) is proposed to match the mean and covariance of source and target distributions.
Alternatively, adversarial domain adaptation methods Ganin & Lempitsky (2015); Tzeng et al.
; Radford et al.
; Hoffman et al.
; Tsai et al.
; Sankaranarayanan et al.
; Luo et al.
(2019) solve this domain discrepancy by training a domain- invariant feature generator which produces the features to fool a discriminator that distinguishes the representations from source and target domains.
However, since the domain discriminator aligns source and target features without considering the class labels, merely aligning the global marginal distribution of the features in the two domains fails to align the class-wise distribution.
Clustering based domain adaptation Entropy minimization (Grandvalet & Bengio (2005); Vu et al.
and self-training (Zou et al. ; Gu et al.
(2020)) are two streams of approaches to realize the class-wise alignment across domains.
However, minimizing the conditional entropy of target prediction based on source biased classiﬁer will harm the intrinsic discriminative target structure.
To this end, prototypical classiﬁer and alignment Xie et al.
(2018); Pan et al.
(2019) have been explored to mitigate the noisy supervision from source biased classiﬁer.
However, the prototypes of target samples are still noisy estimated cluster centers and therefore target samples away from prototypes may still risk of being wrongly classiﬁed.
Similar effort has been made by Tang et al.
(2020) on learning the latent discriminative target structure with respect to learnable clusters via deep clustering framework Ghasedi Dizaji et al.
Consistency Regularization In semi-supervised setting, mainstream methods apply various consistency regularization on unlabeled data with different pre-deﬁned positive counterpart.
Among them, Tarvainen & Valpola (2017) impose consistency between predictions from student network and moving-average based teacher networks.
Virtual Adversarial Training Miyato et al.
(2018) tries to make network invariant to small adversarial permutations around the neighborhood of a sample while DTALee et al.
(2019) enforces the target predictions from the networks with different choice of dropout mask to be consistent.
In un/self-supervised setting, SimCLR (Chen et al.
and
MoCo (He et al.
(2020)) are the two prevalent approaches to conduct contrastive learning on the feature space among unlabeled data, its strong augmentation and other negative samples from either extremely large current batch or momentum updated memory banks. In comparison to those un/self-supervised methods, our method exploreS neighbor samples for consistency regularization.
In cross-domain person re-identiﬁcation, Zhong et al.
(2019) also utilize the neighbor information on target data.
To emphasize the difference, our method focuses on regularizing the classiﬁer to be less biased toward source domain by applying target neighbor class consistency while they focus on the feature representation learning by enforcing neighbor feature invariance.
M ETHOD 3. P ROBLEM D EFINITION
In unsupervised domain adaptation(UDA), source domain data is denoted as D s = { ( x si , y si ) | N s i =1 } where x si and y si denote the i -th training sample and its label, N s is the number of source images.
Target domain data is denoted as D t = { x ti | N t i =1 } where N t is the number of target images.
The objective of UDA is to train a deep neural network G ( ·| θ ) which has access to the source data ( x s i , y s i ) drawn from D s and target data x t i drawn from D t such that the model G ( ·| θ ) can generalize on target domain.
Network G ( ·| θ ) =
C ◦ F ( ·| θ ) is comprised of a feature extractor F ( ·| θ ) and a classiﬁer C ( ·| θ ) where θ denotes network parameters.
3. R EVISIT S UPERVISED P RE -T RAINING FOR S OURCE DOMAIN
In general, UDA pre-trains a network G ( ·| θ ) on source domain by standard cross entropy loss and then the network is transferred to inference the target data.
The source supervised objective function is in the form of,
[Equation]
However, the model trained on source data usually generalizes poorly on target data due to the domain shift between the joint distribution of two domains.
Self-Training (ST) and Entropy Minimization (Ent) methods are proposed to make the network be conﬁdent on its target predictions following the cluster assumption and thus improve the discrim- [Table] inativeness of network.
Their objective functions are calculated as
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] where ˜ y ti is the pseudo label for the i target sample and H is entropy function.
Nonetheless, the pseudo labels of target samples are noisy as the network is biased towards source data.
Trusting noisy label or prediction has high risk of misleading the training and propagate errors.
Inspired by agglomerative clustering methods which claim that features should be clustered with their near neighborhood, we hypothesize that source domain pre-training can provide a feature space where target features are intrinsically discriminative and have high probability of sharing the same labels with their neighbors .
To verify this, we apply k-Nearest Neighbor (kNN) search on target features extracted from source model on Offce-31 A
→ W. We evaluate the pairwise label consistent accuracy between target features and their k-th neighbors.
Table 1 shows that target features from source model have very high accuracy of sharing the same label with their K-th neighbors.
It demonstrates our hypothesis that target features from source model are locally discriminative.
This observation motivates us to leverage this relative ”clean” pairwise pseudo supervision to help the network generalization on target domain over the noisy class-wise pseudo supervision.
3. N EIGHBORHOOD C LASS CONSISTENCY
1) Neighborhood Discovery.
First, we extract target features z t = F ( x t | θ ) from source pre-train model and save them into a target feature memory bank V t
=
{ z ti | N t
i =1 } .
Then we apply kNN on target features V t to explore the neighborhood
N k ( z t ) which is deﬁned as follows:
[Equation]
where s is a similarity metric and k is the number of neighbors.
Note that we update the memory bank V t with the newest features in training and conduct kNN on V t at every epoch.
2) Vanilla Neighborhood Class Consistency (VNC).
Based on our motivation that target samples should share the same class label with their neighbors, we propose VNC to enforce the class assignment consistency between pairwise neighbors by mutual information (MI) maximization between their network predictions.
Formally, we formulate the objective function of VNC in the form of:
[Equation]
It is also worth noting that the memory bank serves as a look-up table to retrieve the target features and feed them into the classiﬁer C for computing the Neighborhood Class Consistency loss.
As the size of k can be potentially large, using memory bank can implicitly enlarge the batch size by k times without introducing extra computational cost and time.
3) Entropy-weighted Neighborhood Class Consistency (ENC).
Table 1 shows that the reliability of pairwise neighborhood supervision is decreased with the rise of neighborhood size k .
In other words, there is a trade-off between large neighborhood diversity and reliability of neighbor supervision.
A straightforward idea is to treat each neighbor pairs differently in terms of loss weight such that hopefully positive neighbor pairs (Correct) will have higher loss weight than negative neighbor pairs (False).
To achieve this, we propose a entropy-based weighting scheme (EW) to assign different loss weight on different neighbor pairs given an anchor.
As the entropy is a measurement of sample prediction certainty, the less entropy is, more conﬁdent the prediction is and thus more weight we will assign on.
Formally, we deﬁne the loss weight and objective of ENC as follows:
[Equation]
where M is the number of class.
We claim that entropy-based weighting helps mitigate the problem of noisy pairwise supervision and it is more robust to the neighborhood size k
as it will down-weight the neighbor pairs if the selected neighbor samples have high entropy value.
3. S ELF C LASS C ONSISTENCY
In addition to neighborhood consistency, target samples should always share the same label with their self-augmentation and this pairwise supervision is 100% clean.
Therefore, Self Class Consistency is a special case of NC where the nearest neighbor given an anchor is its augmentation.
We adopt Cropping, grayscale and color distortion as our data augmentation following Chen et al. (2020) and enforce the self-consistency in terms of class assignment on target samples and their augmentations as follows,
[Equation]
where ˜ x ti is the data augmentation of i -th target sample.
3. F EATURE R ANKING BETWEEN S ELF -A UGMENTATION AND F IRST N EIGHBOR
As the ﬁrst neighbors given target anchors might be negative (do not share the same label with anchors) and the anchors’ self-augmentations might be heavily distorted, enforcing neighbor class consistency alone might not guarantee that the anchors are closer to their self-augmented samples than their ﬁrst neighbors i feature space. We claim that introducing this feature ranking regularization beneﬁts the feature representation learning by ensuring the positive samples (self-augmentations) rank higher than the negative samples (the ﬁrst neighbors) in feature space. When the ﬁrst neighbors are positive, it can also enforce an inductive bias on model training to rank the positive samples with pre-deﬁned variations (such as cropping, grayscale and color distortion) higher than the positive samples with unknown variations (the ﬁrst neighbors).
To impose this feature ranking regularization, we adopt triplet loss as objective function as shown below,
[Equation]
where N 1 ( F ( x t i | θ )) is the feature of the ﬁrst neighbor of target sample i .
3. T OTAL L OSS
The objective of proposed Neighbor Class Consistency is in the form of,
[Equation]
As pre-training on source data only might not provide the optimal initial feature space to explore high-quality neighborhood, we add target self-consistency loss with source supervised loss in pretraining stage and formulate it as:
[Equation]
E XPERIMENT 4. D ATASETS
We conduct experiments on three widely used domain adaptation classiﬁcation benchmarks: Ofﬁce31 Saenko et al.
, VisDA17 Peng et al.
and
ImageCLEF-DA 1 .
Ofﬁce-31 is a commonly used dataset for unsupervised domain adaptation.
It includes 4652 images of 31 classes from three domains: Amazon (A), Webcam (W) and DSLR (D).
ImageCLEF-DA consists of 12 common classes shared by three public datasets (domains): Caltech-256 (C), ImageNet
ILSVRC 2012 (I), and Pascal VOC 2012 (P).
VisDA17 is a large-scale dataset.
It uses 152 ,2D synthetic images from 12 classes as the source training set and 55 , 400 real images from MSCOCO Lin et al.
(2014) as the target set.
12 object categories are shared by these two domains.
4. I MPLEMENTATION DETAILS
We follow the standard protocol of UDA( Ganin et al.
; Zhang et al.
(2019b)
; Xu et al.
Zou et al.
to use all labeled source samples and all unlabeled target samples as training data.
The reported testing results are the average accuracy over three random repeats with center-crop images.
We adopt ResNet-50 (He et al.
on Ofﬁce-31 and ImageCLEF-DA while ResNet on VisDA17 dataset, ﬁne-tuned from the ImageNet (Deng et al.
pre-trained model.
We use Pytorch as implementation framework.
We adopt Stochastic Gradient Descent (SGD) optimizer with learning rate of 1 × 10 − 3 , weight decay 5 × 10 − 4 , momentum 0 .
9 and batch size 32 .
We set λ = 0 .
1 (Eq 10) for Ofﬁce-31 and ImageCLEF-DA while λ = 0 .
5 for VisDA17.
We set k = [-
-]1 (Eq 7) to explore the most local neighbourhoods and triplet margin m = 0 .
1 in (Eq 9).
The analysis on the impact of k is at section 4.5.
4. A BLATION S TUDY
In this section, we investigate the effectiveness of each components of NC in achieving the stateof-the art performance on Ofﬁce-31.
We name each components of NC as: (1) Source Model , which ﬁne-tunes the base network on source labeled samples with Eq 1; (2) Vanilla Neighbor Class
Consistency (VNC) , which denotes the training with non-weighted Neighbor Class consistency with Eq 5; (3) Entropy-Weighted Neighbor Class Consistency (ENC) , which denotes the training with Entropy-Weighted Neighbor Class consistency with Eq 7; (4) Self Class Consistency(SC) , which denotes the training with Self consistency with Eq 8; (5) Feature Ranking (FR) , which denotes the training with feature ranking relations between anchor, its self-augmentation and its ﬁrst neighbor with Eq 9; (6) Self-Supervised Pretraining (SP) , which ﬁne-tunes the base network with source labeled samples and SC with Eq 11.
From the Tab 2, we observe that all proposed components play a contributing role to our ﬁnal performance.
4. C OMPARISONS WITH THE STATE OF THE ART
Results on Ofﬁce-31 and ImageCLEF-DA based on ResNet-50 are shown in Table 3 and Table 4.
We can observe that NC-SP outperforms all the existing methods and more importantly boosts the
[Table]
Please refer to Section 4.3 on what each component represents [Table] performance substantially on difﬁcult transfer tasks such as A → W, D → A and W → A. Notably, comparing to state-of-the-art self-training (Zou et al.
and entropy minimization methods (Xu et al.
based on class-wise pseudo supervision, NC-SP outperforms them by 4.6% and 3.9% respectively, and it demonstrates that following the relatively clean neighborhood pseudo supervision can alleviate the error accumulation problem from self-training.
Results on VisDA17 based on ResNet-101 are reported in Table 5.
NC-SP achieves much better performance than all compared methods.
Comparing to DTA (Lee et al.
(2019)) which uses VAT (Miyato et al.
as self-consistency and Adversarial Dropout as model-consistency, we instead explore neighborhood consistency and demonstrate the effectiveness of NC-SP over other consistency regularization.
Note: The results of existing methods in Table 3,4,5 refer to their respective papers.
4. A NALYSIS
Feature visualization.
We visualize the target feature embeddings of (a) source model, (b) entropy minimization, and (c) NC on Ofﬁce-31 W → A via t-SNE (Maaten & Hinton (2008)) in Fig.2.
We can qualitatively observe that NC could learn more discriminative features than source model and entropy minimization as it preserves the intrinsic target features structure.
Nearest neighbors visualization.
We visualize the top 3 nearest neighbors given an anchor based on the target features from source model on Ofﬁce-31 A → W in Fig.3(a).
We investigate both success and failure cases to get extra insights into our method.
For the ﬁrst two row, features from source model could retrieve the correct neighbors for the mobile phone and backpack with certain level of appearance and pose variations.
However, at the last row where the pose of that laptop sample is unusual, our method might fail in those cases.
The proposed entropy weighting on neighbor class consistency is to alleviate the misleading learning by false neighborhood supervision.
Impact of neighbourhood size k . Neighbourhood size k is an important parameter as it controls the amount of pairwise neighborhood supervision.
However, there is a trade-off between increasing the neighbor diversity and increasing the risk of adding false neighborhood supervision.
Empirically, the optimal k is associated with the size of target dataset and the number of class.
We evaluate k from { 1, 3, 5 } for both Vanilla NC and Entropy-weighted NC on Ofﬁce-31 A → W as shown in Fig 3(b).
We observe that the performance decreases when increasing the k for VNC while ENC is more
[Table]
[Table]
robust to k .
This is consistent with the intuition of Entropy-weighting which aims to alleviate the problem of false neighborhood supervision.
For simplicity, we use k =1 through all the experiments while increasing k deﬁnitely has lots of potential to boost the performance further especially when applying Entropy weighting scheme.
How does entropy-based weighting looks like?
In order to prove our hypothesis that negative neighbor pairs will have lower consistency loss weight, we compute the entropy weights for all the neighbor pairs ( k =1) on Ofﬁce-
A → W and W → A, and take average over entropy weights on positive pairs and negative pairs respectively.
From Fig 3(c), we can see that positive neighbor pairs have much higher consistency loss weight than negative pairs by 20% on average.
It demonstrates our hypothesis that Entropy-weighted NC could alleviate the false neighbor supervision via assigning less loss weight to them . Hyper-parameter sensitivity.
We conduct the hyper-parameter sensitivity analysis on the λ in Eqn.
{+10 and the triplet margin m in Eqn. 9 on Ofﬁce
 A → W. As Fig. 5 shows, the performance of+}
VNC decreases with the increase of λ while the performance of NC is inversely proportional to the triplet margin m .
To conclude, our method is robust to the regional changes of hyper-parameters.
The reason that large margin results in a performance drop might be that there exists a portion of anchors whose ﬁrst neighbors are positive and semantically more similar to the anchors than their self-augmentations.
Setting the margin to a very large value will emphasize the inductive bias too much on the pre-deﬁned variations and thus harm our model to align those positive neighbors with unknown variations to the anchors.
By setting a small margin, our method could ensure the selfaugmentations are more close to their anchors when the ﬁrst neighbors of anchors are negative.
At the same time, we can ensure that the self-augmentations are not too far away from the ﬁrst neighbors when they are positive.
[Figure]
[Figure]
Anchor backpack laptop 1 NN 2 NN 3 NN (
[Figure]
[Figure]
[Figure] [Figure]  C ONCLUSION In this work, we propose a simple but effective appoarch, named Neighbor Class Consistency based on the observation that target features extracted from source pre-trained model are high intrinsically discriminative.
We introduce an entropy-based weighting scheme to improve the robustness of our framework to potential noisy neighbor supervision.
We also incorporate self-supervision and impose metric learning based on feature ranking relationship.
We conduct solid ablation studies to prove each proposed components contributing to the performance and extensive experiments on three UDA image classiﬁcation benchmarks.
Our method outperforms all existing UDA state-of-the-art.
A A PPENDIX A. A DDITIONAL H YPER - PARAMETER S ENSITIVITY A NALYSIS
To select the best neighborhood size k and analyze the inﬂuence of k on the performance we evaluate k from { 1, 3, 5 } for all the tasks on Ofﬁce31 and report the performance in Fig. We observe that our method (NC) achieves the best performance when k = 1 while the performance does not monotonically decrease with the increase of neighborhood size A. C OMPARISON TO THE R ELATED WORK FROM OTHER D OMAIN
To differentiate our method to the related work from other {+domains such as self-supervised learning (Chen et al. knowledge distillation (Yun et al. and person re-identiﬁcation tasks (Zhong et al. , we implement their idea in unsupervised domain adaptation setting on Ofﬁce
+}
A → W in table 6.
Speciﬁcally, 1) for self-supervised learning, we utilize the self consistency loss in Eqn. 8 only as comparison and term it as SC (ours) .
Though our objective function is different, we adopt the same data augmentation rule from from (Chen et al.
For knowledge distillation which is a supervised setting, we replace our objective function (Mutual information maximization) in our ENC to the KL divergence objectives from (Yun et al.
to compute the class consistency loss and term it as CS-KD .
For person re-identiﬁcation, we refer to the ofﬁcial github repository of (Zhong et al.
(2019)) which introduces the neighbors invariance in feature space and implement it in UDA setting.
We term it as ECN .
As table 6 shows, our ENC outperforms other baselines by a signiﬁcant margin.
It demonstrates that 1) the neighbor class consistency contributes more to the domain alignment than self consistency loss.
Our consistency loss based on mutual information maximization is better than KL divergence in terms of enforcing class prediction consistency.
Our neighbor class consistency loss is more effective than enforcing neighbor feature invariance.
We think the features of target neighbors are naturally close to each other in feature space but the source biased classiﬁer fails to provide consistent class predictions between the target neighbor.
Therefore, in our point of view, providing regularization on the classiﬁer via our neighbor class consistency loss is more important than enforcing neighbor feature invariance in UDA problem
[Table]
A. A DDING R ESULTS FOR ENC ON I MAGE CLEF-DA AND V IS DA
To show the effectiveness of our method with ENC only
[Table]
we report the results on ImageCLEFDA and VisDA17 in table 7 and 8.
We could observe that our method with ENC only achieves the third best performance (exclude NC-SP) in Ofﬁce-31, top 1 performance in ImageCLEF-DA and the second best performance in VisDA17.
Compared to the self-training based methods (ENT, MRENT,
SAFN+Ent)
[Table]
our ENC outperforms them by a large margin without bells and whistles
[Table]
[Table]