CDT RANS : C ROSS - DOMAIN T RANSFORMER FOR U NSUPERVISED D OMAIN A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Unsupervised domain adaptation (UDA) aims to transfer knowledge learned from a labeled source domain to a different unlabeled target domain.
Most existing
UDA methods focus on learning domain-invariant feature representation, either from the domain level or category level, using convolution neural networks (CNNs)-based frameworks.
One fundamental problem for the category level based UDA is the production of pseudo labels for samples in target domain, which are usually too noisy for accurate domain alignment, inevitably compromising the UDA performance.
With the success of Transformer in various tasks, we ﬁnd that the cross-attention in Transformer is robust to the noisy input pairs for better feature alignment, thus in this paper Transformer is adopted for the challenging UDA task.
Speciﬁcally, to generate accurate input pairs, we design a two-way center-aware labeling algorithm to produce pseudo labels for target samples.
Along with the pseudo labels, a weight-sharing triple-branch transformer framework is proposed to apply self-attention and cross-attention for source/target feature learning and source-target domain alignment, respectively.
Such design explicitly enforces the framework to learn discriminative domain-speciﬁc and domain-invariant representations simultaneously.
The proposed method is dubbed
CDTrans (cross-domain transformer), and it provides one of the ﬁrst attempts to solve UDA tasks with a pure transformer solution.
Extensive experiments show that our proposed method achieves the best performance on all public UDA datasets including Ofﬁce-Home, Ofﬁce-31, VisDA-2017, and DomainNet.
I NTRODUCTION
Deep neural network have achieved remarkable success in a wide range of application scenarios but it still suffers poor generalization performance to other new domain because of the domain shift problem (Csurka, 2017;
Zhao et al., 2020; Zhang et al., 2020; Oza et al., 2021).
To handle this issue and avoid the expensive laborious annotations, lots of research efforts (Bousmalis et al., 2017;
Kuroki et al., 2019; Wilson & Cook, 2020; VS et al., 2021) are devoted on Unsupervised Domain Adaptation (UDA).
The UDA task aims to transfer knowledge learned from a labeled source domain to a different unlabeled target domain.
In UDA, most approaches focus on aligning distributions of source and target domain and learning domain-invariant feature representations.
One kind of such UDA methods are based on category-level alignment (Kang et al., 2019; Zhang et al., 2019;
Jiang et al., 2020; Li et al., 2021b), which have achieved promising results on public UDA datasets using deep convolution neural networks (CNNs).
The fundamental problems in category-level based alignment is the production of pseudo labels for samples in target domain to generate the input source-target pairs.
However, the current CNNs-based methods are not robust to the generated noisy pseudo labels for accurate domain alignment (Morerio et al., 2020; Jiang et al., 2020).
With the success of Transformer in natural language processing (NLP) (Vaswani et al., 2017;
Devlin et al., 2018) and vision tasks (Dosovitskiy et al., 2020; Han et al., 2020; He et al., 2021; Khan et al., 2021), it is found that cross-attention in Transformer is good at aligning different distributions, even from different modalities e.g. , vision-to-vision (Li et al., 2021e), vision-to-text (Tsai et al., 2019;
Hu & Singh, 2021) and text-to-speech (Li et al., 2019).
And we ﬁnd that it is robust to noise in pseudo labels to some extent.
Hence, in this paper, we apply transformers to the UDA task to take advantage of its robustness to noise and super power for feature alignment to deal with the problems as described above in CNNs.
In our experiment, we conclude that even with noise in the labeling pair, the cross-attention can still work well in aligning two distributions, thanks to the attention mechanism.
To obtain more accurate pseudo labels, we designed a two-way center-aware labeling algorithm for samples in the target domain.
The pseudo labels are produced based on the cross-domain similarity matrix, and a center-aware matching is involved to weight the matrix and weaken noise into the tolerable range.
With the help of pseudo labels, we design the cross-domain transformer (CDTrans) for UDA.
It consists of three weight-sharing transformer branches, of which two branches are for source and target data respectively and the third one is the feature alignment branch, whose inputs are from source-target pairs.
The self-attention is applied in the source/target transformer branches and crossattention is involved in the feature alignment branch to conduct domain alignment.
Such design explicitly enforces the framework to learn discriminative domain-speciﬁc and domain-invariant representations simultaneously.
In summary, our contributions are three-fold:
• We propose a weight-sharing triple-branch transformer framework, namely, CDTrans, for accurate unsupervised domain adaptation, taking advantage of its robustness to noisy labeling data and great power for feature alignment.
• To produce pseudo labels with high quality, a two-way center-aware labeling method is proposed, and it boosts the ﬁnal performance in the context of CDTrans.
• CDTrans achieves the best performance compared to state-of-the-arts with a large margin on Ofﬁce-home (Venkateswara et al., 2017), VisDA-2017 (Peng et al., 2017), Ofﬁce31 (Saenko et al., 2010) and DomainNet (Peng et al., 2019) datasets.
R ELATED W ORK 2. T RANSFORMER FOR V ISION
Transformer is proposed in (Vaswani et al., 2017) to model sequential data in the ﬁeld of NLP.
Many works have shown its effectiveness for computer-vision tasks (Han et al., 2020;
Khan et al., 2021; Li et al., 2021d; Han et al., 2021b;
Yu et al., 2021;
Li et al., 2021c; Yang et al., 2021).
Pure Transformer based models are becoming more and more popular.
For example, ViT (Dosovitskiy et al., 2020) is proposed recently by feeding transformer with sequences of image patches; Touvron et al.
(Touvron et al., 2021) propose DeiT that introduces a distillation strategy for transformers to help with ViT training; many other ViT variants (Yuan et al., 2021a;
Wang et al., 2021b; Han et al., 2021a; Chen et al., 2021a; Ranftl et al., 2021; Liu et al., 2021; Chen et al., 2021b; Zhang et al., 2021; Huang et al., 2021;
Rao et al., 2021;
Zhou et al., 2021; El-Nouby et al., 2021; Wang et al., 2021c; Xu et al., 2021; Wang et al., 2021a; Yuan et al., 2021b) are proposed from then, which achieve promising performance compared with its counterpart CNNs for both image classiﬁcation and downstream tasks, such as object detection (Liu et al., 2021), semantic segmentation (Yuan et al., 2021b) and object ReID (He et al., 2021).
For multi-modal based networks, there are several works (Tsai et al., 2019; Li et al., 2021e; Hu & Singh, 2021) that apply cross-attention for multi-modal feature fusion, which demonstrates that attention mechanism is powerful at distilling noise and feature alignment.
This paper adopts cross-attention in the context of pure transformers for UDA tasks.
2. U NSUPERVISED D OMAIN A DAPTATION
There are mainly two levels for UDA methods: domain-level (Tzeng et al., 2014; Long et al., 2015;
Ghifary et al., 2016; Tzeng et al., 2017; Bousmalis et al., 2017; Hoffman et al., 2018) and categorylevel (Saito et al., 2018;
Kang et al., 2019; Du et al., 2021;
Li et al., 2021a).
Domain-level UDA mitigates the distribution divergence between the source and target domain by pulling them into the same distribution at different scale levels.
The commonly used divergence measures include
Maximum Mean Discrepancy (MMD) (Gretton et al., 2006; Tzeng et al., 2014; Long et al., 2015) and Correlation Alignment (CORAL) (Sun et al., 2016; Sun & Saenko, 2016).
Recently, some works (Saito et al., 2018; Du et al., 2021; Li et al., 2021a) focus on the ﬁne-grained category-level label distribution alignment through an adversarial manner between the feature extractor and two domain-speciﬁc classiﬁers.
Unlike coarse-grained alignment at the domain scale, this approach aligns each category distribution between the source and target domain data by pushing the target samples to the distribution of source samples in each category.
Obviously, the ﬁne-grained alignment results in more accurate distribution alignment within the same label space.
Although the adversarial approach achieves new improvements by fusing ﬁne-grained alignment operations of source and target samples at the category level, it still does not solve the problem of noisy samples in the wrong category.
Our method adopts Transformers for category-level UDA which is robust to noisy inputs.
2. P SEUDO L ABELING
Pseudo labeling (Lee et al., 2013) is ﬁrst introduced for semi-supervised learning and gains popularity in domain adaptation tasks.
It learns to label unlabeled data using predicted probabilities and performs ﬁne-tuning together with labeled data.
In terms of using pseudo labeling for domain adaptation tasks, (Long et al., 2017; 2018) adopt pseudo labels to conduct conditional distribution alignment; (Zhang et al., 2018; Choi et al., 2019) use pseudo labels as a regularization for domain adaptation; (Zou et al., 2018) designs a self-training framework by alternately solving target pseudo labels; (Caron et al., 2018) propose a deep self-supervised learning methods by generating pseudo labels via k -means cluster to progressively train the model; (Liang et al., 2020) develop a selfsupervised pseudo labeling method to alleviate the effects of noisy pseudo labels.
Based on (Liang et al., 2020), in this work, we propose a two-way center-aware labeling algorithm to further ﬁlter the noisy pseudo pairs.
T HE P ROPOSED M ETHOD
We ﬁrst introduce the cross attention module and analyze its robustness to the noise in Section 3.1.
Then the two-way center-aware labeling method is presented in Section 3.2.
With the produced pseudo labels as inputs, our cross-domain transformer (CDTrans) is proposed in Section 3.3, consisting of three weight-sharing transformers.
3. T HE C ROSS A TTENTION IN T RANSFORMER 3.1. P RELIMINARY
Vision Transformer (ViT) (Dosovitskiy et al., 2020) has achieved comparable or even superior performance on computer vision tasks.
One of the most important structures in ViT is the selfattention module (Vaswani et al., 2017).
In ViT, an image I ∈ R H × W × C is reshaped into a sequence of ﬂattened 2D patches x ∈ R N × ( P 2 · C ) , where ( H, W ) is the resolution of the original image, C is the number of channels, ( P, P ) is the resolution of each image patch, and N = HW/P 2 is the resulting number of patches.
For self-attention, the patches are ﬁrst projected into three vectors, i.e. queries Q ∈ R N × d k , keys K ∈ R N × d k and values V ∈ R N × d v .
d k and d v indicates their dimensions.
The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.
The N patches serve as the inputs for the self-attention module, and the process can be formulated as:
[Equation]
From Eq.
1, it can be concluded that the self-attention module aims to emphasize the relationships among patches of the input image.
The cross-attention module is derived from the self-attention module.
The difference is that the input of cross-attention is a pair of images, i.e. I s and I t .
Its query and key/value are from patches of I s
and I t respectively.
The cross-attention module can be calculated as follows:
[Equation]
where Q s ∈ R M ×
d k are queries from M patches of image
I s , and K t ∈ R N × d k , V t ∈ R N ×
d v are keys and values from N patches of image
I t .
The output of the cross-attention module holds the same length M as the number of the queries.
For each output, it is calculated by multiplying V t with attention weights, which comes from the similarity between the corresponding query in I s and
[Figure]
DF
[Figure] ZLWKFURVVDWWQRQDOOSDLUV ZRFURVVDWWQRQDOOSDLUV ZLWKFURVVDWWQRQWUXHSRVSDLUV
[Figure] all the keys in I t . As a result, among all patches in I t , the patch that is more similar to the query of
I s would hold a larger weight and contribute more to the output.
In other words, the output of the cross-attention module manages to aggregate the two input images based on their similar patches.
So far, many researchers have utilized the cross-attention for feature fusion, especially in multimodal tasks (Tsai et al., 2019;
Li et al., 2019; Hu & Singh, 2021;
Li et al., 2021e).
In these works, the inputs of the cross-attention module are from two modalities, e.g. vision-to-text (Tsai et al., 2019; Hu & Singh, 2021), text-to-speech (Li et al., 2019) and vision-to-vision (Li et al., 2021e).
They apply the cross-attention to aggregate and align the information from two modalities.
Given its great power in feature alignment, we propose to use the cross attention module to solve the unsupervised domain adaptation problem.
3.1. R OBUSTNESS TO N OISE
As mentioned above, the input of the cross-attention module is a pair of images, which usually comes from two domains, and the cross-attention module aims to align these two images.
If label noise exists, there would be false positive pairs in the training data.
Images in the false positive pairs would have dissimilar appearance, and forcibly aligning their features would inevitably injure the training and compromise the performance.
We assume that the dissimilar patches in false positive pairs are more harmful to the performance than the similar patches.
In the cross-attention module, two images are aligned based on their patch similarity.
As shown in Fig.
1a, the cross-attention module would assign a low weight to the dissimilar patches in false positive pairs.
Thus it weakens the negative effects of the dissimilar patches on the ﬁnal performance to some extent.
To further analyze this issue, an experiment is carefully designed.
Speciﬁcally, we randomly sample true positive pairs from source and target domain in VisDA-2017 dataset (Peng et al., 2017) as the training data.
Then we manually replace the true positive pairs with random false positive pairs to increase the noise, and watch the changes of the performance as shown in Fig.
1b.
The x-axis indicates the rate of false positive pairs in the training data, and the y-axis shows the performance of different methods on the UDA task.
The red curve represents the results by aligning pairs with the cross-attention module, while the green curve is that without cross-attention, i.e. to directly train the target data with the label of corresponding source data in the pair.
It can be seen that the red curve achieves a much better performance than the green one, which implies the robustness of the cross-attention module to the noise.
We also provide another baseline shown as the blue curve in Fig.
1b, which is to remove the false positive pairs from the training data and train the cross-attention with only true positive pairs.
Without the noisy data, this baseline can be considered as the upper bound to our methods.
We can see the red curve is very close to the blue curve, and both of them are much better than the green one.
It further implies that the cross-attention module is robust to the noisy input pair.
[Equation]
To build the training pairs for the cross-attention module, an intuitive method is that for each image in the source domain, we manage to ﬁnd the most similar image from the target domain.
The set P S of selected pairs is:
[Equation]
where S, T are the source and target data respectively.
d ( f i , f j ) means the distance between features of image i and j .
The advantage of this strategy is to make full use of source data, while its weakness is obvious that only a part of target data is involved.
To eliminate this training bias from target data, we introduce more pairs P T from the opposite way, consisting of all the target data and their corresponding most similar images in the source domain.
[Equation]
As a result the ﬁnal set P is the union of two sets, i.e. P = { P S ∪ P T } , making the training pairs include all the source and target data.
3.2. C ENTER -A WARE F ILTERING
The pairs in P are built based on the feature similarities of images from both domains, thus the accuracy of the pseudo labels of pairs is highly dependent on the feature similarities.
Inspired by
Liang et al.
, we ﬁnd that the pre-trained model of the source data is also useful to further improve the accuracy.
Firstly, we send all the target data through the pre-trained model and obtain their probability distributions δ on the source categories from the classiﬁer.
Similar to Liang et al.
(2020), these distributions can be used to compute initial centers of each category in the target domain by weighted k-means clustering:
[Equation]
where δ kt indicates the probability of image t on category k .
Pseudo labels of the target data can be produced via the nearest neighbor classiﬁer:
[Equation]
where t ∈ T and d ( i, j ) is the distance of features i and j .
Based on the pseudo labels, we can calculate new centers:
[Equation]
In Liang et al.
(2020), Eq. 6 and 7 could be updated for multiple rounds, and we only adopt one round in our paper.
The ﬁnal pseudo labels are then used to reﬁne the selected pairs.
Speciﬁcally, for every pair, if the pseudo label of the target image is consistent with the label of the source image, this pair would be kept for our training, otherwise it will be discarded as a noise.
3. CDT RANS : C ROSS - DOMAIN T RANSFORMER
The framework of the proposed Cross-domain Transformer (CDTrans) is shown in Fig.
2, which consists of three weight-sharing transformers.
There are three data ﬂows and constraints for the weight-sharing branches.
The inputs of the framework are the selected pairs from our labeling method mentioned above.
The three branches are named as source branch , target branch , source-target branch .
As shown in Fig.
2, the source and target images in the input pair are sent to source branch and target branch respectively.
In these two branches, the self-attention module is involved to learn the domain-speciﬁc representations. And the softmax cross-entropy loss is used to train the classiﬁcation. It is worth noting that all three branches share the same classiﬁer due to the same label of two images. L cls
Self-Attention & Cross-Attention
Classifier
Self-Attention & Cross-Attention
L cls
L dtl ddd Multi-Head Self Attention
Add & Norm
Feed Forward
Add & Norm
V S K S Q S
H S
W V W K W Q
Multi-Head Cross Attention
Add & Norm
Feed Forward
Add & Norm
Q S K T V T
Multi-Head Self Attention
Add & Norm
Feed Forward
Add & Norm
V T K T Q T
H T
W V W K W Q
H S+T
H’ S
H’ T H’ S+T
W V W K W Q
Self-Attention & Cross-Attention
Weight-Sharing
Weight-Sharing
The cross-attention module is imported in the source-target branch.
The inputs of the source-target branch are from the other two branches.
In the N -th layer, the query of the cross-attention module comes from the query in the N -th layer of the source branch, while the keys and values are from those of the target branch.
Then the cross-attention module outputs aligned features which are added with the output of the ( N − 1) -th layer.
The features of the source-target branch not only align distributions of two domains, but are robust to the noise in the input pairs thanks to the cross-attention module.
Thus we use the output of the source-target branch to guide the training of the target branch.
Speciﬁcally, the source-target branch and target branch are denoted as teacher and student respectively.
We consider the probability distribution of the classiﬁer in source-target branch as a soft label that can be used to further supervise the target branch through a distillation loss (Hinton et al., 2015):
[Equation]
(8) where q k and p k are the probabilities of category k from the source-target branch and the target branch respectively.
During inference, only the target branch is used.
The input is an image from testing data, and only the target data ﬂow is triggered, i.e. the blue lines in Fig.
Its output of the classiﬁer is utilized as the ﬁnal predicted labels.
E XPERIMENTS 4. D ATASETS AND I MPLEMENTATION
The proposed method is veriﬁed on four popular UDA benchmarks, including VisDA-2017 (Peng et al., 2017), Ofﬁce-Home (Venkateswara et al., 2017), Ofﬁce-31 (Saenko et al., 2010) and
DomainNet (Peng et al., 2019).
The input image size in our experiments is 224 × 224.
Both the DeiT-small and DeiT-base (Touvron et al., 2021) are adopted as our backbone for fair comparison.
We use the mini-batch Stochastic
Gradient Descent (SGD) algorithm with the momentum of 0.9 and weight decay ratio 1e-4 to optimize the training process.
The learning rate is set to 3e-3 for Ofﬁce-Home, Ofﬁce-31 and DomainNet, 5e-5 for VisDA-2017 since it can easily converge.
The batch size is set to 64.
[Table]
[Table]
[Table]
4. C OMPARISON TO S O TA
We compare our method with state-of-the-art methods on UDA tasks, including MinEnt (Grandvalet et al., 2005), DAN (Long et al., 2015), DANN (Ganin & Lempitsky, 2015), ResNet (He et al., 2016), ADR (Saito et al., 2017), CDAN+E (Long et al., 2018), CDAN+BSP
(Chen et al., 2019), CDAN+TN (Wang et al., 2019), rRGrad+CAT (Deng et al., 2019), MCD (Saito et al., 2018), SWD (Lee et al., 2019), AFN (Xu et al., 2019), MSTN+DSBN (Chang et al., 2019), SAFN (Xu et al., 2019), SAFN+ENT (Xu et al., 2019), CADA-P (Kurmi et al., 2019), BNM (Cui et al., 2020),
DCAN (Li et al., 2020), SHOT (Liang et al., 2020), MCC (Jin et al., 2020), ATDOC-NA (Liang et al., 2021), and CGDM (Du et al., 2021).
The results are shown in Table 1, 2, 3 and 4.
[Table]
For Ofﬁce-Home, Ofﬁce-31 and DomainNet, as most of the methods use ResNet-50 as their backbones, we provide results with DeiT-small as our backbone for a fair comparison, which has a comparable model size as ResNet-50, but we also show the results using DeiT-base. And for
VisDA-2017, we adopt the DeiT-base backbone for fair comparisons, where other methods utilize ResNet-101 for their results.
The “Baseline-S/B” indicates directly training a DeiT-small/DeiT-base on the source domain and testing on the target domain.
The baseline shows lower performance than the SoTA methods on
VisDA-2017, Ofﬁce-Home and Ofﬁce-31, while it is higher than the SoTA on DomainNet.
On the one hand, it demonstrates the advantages of the Transformer in some computer vision tasks.
On the other hand, it suggests that the Transformer does not always hold a better generalization ability in its vanilla form.
Our method is also a pure Transformer-based solution, but includes our improvements into the Transformer.
From Table 1, 2, 3 and 4, it can be seen that our method outperforms the baseline with a large margin on all four datasets, e.g. nearly 21% on VisDA.
With our improvements, the new Transformer with cross-attention module shows a much better generalization power, and achieves the best performance on all datasets compared to other SoTAs methods.
It further implies the effectiveness of our method on the UDA task.
It is worth noting that our method on DeiT-base backbone outperforms other SoTA with 5.5%/8.7%/3.3%/9.8% on four datasets, which sets a new benchmark for UDA tasks.
Taking a closer look at the results, for the hard categories, such as “person” in VisDA-2017 dataset, the baseline is very low, which indicates the initial model of our method has a poor classiﬁcation ability on this category, leading to the pseudo labels with more noise.
Even with such a poor baseline and poor quality of pseudo labels, our method can still achieve a much higher performance boost (from 10.3% to 88.6%).
It suggests that our method has a great robustness to the labeling noise and can overcome the noise problem to some extent.
In Table 4, although the average performance of our methods outperforms other approaches by a large margin, it is noticed that the results of qdrto- others is much lower due to the low baseline.
It is because the Transformer is weak in shape modeling caused by its hard patch partition (Naseer et al., 2021).
4. A BLATION S TUDY
In this section, the effectiveness of the pseudo labeling and the losses in the proposed method is analyzed on the VisDa-2017 dataset.
4.3. D IFFERENT P SEUDO L ABELING
We have conducted experiments on different pseudo labeling methods to verify their inﬂuence on the ﬁnal performance.
The results are listed in Table.
Rec s , Rec t means the recall of the selected
[Table]
[Table]
training pairs in the source and target data, while Prec represents the accuracy of the pairs.
“Oneway-source” and “One-way-target” denote only using the pair set P S in Eq.
3 or P T in Eq.
4 for training.
“Two-way” indicates results of using the union of P s and P t without the center-aware strategy.
“Tw+Ca” implies our two-way center-aware labeling method, and “Groundtruth” means all training pairs are from groundtruthes.
By looking at Rec s and Rec t in Table.
5, it can be found that the one-way methods have an apparent bias on either source or target data, and its results are lower than the two-way method.
By comparing “Two-way” and “Tw+Ca”, we can conclude that although the center-aware method ﬁlters the training pair and slightly reduces the recall, it largely improves the precision and leads to a better ﬁnal performance.
We also ﬁnd that our two-way center-aware labeling method achieves a very high result, even very close to the upper bound trained with groundtruth pairs.
4.3. D IFFERENT L OSSES
As there are three losses in our method, we conduct another experiment to verify the effectiveness of each loss, as shown in Table.
“cls” in L s + t denotes that we replace the distillation loss with a classiﬁcation loss for the source-target branch.
We can see that the 3rd row with both L s and L t having classiﬁcation loss achieves a better result than the ﬁrst row where only L s has the cls loss, which means the target branch with the pseudo labels is helpful to improve the UDA result.
With the addition of “cls” in L s + t , the performance is further improved, which demonstrates the advantages of using the cross-attention module for feature alignment.
Using “dtl” instead of “cls” on the sourcetarget branch can further improve the results, showing the effectiveness of our distillation loss.
C ONCLUSION
In this paper, we tackle the problem of unsupervised domain adaptation by introducing the crossattention module into Transformer in a novel way.
We propose a new network structure CDTrans which is a pure transformer-based structure with three branches, and we also propose to generate high-quality pseudo labels using a two-way center-aware labeling method.
Training CDTrans using the generated high-quality pseudo labels yields a robust solution and also achieves state-of-the-art results on four popular UDA datasets, outperforming previous methods by a large margin.
We believe that transformer-based approaches will have great potential in the UDA community, and our work, as one of the ﬁrst attempts along this direction, has pushed forward the frontiers and shed lights for future research.