J OINT
A TTENTION -D RIVEN
D OMAIN
F USION
AND
N OISE -T OLERANT
L EARNING
FOR
M ULTI -S OURCE
D OMAIN
A DAPTATION
A BSTRACT
Multi-source Unsupervised
Domain
Adaptation
(MUDA
) transfers knowledge from multiple source domains with labeled data to an unlabeled target domain.
Recently, endeavours have been made in establishing connections among different domains to enable feature interaction.
However, as these approaches essentially enhance category information, they lack the transfer of the domain-specific information.
Moreover, few research has explored the connection between pseudolabel generation and the framework’s learning capabilities, crucial for ensuring robust MUDA.
In this paper, we propose a novel framework, which significantly reduces the domain discrepancy and demonstrates new state-of-the-art performance.
In particular, we first propose a Contrary Attention-based Domain Merge (CADM) module to enable the interaction among the features so as to achieve the mixture of domain-specific information instead of focusing on the category information.
Secondly, to enable the network to correct the pseudo labels during training, we propose an adaptive and reverse cross-entropy loss, which can adaptively impose constraints on the pseudo-label generation process.
We conduct experiments on four benchmark datasets, showing that our approach can efficiently fuse all domains for MUDA while showing much better performance than the prior methods.
INTRODUCTION
Deep neural networks (DNNs) have achieved excellent performance on various vision tasks under the assumption that training and test data come from the same distribution.
However, different scenes have different illumination, viewing angles, and styles, which may cause the domain shift problem (Zhu et al., 2019;
Tzeng et al., 2017; Long et al., 2016).
This can eventually lead to a significant performance drop on the target task.
Unsupervised Domain Adaptation (UDA) aims at addressing this issue by transferring knowledge from the source domain to the unlabeled target domain (Saenko et al., 2010).
Early research has mostly focused on Single-source Unsupervised Domain Adaptation (SUDA), which transfers knowledge from one source domain to the target domain.
Accordingly, some methods align the feature distribution among source and target domains (Tzeng et al., 2014) while some (Tzeng et al., 2017) learn domain invariants through adversarial learning.
Liang et al.
(2020) use the label information to maintain the robust training process.
However, data is usually collected from multiple domains in the real-world scenario, which arises a more practical task, i.e., Multi-source Unsupervised Domain
Adaptation (MUDA) (Duan et al., 2012).
MUDA leverages all of the available data and thus enables performance gains; nonetheless, it introduces a new challenge of reducing domain shift between all source and target domains.
For this, some research (Peng et al., 2019) builds their methods based on SUDA, aiming to extract common domain-invariant features for all domains.
Moreover, some works, e.g. , Venkat et al.
Zhou et al.
(2021) focus on the classifier’s predictions to achieve domain alignment.
Recently, some approaches (Li et al., 2021; Wen et al., 2020) take advantage of the MUDA property to create connections for each domain.
Overall, since the main challenge of MUDA is to eliminate the differences between all domains, there are two main ways to achieve this.
One is to extract domain invariant features among all domains, i.e., filter domain-specific information for different domains.
The other is by mixing domain-specific information from different domains so that all domains share such
Feature Extractor
Classifier 𝓛 𝑪 r 𝓛 𝑪𝑬 𝓛 𝑨𝑹 − 𝑪𝑬 hard-label soft-label pseudo-label
f 𝒔𝟏 f 𝒔𝟏 h 𝒔f 𝒔𝟏 f 𝒔𝟏 h 𝒔f 𝒔𝟏 f 𝒔𝟏 h 𝒔f 𝒔𝟏 f 𝒔𝟏 h 𝒕 real-label
Memory
CADM h 𝑡
SourceTarget . . .
Source- . . .
Source- . .
mixed information and thus fuse into one domain .
Previous approaches have mostly followed the former, however, filtering the domain-specific information for multiple domains can be difficult and often results in losing discrimination ability.
For the latter, few methods have been proposed to address MUDA in this way, and there is a lack of effective frameworks to achieve such domain fusion, which is the main problem to be addressed by our proposed approach.
Moreover, existing methods ignore the importance of generating reliable pseudo labels as noisy pseudo labels could lead to the accumulation of prediction errors.
Consequently, it is imperative to design a robust pseudo-label generation based on the MUDA framework.
In this paper, we propose a novel framework that better reduces the domain discrepancy and shows the new state-of-the-art (SoTA) MUDA performance, as shown in Fig.
Our method enjoys two pivotal technical breakthroughs.
Firstly, we propose a Contrary Attention-based Domain Merge (CADM) module (Sec. 4.2), whose role is to perform the domain fusion.
Self attention (Vaswani et al., 2017; Dosovitskiy et al., 2020) can capture the higher-order correlation between features and emphases more relevant feature information, e.g. , the semantically closest information.
Differently, our CADM proposes the contrary attention, enabling each domain to pay more attention to semantically different domain-specific information of other domains.
Then, by integrating these domain-specific information, each domain can achieve movement to other domains, thus resulting in domain fusion.
Secondly, to enable the network to correct the pseudo labels during training, we take the pseudo-label generation as the optimization objective of the network by proposing an adaptive and reverse cross-entropy (AR-CE) loss (Sec. 4.3).
It imposes the optimizable constraints for pseudo-label generation, enabling the network to correct pseudo labels that tend to be wrong and reinforce pseudo labels that tend to be correct.
We conduct extensive experiments on four benchmark datasets.
The results show our method achieves new state-of-the-art performance and especially has significant advantages in dealing with the large-scale dataset.
In summary, we have made the following contributions:
• We propose a CADM to fuse the features of source and target domains, bridging the gap between different domains and enhancing the discriminability of different categories.
• We propose a loss function, AR-CE loss, to diminish the negative impact of the noisy pseudo label during training.
• Our method demonstrates the new state-of-the-art (SoTA) MUDA performance on multiple benchmark datasets.
RELATED WORK 2. U NSUPERVISED D OMAIN A DAPTATION (UDA)
According to the number of source domains, UDA can be divided into Single-source Unsupervised Domain Adaptation (SUDA) and Multi-source Unsupervised Domain Adaptation (MUDA).
In SUDA, some methods use the metrics, such as Maximum Mean Discrepancy (Long et al., 2017;
Wang et al., 2018), to define and optimize domain shift.
Some methods (Sankaranarayanan et al., 2018; Ma et al., 2019) learn the domain-invariant representations through adversarial learning.
Other methods are based on the label information.
Liang et al.
(2020) aligns domains implicitly by constraining the distribution of prediction.
Following Liang et al.
(2020), Liang et al.
(2021) introduces a label shifting strategy as a way to improve the accuracy of low confidence predictions.
For MUDA, previous methods eliminate the domain shift while aligning all source domains by constructing multiple source-target pairs, where (Peng et al., 2019; Zhu et al., 2019) are based on the discrepancy metric among different distributions and (Venkat et al., 2021;
Ren et al., 2022) are based on the adversarial learning.
Some methods (Venkat et al., 2021; Nguyen et al., 2021a; Li et al., 2022; Nguyen et al., 2021b) explore the relation between different classifiers and develop different agreements to achieve domain alignment.
Some (Wen et al., 2020; Ahmed et al., 2021) achieve an adaptive transfer by a weighted combination of source domains.
Current methods (Wang et al., 2020; Li et al., 2021) mainly construct connections between different domains to enable feature interaction and explore the relation of category information from different domains.
Our approach also constructs connections between different domains; however, it mainly transfers domain-specific information to achieve the fusion of all domains.
2. R OBUST L OSS F UNCTION UNDER N OISY L ABELS
A robust loss function is critical for UDA because the unlabeled target domain requires pseudo labels, which are often noisy.
Previous work (Ghosh et al., 2017) demonstrates that some loss functions such as Mean Absolute Error (MAE) are more robust to noisy labels than the commonly used loss functions such as Cross Entropy (CE) loss.
(Wang et al., 2019b) proposes the Symmetric
Cross Entropy (SCE) combining the Reverse Cross Entropy (RCE) with the CE loss.
Moreover, (Wang et al., 2019a) shows that directly adjusting the update process of the loss by the weight variance is effective.
Some methods (Venkat et al., 2021; Zhou et al., 2021) mentioned in Section 2.
focus on exploring a learning strategy to handle prediction values that have high or low confidence levels, respectively.
Differently, we design an adaptive loss function for the network to learn robust pseudo-label generation by self-correction.
PROBLEM SETTING
For MUDA task, there are N source distributions and one target distribution, which can be denoted as { p s j ( x, y ) } Nj =1 and { p t ( x, y ) } .
The labeled source domain images { X s j , Y s j } Nj =1 are obtained from the source distributions, where X s j = { x is j } |X sj | i =1 are the images in the source domain j and
Y s j = { y is j } |X sj | i =1 represents the corresponding ground-truth labels.
As for the unlabeled data in the target domain, there are target images X t = { x it } |X t | i =1 from target distribution.
In this paper, we uniformly set the number of samples in a batch as B , with b = BN +1 for every domain (including the target domain).
For a single sample x is j , the subscript s j represents the j -th source domain, while the superscript i indicates that it is the i -th sample in that domain.
PROPOSED METHOD 4. O VERALL F RAMEWORK
The general MUDA pipeline consists of two phases: 1) pre-training on the source domain; 2) training when transferring to the target domain.
The base structure of the model consists of a feature extractor
F ( · ) and a classifier C ( · ) .
Previous approaches mostly use different extractors or classifiers, unlike
Contrary Attention 𝐵×𝐷 𝐴𝑣𝑔()
∙U 𝑞
V
K
Q 𝐵×𝐷 𝐵×𝐷 (𝑁+1)×𝐷 (𝑁+1)×𝐵 𝐴′ 𝑠𝐴′ 𝑠𝐴′ 𝑡 ∙U 𝑘𝑣 Φ 𝑠Φ 𝑠Φ 𝑡 Φ 𝑛− Φ 𝑛
MLP (𝑁 + 1) × 𝐷 f 𝑠 1 f 𝑠f 𝑡 h
𝑠h 𝑠h 𝑡 𝑠𝑠𝑡 them, our approach expects to achieve a deep fusion of all domains, i.e., belonging to one domain, and therefore F and C are shared among domains.
In the pre-training phase, we use the source domain data { X s j , Y s j } Nj =1 for training to obtain the pre-trained F ( · ) and C ( · ) .
In the training phase for the target domain, we firstly feed the source domain data { X s j , Y s j } Nj =1 and target domain data
X t into F ( · ) to get the features f , and perform the feature fusion by CADM to get the fused feature h .
Then, we use the fused features of the target domain for the pseudo-label generation and feed the fused features of all domains into C ( · ) to get prediction values.
Finally, we compute the loss on the labels.
Since source training is general, training in this paper typically refers to target domain training.
4. C ONTRARY A TTENTION - BASED D OMAIN M ERGE M ODULE
The proposed CADM enables the passing of domain-specific information to achieve the deep fusion of all domains.
For each domain, we expect it can receive domain-specific information from other domains by the information transfer to move towards other domains as shown in Fig.
To achieve this goal, the transfer of semantically distinct domain-specific information should be encouraged.
It is because receiving semantically close domain-specific information primarily strengthens the original domain information and does not help eliminate the domain shift.
Instead, receiving different domain-specific information can contribute to the mixing of domain-specific information and ‘push’ the domain to other domains, thus achieving domain fusion.
As a result, we construct contrary attention to focus on semantic differences among domains.
The specific structure is shown in Fig.
Specifically, features f are first extracted from the samples via F , and then be mapped into the latent space to obtain query, key, and value feature:
[Equation]
where U q and U kv are learnable projection matrices, D is the feature dimension, f 1 s 1 is the feature obtained from the sample x 1 s 1 , and f s 1 is the domain prototype representing the domain information of s 1 .
Let g represent any domain, the domain prototype is obtained as follows:
[Equation]
Since the query of each domain and the key of all samples are obtained in Eq.
1,2, then we use the query feature q g in Q to compute the correlation with the samples of different domains:
[Equation]
We use the self-attention mechanism to model the correlation because of its powerful ability to capture dependencies, and A g is the obtained attention map.
The value in A g reflects the correlation between the domain g and the samples from other domains, while, as mentioned before, it is more meaningful to focus on semantically different domain-specific information.
Consequently, we ‘reverse’ the obtained A g to generate a new attention map, which we refer to as the contrary attention map:
[Equation]
where 1 is an all-one vector.
The A ′ g thus obtained can achieve the opposite effect of the previous
A g , where those features that are more different from the domain g in terms of domain-specific information are assigned higher weights.
Subsequently, we use A ′ g to perform the feature fusion for domain g , which allows the merging of information from features according to their degree of dissimilarity with domain g :
[Equation]
where a ′ g ˆ
g
i represents the contrary attention weight between the prototype of g and the i -th feature in ˆ g .
Φ g is obtained by weighting features from different domains and summing them according to their discrepancies from g .
Given its greater emphasis on the merging of knowledge with distinct styles, we call Φ g the domain style centroid with respect to g .
In addition, to guarantee the robustness of Φ g , we use exponential moving averages (EMA) to maintain it:
[Equation]
[Equation]
where n denotes the n -th mini-batch.
The recorded ¯Φ g will be used in the inference phase.
Finally, we integrate the domain style centroid ¯Φ g into the original features.
A fundamental MLP is used to simulate this process:
[Equation]
8) where h ig is the fused feature and ¯Φ g is integrated into every feature in g , thus creating the overall movement of domain.
Through the process described above, the original features achieve the merging of different domain-specific information by integrating the domain style centroid.
Meanwhile, to maintain a well-defined decision boundary during such domain fusion, we use a metric-based loss (Wen et al., 2016) to optimize the intra-class distance.
It allows samples from the same categories to be close to each other and shrink around the class center.
[Equation]
where y ig is the label of h ig and µ iy g ∈ R 1 × D denotes the y ig -th class center.
For the robustness of class center, we maintain a memory M to store features at the class level and update them batch by batch during training, which is widely applied in UDA (Saito et al., 2020).
The class-level features can be regarded as the class center, and we then apply them in the computation of L Cr .
Specifically, we traverse all the datasets to extract features and obtain different class centers before the training process.
In the target domain, we use the classification result as the category since there is no ground-truth label:
[Equation]
where p j ( x ) represents the confidence level of predicting x as the j -th class.
Then after each backpropagation step, we update the memory M with the following rule:
[Equation]
where k is the label of h and µ k is the k -th class center in M .
Under the supervision of L Cr , the features are able to maintain intra-class compactness in domain movement.
Overall, through CADM, we establish connections among domains and enable the deep fusion of different domains.
Eventually, with the L Cr , the model is able to reach a tradeoff between domain merging and a well-defined decision boundary.
4. A DAPTIVE AND R EVERSE C ROSS E NTROPY L OSS
General UDA methods use the K-means-like algorithm to generate pseudo labels for the entire unlabeled target dataset.
However, it has been observed that the pseudo labels generated in this way are often misassigned due to the lack of constraints, which introduces noise and impairs performance.
To address this issue, we creatively propose to assign gradients to the generation process of pseudo labels and constrain it with a well-designed rule, so that the network can optimize both the prediction values and pseudo labels simultaneously.
To begin, we expect to obtain the pseudo labels with gradient flow.
Since the dynamically updated memory M is introduced in Section 4.2, we can obtain both soft labels and hard labels by computing the distance between features and the class-level features in M in every batch:
[Equation]
where q ( y | x ) is the soft label and ˆ y is the hard label.
The soft label makes the gradient backpropagation possible.
Specifically, previous experimental observations (Venkat et al., 2021; Wang et al., 2019b) demonstrate that the trained classifier has important implications, which is key to preventing the model from collapsing in the unlabeled target domain.
Therefore, based on the output of classifier, we establish constraints on the pseudo label by a reverse cross entropy loss:
[Equation]
[Equation]
where k represents the k -th class, and p k ( x ) represents the classifier’s output at the k -th class.
As shown in the Eq.
14, unlike the cross entropy loss, which uses labels to supervise the output of the classifier, ℓ rce uses the output of the classifier to supervise the generated soft labels.
Then the gradient can be back-propagated through the soft label, thus preventing the accumulation of errors from pseudo-label generation.
However, it is unfair to apply the same rule to those pseudo labels that tend to be accurate.
An ideal situation is that the loss can be adaptively adjusted to correspond to different pseudo labels.
Therefore, we propose the adaptive factor a ( x ) to achieve the adaptive adjustment of ℓ rce , which is denoted as:
[Equation]
where ˆ y is the generated hard label of the target domain sample x .
We consider the classifier’s output other than the class
ˆ y as a distribution and the probability of j -th class is denoted as δ j ( x, ˆ y ) in Eq.15.
The entropy of this distribution then serves as the adaptive factor a ( x ) .
It is because that pseudo label essentially arises from the feature extracted by the network, so its accuracy is closely related to the ability of the network to distinguish this sample.
Liang et al.
(2021) demonstrates that when the model has a good distinguishing ability for one sample, the output of classifier should be close to the one-hot vector.
Consequently, the value of a ( x ) , in this case, is larger than the case of poor discriminatory ability.
Based on a ( x ) and ℓ rce , we design an adaptive and reverse cross entropy loss:
[Equation]
The exponential form and τ are used to amplify its effect.
Since L AR − CE aims to focus on the optimization of pseudo label, we freeze the gradient of the prediction values in Eq.
The backpropagation of L AR − CE can eventually enable the optimization of pseudo-label generation.
Furthermore, the cross entropy loss is denoted as follows:
[Equation]
where y i is the ground-truth label for source domain sample and is the hard pseudo label ˆ y i obtained by Eq. 13 when x i is in the target domain.
Ultimately, the loss function for the entire framework can be represented as follows:
[Equation]
where L CE is the normal cross entropy loss, L Cr is for feature fusion, L AR − CE is used for noise tolerant learning, and the latter one item acts only on the target domain.
E XPERIMENTS 5. D ATASETS AND I MPLEMENTATION
Office-31 (Saenko et al., 2010) is a fundamental domain adaptation dataset comprised of three separate domains: Amazon (A), Webcam (W), and DSLR (D).
Office-Caltech (Gong et al., 2012) has an additional domain Caltech-256 (C) than Office-31.
Office-Home (Venkateswara et al., 2017) is a medium-size dataset which contains four domains: Art (Ar), Clipart (Cl), Product (Pr), and RealWorld (Rw).
DomainNet (Peng et al., 2019) is the largest dataset available for MUDA.
It containsdifferent domains: Clipart (Clp), Infograph (Inf), Painting (Pnt), Quickdraw (Qdr), Real (Rel), and
Sketch (Skt).
For a comparison with our method, we introduce the SoTA methods on MUDA, including DCTN (Xu et al., 2018),
M 3 SDA (Peng et al., 2019), MFSAN (Zhu et al., 2019), MDDA (Zhao et al., 2020), DARN
(Wen et al., 2020), Ltc-MSDA (Wang et al., 2020), SHOT++ (Liang et al., 2021),
SImpAl (Venkat et al., 2021), T-SVDNet (Li et al., 2021), CMSS (Ahmed et al., 2021), DECISION (Ahmed et al., 2021), DAEL (Zhou et al., 2021), PTMDA (Ren et al., 2022) and DINE (Liang et al., 2022).
For Office-31, Office-Caltech and Office-Home, we use an Imagenet (Deng et al., 2009) pre-trained ResNet-50 (He et al., 2016) as the feature extractor.
As for the DomainNet, we use
ResNet-101 instead.
In addition, a bottleneck layer (consists of two fully-connected layers) is used as the classifier.
We use the Adam optimizer with a learning rate of 1 × 10 − 6 and a weight decay of 5 × 10 − 4 .
We set the learning rate of extractor to ten times that of the classifier.
The hyperparameters α and β are set to 0.05 and 0.005, respectively.
All the framework is based on the Pytorch (Paszke et al., 2019).
5. R ESULTS
We compare the ADNT with different approaches on four datasets, and the results are shown in the following tables.
Note that the one pointed by the arrow is the target domain and the others are the source domains.
As shown in Table 1, the experiments on Office-31 show optimal results.
In particular, it is second only to PTMDA in the ResNet50-based approach and achieves the best results in all scenarios, except for transferring W and D to A. On the Office-Caltech dataset, our method not only outperforms all current approaches, but also achieves an impressive 100% accuracy on the task of transferring to W and to D.
In addition, our method achieves the best results on all tasks except transfer to A, and has a significant advantage over the ResNet-101-based method.
As shown in Table 3, on the medium-sized Office-Home dataset, our proposed ADNT is the best performer and has a 0 .
9% lead over the best ResNet-50-based method.
Furthermore, compared with the ResNet-101-based methods (DINE and SImpAl 101 ), our approach achieves leading performance with fewer parameters and a simpler model structure.
As shown in Table 4, ADNT also demonstrates its strength on the largest and most challenging dataset.
The total result outperforms the current optimal method by 2 . 3% , which is highly impressive on DomainNet.
Overall, our approach not
[Table]
[Table] [Table]
only achieves the best results
[Table]
but also has better performance on large datasets with more domains, especially DomainNet.
For tasks with degraded performance, in addition to gaps in model structure (compared to ResNet-101), they mainly appear when transferring to some less stylized domains (Amazon, Clipart, Quickdraw).
Further analysis is presented in the Appendix
[Table] [Table]
5. A BLATION S TUDY
To verify the effectiveness of the proposed components
[Table]
we conduct ablation experiments on the
Office-Home dataset, as shown in Table 5.
In order to determine whether the attention-driven module actually achieves domain fusion, we conduct experiments with only the domain fusion without
L Cr , which is shown as ‘+ CADM (w/o L Cr )’ in the table.
Additionally, due to the fact that our proposed contrary attention differs from general attention by paying attention to the fusion of domainspecific information, a comparison with general attention can evaluate the viability of our strategy.
[Table]
[Figure]
Therefore, we replace the contrary attention map in the proposed CADM with a general attention map,
i
[Table]
e., without Eq. 5.
The result of general attention is shown as ‘+ ADM’ in Table 5.
The significant performance improvement powerfully demonstrates the effectiveness of our proposed CADM.
We also use t-sne (Van der Maaten & Hinton, 2008) to visualize the results with and without CADM, as shown in Fig.
We can observe that with the designed CADM, the different distributions are mixed with each other, achieving a higher degree of fusion.
Meanwhile, the decision boundaries of different categories are more distinct.
Table 5 verifies the effectiveness of our AR-CE loss.
Additionally, we conduct experiments using only reverse cross entropy loss, denoted as + L R − CE .
As shown in the Table 5, L R − CE brings the performance improvement, and the adaptive factor is able to amplify this improvement.
This demonstrates the positive and effective role of adaptive factor.
Due to the space limit, additional analytical experiments, including the visualization of the training process and analysis of hyperparameters, can be found in the appendix.
C ONCLUSION
This paper proposes the ADNT, a framework that combines the domain fusion module and robust loss function under noisy labels.
Firstly, We construct a contrary attention-based domain merge module, which can achieve a mixture of different domain-specific information and thus fuse features from different domains.
In addition, we design a robust loss function to avoid the error accumulation of pseudo-label generation.
Substantial experiments on four datasets have shown the superior performance of ADNT, which strongly demonstrates the effectiveness of our approach.
A A PPENDIX A. A DDITIONAL E XPERIMENTS ABOUT CADM
Our proposed CADM aims to achieve the domain fusion through the transfer of domain-specific information, where the coefficients in the Contrary Attention Map
A ′ have an important role to focus on the semantically different domain-specific information.
To verify the action of contrary attention, we perform the visualization by Grad-CAM.
Specifically, we used Grad-CAM to visualize the weights in A ′ , as the column ‘Painting’ in Fig. 9, reflecting the attention of other domains to that sample in Painting domain.
Since our approach emphasizes attention to domain-specific information from other domains, it is reasonable to focus on texture and background, etc., rather than content.
As shown in Fig 9, the designed contrary attention is able to focus on domain-specific information, such as texture, color, background, and other domain characteristics ( e.g. , handwritten text in Sketch), rather than content.
This makes the domain fusion possible.
Meanwhile, the visualization results on domain Quickdraw show that since this domain is composed of simple lines, there is not enough information related to texture, background, etc., and the attention can be limited.
This also explains why our approach does not achieve the best performance on some tasks, except for the reason of different model structure (ResNet-50 and ResNet-101).
This is because domains such as A in Office-31 and Qdr in DomainNet lack sufficient domain information to support contrary attention.
Nevertheless, our ADNT still has a top performance on these domains, which reflects the robustness of our method.
[Figure]
Furthermore, the weight value in A ′ actually reflects the correlation among domains (Here, we use the domain to refer to the features in the domain).
Therefore, the relation of different domains can be estimated by A ′ .
In this paper, we expect the following domain relationship: Initially, they should have very different styles, and the weights in A ′ will be quite uncertain.
Then, as the optimization of the model parameters, different domains will gradually be mixed.
Finally, all domains should have a consistent style and maintain a balanced relationship.
Therefore, the merging of all domains together means that there should be little difference among the final correlation coefficients in A ′ .
So we calculate the standard deviation of A ′ during the training.
As shown in the Fig.
5, during the training process, the standard deviation gradually becomes smaller and eventually tends to be stable, which indicates that the features from different domains are distributed in the space with a uniform state.
[Figure]
t a n d a
[Figure]
D e v i a t i o n
[Figure]
t a n d a r d
D e v i a t i o n
[Figure] [Figure]
However, the above results alone do not demonstrate that features from different domains are deeply fused because the previous effect in Fig.
5 can also be achieved if features are sufficiently separated from each other.
Consequently, we map the fused features h into the low-dimensional space and visualize the distribution during the training process, divided by different domains and categories.
As shown in Fig.
6, at the start of training, the fused features are distributed haphazardly in space, and the network does not have capacity to process them.
Then, different domains gradually move toward each other and merge together as the updating of model parameters.
Finally, the features of the same category become more compact while preserving domain fusion.
This powerfully demonstrates that through CADM, different domains embrace each other and achieve deep fusion.
We make the model learn how to handle the fused features and show its excellent results.
A. A DDITIONAL E XPERIMENTS ABOUT L AR − CE
First, as shown in the Fig.
7, we show the specific structure of the pseudo-label generation and the gradient back-propagation.
Through L AR − CE , the soft label generation can be optimized during gradient descent to weaken the influence from the noise in the pseudo-labels.
Furthermore, the accuracy of the generated pseudo labels can reflect the effectiveness and stability of the pseudo-label generation.
Therefore, we compare the average accuracy of pseudo labels generated with and without AR-CE loss on Office-31, Office-Caltech and Office-Home, which is shown in
Fig.
Specifically, we calculate the mean accuracy of the final five epochs during training.
It can be seen that after the introduction of the AR-CE loss, the model can derive noisy labels through self-correction and achieve the high accuracy of pseudo labels.
Finally, we have conducted additional experiments to vertify the effect of adaptive factors.
Specifically, we counted 1000 samples with correct and wrong pseudo labels separately and calculated their distribution entropy as described in Sec. 4.3.
The results are shown in Fig.
in the appendix.
It can be seen that the distribution entropy of the correct samples has values around 4 , while the incorrect samples have smaller values, mostly around 2 . 5 .
This difference proves the feasibility of
𝓛 𝑪𝑬 𝓛 𝑨𝑹−𝑪𝑬
Classifier h 𝒕
Memory
Softmax soft-label pseudo label
Class Score . . .
𝒅(𝒙,𝒚) 𝐚𝐫𝐠𝐦𝐚𝐱 𝒅(𝒙,𝒚)
Forward Propagation
Backward Propagation of 𝓛 𝑨𝑹−𝑪𝑬 Backward Propagation of 𝓛 𝑪𝑬
Compute distance with memory our proposed adaptive factor.
Although it is difficult to be completely correct because the pseudo labels depend on the generation, our method can significantly achieve the correction of pseudo-label generation.
As a result, the impact of noise on the model performance is reduced.
→D →W →A
[Figure]
cc u r ac y ( % )
[Figure]
[Figure] [Figure]
w/
[Figure]
AR
[Figure]
A cc
[Figure]
4050607080→Ar
w/o AR-CE loss w/ AR-CE loss
A cc u r ac y ( % )
[Figure]
A. M ODEL A RCHITECTURE
Since the feature extractor F () in our method is the pre-trained ResNet-50, in order to explore the performance of our method using a larger network structure, we conducted experiments on Office31, Office-Caltech, and Office-Home using ResNet-101 as the feature extractor, as shown in Table 6, 7, 8.
The results show that the performance of the network does improve somewhat, however, there is a drop in some tasks (especially some tasks that have close to 100% accuracy with ResNet-50).
On the Office-Home dataset, there is a significant improvement.
This indicates that data size is an important factor limiting the application of large model to our approach.
Moreover, since we simply perform model replacement in this experiment, there is further room to optimize the way the larger model is combined with our proposed ADNT.
[Table]
[Table] [Table]
P ARAMETER
[Table]
Exponential moving average (EMA) is used in the experiments to maintain the domain style centers and class-level features, so the selection of hyperparameters α and β requires additional analysis and verification.
Therefore, we perform an experimental comparison of their values and present them in
Fig.
[Figure]
As can be seen, the value of the hyperparameter α is somewhat greater and the β is taken at an overall smaller value.
This indicates that the domain style centroid is updated more vigorously and provides a gradient sufficient for the backpropagation of the network.
Memory M , on the other hand, requires a slower update to maintain network robustness because it involves the pseudo-label generation.