H OMEOMORPHISM A LIGNMENT IN T WO S PACES FOR U NSUPERVISED D OMAIN A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
The existing unsupervised domain adaptation methods always align the features from the source and target domains explicitly or implicitly in a common space, i.e., the domain invariant space.
Explicit distribution matching always ignores the discriminability of the learned features, while implicit distribution matching such as self-supervised learning suffers from the pseudo-label noises.
It is difficult to find a common space which maintains discriminative structure of the source and target domain data when aligning the data distributions.
We propose a novel approach dubbed as HomeomorphisM Alignment (HMA) so that the source and target features can be aligned in two different spaces.
Specifically, an invertible neural network based homeomorphism is constructed.
Distribution matching method is used as a sewing up tool for connecting homeomorphism mapping between the source and target feature spaces.
Theoretically, we show this mapping can preserve data topological structure, i.e., the samples in the same cluster are still in the same projected cluster.
Based on this property, we adapt the model by the cross entropy of transformed and original source features and prediction consistency between target features and transformed target features.
Extensive experiments demonstrate that our method can achieve the state-of-the-art results.
I NTRODUCTION
Deep learning has revolutionized the progress of machine learning and computer vision ( e.g. , object recognition (He et al., 2016)).
However, this advance relies heavily on a large quantity of manually labeled data, which could be prohibitively expensive or even impossible to collect in many scenarios.
To solve this scalability issue, there is a strong motivation to exploit pre-existing labeled data ( i.e. , the source domain) for a target domain without any label annotation.
Due to the domain shift challenge (Pan & Yang, 2009), a model pretrained on a source domain often suffers from drastic performance degradation when directly applied on a target domain.
This gives rise to the research attention of Unsupervised Domain Adaptation (UDA).
Existing UDA methods can be roughly divided into two categories.
One is based on distribution alignment (Long et al., 2015;
Kang et al., 2019; Ganin et al., 2016; Long et al., 2018) which minimizes domain discrepancy by aligning the distributions between two domains.
They usually match two different distributions to a single distribution.
Doing so could distort the original structural information, potentially hurting the final model generalization (Chen et al., 2019;
Ge et al., 2022; Tang et al., 2020).
Another is self-supervised learning (French et al., 2018; Liang et al., 2021a; Sun et al., 2019) which also learns a common space by using pseudo labels or other supervision information.
The self-supervised learning faces the same situation.
Since the label noise is inevitable, it is difficult to obtain such a common feature space while keeping discriminative structure.
As shown in
Fig.
1(a), adapting model in a common space cannot guarantee classification performance.
So it is a natural thought to align the target and source data in two spaces.
There do not exit much works in this route.
CyCADA (Hoffman et al., 2018) uses two different networks to transform the image in the source domain to the target domain and vice versa, which are trained by cycleconsistent constraint and adversarial loss.
However, the learned two networks are not strictly inverse mappings, the transformed images are not in control by category.
It causes that data topological structure is not well preserved.
Homeomorphism is a concept from Topology (Munkres, 2000).
If a bijection satisfies the definition of homeomorphism, i.e., one-to-one correspondence and continuous,
[Figure]
theoretically, we prove that the data topological structure is well preserved in the projected space, i.e., the samples in the same cluster is still in the same projected cluster.
As shown in Fig.1(b), by homeomorphism mapping, the adapted model works well in both source and target feature spaces.
Based on the above analysis, we propose a novel unsupervised domain adaptation method, called
HomeomorphisM Alignment (HMA).
Our method consists of three parts.
The first is the construction of homeomorphism mapping to connect the source and target feature spaces.
Fortunately, the recently proposed Invertible Neural Network (INN) (Kingma & Dhariwal, 2018) can naturally help us find a pair of mutually invertible functions through the forward process and invertible process of the network, which also greatly saves memory space, so it is suitable for us to construct a homeomorphism.
Then, distribution matching method is used as a sewing up tool.
At both ends of the homeomorphism mapping, we require that the transformed features aligned with the features of the corresponding domain.
Obviously, if we can stitch them by category, the homeomorphic mapping will be able to better implement the transformation between two spaces while keeping topological structure.
In the end, the pretrained source model is retrained in the source and target feature spaces by the preserved topological structure, which utilizes that the transformed source features have same labels as original source features and the same prediction should exist in target features and transformed target features.
Our contribution can be summarized as follows: (1) We theoretically proved that homeomorphism mapping can guarantee topological structure of the mapped data, and we also show that INN is actually a homeomorphism.
This is an important property that is ignored in current researches.
Compared with the previous methods which try to extract a common space to align the source and target features, we analyzed its difficulty and proposed a novel UDA method based on homeomorphism, which is the first time to consider the UDA problem from the viewpoint of Topology and the alignment is done in two spaces.
(3) Extensive experiments demonstrate the effectiveness of our method.
Ablation analysis also demonstrates the superiority of using homeomorphism alignment.
R ELATED WORK
Unsupervised Domain Adaptation.
UDA aims to improve the generalization ability of a model on an unlabeled target domain by leveraging the labeled source domain.
Existing methods can be roughly divided into two categories.
The first category adopts the idea of distribution alignment that trains the model by minimizing the source error and the discrepancy between source and target domains concurrently.
There exist two main strategies: statistic moment matching and adversarial learning .
The methods based on statistic moment matching minimize the statistic discrepancy to align the distributions between two domains.
DAN (Long et al., 2015) proposes multiple kernels Maximum Mean Discrepancy (MK-MMD) for adapting marginal distribution between two domains.
CORAL (Sun et al., 2017) minimizes the domain shift by aligning the second-order statistics of the source and target distributions.
CAN (Kang et al., 2019) proposes Constrastive Domain Discrepancy (CDD) to minimize the intra-class discrepancy and maxmize the inter-class discrepancy, which aligns the conditional distributions between two domains.
The methods based on adversarial learning are inspired by GAN (Goodfellow et al., 2014), which plays a minimax game between feature extractor and discriminator to learn domain invariant features.
DANN (Ganin et al., 2016) directly uses the features of the source and target domains from the same feature extractor as the input of the discriminator for domain classification.
ADDA (Tzeng et al., 2017) uses two different feature extractors for source and target domains respectively and uses a discriminator to identify the domain labels of the features.
Both of discriminators used in DANN and ADDA only focus on the domain information of features and ignore the category information, which only achieves the marginal distribution alignment.
Therefore, CDAN (Long et al., 2018) takes the prediction of classifier and features as the input of the discriminator, which makes conditional distribution alignment between two domains.
The second category of methods regards domain adaptation as a self-supervised learning problem, which tries to use more accurate pesudo-labels or supervision information to retrain the source model.
In fact, in the training process, such methods are also trying to find a common space to implicitly align the source and target features such that the source and target domain features projected by the feature extractor have better discriminability.
For example, SE (French et al., 2018) uses the mean teacher framework, which consists of a student network and a teacher network.
For the update of the student network, it uses the cross-entropy of the source samples and the consistency constraints of the target samples.
While the teacher network is updated by exponential moving average of student network.
ATDOC (Liang et al., 2021a) assigns pseudo-label for each target samples by employing a memory mechanism.
ssUDA (Sun et al., 2019) performs self-supervised tasks, such as rotation, flip and patch location predictions, to improve the generalization of the model.
A NALYSIS OF D ISTRIBUTION A LIGNMENT
Problem statements.
In the UDA setting, there is a source domain D s = { ( x si , y si ) } n s i =1 consisted of n s labeled samples and a target domain D t
= { ( x ti ) } n t i =1 consisted of n t unlabeled samples.
The source domain and target domain share the same label space { 1 , 2 , · · · , K } , but with different distributions.
Suppose the source model Γ s , pretrained by labeled source data, is composed of a feature extractor F and a classifier C .
The goal of UDA is to adapt the source model such that it works well in the unlabeled target domain.
Ideally, given the ground-truth labels, the trained model can work very well in the source and target domains.
To verify this, we conduct experiments on the Office-31 dataset.
Based on the real labels in both source and target domains, supervised training is performed.
100% accuracy on the source and target domains can be achieved.
Now, we will check the adapted model based on distribution alignment approaches.
See Appendix for algorithm details.
Can previous distribution alignment methods really achieve 100% accuracy given groundtruth labels?
As mentioned earlier, current domain adaptation methods usually adapt the source domain model to the target domain through two technical routes: explicit distribution alignment based on statistic moment matching or adversarial learning, and implicit alignment based self-supervised learning.
Essentially, these alignment methods implement domain adaptation by projecting source and target domain samples into a common space, i.e., the domain invariant space.
We do an experiment on the top-4 challenging adaptation tasks of Office-31 dataset which uses the ground-truth target domain labels to perform different kinds of distribution alignments to show what happens.
The results are shown in Table 1.
The first row shows the results of CAN (Kang et al., 2019) which aligns feature distributions based on the real target labels.
The second row in Table 1 shows the performance based on adversarial learning method CDAN (Long et al., 2018).
From these two rows, it can be observed that the explicit distribution alignment methods cannot achieve 100% accuracy which means that the discriminative data structure is not well preserved when they align the source and target domain samples in a common space.
The third row shows the results of self-supervised
[Table]
learning approach using the real target sample labels.
It works very well which means the feature extractor can find the domain invariant space while keeping data discriminative structure.
But the label noisy cannot be eliminated in the actual situation.
Can double mapping achieve 100% accuracy?
Since it is difficult to find a feature extractor to obtain a common space while keeping data discriminative structure, naturally whether the source and target domain samples can be aligned in two feature spaces in a way of double mapping, so as to achieve accurate classification for target domain samples.
The fourth row in Table 1 shows the performance that uses two different networks to map the source features to the target feature space and vice versa.
We use the real target sample labels to train these two networks and align the conditional distributions between the transformed features and original features.
This bijection composed of these two networks is not a homeomorphism.
The data topological structure can not be preserved in the process of bidirectional projections between the source and target domains.
Although we know all the real labels of the target domain, we still cannot achieve 100% accuracy.
Homeomorphism alignment can achieve 100% accuracy.
A homeomorphism, also called a continuous transformation, is a one-to-one correspondence mapping between the points in two topological spaces that is continuous in both directions.
More precisely refer to (Munkres, 2000).
Let M and N be two topological spaces, and g : M → N be a bijection.
If both the function g and its inverse function g − 1 : N → M are continuous, then g is called a homeomorphism .
That is to say that a homeomorphism is a bijective correspondence g : M → N such that g ( U ) is open if and only if U is open.
By the definition above, it is easy to say that g is a homeomorphism if and only if g −is a homeomorphism.
Based on this definition, the following theorem can be easily derived.
Theorem 1.
The set boundary corresponds to the set boundary by homeomorphism.
More precisely, let ( M, d M ) and ( N, d N ) be two metric spaces where d M and d N are the metrics on
M, N respectively.
Suppose there is a homeomorphism g : M → N , and A is an open subset in ( M, d M ) , we have that its image B :=
g ( A ) is an open subset in ( N, d N ) , and
[Equation]
where ∂ means the boundary.
As shown in Theorem 1, data topological structure is preserved by homeomorphism mapping, i.e., the samples in the same cluster are still in the same projected cluster.
Fortunately, in the community of machine learning, there exists a network satisfying homeomorphism definition which is called
Invertible Neural Network (INN) (Kingma & Dhariwal, 2018).
It is easy to validate that INN satisfies the following theorem.
Theorem 2.
Invertible Neural Network is a homeomorphism .
As shown in the last row of Table 1, we use INN to connect two domains at the feature level.
With the real target sample labels, the homeomorphism alignment can achieve 100% accuracy because of topological structure preserving.
M ETHOD OF H OMEOMORPHISM A LIGNMENT
Overview: Based on the analysis of distribution alignment, as shown in Fig. 2, the proposed HomeomorphisM Alignment (HMA) method consists of three parts.
The first part is about homeomorphism mapping construction based on INN.
Another part is sewing up which uses the homeomorphism mapping to connect the source and target feature spaces.
The final part is retraining the pretrained source model in the source and target feature spaces by using the property of homeomorphism.
Source data
Target data
Forward
Inverse
Split Split
Concatenation Concatenation
Addition
Reduction 𝒕 𝒔
Invertible Neural Network (INN)
INN
INN …
Source feature space
Target feature space 𝒇 𝒔 , 𝒇 𝒕𝟐𝒔 𝒇 𝒕 , 𝒇 𝒔𝟐𝒕 Classifier 𝑳𝒐𝒔𝒔 𝑺 + 𝑳𝒐𝒔𝒔 𝑻
𝒇
𝒔𝟐𝒕 : Transformed source feature 𝒇 𝒕𝟐𝒔 : Transformed target feature 𝒇 𝒔 : Source feature 𝒇 𝒕 : Target feature
Split feature mFeature extractor
Sewing up 𝑫𝑴(𝒇 𝒕 ,𝒇 𝒔𝟐𝒕 )
Sewing up 𝑫𝑴(𝒇 𝒔 ,𝒇 𝒕𝟐𝒔 ) (a) (b)
[Figure]
Homeomorphism
4. H OMEOMORPHISM IMPLEMENTED BY AN INVERTIBLE NEURAL NETWORK (INN)
In each iteration, we randomly sample a batch of source and target samples.
We use ResNet (He et al., 2016) as the feature extractor F to map a sample x to the feature space: f s/t = F ( x s/t ) where s/t represents the source and target domain respectively.
Due to the distribution discrepancy between the source and target domains, we consider that the source and target features reside on two different spaces (manifolds) respectively.
A homeomorphism g consists of m blocks of INN.
m is a hyperparameter discussed in Appendix.
In each block, we use an affine network to implement the INN (Dinh et al., 2017), as shown in Fig.
For the i -th block, we denote the input µ i 1:2 d with 2 d the dimension.
In the forward process, we transform from the source feature space f
s to the target feature space f t .
Specifically, we split evenly µ i 1:2 d to two parts [ µ i 1: d , µ id +1:2 d ] , and further transform them with two respective linear neural networks s ( · ) , t ( · ) .
The output of i -th block µ i +11:2 d is then obtained with residual as follows:
[Equation]
The output µ i +11:2 d will be set as the input of the next block.
In the inverse projection process, we map µ i +11:2 d to µ i 1:2 d in the opposite away around.
According to equation 1, we can get the following equation:
[Equation]
We similarly split µ i +11:
d into two parts [ µ i +11: d , µ i +1 d +1:2 d ] and follow equation 2 to get the original input µ i 1:2 d .
Obviously equation 1 and equation 2 are inverse functions of each other.
We denote the forward process of the m INNs as the function g , while the inverse process as g − 1 .
Hence, the function g is a bijection.
Since the functions s ( · ) , t ( · ) are implemented by two linear connected neural networks, they are continuous; This means both g and g − 1 are continuous, too.
According to the definition of homeomorphism, this INN is a homeomorphism.
4. S EWING UP
Now we will sew up homeomorphism mapping g to the source and target feature spaces such that the corresponding classes are aligned.
Suppose the transformed feature f s 2 t
=
g
( f s ) according to the source feature f s and the transformed feature f t 2 s = g − 1 ( f t ) according to target feature f t .
According to the property of homeomorphism mapping g , f s = g − 1 ( f s 2 t ) and f t =
g ( f t 2 s ) .
To guarantee that the transformed features are the correct places, i.e., they are aligned with the corresponding classes, the distribution matching method is used for sewing up.
The loss function is defined as follows,
[Equation]
where DM ( · , · ) refer to the existing distribution matching methods, such as DAN (Long et al., 2015), CAN (Kang et al., 2019), DANN (Ganin et al., 2016) and CDAN (Long et al., 2018), etc.
It should be noted that only using marginal distribution matching method, such as DAN and DANN, cannot achieve good results, because these methods only focus on overall distribution alignment instead of class-wise alignment.
Although our homeomorphism mapping g can ensure that the transformed features retain topological structure, if not stitched correctly according to the corresponding category, it will lead to that the source domain and target domain do not achieve the discriminate feature lossless transformation.
So class conditional distribution matching is a more better choice.
This is also confirmed in the experiment section.
4. T RAINING M ODEL
A model often suffers performance degradation from the domain shift.
To address this problem, we leverage the homeomorphic property.
Specifically, after training the homeomorphism mapping, we perform distribution alignment between f s and f t 2 s , and between f t and f s 2 t .
As proved by
Theorem 1, f s has the same structural information as f s 2 t .
Concretely, for a specific labeled source sample x , the corresponding feature f s and f s 2 t share the same label.
The following loss function is applied to the supervised training of feature extractor F and classifier C ,
[Equation]
where y s is the corresponding label of the source sample x s , and L ce ( · , · ) denotes the cross entropy function.
In particular, the term L ce ( C ( f s ) , y s ) focuses on the classification of the source domain, whilst L ce
( C ( f s 2 t ) , y s ) is concerned with the classification of the target domain since f s 2 t and f t have been aligned.
Considering that our homeomorphism preserves the structure across the mapping and no label information in the target domain, unsupervised consistency constraint is a natural strategy for optimization.
Formally, for an unlabeled target sample x t , we formulate the consistency constraint on the predictions between f t and f t 2 s as:
[Equation]
where L C ( · , · ) is a consistency constraint such as L 1 -Norm and L 2 -Norm.
In practice, we found
L 2 -Norm suffices.
By combining equation 4 and equation 5, the overall loss is defined as follows,
[Equation]
Remark.
The model is trained in the source and target feature spaces.
Compared with the alignment based UDA methods in a common space, our method overcomes the difficulty of projecting the source and target domain samples into a common feature space using a feature extraction network while keeping data discriminative structure.
When the distributions between the feature spaces are not aligned, homeomorphism can transform their features from each other to further align the transformed features.
The alignment in two spaces helps keeping more discriminative information.
Compared with the self-supervised learning approach, which can not guarantee the accuracy of pseudo-labels, the transformed source features in the target domain have the completely correct label information and topological structure from the source domain.
For the unlabeled target features, the consistency constrain also utilizes the transformed target features whose topological structure is guaranteed by homeomorphism mapping.
All in all, our method HMA is divided into two parts.
At the training phase, in each iteration, the first step is to train an INN based homeomorphism mapping.
Next step is retraining the source model.
At the inference phase, both the target features f t and the transformed target features f t 2 s can be used to get the predictions.
In our method, we use an ensemble strategy which averages these two types of predictions as the final prediction.
[Table] [Table]
[Table]
E XPERIMENTS 5. E XPERIMENTAL S ETUP
Datasets: In our experiments, the three standard datasets are used.
Office-31 (Saenko et al., 2010)is a popular benchmark, which contains 4110 images total within 31 office environment objects from 3 domains: Amazon (A), Webcam (W), Dslr(D).
Office-Home (Venkateswara et al.
, 2017)is a more challenging dataset which contains 15588 images within 65 classes from 4 domains: Artistic images (A), Clip-Art images (C), Product images (P) and RealWorld images (R).
Visda-17 (Peng et al.
2017)is a widely used benchmark for domain adaptation which focuses on the 12-class synthesis-toreal object classification task.
The source domain contains 152,397 synthetic images and the target domain has 55,388 real object images.
[Table]
Implementation details: Our experiment is performed on Pytorch platform.
Each task is runtimes to enhance the robustness of the results.
The same backbone network is selected as other compared methods for fair comparison.
Specifically, the Resnet-50 is selected as the backbone on Office-31 and Office-Home
[Table]
and the Resnet-101 is selected as the backbone on Visda-17 .
It is worth noting that the output dimension of the classifier in the original backbone is replaced by the number of categories to suit our task.
The SGD optimizer is chosen to update the network and the
CosineAnnealingLR (Loshchilov & Hutter, 2016) is used to update the learning rate of the SGD optimizer.
Competitors: To verify the effectiveness of our method, we compare it with the following three types of state-of-the-art methods.
The first category methods are based on distribution alignment, such as statistic moment matching methods DAN
(Long et al., 2015), CAN (Kang et al., 2019), TSA (Li et al., 2021c).
The second category methods are based on adversarial learning including DANN (Ganin et al., 2016), CDAN (Long et al., 2018), MDD+IA (Jiang et al., 2020),
DADA (Tang & Jia, 2020), BCDM (Li et al., 2021a), CLS (Liu et al., 2021), ILA (Sharma et al., 2021), MetaAlign (Wei et al., 2021), DWL (Xiao & Zhang, 2021), DALN (Chen et al., 2022).
The third category methods are based on self-supervised learning.
They are ALDA (Chen et al., 2020),
ATDOC (Liang et al., 2021a), CaCo (Huang et al., 2022), SUDA (Zhang et al., 2022).
5. C OMPARISONS TO S TATE - OF - THE -A RT
The performance comparison with other state-of-the-art methods on Office-31, Office-home and
Visda-17 are shown in Table 2, Table 3 and Table 4 respectively.
The methods HMA(DANN) and
HMA(CDAN) mean the sewing up tool is the distribution matching method based on adversarial learning where DANN and CDAN focus on marginal distribution alignment and conditional distribution alignment respectively.
While HMA(DAN) and HMA(CAN) apply the statistic moment matching methods as the sewing up tool, where DAN and CAN focus on marginal distribution alignment and conditional distribution alignment respectively.
It can be observed that HMA(CAN) yields the best average performance on both three datasets.
This also confirms our previous analysis.
Different sewing up methods will affect the final performance.
In general, conditional distribution alignment is better than marginal distribution alignment on the two kinds of methods based on adversarial learning strategy and statistic moment matching strategy because conditional distribution alignment can stitch homeomorphism mapping with two spaces by category.
Furthermore, for condition distribution matching method, statistic moment matching strategy is better than adversarial learning strategy.
The reason is that statistic moment matching strategy stitches homeomorphism mapping with two feature spaces explicitly by category.
Interestingly, the alignment method CAN published in 2019 still achieves SOTA results.
It can be seen that alignment
[Table] [Table]
by category is very important for extracting domain invariant features.
Our method HMA(CAN) further boosts the CAN performance
[Table]
because we realize the difficulty of extracting a domain invariant space and we do alignment in the two spaces by homeomorphism mapping.
5. A BLATION A NALYSIS AND D ISCUSSION
Homeomorphism map is better than double mapping.
As mentioned in Section 3, there does not exist many methods which do alignments in two feature spaces based on bijection.
In this experiment, we will apply the double mapping method in Section 3 to top-4 challenging tasks on Officedataset, the difference between INN and double mapping method is mainly reflected in topological structure maintenance.
The results are shown in Table 5.
It is obvious that homeomorphism mapping is superior to the normal double mapping method in accuracy and model parameters.
For the model size, the INN based homeomorphism mapping uses almost half of the parameters compared to using two neural networks.
Ablation study.
To show the effectiveness of alignment in two spaces, we conduct an experiment on top-4 challenging tasks of Office-31 dataset.
The results are shown in Table 6.
The method CAN is considered as the baseline.
INN(CAN) means just sewing up homeomorphism mapping to the two feature spaces and the source model is retrained based on the source labels.
INN(CAN)+S2T means the transformed source features are used to learn the model in the target feature space compared with INN(CAN).
HMA(CAN) uses all features in two spaces.
From Table 6, we can find that simply using INN can achieve similar performance as the feature distribution alignment method CAN.
As shown in the third row in Table 6, by transferring the source features to the target feature space, the performance is greatly improved.
For this case, the data topological structure and label information from source domain can be correctly transformed to the target domain, which allows that the learned model works well in the target domain.
The last row in Table 6 shows that the performance can be further improved if both transformed features are used.
Because our homeomorphism mapping keeps the corresponding relationship by category, with the help of supervision information in the source feature spaces, the generalization performance of the model in the two domains is improved.
C ONCLUSION
In this paper, we have proposed a new unsupervised domain adaptation method, termed as HomeomorphisM Alignment in two spaces (HMA).
By analyzing previous alignment base methods, we argue that it is difficult to find a common space or domain invariant space to adapt the pretrained source model.
So the alignment is performed in two spaces.
The extracted source and target features can be further transformed respectively by a homeomorphism mapping so that they can be aligned semantically.
Our method consists of three steps, i.e., constructing an INN based homeomorphism mapping, sewing up by category and retraining iteratively training the model in two spaces.
In this way, the source labels can be fully used even in the target feature space for improving the model generalization for the target domain.
Extensive experimental results demonstrate the effectiveness of our method.
A A PPENDIX
A M ORE R ELATED
W ORKS
Unsupervised Domain Adaptation.
In addition to the statistic moment matching and adversarial learning mentioned in the text, the methods based on distribution alignment also have the following strategies: adversarial generation framework , bi-classifier adversarial learning and optimal transport .
Specifically, the methods based on adversarial generation framework combine the domain discriminator and a generator, and generate fake data to align the distribution between two domains at pixel-level.
The methods with high attention are CoGAN (Liu & Tuzel, 2016), SimGAN (Shrivastava et al., 2017) and CycleGAN (Zhu et al., 2017).
The methods based on bi-classifier adversarial learning play a minimax game with a single feature extractor and two distinct classifiers during domain adaptation (Saito et al., 2018; Lee et al., 2019; Li et al., 2021b;
Lu et al., 2020;
Zhou et al., 2022).
These methods maximize the prediction discrepancy when training classifiers and minimize the prediction discrepancy when training feature extractor.
Specifically, MCD (Saito et al., 2018) uses L 1 -Norm to calculate prediction discrepancy, SWD (Lee et al., 2019) proposes slide wasserstein distance and BCDM (Li et al., 2021b) proposes classifier determinacy disparity distance.
CDAL (Zhou et al., 2022) proposes the ECI strategy to solve the ambiguous samples during domain adaptation.
STAR (Lu et al., 2020) integrates an approximately infinite number of classifiers by sampling from the distribution, whilst keeping the model size the same as having two classifiers.
The methods based on optimal transport aims to learn a transformation between two domains (Damodaran et al., 2018;
Xu et al., 2020; Li et al., 2020).
These methods generally can be divided into two steps: the first step is to find the coupling matrix γ which used to connect each source data samples and target samples.
And the second step is to minimize the cost of pair-wise.
DeepJDOT (Damodaran et al., 2018) minimizes the discrepancy of the features and predictions simultaneously based on Wasserstein distance.
RWOT (Xu et al., 2020) exploits spatial prototypical information and intra-domain structure, and then achieves the precise-pair-wise optimal transport procedure.
ETD (Li et al., 2020) builds an attention-aware transport distance, which can be viewed as the prediction feedback of the iteratively learned classifier, to measure the domain discrepancy.
The method based on the optimal transport has some similarities with our method, both by finding a transformation to align the distributions.
However, there are the following differences.
First, the methods based on the optimal transport obtain transformation by solving a linear programming problem to obtain the coupling matrix γ , while our method constructs a homeomorphism mapping based on invertible neural network.
Second, the methods based on the optimal transport focus on the distribution alignment on pair-wise samples, while our method now focuses on overall alignment or class-wise alignment between two distributions during the sewing up process.
Of course, our method can also use this pair-wise samples alignment.
Third, the methods based on the optimal transport still need to find a common space for distribution alignment, while our method aligns the distributions in two spaces while preserving their original structures.
Invertible Neural Network (INN).
INN is a flow-based model, which transforms a probability distribution into another distribution by a sequence of invertible and differentiable mappings.
Invertible neural networks has been applied in the field of image super-resolution, lossless compression, style transfer, privacy protection and so on.
HCFlow (Liang et al., 2021b) utilizes the hierarchical conditional flow as a unified framework for image super-resolution and image rescaling.
NCSR (Kim & Son, 2021) proposes noise conditional flow model for super-resolution, which increases the visual quality and diversity of images through noise conditional layer.
In the field of lossless compression, derived from general volume preserving flows, iVPF (Zhang et al., 2021b) achieves an exact bijective mapping without any numerical error and then propose a lossless compression algorithm.
iFlow (Zhang et al., 2021a) achieves efficient lossless compression by the proposed modular scale transform combining numerically invertible flow transformations.
DIST (Chen et al., 2021) designs a diverse image style transfer framework by enforcing an invertible cross-space mapping.
Additionally, invertible networks play an important role in protecting privacy, such as invertible de-identification and image hiding (Jing et al., 2021; Cao et al., 2021).
Although invertible neural networks have been widely and effectively used in image processing, no work has applied them in domain adaptation.
A. P ROOF OF T HEOREM
Theorem 1.
Let ( M, d M ) and ( N, d N ) be two metric spaces with a homeomorphism g : M → N, and A is an open subset in ( M, d M ) , we have that its image B :=
g ( A ) is an open subset in ( N, d N ) , and
[Equation]
where ∂ means the boundary.
Proof .
It is sufficient to show that
[Equation]
If this is true, then we can implied equation 7 to g − 1 , and obtain g − 1 ( ∂B ) ⊂ ∂A.
Hence we have ∂B ⊂ g ( ∂A ) .
Combing this with equation 7, we have g ( ∂A ) =
∂B.
Now we want to show equation 7, that is, for any x ∈ ∂A , we have g ( x ) ∈ ∂B. Since x ∈ ∂A , but x / ∈ A , then g ( x ) / ∈ B , and there is a sequence { x i } ⊂
A such that lim
i →∞ x i = x .
By the continuity of the function g , we have
[Equation]
Noting that g ( x i ) ∈ B , we get g ( x ) ∈ ∂B. This completes the proof.
□
A. P ROOF OF T HEOREM
Theorem 2.
Invertible Neural Network is a homeomorphism.
Proof .
The definition of homeomorphism is that a function g : M → N between two topological spaces is a homeomorphism if it has the following properties: 1.
g is a bijection; 2.
g is continuous; 3.
the inverse function g − 1 is continuous.
For any invertible neural network, assuming that its forward process is g , then its invertible process can be represent g − 1 , so g is a bijection.
Because the function of each part of the invertible neural network is continuous, such as s ( · ) and t ( · ) in our method, so both g and g − 1 are continuous.
To sum up, the invertible network is a homeomorphism.
□
A. A LGORITHM
Our method is summarized in Algorithm 1.
In each iteration, the INN and backbone network, which consists of feature extractor and classifier, are both trained.
The loss functions are shown in equation 3 and equation 6 respectively.
A. I MPLEMENTATION DETAILS OF METHODS IN T ABLE
In Table 1, we report the results of several classical domain adaptation strategies given ground-truth labels.
Since the real labels are used, so we need to make simple modifications to these algorithms, which are shown below.
For the first line in Table 1, CAN (Kang et al., 2019) is selected to test the statistic moment matching strategy, which is almost the best statistical moment matching method in recent years.
It uses the clustering algorithm to pseudo-label the all target domain samples, and then uses the CAS strategy to sample target domain samples with high-confidence pseudo-label and source samples, finally, it minimizes the inter-class cross domain discrepancy and maximizes the intra-class cross domain discrepancy, which is shown as follows:
[Equation]
where MMD ( A, B ) represents the MMD discrepancy between A and B , f s,c represents the source features with true label c and f t, ˆ c represents the target features with pesudo label c .
When we giving the ground-truth labels to target domain, we do not need to pseudo label target samples, and directly sample all target samples to perform distribution alignment as follows:
[Equation]
where f t,c represents the target features with true label c .
In this case, the feature extractor is retrained by equation 9; the source classifier is retrained by source samples.
For the second line in Table 1, CDAN (Long et al., 2018) is selected to test adversarial learning strategy.
CDAN thinks the prediction of the classifier carry the discriminative information which can be used to align the conditional distribution between two domains.
Specifically, it first introduces a domain discriminator D to perform domain classification.
The input of the domain discriminator is the outer product of features and predictions and the loss function is defined as follows:
[Equation]
where ⊗ is the outer product, p si is the prediction of i -th source sample and p ti is the prediction of i -th target sample.
While in our test, ground-truth label are available during the training, we perform an one-hot operation on the ground-truth labels y si and y ti to get l si and l ti , and train the feature extraction network and the discrimination network in the following way:
[Equation]
In this case, the feature extractor is retrained by equation 11; the source classifier is retrained by source samples.
For the third line in Table 1, it reports the self-supervised training strategy.
Tradition methods based on this strategy (Liang et al., 2021a) usually assign target sample a pseudo-label ˆ y ti , and use pseudolabel to train the model as follows:
[Equation] [Equation]
where C means classifier.
When the true target labels y ti are given, it can directly supervised train the model as follows:
[Equation]
In this case, the feature extractor and source classifier are retrained by equation 13.
For the fourth line in Table 1, which uses two different networks to learn two transformations, which maps the source features to the target feature space and vice versa.
Specifically, two linear networks
F s 2 t ( · ) and F t 2 s ( · ) are introduced, and we have f s 2 t
= F s 2 t ( f s ) , f t 2 s = F t 2 s ( f t ) .
We hope the transformed features can be aligned to original features in their feature spaces respectively.
In this
[Table]
test, we also have true labels from both source domain and target domain and use CAN to align the distributions between transformed features and original features, which is shown as follows:
[Equation]
In this case, the feature extractor is retrained by equation 14; the source classifier is retrained by source samples.
For the fifth line in Table 1, which is our method based on ground-truth label, we introduce invertible neural network g to connect two feature spaces.
Specifically, the transformed features can be obtained by g as f s 2 t =
g
( f s ) and f t 2 s = g − 1 ( f t ) .
Due to the ground-truth label are available.
Therefore, We just need to modify our sewing up operation to the following:
[Equation]
In this case, the feature extractor and source classifier are retrained in two spaces.
A. C OMPARISONS TO S TATE - OF - THE -A RT ON D OMAIN N ET
DomainNet (Peng et al., 2019) is one of the most challenging datasets in domain adaptation.
It contains about 600 thousand images in 345 categories from 6 domains: Clipart (C), Infograph (I),
Painting (P), Quickdraw (Q), Real (R) and Sketch (S).
We compare our method HMA(CAN) with the following state-of-the-art methods: MCD (Saito et al., 2018), BNM (Cui et al., 2020), SWD (Lee et al., 2019) and CGDM (Du et al., 2021).
ResNet-50 is used as backbone.
As shown in Table 7, our method achieves the SOTA results.
A. M ODEL ANALYSIS Loss function for consistency constraint.
In equation 5, we use L 2 -Norm for consistency constraint for unlabeled target feature f t and f t 2 s .
In order to further verify the scalability of our method, we use cross entropy to calculate consistency constraints.
The experimental results on office-31 are shown in the Table 8.
From the experimental results, although different strategies are used, there does not exist a big gap in performance.
[Table] [Table]
How many blocks of INN do we need?
The forward and invertible process of INN for each block are shown in equation 1 and equation 2, so we need to discuss how many INN blocks we need.
As shown in Table 9, the average accuracy on Office-31 are reported.
It can be found that when the block number is changed from 1 to 2, the performance of both HMA(DAN) and HMA(CAN) has been greatly improved, while when the number of blocks is increased from 2 to 5, the performance increase is relatively slow.
This is because when the block number is 1, the y 1 in the output of the INN and the x 1 in the input are linearly related, i.e. ∂ y 1 ∂ x 1 = I where I is the identity matrix.
When the block number becomes 2, there is no such linear relationship, which makes the network has more capacity.
In addition, as the number of blocks in the network increases, the nonlinearity of the network also becomes stronger, resulting in better results.
Of course, before the learning ability is saturated, more blocks will definitely have better learning ability, but considering the computational overhead, we finally chose 5 blocks.
[Figure]
sewing up strategies which are performed in the target feature space and source feature space respectively.
The curve named Bi-alignment means the bilateral sewing up strategy.
Unilateral sewing up or bilateral sewing up?
Obviously, in our method, when the distributions between f t and f s 2 t are aligned, the discrepancy between f s and f t 2 s can also be minimized due to the reversibility of the INN, and vice versa.
But in our method, we do not use this unilateral sewing up but bilateral sewing up, i.e., f s and f t 2 s ; f t and f s 2 t are aligned as shown in equation 3.
We compare three strategies: unilateral sewing up: only alignment between f t and f s 2 t in target feature space or only alignment f s and f t 2 s in source feature space; and bilateral sewing up where the above mentioned pairs are all aligned.
We select
HMA(CAN) as the baseline and conduct experiments on A → D and D
→ A tasks of Office-31.
From the experimental results, the bilateral sewing up can make training faster than other two strategies.
In addition, we also find that bilateral sewing up can get better performance compared with unilateral sewing up.
[Table]
How to use our model?
Our method do alignment in two spaces, it is natural to ask a problem in which space using our model.
There are three strategies: using our model in the target feature space f t , or in the source feature space f t 2 s or in both source and target feature spaces where the average prediction is considered as the final result.
We test these three strategies on Office-31 dataset using HMA(CAN), which is shown in Table 10.
From the experimental results, the effect of adopting the ensemble strategy is slightly better than others, so for using our model, we adopt this ensemble strategy.
A. V ISUAL ANALYSIS BY T -SNE
To intuitively understand the proposed HMA, we use t-SNE (Van der Maaten & Hinton, 2008) to visualize the classification results on Office-31 based on two baselines, DAN and CAN, as shown in Fig.
4 and Fig.5, respectively.
For both figures, the first row represents the results on tasks
A → D , and the second row shows the results on tasks W
→ A .
From left to right, the visualization images represent the visualization results of the baseline method, the alignment results using INN on the baseline method, and the visualization results of our final proposed method, respectively.
From Fig.
4 and Fig.5, it can be seen that only using INN to sew up two domains can achieve similar results with the previous alignment method.
HMA shows a huge improvement over other visualization results.
This is because in addition to the distribution alignment using INN, our HMA approach further applies the property of INN to train the feature extractor and classifier and yields better performance.
[Figure] [Figure]
A. V ISUAL ANALYSIS BY G RAD -CAM
We show the visualization of Grad-CAM (Selvaraju et al., 2017) on Office-31 task W
→ A , shown in Fig.6 and Fig.7.
We randomly select 8 categories.
For each category, one image is showed for activation mapping visualization.
For both figures, from top to bottom, the images represent the results of original image, HMA(DAN), doublemap, CAN, HMA(CAN), respectively.
From the visualization results, the focus of HMA(DAN) and doublemap is mostly on local points, while ignoring the characteristics of the whole object.
Compared with the above two methods, CAN is slightly improved, but it still lacks certain accuracy.
Our proposed HMA(CAN) better estimates the attention.
A. NETWORK STRUCTURE
In this section, we will go into detail about the neural network we use.
For the feature extractor,
Resnet (He et al., 2016) is used, but its original last layer which is a fully connected linear layer for classification is removed.
It is worth noting that the dimension size of features yielded by feature extractor of both Resnet-50 and Resnet-101 is 2048.
The structure is shown as follows.
For classifier, a fully connected linear layer is constructed for suit our tasks, which maps features to predictions.
The dimension size of predictions are category number which are different in different datasets.
Specifically, the dimensions of prediction are 31, 65, 12 in Office-31, Office-home and
Visda-17 respectively.
For the INN, the affine network is used.
Specifically, it consists of two twolayers linear networks s ( · ) and t ( · ) .
The structure of s ( · ) and t ( · ) are the same.
Specifically, the network s ( · ) consists of two fully connected neural networks and a ReLU function.
The detail is shown in Fig.9(a).
We also discuss that using two different linear networks F s 2 t ( · ) and F t 2 s ( · ) to learn the mappings between two feature spaces.
The structures of F s 2 t ( · ) and F t 2 s ( · ) are same, which consists of four blocks.
Each block consists of a fully connected neural networks a batchnorm and a relu function.
The specific structure is shown in Fig.9(b).
Original image
HMA(DAN)
Doublemap
CAN
HMA(CAN)
[Figure] [Figure]
keyboard letter_tray
Original image
HMA(DAN)
Doublemap
CAN
HMA(CAN)
I np u t
C o n v d
B a t c h
N o r m
R e L U
M a x P oo l
C o n v
B l o c k
I d e n t i t y B l o c k
C o n v
B l o c k
I d e n t i t y B l o c k
C o n v
B l o c k
I d e n t i t y B l o c k
C o n v
B l o c k
I d e n t i t y B l o c k
A v er ag e P oo l
O u t pu t
C o n v 1x
B a t c h
N o r m
R e L U
C o n v 3x
B a t c h
N o r m
R e L U
C o n v 1x
B a t c h
N o r m
C o n v 1x
B a t c h
N o r m
C o n v 1x
B a t c h
N o r m
R e L U
C o n v 3x
B a t c h
N o r m
R e L U
C o n v 1x
B a t c h
N o r m
A dd R e L U A dd R e L U
I d e n t i t y B l o c k I d e n t i t y B l o c k
I d e n t i t y B l o c k I d e n t i t y B l o c k
I d e n t i t y B l o c k I d e n t i t y B l o c k
I d e n t i t y B l o c k I d e n t i t y B l o c k
[Figure] [Figure]
BatchNorm
ReLU (2048) (2048)
[Figure]
[Figure] [Figure]
[Figure] [Figure]
ReLU
[Figure]