D OMAIN A DAPTATION WITH F EATURE AND L ABEL D ISTRIBUTION C O -A LIGNMENT
Anonymous authors
Paper under double-blind review
A BSTRACT
Unsupervised knowledge transfer has a great potential to improve the generalizability of deep models to novel domains.
Yet the current literature assumes that the label distribution is domain-invariant and only aligns the Shift in Feature Distribution or vice versa .
In this paper, we explore the task of Domain Adaptation with Shift in the Covariate and Label distribution : How to transfer knowledge across different domains in the presence of both Shift in Feature Distribution and Shift in Label Distribution?
We propose a feature and target distribution CO-ALignment (COAL) model to tackle this problem.
Our model leverages prototype-based conditional alignment and label distribution estimation to alleviate the negative effect of the the types of domain shift respectively.
We demonstrate experimentally that when both types of shift exist in the data, COAL leads to state-of-the-art performance on several cross-domain benchmarks.
I NTRODUCTION
The success of deep learning models is highly dependent on the assumption that the training and testing data are i.i.d and sampled from the same distribution.
In reality, they are typically collected from different but related domains, leading to a phenomenon known as domain shift (QuioneroCandela et al., 2009).
To bridge the domain gap, Unsupervised Domain Adaptation (UDA) transfers the knowledge learned from a labeled source domain to an unlabeled target domain by statistical distribution alignment (Long et al., 2015; Tzeng et al., 2014a) or adversarial alignment (Tzeng et al., 2017; Ganin & Lempitsky, 2015a;
Saito et al., 2018).
Though recent UDA work has made great progress, it has mostly failed to address the case of Shift in Label Distribution, i.e. , a changing prior over the labels.
Denote the input data as x and output labels as y , and let the source and target domain be characterized by probability distributions p and q , respectively.
The majority of methods assume that the conditional label distribution is invariant ( p ( y | x ) =
q ( y | x ) ), and only the Shift in Feature Distribution (SFD) ( p ( x ) (cid:54) =
q ( x ) ) needs to be tackled, neglecting any potential Shift in Label Distribution (SLD) ( p ( y ) (cid:54) =
q ( y ) ).
However, recent theoretical work (Zhao et al., 2019a) has demonstrated that tackling SLD is crucial to solving the domain adaptation problem.
SLD also occurs in real applications; for example, an autonomous driving system should be able to handle changing frequencies of pedestrians and cars when adapting from a rural to a downtown area.
In this paper, we propose Generalized Domain Adaptation ( GDA ), a more challenging but practical domain adaptation setting where the conditional Shift in Feature Distribution and Shift in Label Distribution are required to be minimized simultaneously.
Speciﬁcally, in addition to the Covariate Shift assumption ( p ( x ) (cid:54) = q ( x ) , p ( y | x ) = q ( y | x ) ), we further assume p ( x | y )
(cid:54) =
q ( x | y ) and p ( y ) (cid:54) =
q ( y ) .
The main challenges of GDA are: ( 1 ) SLD exists between the source and target domain, which hampers the effectiveness of mainstream domain adaptation methods that only minimize SFD, ( 2 ) aligning the conditional feature distributions ( p ( x | y ) , q ( x | y ) ) is difﬁcult in the presence of SLD, and ( 3 ) when the data in one or both of the domains are unequally distributed across different categories, it is difﬁcult to train an unbiased classiﬁer.
An overview of GDA is shown in Figure 1.
Mainstream unsupervised domain adaptation aligns the marginal feature distributions of two domains by methods that include minimizing the Maximum Mean Discrepancy (Long et al., 2015; Tzeng et al., 2014a), aligning high-order moments (Zellinger et al., 2017; Peng et al., 2019a), or adversarial training (Tzeng et al., 2017; Ganin & Lempitsky, 2015a).
These methods have achieved state-of-the-art performance on several domain adaptation benchmarks (Saenko et al.,
[Figure] ; Venkateswara et al., 2017; Peng et al., 2017) which have signiﬁcant SFD but limited SLD.
These models are limited when applied to the GDA task as they only align the feature distribution, ignoring the real issue of SLD which exists in many real-world applications.
Another line of works (Lipton et al., 2018a; Azizzadenesheli et al., 2019a) assume that only SLD exists ( p ( y ) (cid:54) =
q ( y ) ) between two domains and the conditional feature distribution is invariant ( p ( x | y )
=
q ( x | y ) ).
These methods have achieved good performance when the data in both domains are sampled from the same data distributions but under different label distributions.
However, these models cannot handle the GDA task as the label distribution is not well aligned.
Taking steps towards minimizing the SFD and SLD simultaneously, several works (Zhang et al., 2013; Gong et al., 2016; Wu et al., 2019) provide a theoretical analysis with additional constraints on distributions p and q , such as a linearity assumption between p ( x | y ) and q ( x | y )
(Zhang et al., 2013), which does not necessarily hold in many real applications.
In addition, no practical algorithm which can solve real-world cross-domain problems has been proposed by these papers.
We postulate that it is essential to align the conditional feature distributions as well as the label distributions to tackle the GDA task.
In this work, we address GDA with SFD and SLD CO-ALignment ( COAL ).
Speciﬁcally, our approach diminishes the negative effect of SFD and SLD with prototypebased conditional distribution alignment and label distribution estimation , respectively.
First, to reduce the SFD in the context of SLD, it is essential to align the conditional rather than marginal feature distributions, to avoid the negative transfer effects which are caused by matching marginal feature distributions, according to a theoretical proof from Zhao et al. (2019a) (illustrated in Figure 2).
To this end, we propose a prototype-based method to align the conditional feature distributions of the two domains.
The source prototypes are computed by learning a similarity-based classiﬁer and the target prototypes are estimated by a minimax entropy algorithm
(Saito et al., 2019b).
Second, we align the label distributions in the context of SFD by estimating the target label distribution.
We incorporate feature distribution and label distribution alignment into an end-to-end deep learning framework, as illustrated in Figure 1.
Comprehensive experiments on standard crossdomain recognition benchmarks demonstrate that COAL achieves signiﬁcant improvements over the state-of-the-art methods on the task of GDA.
The main contributions of this paper are highlighted as follows: (1) to the best of our knowledge, we provide the ﬁrst practical solution and set of benchmarks for joint SFD and SLD in deep learning; (2) we develop an end-to-end CO-ALignment (COAL) framework which leverages conditional prototype-based alignment to solve the problem caused by SFD and label distribution estimation to solve the problem caused by the SLD; and (3) we deliver extensive experiments and analysis to show the effectiveness of COAL, and importantly, we empirically demonstrate that state-of-the-art methods fail to align conditional feature distribution in the presence of label distribution.
R ELATED W ORK
Unsupervised Domain Adaptation Domain adaptation aims to transfer the knowledge learned from one or more source domains to a target domain.
Recently, many unsupervised domain adaptation methods have been proposed.
These methods can be taxonomically divided into three categories (Wang & Deng, 2018).
The ﬁrst category is the discrepancy-based approach, which leverages
=
Feature Extractor
=
Classifier
Gradient for labeled and pseudo-labeled samples 𝑯 = Entropy 𝑳 𝑪𝑬 =
Cross Entropy Loss 𝒙 𝑻 𝒂𝒓𝒈𝒎𝒂𝒙 top −𝒌 ?
𝒎 ෝ𝒚 𝑻
Step A
Softmax
Step B 𝑪 𝑭 𝒙 𝑺 ,𝒙 𝑻 𝑷 𝑳 𝑪𝑬 (𝑷,𝒚 𝑺 ,ෝ𝒚 𝑻 ,𝒎 ）
Gradient Reverse −𝑯(𝑷)
Softmax = Gradient Path 𝑭 𝑪 𝑪 𝑭 different measures to align the marginal feature distributions between source and target domains.
Commonly used measures include Maximum Mean Discrepancy (MMD) (Long et al., 2017; Tzeng et al., 2014b), H -divergence (Ben-David et al., 2010), Kullback-Lerbler (KL) divergence (Zhuang et al., 2015), and Wasserstein distance (Lee & Raginsky, 2017; Shen et al., 2017).
The second category is the adversarial-based approach (Tzeng et al., 2017; Liu & Tuzel, 2016; Peng et al., 2019b) which uses a domain discriminator to encourage domain confusion via an adversarial objective.
The third category is the reconstruction-based approach.
Data are reconstructed in the new domain by an encoder-decoder (Bousmalis et al., 2016; Ghifary et al., 2016) or a GAN discriminator, such as dual-GAN (Yi et al., 2017), cycle-GAN (Zhu et al., 2017),
disco-GAN (Kim et al., 2017), and
CyCADA (Hoffman et al., 2018).
However, these methods mainly consider aligning the marginal distributions to decrease feature shift, neglecting label shift.
To the best of our knowledge, we are the ﬁrst the propose an end-to-end deep model to tackle both feature and label shift between the source and target domains.
Learning with Label Shift Despite its wide applicability, learning under label shift remains underexplored.
Existing works tackle this challenge by importance reweighting or target distribution estimation.
Speciﬁcally, Zhang et al.
(2013) exploit importance reweighting to enhance knowledge transfer under label shift.
Recently, Lipton et al.
(2018b) introduce a test distribution estimator to detect and correct for label shift.
Based on it, Azizzadenesheli et al.
(2019b) propose a regularized algorithm to better estimate the target label distribution.
These methods assume that the source and target domains share the same generative distributions and only differ in the marginal label distribution.
In this work, we explore transfer learning between domains under domain and label shifts.
In this direction, Zhao et al.
(2019b) introduce a theoretical analysis to show that only learning domain-invariant features is not sufﬁcient to solve domain adaptation task since the label priors are not aligned.
In related work, Wu et al.
(2019) propose asymmetrically-relaxed distribution alignment to overcomes the limitations of standard domain adaptation algorithms which aims to extract domain-invariant representations.
Panareda Busto & Gall (2017) propose open set domain adaptation where the categories in the training domain and testing domain are not fully overlapped.
Cao et al.
(2018b) introduce partial domain adaptation where the categories in the target domain are a subset of those in the source domain.
Domain adaptation with self-training Self-training methods utilize pseudo-labels to compensate for the lack of categorical information in the target domain.
The intuition is to assign pseudo-labels to unlabeled samples based on the predictions of one or more classiﬁers.
Long et al.
(2013) jointly align the marginal and conditional distributions using an MMD loss with pseudo-label reﬁnement.
Saito et al.
(2017) leverage an asymmetric tri-training strategy to assign pseudo-labels to the unlabeled target domain.
Xie et al.
(2018) propose to assign pseudo-labels to all target samples and use them to achieve semantic alignment across domains.
Zhang et al.
(2018) progressively enlarge the training data with pseudo-labeled target samples assigned by the classiﬁer from the previous training epoch and retrain the model with the enlarged training set.
Recently, Chen et al.
(2019a) propose to progressively label the target samples and align the prototypes of source domain and target domain to achieve domain alignment.
However, to the best of our knowledge, self-training has not been applied for DA with label shift.
CO-AL IGNMENT OF F EATURE AND L ABEL D ISTRIBUTION
In domain adaptation with shift in the feature and label distribution, we are given a source domain D S
= { ( x si , y si ) N s i =1 } with N s labeled examples, and a target domain D T
= { ( x ti ) N t
i =1 } with N t unlabeled examples.
We assume that p ( x | y ) (cid:54) =
q ( x | y ) and p ( y ) (cid:54) =
q ( y ) .
We aim to construct a end-to-end deep neural network which is able to transfer the knowledge learned from D S to D T , and train a classiﬁer y
= θ ( x ) which can minimize task risk (cid:15) T ( θ ) =
Pr ( x,y ) ∼ q [ θ ( x ) (cid:54) =
y ] .
Previous works either focus on aligning the marginal feature distributions (Long et al., 2015; Tzeng et al., 2017) or aligning the label distributions (Lipton et al., 2018b).
These approaches are not able to fully tackle the domain adaptation with shift in the feature and label distribution as they only align one of the two marginal distributions.
In this work, we tackle GDA with prototype-based conditional alignment and label distribution estimation .
3. P ROTOTYPE - BASED C ONDITIONAL A LIGNMENT OF F EATURE D ISTRIBUTION
The mainstream idea in covariate-shift oriented methods is to learn domain-invariant features by aligning the marginal feature distributions, which was proved to be inferior in the presence of label shift (Zhao et al., 2019a).
Instead, we propose to align the conditional feature distributions.
To this end, we leverage a similarity-based classiﬁer to estimate p ( x | y ) , and a minimax entropy algorithm to align it with q ( x | y ) .
We achieve conditional feature distribution alignment by aligning the source and target prototypes in an adversarial process.
Similarity-based Classiﬁer
The architecture of our COAL model contains a feature extractor F and a similarity-based classiﬁer C .
Similarity- or prototype-based classiﬁers perform well in fewshot learning settings (Chen et al., 2019b), which motivates us to adopt them since in label-shift settings some categories can have low frequencies.
Speciﬁcally, C is composed of a weight matrix
W ∈ R d × c and a temperature parameter T , where d is the dimension of feature generated by F , and c is the total number of classes.
Denote W as [ w 1 , w 2 , ..., w c ] , this matrix can be seen as c d -dimension vectors, one for each category.
For each input feature F ( x ) , we compute its similarity with the i th weight vector as s i
= F ( x ) w i T (cid:107) F ( x ) (cid:107) .
Then, we compute the probability of the sample being labeled as class i by h i ( x )
= σ ( F ( x ) w i T (cid:107) F ( x ) (cid:107) ) , normalizing over all the classes.
Finally, we can compute the prototype-based classiﬁcation loss for D S with standard cross-entropy loss:
[Equation]
The intuition behind this loss is that the higher the conﬁdence of sample x being classiﬁed as class i , the closer the embedding of x is to w i .
Hence, when optimizing Equation 1, we are reducing the intra-class variation by pushing the embedding of each sample x closer to its corresponding weight vector in W .
In this way, w i can be seen as a representative data point (prototype) for p ( x | y
=
i ) .
Prototype-based Conditional Alignment by Minimax Entropy
Due to the lack of categorical information in the target domain, it is infeasible to utilize Equation 1 to obtain target prototypes.
Instead, we propose to tackle this problem by 1) moving each source prototype to be closer to its nearby target samples, and 2) clustering target samples around this moved prototype.
Inspired by
Saito et al. (2019a), we propose to achieve these two objectives jointly by entropy minimax learning.
Speciﬁcally, for each sample x t ∈ D T fed into the network, we can compute the entropy of the classiﬁer’s output by
[Equation]
Larger H indicates the target samples are similar to all the weight vectors (prototypes) of C .
We achieve conditional feature distributions alignment by aligning the source and target prototypes in an adversarial process: (1) we train C to maximize H , aiming to move the prototypes from the source samples towards the neighboring target samples; (2) we train F to minimize H , aiming to make the embedding of target samples closer to their nearby prototypes.
By training with these two objectives as a min-max game between C and F , we can align source and target prototypes.
Practically, we add a gradient-reverse layer (Ganin & Lempitsky, 2015a) between C and F to ﬂip the sign of gradient.
Input: F , C , D S , D T , N iter , N epoch , N b , k = k
Output: ˆ F , ˆ C 1 train F and C with D S using Equation2 for i = 0 , ..., N epoch do
Step A : compute ˆ D T = { ( x ti , ˆ y ti , m i )
N t
i =1 } with k as in Section 3.
Step B : for j = 0 , ..., N iter do randomly sample mini-batch { ( x si , y si ) N b
i =1 } ∈ D S , { ( x ti , ˆ y ti , m i ) N b i =1 } ∈ ˆ D T train F and C with Equation
k = min( k + k step , k max )
3. L ABEL D ISTRIBUTION E STIMATION WITH S ELF - TRAINING
As the source label distribution p ( y ) is different from that of the target q ( y ) , it is not guaranteed that the classiﬁer C which has low risk on D S will have low error on the target domain.
Intuitively, if the classiﬁer is trained with imbalanced source data, the decision boundary will be dominated by the most frequent categories in the training data, leading to a “biased” classiﬁer.
When the classiﬁer is applied to target domain with a different label distribution, its accuracy will degrade as it is highly biased.
To tackle this problem, we employ the self-training method to estimate the target label distribution and reﬁne the decision boundary.
In addition, we leverage balanced sampling of the source data to further diminish the effect of label shift.
Self-training In order to reﬁne the decision boundary, we propose to estimate the target label distribution with self-training (Zou et al., 2018).
We will assign pseudo labels ˆ y to all the target samples according to the output the classiﬁer C .
As we have aligned the conditional feature distributions ( p ( x | y ) and q ( x | y ) ), we assume that the pseudo label distribution q (ˆ y ) is the approximation of the real label distribution q ( y ) for the target domain.
For each category, we select top- k percent of the target samples with the highest conﬁdence scores belonging to that category.
We utilize the highest probability in h ( x ) as the classiﬁer’s conﬁdence on sample x .
Speciﬁcally, for each pseudo-labeled sample ( x, ˆ y ) , we set its selection mask m
=if h ( x ) is among the top- k percent of all the target samples with the same pseduo-label, otherwise m
= 0 .
Denote the pseudo-labeled target set as ˆ D T =
( x ti , ˆ y ti , m i )
N t
i =1 } , we leverage the data and pseudo labels from ˆ D T to train the classiﬁer C , aiming to reﬁne the decision boundary with target label distribution.
The total loss function for classiﬁcation is:
[Equation]
where ˆ y indicates the pseudo labels and m indicates selection masks.
In our approach, we choose the top-k percent of the highest conﬁdence target samples within each category, instead of universally.
This is crucial to estimate the real target label distribution, otherwise, the easy-to-transfer category(ies) will dominate ˆ D T , leading to inaccurate estimation of the target label distribution.
Balanced Sampling of Source Data
In addition to estimating the target label distribution, we propose a balanced sampling process to enhance label shift alignment.
When coping with label shift, the label distribution of the source domain could be highly unbalanced.
A classiﬁer trained on unbalanced categories will make highly-biased predictions for the samples from the target domain (He & Garcia, 2009).
This effect also hinders the self-training process discussed above, as the label distribution estimation will also be biased.
To tackle these problems, we apply a balanced mini-batch sampler to generate training data from the source domain and ensure that each source mini-batch contains roughly the same number of samples for each category.
3. T RAINING P ROCESS
In this section, we combine the above ideas into an end-to-end training process.
Given input samples from source domain D S and target domain D T , we ﬁrst initialize our network F and C with only labeled data D S .
Then, we iterate between the assignment of pseudo labels and adaptive learning until convergence or reaching the maximum number of iterations.
An overview of this process can be seen in Algorithm 1.
The overall training objective is as follows:
[Equation]
3. T HEORETICAL I NSIGHTS
Conditional Feature Alignment
According to Zhao et al. (2019b), the target error in domain adaptation is bounded by three terms: 1) source error, 2) the discrepancy between the marginal distributions and 3) the distance between the source and target optimal labeling functions.
Denote h ∈ H as the hypothesis, (cid:15) S ( · ) and (cid:15) T ( · ) as the expected error of a labeling function on source and target domain, and f S and f T as the optimal labeling functions in the source and target domain.
Then, we have:
[Equation]
where d ˆ H denote the discrepancy of the marginal distributions (Zhao et al., 2019b).
The bound demonstrates that the optimal labeling functions f S and f T need to generalize well in both domains, such that the term min { (cid:15) S ( f T ) , (cid:15) T ( f S ) } can be bounded.
Conventional domain adaptation approaches which only align marginal feature distribution cannot guarantee that min { (cid:15) S ( f T ) , (cid:15) T ( f S ) } is minimized.
This motivates us to align the conditional feature distribution, i.e. p ( x | y ) and q ( x | y ) .
Our model COALestimates and aligns the source and target prototypes to align p ( x | y ) and q ( x | y ) .
Decision Boundary Reﬁnement Theorem 4.3 in Zhao et al.
(2019b) indicates that the target error (cid:15) T ( h ) can not be minimized if we only align the feature distributions and neglect the shift in label distribution.
Denote d JS as the Jensen-Shannon(JS) distance between two distributions, Zhao et al.
(2019b) propose:
[Equation]
This theorem demonstrates that when the divergence between label distributions d JS ( p ( y ) , q ( y )) is signiﬁcant, minimizing the divergence between marginal distributions d JS ( p ( x ) , q ( x )) and the source task error (cid:15) S ( h ) will enlarge the target task error (cid:15) T ( h ) .
Motivated by this, we propose to align the empirical label distributions with self-training algorithm.
E XPERIMENTS
We evaluate our CO-ALignment ( COAL ) model with three popular benchmarks, i.e. , Digits , OfﬁceHome (Venkateswara et al., 2017) and DomainNet (Peng et al., 2019a).
Sample images of these datasets can be found in Figure 3a.
As Digits and Ofﬁce-Home dataset contains limited SLD, we artiﬁcially create considerable SLD by random sub-sampling the data for each category.
4. S ETUP
Sub-sampling Protocol.
To create signiﬁcant SLD between source and target domains, we subsample the current datasets with R eversely-unbalanced S ource and U nbalanced T arget ( RS-UT ) protocol.
In this setting, both the source and target domains have unbalanced label distribution, while the label distribution of the source domain is a reversed version of that of the target domain.
An illustration of this setting can be found in Figure 3(a).
We refer our reader to supplementary material for more details!
Digits We select four digits datasets: MNIST (LeCun et al., 1998), USPS (Hull, 1994), SVHN (Netzer et al., 2011) and Synthetic Digits (SYN) (Ganin & Lempitsky, 2015b).
In this work, we investigate four domain adaptation tasks: MNIST → USPS , USPS → MNIST , SVHN → MNIST , and SYN → MNIST .
Ofﬁce-Home (Venkateswara et al., 2017) is a dataset collected in ofﬁce and home environment with 65 object classes and four domains: Real World ( Rw ), Clipart ( Cl ), Product ( Pr ), Art ( Ar ).
Since
[Figure]
v mm m
[Figure] [Figure]
Methods
USPS → MNIST MNIST → USPS
SVHN → MNIST SYN → MNIST
AVG
Source Only 75.31 ± 0.
87.92 ± 0.
50.25 ± 0.
85.74 ± 0.
F-DANN 72.59 ± 1.
81.62 ± 2.
45.65 ± 2.
[Table]
the “Art” domain is too small to sample an imbalanced subset, we focus on the remaining domains and explore all the six adaptation scenarios.
DomainNet (Peng et al., 2019a) is a large-scale testbed for domain adaptation, which contains six domains with about 0.6 million images distributed among 345 classes.
Since some domains and classes contains many outliers, we select 40 commonly-seen classes from four domains: Real ( R ),
Clipart ( C ), Painting ( P ), Sketch ( S ).
Different from two datasets above, the SLD in DomainNet is signiﬁcant enough, as illustrated in Figure 3.
Moreover, the images are directly collected from image search engine and they are in consistence with the natural label distribution in the real world.
Baselines We compare covarite and SLD CO-AL ignment ( COAL ) with state-of-the-art domain adaptation methods that mainly aligns feature distribution: Deep Adaptation Network ( DAN ) (Long et al., 2015), Joint Adaptation Network (JAN) (Long et al., 2017), Domain Adversarial Neural Network ( DANN ) (Ganin & Lempitsky, 2015a), Batch Spectral Penalization ( BSP ) (Chen et al., 2019c), f-divergence Domain Adversarial Neural Network ( F-DANN )
(Wu et al., 2019), Maximum Classiﬁer Discrepancy ( MCD ) (Saito et al., 2018).
We also compare our model with methods that aligns the marginal label distribution: Partial Adversarial Domain Adaptation ( PADA ) (Cao et al., 2018a),
Black Box Shift Estimation ( BBSE ) (Lipton et al., 2018b).
Implementation Details.
We implement all our experiments in Pytorch 1 platform.
For the Digits dataset, we adopt the network architecture proposed by Saito et al.
We adopt SGD with the momentum of 0.9 and learning rate of 0.01 for the linear classiﬁer and 0.001 for all other layers.
The batch size is set as 32 for samples from each domain.
For the other two datasets, we utilize ResNet-
[Table] [Table]
50 (He et al., 2015) as our backbone network, and replace the last
-connected layer with a randomly initialized N-way classiﬁer layer (for N categories).
We also use SGD with momentum of 0.9 while setting the learning rate to be 0.001 for linear layers and 0.0001 for all the other layers.
The batch size is set as 16 for each domain.
We set the α = 1 in Equation 4 and set k 0 = 5 , k step = 5 , k max = 50 as the parameters for self-training selection policy.
For all the compared methods, we decide their hyper-parameters on the validation set of Painting → Clipart task in DomainNet.
Evaluation metric.
When the target domain is highly unbalanced, conventional overall average accuracy that treats every class uniformly is not an appropriate performance metric (He & Garcia, 2008).
Therefore, we follow Dong et al.
(2019) to use the per-class mean accuracy in our main results.
Formally, we denote S i = n
( i,i ) n i as the accuracy for class i , where n ( i,j ) represents the number of class i samples labeled as class j , and n i = (cid:80) cj =1 n ( i,j ) represents the number of samples in class i .
Then, the per-class mean accuracy is computed as S = 1 c (cid:80) ci
=
1 S i .
4. R ESULTS
We ﬁrst show the experimental results of our model and the compared baselines on Digits dataset in Table 1.
The covariate shift between the source and target domains is related small on Digits dataset.
For fair comparison, we leverage the same backbones for all the models.
From the results, we can make the following observations.
(1) Our model achieves 84.33 % average accuracy across four experimental setting, outperforming the best-performing baselines by 8.4 %!
Within the eight selected baselines, only DAN and DANN have marginal improvement (less than 2%) over the source-only baseline.
The other six compared methods encounter the negative transfer (Pan &
Yang, 2010) issue and perform worse than the source-only baselines.
These results demonstrate that aligning only the marginal feature distributions or only the label distributions can not fully tackle GDA task.
In contrast, our model co-aligns the conditional feature distributions and label distributions and outperforms the baselines by a large margin.
Next, we show the experimental results on more challenging real-image datasets, i.e. Ofﬁce-Home and DomainNet, in Table 2 and Table 3, respectively.
In Ofﬁce-Home experiments, our model
[Figure]
achieves 58.40 % average accuracy across the six GDA tasks, outperforming other compared baselines.
Our model has 5.59 % improvement from the source-only result.
We also notice that the BSP ,
PADA , BBSE and MCD models perform worse than the source-only baselines, which is in consistent with the results on Digits dataset.
The empirical results on Ofﬁce-Home dataset demonstrate the effectiveness of co-alignment of conditional covariate shift and SLD.
In DomainNet experiments, our model get 74.96 % average accuracy across the 12 experiments, outperforming the compared baselines.
Note we have carefully tuned the hyper-parameters for the compared domain adaptation methods.
Without the hyper-parameters tuning, i.e. directly applying the model released by the authors, these models perform worse than the source-only baselines.
In addition, our model improves the performance of source-only models by 12.4 %.
4. A NALYSIS
Ablation Study.
Our COAL method has mainly two objectives: 1) alignment of conditional feature distribution and 2) alignment of label distribution.
To show the importance of these two objectives in GDA, we show the performance of our method without each of these objectives respectively on multiple tasks.
The results in Table 4 showed the importance of both objectives.
For example, for task USPS → MNIST, if we remove the conditional feature distribution alignment objective, the accuracy of our model will drop by 2 .
Similarly, if we remove the label distribution alignment objective, the accuracy will drop by 2 . 9% .
These results demonstrate that both the alignment of conditional feature distribution and label distribution are important to GDA task.
Feature Visualization.
In this section, we plot the learned features with t-SNE (van der Maaten &
Hinton, 2008) in Figure 4.
We investigate the Real to Clipart task in DomainNet experiment with
ResNet-50 backbones.
From (a)-(d), we can observe that our method has better ability to cluster each sample towards its class centroid.
This conﬁrms the effectiveness of the entropy minimization in Equation 4.
From (e)-(h), we observe that our method can better align source and target features in each category, while other methods either leave the feature distributions unaligned, or incorrectly aligned samples in different categories.
These results further show the importance of prototypebased conditional feature alignment for generalized domain adaptation task.
Effect of Source Balanced Sampler.
Source balanced samplers described in Section 3.2 can help us tackle the bias-classiﬁer problem caused by the imbalanced data distribution of source domain.
A signiﬁcant performance boost can be observed after applying the balanced sampler for our
COAL model, as well as the compared baselines.
In this section, we speciﬁcally show the effect of using source balanced samplers.
We show in Table 5 the performance of several methods with and without source balanced samplers on 5 adaptation tasks from multiple datasets.
We observe that for 22 of the total 25 tasks (5 models on 5 adaptation tasks), using source balanced samplers
[Table]
[Table]
balanced sampler.
We observe a signiﬁcant performance boost when the source balanced sampler is applied, both for our model and the compared baselines, demonstrating the effectiveness of source balanced sampler to GDA task.
will improve the domain adaptation performance.
Furthermore, in USPS → MNIST and SVHN →
MNIST, source balance samplers can prevent the baseline methods from negative transfer .
These results show the effectiveness of having a source balanced sampler when tackling GDA task.
[Figure]
Different Degrees of SLD.
In Section 4, we only investigated the performance of each method under a certain kind of SLD in each dataset, i.e. , RS-UT.
In this section, we investigate the effect of different degrees of SLD on the performance of domain adaptation methods.
Specifically, we create 4 interval degrees of SLD between the BS-BT ( B lanced S ource and B lanced T arget) and RS-UT setting.
To this end, we compute the proportions of each category by linear interpolation between its proportions in BS-BT and RS-UT.
We denote BS-BT and RS-UT as 0% and 100% degree of SLD respectively, and create datasets with SLDs of 20% , 40% , 60% and 80% on the linear interpolation.
For fair comparison, all the datasets have the same total amount of samples.
With these datasets, we evaluated the performances of different methods on the
USPS → MNIST task of Digits.
The results in Figureshow that the performances of previous domain adaptation methods will be signiﬁcantly affected by SLD.
For example, the accuracy of MCD drastically drops from 91.45 % to 77.18 %.
In contrast, the performance our method is much more stable, which ranges between 93.42 % and 88.12 %.
It shows that our method are robust to different degrees of SLD.
C ONCLUSION
In this paper, we ﬁrst propose a generalized domain adaptation ( GDA ) learning schema and demonstrate the importance of GDA in practical scenarios.
Then we deliver theoretically analysis to demonstrate that aligning conditional feature distributions ( p ( x | y ) , q ( x | y ) ) and aligning the label distributions ( p ( y ) , q ( y ) ) are essential for GDA.
Towards tackling GDA task, we have proposed conditional feature distribution and label distribution CO-ALignment ( COAL ) approach, which leverages prototype-based conditional alignment and self-training method.
Empirically, we demonstrate that previous UDA models which learn domain-invariant features by aligning the marginal distributions fail to tackle GDA task, due to the presence of shift in the label distribution.
An extensive empirical evaluation on GDA benchmarks demonstrate the efﬁcacy of the proposed model against several state-of-the-art domain adaptation algorithms.
A A DDITIONAL R ESULTS
We have shown in the paper the performance of different domain adaptation methods when conditional covariant shift and label shift exist simultaneously.
To analysis the inﬂuence of label shift in unsupervised domain adaptation, in this appendix we also include the results on datasets without label shift.
To this end, we also created datasets with the Balanced Source Balanced Target (BS-BT) setting.
Speciﬁcally, under BS-BT every class has the same number of samples.
We use this setting to sub-sample the BS-BT datasets from Digits and Ofﬁce-Home.
For fair comparison with the results in Section 4, we constrain that the total amount of samples of each domain is the same in
BS-BT and RS-UT.
We show the results in Table
[Table]
and 2.
[Table] [Table]
We can observe that conventional methods that focus on covariate shift can achieve decent improvement over the Source Only baseline in BS-BT setting.
For example, MCD is largely improving the performance when there is no label shift between the source and target domain.
However, as we observed in Table 1 of our paper, MCD is actually having negative transfer effect when applied to the more general GDA setting.
This comparison demonstrates the importance of considering both covariate and label shift in our generalized setting.
Please note that the results in this section are produced by models with hyper-parameters tuned on unbalanced DomainNet.
B C REATION D ETAILS FOR L ABEL S HIFT
In order to create unbalanced label distribution in each dataset, inspired by Liu et al.
(2019), we follow the Paredo distribution set different proportions for each category.
By using this distribution, we can create long-tailed label distribution, which is frequently seen in real applications and benchmarks (Van Horn et al., 2018; Peng et al., 2019a).
The shape of Paredo distribution (Reed, 2001) is controlled by parameter α .
Because different datasets have different amount of samples, to avoid making some classes in the unbalanced dataset to have too few samples, we use a different parameter for each dataset.
Speciﬁcally, we set α =for Digits dataset, and α = 100 for Ofﬁce-Home dataset.
We further assign each computed proportion to each category by following the descending order of the original class index.
Speciﬁcally, in the target domain of RS-UT, we assign the k th largest propotion to class k − 1 , with class index starting from 0.
For Digits, we set the index of each class directly as the digit it represents.
For Ofﬁce-Home, we set the class index in alphabetical ascending order.
C I MPLEMENTATION D ETAILS FOR C OMPARED M ETHODS
In our experiments, we use the authors’ ofﬁcial implementations for DAN (Long et al., 2015), and
JAN (Long et al., 2017), PADA (Cao et al., 2018b), MCD (Saito et al., 2018), and BSP (Chen et al., 2019c).
For DANN (Ganin & Lempitsky, 2015a), BBSE (Lipton et al., 2018b) and FDANN (Wu et al., 2019), we implement their method by ourselves.
We tune the hyper-parameters of each method on Painting → Clipart task in DomainNet.
Speciﬁcally, for PADA, BSP, DAN, JAN and DANN we tune the weight α of the marginal feature alignment loss.
We empirically ﬁnd that these method achieve better performance when we set α to be 5times lower than default.
Intuitively, it means that we can achieve better performance under generalized domain shift setting if we relax the strength of marginal feature alignment.
For MCD we tune the number of feature generator updating times n .
D D ETAILED INFORMATION FOR DATASETS
We provide detailed information for datasets in Table 3.
Digits
Splits
USPS
MNIST
SVHN
SYN
Total
Train 12,
[Table]