U NIFIED P RINCIPLES F OR M ULTI -S OURCE T RANSFER L EARNING U NDER L ABEL S HIFTS
Anonymous authors
Paper under double-blind review
A BSTRACT
We study the label shift problem in multi-source transfer learning and derive new generic principles.
Our proposed framework uniﬁes the principles of conditional feature alignment, label distribution ratio estimation and domain relation weights estimation.
Based on inspired practical principles, we provide uniﬁed practical framework for three multi-source label shift transfer scenarios: learning with limited target data, unsupervised domain adaptation and label partial unsupervised domain adaptation.
We evaluate the proposed method on these scenarios by extensive experiments and show that our proposed algorithm can signiﬁcantly outperform the baselines.
I NTRODUCTION
Transfer learning (Pan & Yang, 2009) is based on the motivation that learning a new task is easier after having learned several similar tasks.
By learning the inductive bias from a set of related source domains ( S 1 , . . .
, S T ) and then leveraging the shared knowledge upon learning the target domain
T , the prediction performance can be signiﬁcantly improved.
Based on this, transfer learning arises in deep learning applications such as computer vision (Zhang et al., 2019; Tan et al., 2018; Hoffman et al., 2018b), natural language processing (Ruder et al., 2019;
Houlsby et al., 2019) and biomedical engineering (Raghu et al., 2019;
Lundervold & Lundervold, 2019; Zhang & An, 2017).
To ensure a reliable transfer, it is critical to understand the theoretical assumptions between the domains.
One implicit assumption in most transfer learning algorithms is that the label proportions remain unchanged across different domains (Du Plessis & Sugiyama, 2014)
(i.e., S ( y ) =
T ( y ) ).
However, in many real-world applications, the label distributions can vary markedly (i.e. label shift)
(Wen et al., 2014; Lipton et al., 2018; Li et al., 2019b), in which existing approaches cannot guarantee a small target generalization error, which is recently proved by Combes et al.
Moreover, transfer learning becomes more challenging when transferring knowledge from multiple sources to build a model for the target domain, as this requires an effective selection and leveraging the most useful source domains when label shift occurs.
This is not only theoretically interesting but also commonly encountered in real-world applications.
For example, in medical diagnostics, the disease distribution changes over countries (Liu et al., 2004; Geiss et al., 2014).
Considering the task of diagnosing a disease in a country without sufﬁcient data, how can we leverage the information from different countries with abundant data to help the diagnosing?
Obviously, na¨ıvely combining all the sources and applying one-to-one single source transfer learning algorithm can lead to undesired results, as it can include low quality or even untrusted data from certain sources, which can severely inﬂuence the performance.
In this paper, we study the label shift problem in multi-source transfer learning where S t ( y ) (cid:54) =
T ( y ) .
We propose uniﬁed principles that are applicable for three common transfer scenarios: unsupervised
Domain Adaptation (DA) (Ben-David et al., 2010), limited target labels (Mansour et al., 2020) and partial unsupervised DA with supp ( T ( y )) ⊆
supp ( S t ( y ))
(Cao et al., 2018), where prior works generally treated them as separate scenario.
It should be noted that this work deals with target shift without assuming that semantic conditional distributions are identical (i.e., S t ( x | y ) (cid:54) =
T ( x | y ) ), which is more realistic for real-world problems.
Our contributions in this paper are two-folds: ( I )
We propose to use Wasserstein distance (Arjovsky et al., 2017) to develop a new target generalization risk upper bound (Theorem 1), which reveals the importance of label distribution ratio estimation and provides a principled guideline to learn the domain relation coefﬁcients.
Moreover, we provide a theoretical analysis in the context of representation learning (Theorem 2), which guides to learn a feature function that minimizes the conditional Wasserstein distance as well as controls the weighted source risk.
We further reveal the relations in the aforementioned three scenarios lie in the different assumptions for estimating label distribution ratio.
( II ) Inspired by the theoretical results, we propose Wasserstein Aggregation Domain Network (WADN) for handling label-shift in multi-source transfer learning.
We evaluate our algorithm on three benchmark datasets, and the results show that our algorithm can signiﬁcantly outperform stateof-the-art principled approaches.
R ELATED W ORK
Multi-Source Transfer Learning Theories have been investigated in the previous literature with different principles to aggregate source domains.
In the popular unsupervised DA, (Zhao et al., 2018;
Peng et al., 2019; Wen et al., 2020; Li et al., 2018b) adopted H -divergence
(Ben-David et al., 2007), discrepancy (Mansour et al., 2009) and Wasserstein distance (Arjovsky et al., 2017) of marginal distribution d ( S t ( x ) , T ( x )) to estimate domain relations and dynamically leveraged different domains.
These algorithms generally consists source risk, domain discrepancy and an un-observable term η , the optimal risk on all the domains, which are ignored in these approaches.
However, as Combes et al.
(2020) pointed out, ignoring the inﬂuence of η will be problematic when label distributions between source and target domains are signiﬁcantly different.
Therefore it is necessary to take η into consideration by using a small amount of labelled data is available for the target domain (Wen et al., 2020).
Following this line, very recent works (Konstantinov & Lampert, 2019; Wang et al., 2019a;
Mansour et al., 2020) started to consider measure the divergence between two domains given label information for the target domain by using Y -discrepancy (Mohri & Medina, 2012).
However, we empirically showed these methods are still unable to handle label shift.
Label-Shift Label-Shift (Zhang et al., 2013;
Gong et al., 2016) is a common phenomena in the transfer learning with S ( y ) (cid:54) =
T ( y ) and generally ignored by the previous multi-source transfer learning practice.
Several theoretical principled approaches have been proposed such as (Azizzadenesheli et al., 2019; Garg et al., 2020).
In addition, (Combes et al., 2020; Wu et al., 2019) analyzed the generalized label shift problem in the one-to-one single unsupervised DA problem but did not provide guidelines of levering different sources to ensure a reliable transfer, which is more challenging.
(Redko et al., 2019) proposed optimal transport strategy for the multiple unsupervised DA under label shift by assuming identical semantic conditional distribution.
However they did not consider representation learning in conjunction with their framework and did not design neural network based approaches.
Different from these, we analyzed our problem in the context of representation learning and propose an efﬁcient and principled strategies.
Moreover, our theoretical results highlights the importance of label shift problem in a variety of multi-source transfer problem.
While the aforementioned work generally focus on the unsupervised DA problem, without considering uniﬁed rules for different scenarios (e.g. partial multi-source DA).
T HEORETICAL I NSIGHTS : T RANSFER R ISK U PPER B OUND
We assume a scoring hypothesis deﬁned on the input space X and output space Y with h : X ×Y → R , is K -Lipschitz w.r.t.
the feature x (given the same label), i.e. for ∀ y , (cid:107) h
( x 1 , y ) − h ( x 2 , y ) (cid:107) 2 ≤
K (cid:107) x 1 − x 2 (cid:107) 2 , and the loss function (cid:96) : R × R → R + is positive, L -Lipschitz and upper bounded by L max .
We denote the expected risk w.r.t distribution D : R D ( h ) =
E ( x,y ) ∼D (cid:96) ( h ( x, y )) and its empirical counterpart (w.r.t.
ˆ D )
ˆ R D ( h )
= (cid:80) (
x,y ) ∈
ˆ D (cid:96) ( h ( x, y )) .
We adopted Wasserstein-1 distance (Arjovsky et al., 2017) as a metric to measure the similarity of the domains.
Compared with other divergences, Wasserstein distance has been theoretically proved tighter than TV distance (Gong et al., 2016) or Jensen-Shnannon divergence (Combes et al., 2020).
Based on previous work, the label shift is generally handled by label-distribution ratio weighted loss:
R α S ( h )
=
E ( x,y ) ∼S α ( y ) (cid:96) ( h ( x, y )) with α ( y )
=
T ( y ) / S ( y ) .
We also denote ˆ α t as its empirical counterpart, estimated from samples.
Besides, to measure the task relations, we deﬁne a simplex λ with λ [ t ] ≥ 0 , (cid:80)
Tt =1 λ [ t ] = 1 as the task relation coefﬁcient vector by assigning high weight to the similar task.
Then we ﬁrst present Theorem 1, which proposed theoretical insights about how to combine source domains through properly estimating λ .
Theorem 1.
Let { ˆ S t = { ( x i , y i ) } N S t
i =1 } Tt =1 and ˆ T = { ( x i , y i ) } N T i =1 , respectively be T source and target i.i.d.
samples.
For ∀ h ∈ H with H the hypothesis family and ∀ λ , with high probability ≥ 1 − 4 δ , the target risk can be upper bounded by:
[Equation]
t =
[Equation]
[ t ]β t [Equation] log
[Equation]
1 /δ ) 2 N
[Equation]
where N = (cid:80) Tt =1 N S t and β t = N S t /N and d sup ∞ =
max t ∈ [1 ,T ] ,y ∈ [1 , Y ] α t ( y ) the maximum true label distribution ratio value.
W 1 ( ·(cid:107)· ) is the Wasserstein-1 distance with L 2 -distance as cost function.
Comp ( N S 1 , . . . , N S T , N T , δ ) is a function that decreases with larger N S 1 , . .
, N T , given a ﬁxed δ and hypothesis family H .
(See Appendix E for details)
Remarks (1)
In the ﬁrst two terms, the relation coefﬁcient λ is controlled by α t -weighted loss ˆ
R ˆ α t S t ( h ) and conditional Wasserstein distance E y ∼ ˆ T ( y ) W 1 ( ˆ T ( x | Y =
y ) (cid:107) ˆ S t ( x | Y = y )) .
To minimize the upper bound, we need to assign a higher λ [ t ] to the source t with a smaller weighted prediction loss and a smaller weighted semantic conditional Wasserstein distance .
Intuitively, we tend to leverage the source task which is semantic similar to the target and easier to learn.
(2) If each source have equal observations with β t = 1 , then the third term will become (cid:107) λ (cid:107) 2 , a L 2 norm regularization, which can be viewed as an encouragement of uniformly leveraging all the sources.
Combing these three terms, we need to consider the trade-off between assigning a higher λ [ t ] to the source t that has a smaller weighted prediction loss and conditional Wasserstein distance, and assigning balanced λ [ t ] for avoiding concentrating on only one source.
(3) (cid:107) ˆ α t − α t (cid:107) 2 indicates the gap between ground-truth and empirical label ratio.
Therefore if we can estimate a good ˆ α t , these terms can be small.
In the practice, If target labels are available, ˆ α t can be computed from the observed data and ˆ α t → α t .
If target labels are absent (unsupervised DA), we need to design methods and to properly estimate ˆ α t (Sec. 4).
(4) Comp ( N S 1 , . . .
, N S T , N T , δ ) is a function that reﬂects the convergence behavior, which decreases with larger observation numbers.
If we ﬁx H , δ ,
N and N T , this term can be viewed as a constant.
Insights in Representation Learning Apart from Theorem 1, we propose a novel theoretical analysis in the context of representation learning , which motivates practical guidelines in the deep learning regime.
We deﬁne a stochastic feature function g and we denote its conditional distribution w.r.t.
latent variable Z (induced by g ) as S ( z | Y = y )
= (cid:82) x g ( z | x ) S ( x | Y
=
y ) dx .
Then we have:
Theorem 2.
We assume the settings of loss, the hypothesis are the same with Theorem 1.
We further denote the stochastic feature learning function g : X → Z , and the hypothesis h :
Z × Y → R .
Then ∀ λ , the target risk is upper bounded by:
[Equation]
where R T ( h, g ) =
E ( x,y ) ∼T
( x,y ) E
z ∼ g
( z | x ) (cid:96) ( h ( z, y )) .
Theorem 2 reveal that to control the upper bound, we need to learn g that minimizes the weighted conditional Wasserstein distance and learn ( g, h ) that minimizes the weighted source risk.
Comparison with previous Theorems.
Our theory proposed an alternative prospective to understand transfer learning.
The ﬁrst term is α -weighted loss.
And it will recover the typical source loss minimization if there is no label shift with α t ( y ) ≡ 1 (Li et al., 2019a; Peng et al., 2019; Zhao et al., 2018; Wen et al., 2020).
Beside, minimizing the conditional Wasserstein distances has been shown to be advantageous, compared with W 1 ( S t ( z ) (cid:107)T ( z ))
(Long et al., 2018).
Moreover, Theorem 2 explicitly proposed the theoretical insights about the representation learning function g , which remains elusive for previous multi-source transfer theories such as (Wang et al., 2019a; Mansour et al., 2020;
Konstantinov & Lampert, 2019; Li et al., 2019a; Peng et al., 2019).
[Table]
U NIFIED P RACTICAL F RAMEWORK IN D EEP L EARNING
The theoretical results in Section 3 motivate general principles to follow when designing multisource transfer learning algorithms.
We summarize those principles in the following rules.
(I) Learn a g that minimizes the weighted conditional Wasserstein distance as well as learn ( g, h ) that minimizes the ˆ α t -weighted source risk (Sec. 4.1).
(II) Properly estimate the label distribution ratio ˆ α t (Sec. 4.2).
(III) Balance the trade-off between assigning a higher λ [ t ] to the source t that has a smaller weighted prediction loss and conditional Wasserstein distance, and assigning balanced λ [ t ] .
(Sec. 4.3).
We instantiate these rules with a uniﬁed practical framework for solving multi-source transfer learning problems, as shown in Tab 1.
We would like to point out that our original theoretical result is based on setting with the available target labels.
The proposed algorithm can be applied to unsupervised scenarios under additional assumptions.
4. G UIDELINES IN THE REPRESENTATION LEARNING
Motivated by Theorem 2, given a ﬁxed label ratio estimation ˆ α t and ﬁxed λ , we should ﬁnd a representation function g : X → Z and a hypothesis function h : Z × Y → R such that:
[Equation]
Explicit Conditional Loss
When target label information is available, one can explicitly solve the conditional optimal transport problem with g and h for a given Y =
y .
However, due to the high computational complexity in solving T × |Y| optimal transport problems, the original form is practically intractable.
To address this issue, we propose to approximate the conditional distribution on latent space Z as Gaussian distribution with identical Covariance matrix such that ˆ S t ( z | Y =
y )
≈
N ( C yt , Σ ) and ˆ T ( z | Y =
y )
≈ N ( C y , Σ ) .
Then we have W 1 ( ˆ S t ( z | Y =
y ) (cid:107) ˆ T ( z | Y = y )) ≤ (cid:107) C yt − C y (cid:107) 2 (see Appendix G for details).
Intuitively, the approximation term is equivalent to the well known feature mean matching (Sugiyama & Kawanabe, 2012), which computes the feature centroid of each class (on latent space Z ) and aligns them by minimizing their L 2 distance.
Implicit Conditional Loss When target label information is not available (e.g. unsupervised
DA and partial DA), the explicit matching approach can adopt pseudo-label predicted by the hypothesis h as a surrogate of the true target label.
However, in the early stage of the learning process, the pseudo-labels can be unreliable, which can lead to an inaccurate estimate of
W 1 ( ˆ S ( z | Y =
y ) (cid:107) ˆ T ( z | Y = y )) .
To address this, the following Lemma indicates that estimating the conditional Wasserstein distance is equivalent to estimating the Wasserstein adversarial loss weighted by the label-distribution ratio.
Lemma 1.
The weighted conditional Wasserstein distance can be implicitly expressed as:
[Equation]
)] , where ¯ α t ( z ) = 1 { ( z,y ) ∼S t } α t ( Y = y ) , and d 1 , . . .
, d T : Z → R + are the 1 -Lipschitz domain discriminators (Ganin et al., 2016; Arjovsky et al., 2017).
Lemma 1 reveals that instead of using pseudo-labels to estimate the weighted conditional Wasserstein distance, one can train T domain discriminators with weighted Wasserstein adversarial loss, which does not require the pseudo-label of each target sample during the matching.
On the other hand, ¯ α t can be obtained from ˆ α t , which will be elaborated in Sec. 3.2.
In practice, we adopt a hybrid approach by linearly combining the explicit and implicit matching strategies for all the scenarios, in which empirical results show its effectiveness.
4. E STIMATE LABEL DISTRIBUTION RATIO ˆ α t
Multi-Source Transfer with target labels When the target labels are available, ˆ α t can be directly estimated from the data without any assumption and ˆ α t → α t can be proved from asymptotic statistics.
Unsupervised Multi-Source DA
In this scenario, it is impossible to estimate a good ˆ α t without imposing any additional assumptions.
Following (Zhang et al., 2013; Lipton et al., 2018;
Azizzadenesheli et al., 2019; Combes et al., 2020), we assume that the conditional distributions are aligned between the target and source domains (i.e., S t ( z | y )
=
T ( z | y ) ).
Then, we denote ¯
S t ( y ) , ¯ T ( y ) as the predicted t -source/target label distribution through the hypothesis h , and also deﬁne C ˆ S t [ y, k ] =
ˆ S t [ argmax y (cid:48) h
( z, y (cid:48) )
=
y, Y = k ] is the t -source prediction confusion matrix .
We can demonstrate that if the conditional distribution is aligned, we have ¯ T ( y ) =
¯ T ˆ α t ( y ) , with ¯ T ˆ α t ( Y = y ) = (cid:80)
Y k
=1 C ˆ S t
[ y, k ]ˆ α t ( k ) the constructed target prediction distribution from the t -source information.
(See Appendix I for the proof).
Then we can estimate ˆ α t through matching these two distributions by minimizing D KL ( ¯ T ( y ) (cid:107) ¯ T
ˆ α t ( y )) , which is equivalent to:
[Equation]
In the aforementioned part, we have assumed the conditional distribution is aligned, which is a feasible requirement in our algorithm, since the goal of g exactly aims at gradually achieving this.
In the experiments, we iteratively estimate ˆ α t and learn g .
Unsupervised Multi-Source Partial DA
When supp ( T ( y )) ⊆
supp ( S t ( y )) , α t is sparse due to the non-overlapped classes.
Accordingly, in addition to the assumption of S t ( z | y )
=
T ( z | y ) as in unsupervised DA, we also impose such prior knowledge by adding a regularizer (cid:107) ˆ α t (cid:107) 1 to the objective of Eq.
(2) to induce sparsity in ˆ α t (See Appendix J for more details).
In training the neural network, since the non-overlapped classes will be automatically assigned with a small or zero ˆ α t , ( g, h ) will be less affected by the classes with small ˆ α t .
Our empirical results effectively validate its capability in detecting non-overlapping classes and show signiﬁcant improvements over other baselines.
4. E STIMATE TASK RELATION COEFFICIENT λ
Inspired by Theorem 1, given ﬁxed ˆ α t and ( g, h ) , we estimate λ through optimizing the derived upper bound.
[Equation]
∀ t, λ [ t ] ≥ 0 ,
T (cid:88) t = λ [ t ] =(3)
In practice, ˆ R ˆ α t S t ( h, g ) is the weighted empirical prediction error and E y ∼ ˆ T ( y ) W 1 ( ˆ T ( z | Y
=
y ) (cid:107) ˆ S ( z | Y = y )) is approximated by the dynamic feature centroid distance (cid:80) y ¯ T ( y ) (cid:107)
C y t − C y (cid:107)(See Appendix L for details).
Thus, solving λ is a standard convex optimization problem.
4. A LGORITHM D ESCRIPTION
Based on the aforementioned components, we present the description of WADN (Algorithm 1) in the unsupervised scenarios (UDA and Partial DA), which iteratively updates ( g, h ) , ˆ α t , and λ .
When
Require:
Labeled source samples ˆ S 1 , . .
, ˆ S T , Target samples
ˆ T
Ensure: Label distribution ratio ˆ α t , task relation simplex λ .
Feature Function g , Classiﬁer h , Domain critic function d 1 , . . .
, d T , class centroid for source C yt and target C y ( ∀ t =
[1 , T ] , y ∈ Y ).
1: (cid:46) (cid:46) (cid:46) DNN Parameter Training Stage (ﬁxed α t and λ ) (cid:47) (cid:47) (cid:47) 2: for mini-batch of samples ( x S 1 , y S 1 ) ∼ ˆ S 1 , . .
, ( x S T , y S T ) ∼ ˆ S T , ( x T ) ∼ ˆ T do 3:
Predict target pseudo-label ¯ y T =
argmax y h
( g ( x T ) , y ) 4:
Compute source confusion matrix for each batch (un-normalized)
C ˆ S t =
#[ argmax y (cid:48) h
( z, y (cid:48) )
=
y, Y =
k ] ( t = 1 , . .
, T ) 5:
Compute the batched class centroid for source C yt and target C y .
Moving Average for update source/target class centroid: ( (cid:15) 1 = 0 . 7 ) 7:
Update Source class centroid
C yt =
(cid:15) 1 × C yt
+ (1 − (cid:15) 1 ) × C yt 8:
Update Target class centroid
C y =
(cid:15) 1 × C y
+ (1 − (cid:15) 1 ) ×
C y 9:
Updating g, h, d 1 , . .
, d T (SGD and Gradient Reversal), by solving: min g,h max d 1 ,...,
d T (cid:88) t λ [ t ] ˆ R ˆ α t S t ( h, g ) (cid:124) (cid:123)(cid:122) (cid:125)
ClassiﬁcationLoss
+ (cid:15)C 0 (cid:88)
t λ [ t ] E y ∼ ¯ T ( y ) (cid:107) C yt − C y
(cid:107)(cid:124) (cid:123)(cid:122) (cid:125) ExplicitConditionalLoss
+ (1 − (cid:15) )
C 0 (cid:88) t λ [ t ][ E z ∼ ˆ S t ( z ) ¯
α t ( z ) d ( z ) − E z ∼ ˆ T ( z ) d ( z )] (cid:124) (cid:123)(cid:122) (cid:125) ImplicitConditionalLoss 10: end for 11: (cid:46) (cid:46)
(cid:46) Estimation ˆ α t and λ (cid:47) (cid:47) (cid:47) 12: Compute the global(normalized) source confusion matrix
C
[Equation]
argmax
[Equation] [Equation]
) 13: Solve denoted
} Tt
UDA).
14: Update by average
× α
+ (
(cid:15)
) × t 15: Compute the weighted loss and weighted centroid distance, then solve λ (denoted as λ (cid:48) ) from
Sec. 4.3.
And updating λ by moving average: λ = (cid:15) 1 × λ
+ (1 − (cid:15) 1 ) × λ (cid:48) updating λ and α t , we used package CVXPY to optimize the two standard convex losses after each training epoch, then we updating them by using the moving average.
As for WADN under target label information, we did not require pseudo-label and directly compute ˆ α t , shown in Appendix L.
E XPERIMENTS
In this section, we compare proposed approaches with several baselines for the popular tasks.
For all the scenarios, the following baselines are evaluated: (I) Source method applied only labelled source data to train the model.
(II) DANN (Ganin et al., 2016).
We follow the protocol of Wen et al.
(2020) to merge all the source dataset as a global source domain.
(III) MDAN
(Zhao et al., 2018); (IV) MDMN
(Li et al., 2018b); (V) M 3 SDA (Peng et al., 2019) adopted maximizing classiﬁer discrepancy (Saito et al., 2018) and (VI) DARN
(Wen et al., 2020).
For the conventional multi-source transfer and partial unsupervised multi-source DA, we additionally compare speciﬁc baselines.
All the baselines are re-implemented in the same network structure for fair comparisons.
The detailed network structures, hyper-parameter settings, training details are put in Appendix M.
We evaluate the performance on three different datasets: (I) Amazon Review.
(Blitzer et al., 2007)
It contains four domains (Books, DVD, Electronics, and Kitchen) with positive and negative product reviews.
We follow the common data pre-processing strategies as (Chen et al.
(2012)) to form a 5000 -dimensional bag-of-words feature.
Note that the label distribution in the original dataset is uniform.
To enhance the beneﬁts of the proposed approach, we create a label distribution drifted task by randomly dropping 50% negative reviews of all the sources while keeping the target identical.
(show in Fig.3 (a)).
(II) Digits .
It consists four digits recognition datasets including MNIST, USPS (Hull, 1994), SVHN (Netzer et al., 2011) and Synth (Ganin et al., 2016).
We also create a slight label distribution drift for the sources by randomly dropping 50% samples on digits 5-9 and keep target identical.
(showed in Fig.(3)(b)).
(III) Ofﬁce-Home Dataset (Venkateswara et al., 2017).
It contains 65 classes for four different domains: Art, Clipart, Product and Real-World.
We used the ResNet50 (He et al., 2016) pretrained from the ImageNet in PyTorch as the base network for feature learning and put a MLP for the classiﬁcation.
The label distributions in these four domains are different and we did not manually create a label drift (showed in Fig.3 (c)).
5. U NSUPERVISED M ULTI -S OURCE DA
In the unsupervised multi-source DA, we evaluate the proposed approach on all the three datasets.
We use a similar hyper-parameter selection strategy as in DANN (Ganin et al., 2016).
All reported results are averaged from ﬁve runs.
The detailed experimental settings are illustrated in Appendix
M. The empirical results are illustrated in Tab. 7, 2 and 3.
Since we did not change the target label distribution throughout the whole experiments, then we still use the target accuracy as the metric.
We report the means and standard deviations for each approach.
The best approaches based on a two-sided Wilcoxon signed-rank test (signiﬁcance level p = 0 . 05 ) are shown in bold.
[Table] [Table]
The empirical results reveal a signiﬁcantly better performance ( ≈ 3% ) on different datasets.
For understanding the working principles of WADN, we evaluate the performance under different levels of source label shift in Amazon Review dataset (Fig.1(a)).
The results show strong practical beneﬁts for WADN during a gradual larger label shift.
In addition, we visualize the task relations in digits (Fig.1(b)) and demonstrate a non-uniform λ , which highlights the importance of properly choosing the most related source rather than simply merging all the data.
E.g. when the target domain is
SVHN, WADN mainly leverages the information from SYNTH, since they are more semantically similar and MNIST does not help too much for SVHN (observed by Ganin et al.
The additional analysis and results can be found in Appendix O.
5. M ULTI -S OURCE T RANSFER L EARNING WITH L IMITED T ARGET S AMPLES
We adopt Amazon Review and Digits in the multi-source transfer learning with limited target samples, which have been widely used.
In the experiments, we still use shifted sources.
We randomly sample only 10% labeled samples (w.r.t.
target dataset in unsupervised DA) as training set and the rest 90% samples as the unseen target test set.
(See Appendix M for details).
We adopt the same
[Figure]
(c) hyper-parameters and training strategies with unsupervised DA.
We speciﬁcally add a recent baseline RLUS (Konstantinov & Lampert, 2019) and MME (Saito et al., 2019), which also considered transfer learning with the labeled target.
[Table] [Table]
The results are reported in Tabs.
4, 5, which also indicates strong empirical beneﬁts.
To show the effectiveness of WADN, we select various portions of labelled samples ( 1% ∼ 10% ) on the target.
The results in Fig.1(c) on USPS dataset shows a consistently better than the baselines, even in the few target samples.
5. P ARTIAL U NSUPERVISED M ULTI -S OURCE DA
In this scenario
[Table]
we adopt the Ofﬁce-Home dataset to evaluate our approach, as it contains large (65) classes.
We do not change the source domains and we randomly choose 35 classes from the target.
We evaluate all the baselines on the same selected classes and repeat 5 times.
All reported results are averaged from 3 different sub-class selections (15 runs in total), showing in Tab.6.
(See
Appendix M for details.)
We additionally compare PADA (Cao et al., 2018) approach by merging all the sources and use one-to-one partial DA algorithm.
We adopt the same hyper-parameters and training strategies with unsupervised DA scenario.
[Table]
[Figure]
The reported results are also signiﬁcantly better than the current multi-source DA or one-to-one partial DA approach, which veriﬁes the beneﬁts of WADN: properly estimating ˆ α t and assigning proper λ for each source.
[Figure]
Besides, we change the number of selected classes (Fig 2(a)), the proposed WADN still indicates consistent better results by a large margin, which indicates the importance of considering ˆ α t and λ .
In contrast, DANN shows unstable results in less selected classes.
(See Appendix P for details)
Beside, WADN shows a good estimation of the label distribution ratio (Fig 2(b)) and has correctly detected the non-overlapping classes, which indicates its good explainability.
C ONCLUSION
In this paper, we proposed a new theoretical principled algorithm WADN (Wasserstein Aggregation
Domain Network) to solve the multi-source transfer learning problem under target shift.
WADN provides a uniﬁed solution for various deep multi-source transfer scenarios: learning with limited target data, unsupervised
DA, and partial unsupervised DA.
We evaluate the proposed method by extensive experiments and show its strong empirical results.
A A DDITIONAL E MPIRICAL R ESULTS
[Table]
B A DDITIONAL R ELATED W ORK
Multi-source transfer learning Practice has been proposed from various prospective.
The key idea is to estimate the importance of different sources and then select the most related ones, to mitigate the inﬂuence of negative transfer.
In the multi-source unsupervised DA, (Sankaranarayanan et al., 2018;
Balaji et al., 2019; Pei et al., 2018;
Zhao et al., 2019; Zhu et al., 2019; Zhao et al., 2020; 2019; Stojanov et al., 2019; Li et al., 2019b; Wang et al., 2019b; Lin et al., 2020) proposed different practical strategies in the classiﬁcation, regression and semantic segmentation problems.
In the presence of target labels, Hoffman et al.
(2012); Tan et al.
; Wei et al.
; Yao & Doretto (2010); Konstantinov & Lampert (2019) used generalized linear model to learn the target.
Christodoulidis et al.
; Li et al.
(2019a)
; Chen et al.
(2019) focused on deep learning approaches and Lee et al.
(2019) proposed an ad-hoc strategy to combine to sources in the few-shot target domains.
These ideas are generally data-driven approaches and do not analyze the why the proposed practice can control the generalization error.
Label-Partial Transfer Learning Label-Partial can be viewed as a special case of the label-shift.
Most existing works focus on one-to-one partial transfer learning (Zhang et al., 2018;
Chen et al., 2020; Bucci et al., 2019; Cao et al., 2019) by adopting the re-weighting training approach without a formal understanding.
In our paper, we ﬁrst rigorously analyzed this common practice and adopt the label distribution ratio as its weights, which provides a principled approach in this scenario.
B. O THER SCENARIOS RELATED TO M ULTI -S OURCE T RANSFER L EARNING Domain
Generalization The domain generalization (DG) resembles multi-source transfer but aims at different goals.
A common setting in DG is to learn multiple source but directly predict on the unseen target domain.
The conventional DG approaches generally learn a distribution invariant features (Balaji et al., 2018;
Saenko et al., 2010; Motiian et al., 2017; Ilse et al., 2019) or conditional distribution invariant features (Li et al., 2018a; Akuzawa et al., 2019).
However, our theoretical results reveal that in the presence of label shift (i.e α t ( y ) (cid:54) = 1 ) and outlier tasks then learning conditional or marginal invariant features can not guarantee a small target risk.
Our theoretical result enables a formal understanding about the inherent difﬁculty in DG problems.
Few
-Shot Learning
The few-shot learning (Finn et al., 2017; Snell et al., 2017; Sung et al., 2018) can be viewed as a very speciﬁc scenario of multi-source transfer learning.
We would like to point out the differences between the few-shot learning and our paper.
(1) Few-shot learning generally involves a very large set of source domains T (cid:29) 1 and each domain consists a modest number of observations N S t .
In our paper, we are interested in the a modest number of source domains T but each source domain including a sufﬁcient large number of observations ( N S t (cid:29) 1 ).
In the target domain, the few-shot setting generally used K-samples ( K is very small) for each class for the ﬁne-tuning.
We would like to point out this setting generally violates our theoretical assumption.
In our paper, we assume the target data is i
[Figure]
i.d. sampled from D ( x, y ) .
It is equivalently viewed that we ﬁrst i.i.d.
sample y ∼ D ( y ) , then i.i
[Figure]
d. sample x ∼ D ( x | y ) .
Generally the D ( y ) is non-uniform , thus few-shot setting are generally not applicable for our theoretical assumptions.
Multi-Task Learning The goal of multi-task learning (Zhang & Yang, 2017) aims to improve the prediction performance of all the tasks.
In our paper, we aim at controlling the prediction risk of a speciﬁed target domain.
We also notice some practical techniques are common such as the shared parameter (Zhang & Yeung, 2012), shared representation (Ruder, 2017)
[Figure]
etc.
C A DDITIONAL F IGURES RELATED TO THE MAIN PAPER
[Figure]
randomly drop 50% of the negative reviews in all the sources while keeping target label distribution unchanged.
(b) One example in Digits dataset with
Sources: MNIST, USPS, SVHN and Target Synth.
We randomly drop 50% data on digits 5-9 in all sources while keeping target label distribution unchanged.
(c) Ofﬁce-Home dataset.
The original label distribution is non-uniform.
See Appendix M for details.
D T ABLE OF N OTATION
[Table]
E P ROOF OF T HEOREM
Proof idea Theorem 1 consists three steps in the proof:
Lemma 2.
If the prediction loss is assumed as L -Lipschitz and the hypothesis is K -Lipschitz w.r.t.
the feature x (given the same label),
i
[Table]
e. for ∀ Y =
y , (cid:107) h ( x 1 , y ) − h
( x 2 , y ) (cid:107) 2 ≤ K (cid:107) x 1 − x 2 (cid:107) 2 .
Then the target risk can be upper bounded by:
[Equation]
Proof.
The target risk can be expressed as:
[Equation]
By denoting α ( y ) =
T ( y ) S
( y ) , then we have:
[Equation]
Then we aim to upper bound E x ∼T ( x | y ) (cid:96) ( h ( x, y )) .
For any ﬁxed y ,
[Equation]
y )
[Equation]
Then according to the Kantorovich-Rubinstein duality, for any distribution coupling γ ∈ Π( T ( x | y ) , S ( x | y )) , then we have:
[Equation]
The ﬁrst inequality is obvious; and the second inequality comes from the assumption that (cid:96) is LLipschitz; the third inequality comes from the hypothesis is K -Lipschitz w.r.t.
the feature x (given the same label), i.e. for ∀ Y =
y , (cid:107) h ( x 1 , y ) − h
( x 2 , y ) (cid:107) 2 ≤ K (cid:107) x 1 − x 2 (cid:107) 2 .
Then we have:
[Equation]
Supposing each source S t we assign the weight λ
[ t ] and label distribution ratio α t ( y ) =
T ( y ) S t ( y ) , then by combining this T source target pair, we have:
[Equation]
Then we will prove Theorem 1 from this result, we will derive the non-asymptotic bound, estimated from the ﬁnite sample observations.
Supposing the empirical label ratio value is ˆ α t , then for any simplex λ we can prove the high-probability bound.
E. B OUNDING THE EMPIRICAL AND EXPECTED PREDICTION RISK
Proof.
We ﬁrst bound the ﬁrst term, which can be upper bounded as:
[Equation] [Equation]
(cid:124)
[Equation]
(cid:125) ( II )
Bounding term ( I )
According to the McDiarmid inequality, each item changes at most
| .
Then we have:
[Equation]
By substituting δ , at high probability 1 − δ we have:
[Equation]
Where L max = sup h ∈H (cid:96) ( h ) and N = (cid:80) Tt =1 N S t the total source observations and β t = N
S t N the frequency ratio of each source.
[Equation]
constant
[Equation]
Bounding E sup( I ) , the expectation term can be upper bounded as the form of Rademacher Complexity:
[Equation]
Where ¯ R ( (cid:96), H ) =
sup t R t ( (cid:96), H ) = sup
[Equation]
sup
[Equation]
∼H E
[Equation]
S t ,
[Equation]
(cid:80) ( x t
[Equation]
y t )
[Equation]
S t
[Equation]
TN [ α t ( y ) (cid:96) ( h ( x t , y t ))] , represents the Rademacher complexity w.r.t.
the prediction loss (cid:96) , hypothesis h and true label distribution ratio α t .
Therefore with high probability 1 − δ , we have:
[Equation]
Bounding Term ( II ) For all the hypothesis h , we have:
[Equation]
= y ))
¯ (cid:96) ( Y = y ) |
Where ¯ (cid:96) ( Y = y ) = (cid:80) N S t i (cid:96) ( h ( x i , y i = y )) , represents the cumulative error, conditioned on a given label Y =
y .
According to the Holder inequality, we have:
[Equation]
Therefore
[Equation]
∀ h ∈ H
[Equation]
with high probability 1 − δ we have:
[Equation]
E. B OUNDING EMPIRICAL W ASSERSTEIN D ISTANCE
Then we need to derive the sample complexity of the empirical and true distributions, which can be decomposed as the following two parts.
For any t , we have:
[Equation]
Bounding ( I )
We have:
[Equation]
The ﬁrst inequality holds because of the Holder inequality.
As for the second inequality, we use the triangle inequality of Wasserstein distance.
W 1 ( P (cid:107) Q
[Equation]
≤
[Equation]
P
[Equation]
1 ( P 1 (cid:107) P 2 ) + W 1 ( P 2 (cid:107) Q ) .
According to the convergence behavior of Wasserstein distance (Weed et al., 2019)
[Equation]
with high probability ≥ 1 − 2 δ we have:
[Equation]
Where k ( δ, N y S t , N y T ) =
C t,y ( N y S t ) − s t,y + C y ( N y T ) − s y + (cid:113) 12 log( 2 δ )( (cid:113) 1 N y S t + (cid:113) 1 N yt ) , where N y S t is the number of Y =
y in source t and N y T is the number of Y =
y in target distribution.
C t,y , C y s t,y > 2 , s y > 2 are positive constant in the concentration inequality.
This indicates the convergence behavior between empirical and true Wasserstein distance.
If we adopt the union bound (over all the labels) by setting δ ← δ/ |Y| , then with high probability ≥ 1 − 2 δ , we have:
[Equation]
where κ ( δ, N y S t , N y T ) =
C t,y ( N y S t ) − s t,y +
[Equation]
Again by adopting the union bound (over all the tasks) by setting δ ← δ/T , with high probability ≥ 1 − 2 δ , we have:
[Equation]
Where κ ( δ, N y S t , N y T ) =
C t,y ( N y S t ) − s t,y + C y ( N y T ) − s y + (cid:113) 12 log(
[Equation]
Bounding ( II ) We can bound the second term:
[Equation]
Where C t max = sup y W 1 ( ˆ T ( x | Y =
y ) (cid:107) ˆ S ( x | Y = y )) is a positive and bounded constant.
Then we need to bound | (cid:80) y T ( y ) − ˆ T ( y ) | , by adopting MicDiarmid’s inequality, we have at high probability 1 − δ :
[Equation]
Then we bound E σ E ˆ T (cid:80) y
σ ˆ T ( y ) .
We use the properties of Rademacher complexity [Lemma 26.11, (Shalev-Shwartz & Ben-David, 2014)] and notice that ˆ T ( y ) is a probability simplex, then we have:
[Equation]
Then we have | (cid:80) y T ( y ) − ˆ
[Equation]
( y ) | ≤
[Equation]
|
[Equation] [Equation]
Then using the union bound and denoting δ ← δ/T , with high probability ≥ 1 − δ and for any simplex λ , we have:
[Equation]
where C max = sup t C t max .
Combining together
[Equation]
we can derive the PAC-Learning bound
[Equation]
which is estimated from the ﬁnite samples (with high probability 1 − 4 δ ):
[Equation]
Then we denote Comp ( N S 1 , . .
, N T , δ )
= 2 ¯ R ( (cid:96), h ) + sup t κ ( δ, N y S t , N y T ) + C max ( (cid:113) 2log(2 |Y| ) N T + (cid:113) log( T/δ ) 2 N T ) as the convergence rate function that decreases with larger N S 1 , . . .
, N T .
Bedsides, ¯ R ( (cid:96), h ) =
sup t R t ( (cid:96), H ) is the re-weighted Rademacher complexity.
Given a ﬁxed hypothesis with ﬁnite VC dimension 2 , it can be proved ¯ R ( (cid:96), h ) =
min N S 1 ,...,N S T O ( (cid:113) 1 N S t )
i.e (Shalev-Shwartz &
Ben-David, 2014).
F P ROOF OF T HEOREM
We ﬁrst recall the stochastic feature representation g such that g : X → Z and scoring hypothesis h h :
Z × Y → R and the prediction loss (cid:96) with (cid:96) : R → R .
Proof.
The marginal distribution and conditional distribution w.r.t.
latent variable Z that are induced by g , which can be reformulated as:
[Equation]
In the multi-class classiﬁcation problem, we additionally deﬁne the following distributions:
[Equation]
Based on (Nguyen et al., 2009) and g ( z | x ) is a stochastic representation learning function, the loss conditioned a ﬁxed point ( x, y ) w.r.t.
h and g is E z ∼ g
( z | x ) (cid:96) ( h ( z, y )) .
Then taking the expectation over the S ( x, y ) we have:
[Equation]
Intuitively, the expected loss w.r.t.
the joint distribution S can be decomposed as the expected loss on the label distribution S ( y ) (weighted by the labels) and conditional distribution S ( ·| y )
(real valued conditional loss).
Then the expected risk on the S and T can be expressed as:
[Equation]
g ←− X .
(S is the output of the scoring function).
Then the expected loss over the all random variable can be equivalently written as (cid:82) P ( x, y, z, s ) (cid:96) ( s ) d ( x, y, z, s ) =
(cid:82) P ( x ) P ( y | x ) P ( z | x ) P ( s |
z, y ) (cid:96) ( s ) =
(cid:82) P ( x, y ) P ( z | x ) P ( s |
z, y ) (cid:96) ( s ) d
( x, y ) d ( z ) d ( s ) .
Since the scoring S is determined by h ( x, y ) , then P ( s | y, z )
= 1 .
According to the deﬁnition we have P ( z | x ) =
g ( z | x ) , P ( x, y ) =
S ( x, y ) , then the loss can be ﬁnally expressed as E S ( x,y ) E g
( z | x ) (cid:96) ( h ( z, y ))
By denoting α ( y ) =
T ( y ) S
( y ) , we have the α -weighted loss:
[Equation]
Then we have:
[Equation]
Under the same assumption, we have the loss function (cid:96) ( h
[Equation]
z, Y = k )) is KL-Lipschitz w.r
[Equation]
t. the cost (cid:107)·(cid:107) 2 (given a ﬁxed k ).
Therefore by adopting the same proof strategy (Kantorovich-Rubinstein duality) in Lemma 2, we have ≤
[Equation]
Therefore, we have:
[Equation]
Based on the aforementioned result, we have ∀ t = 1 , . .
, T and denote S
= S t and α ( y )
=
α t ( y ) =
T ( y ) / S t ( y ) :
[Equation]
Summing over t = 1 , . .
, T , we have:
[Equation]
G A PPROXIMATION W 1 DISTANCE
According to Jensen inequality
[Equation]
we have
[Equation]
Supposing ˆ S t ( z | Y =
y )
≈ N ( C yt
[Equation]
Σ ) and ˆ T ( z | Y =
y )
≈ N ( C y , Σ )
[Equation]
then we have:
[Equation]
We would like to point out that assuming the identical covariance matrix is more computationally efﬁcient during the matching.
This is advantageous and reasonable in the deep learning regime: we adopted the mini-batch (ranging from 20-128) for the neural network parameter optimization, in each mini-batch the samples of each class are small , then we compute the empirical covariance/variance matrix will be surely biased to the ground truth variance and induce a much higher complexity to optimize.
By the contrary, the empirical mean is unbiased and computationally efﬁcient, we can simply use the moving the moving average to efﬁciently update the estimated mean value (with a unbiased estimator).
The empirical results verify the effectiveness of this idea.
H P ROOF OF L EMMA
For each source S t , by introducing the duality of Wasserstein-1 distance, for y ∈ Y , we have:
[Equation]
Then by deﬁning ¯
α t ( z ) =
( z,y ) ∼S t } T ( Y = y ) S t ( Y = y )
=
1 { ( z,y ) ∼S t } α t ( Y = y ) , we can see for each pair observation ( z, y ) sampled from the same distribution, then ¯ α t ( Z = z )
=
α t ( Y = y ) .
Then we have:
[Equation]
We propose a simple example to understand ¯ α t : supposing three samples in S t
= { ( z 1 , Y = 1) , ( z 2 , Y = 1) , ( z 3 , Y = 0) } then ¯ α t ( z 1 ) =
¯ α t ( z 2 )
=
α t (1) and ¯ α t ( z 3 ) =
α t (0) .
Therefore, the conditional term is equivalent to the label-weighted Wasserstein adversarial learning.
We plug in each source domain as weight λ [ t ] and domain discriminator as d t , we ﬁnally have Lemma 1.
I D ERIVE THE LABEL RATIO LOSS
We suppose the representation learning aims at matching the conditional distribution such that
T ( z | y )
≈ S t ( z | y ) , ∀ t , then we suppose the predicted target distribution as ¯ T ( y ) .
By simplifying the notation, we deﬁne f ( z ) =
argmax y h
( z, y ) the most possible prediction label output, then we have:
[Equation]
The ﬁrst equality comes from the deﬁnition of target label prediction distribution, ¯ T ( y ) =
E T
[Equation]
z
[Equation]
1 { f ( z
[Equation]
= y
[Equation]
= T
[Equation]
f
[Equation]
z
[Equation]
= y
[Equation]
Y k =
[Equation]
Y
[Equation]
k
[Equation] [Equation]
f
[Equation]
y | Y
[Equation]
k ) T ( Y = k ) .
The second equality T ( f ( z ) = y | Y = k ) =
S t ( f ( z ) =
y | Y = k ) holds since ∀ t , T ( z | y )
≈
S t ( z | y ) , then for the shared hypothesis f , we have T ( f ( z ) =
y | Y = k ) =
S t ( f ( z ) =
y | Y = k ) .
The term S t ( f ( z ) =
y, Y = k ) is the (expected) source prediction confusion matrix, and we denote its empirical (observed) version as ˆ S t ( f ( z ) =
y, Y = k ) .
Based on this idea, in practice we want to ﬁnd a ˆ α t to match the two predicted distribution ¯ T and ¯
T ˆ α t .
If we adopt the KL-divergence as the metric, we have:
[Equation]
We should notice the nature constraints of label ratio: { ˆ α t ( y ) ≥ 0 , (cid:80)
y
ˆ α t ( y ) ˆ S t ( y ) = 1 } .
Based on this principle, we proposed the optimization problem to estimate each label ratio.
We adopt its empirical counterpart, the empirical confusion matrix C ˆ S t [ y, k ]
= ˆ S t [ f ( z ) =
y, Y =
k ] , then the optimization loss can be expressed as:
[Equation]
[Figure]
J L ABEL P ARTIAL M ULTI - SOURCE UNSUPERVISED DA
The key difference between multi-conventional and partial unsupervised DA is the estimation step of ˆ α t .
In fact, we only add a sparse constraint for estimating each ˆ α t :
[Equation]
Where C 2 is the hyper-parameter to control the level of target label sparsity
[Equation]
to estimate the target label distribution.
In the paper
[Equation]
we denote C 2 = 0 .
K E XPLICIT AND I MPLICIT CONDITIONAL LEARNING
Inspired by Theorem
[Equation]
we need to learn the function g : X → Z and h : Z × Y → R to minimize:
[Equation]
This can be equivalently expressed as:
[Equation]
Due to the explicit and implicit approximation of conditional distance
[Equation]
we then optimize an alternative form:
[Equation]
Where
• C yt = (cid:80) ( z t ,y t ) ∼ ˆ S t 1 { y t =
y } z t the centroid of label Y =
y in source S t .
• C y
=
(cid:80) ( z t ,y p )
∼ ˆ T 1 { y p =
y } z t the centroid of pseudo-label Y =
y p in target S t .
(If it is the unsupervised DA scenarios).
• ¯ α t ( z ) =
1 { ( z,y ) ∼S t } ˆ α t ( Y = y ) , namely if each pair observation ( z, y ) from the distribution, then ¯ α t ( Z = z )
=
ˆ α t ( Y = y ) .
• d 1 , · · · , d T are domain discriminator (or critic function) restricted within 1 -Lipschitz function.
• (cid:15) ∈
[0 , 1] is the adjustment parameter in the trade-off of explicit and implicit learning.
Based on the equivalence form, our approach proposed a theoretical principled way to
tuning its weights.
In the paper
, we assume (cid:15) = 0 .
• ˆ T ( y ) empirical target label distribution.
(In the unsupervised DA scenarios, we approximate it by predicted target label distribution ¯ T ( y ) .)
Gradient Penalty
In order to enforce the Lipschitz property of the statistic critic function, we adopt the gradient penalty term (Gulrajani et al., 2017).
More concretely, given two samples z s ∼
S t ( z ) and z t ∼ T ( z ) we generate an interpolated sample z int = ξz s + (1 − ξ )
z t with ξ ∼ Unif [0 , 1] .
Then we add a gradient penalty (cid:107)∇ d ( z int ) (cid:107) 22 as a regularization term to control the Lipschitz property w.r.t.
the discriminator d 1 , · · · , d T .
L A LGORITHM D ESCRIPTIONS
We propose a detailed pipeline of the proposed algorithm in the following, shown in Algorithmand 3.
As for updating λ and α t , we iteratively solve the convex optimization problem after each training epoch and updating them by using the moving average technique.
For solving the λ and α t , we notice that frequently updating these two parameters in the mini-batch level will lead to an instability result during the training.
5 As a consequence, we compute the accumulated confusion matrix, weighted prediction risk, and conditional Wasserstein distance for the whole training epoch and then solve the optimization problem.
We use CVXPY to optimize the two standard convex losses.
Comparison with different time and memory complexity.
We discuss the time and memory complexity of our approach.
Time complexity: In computing each batch we need to compute T re-weighted loss, T domain adversarial loss and T explicit conditional loss.
Then our computational complexity is still ( O )( T ) during the mini-batch training, which is comparable with recent SOTA such as MDAN and DARN.
In addition, after each training epoch we need to estimate α t and λ , which can have time complexity
O ( T |Y| ) with each epoch.
(If we adopt SGD to solve these two convex problems).
Therefore, the our proposed algorithm is time
complexity
O
( T |Y| ) .
The extra Y term in time complexity is due to the approach of label shift in the designed algorithm.
Memory Complexity: Our proposed approach requires O ( T ) domain discriminator and O ( T |Y| ) class-feature centroids.
By the contrary, MDAN and DARN require O ( T ) domain discriminator and M3SDA and MDMN require O
( T 2 ) domain discriminators.
Since our class-feature centroids are deﬁned in the latent space ( z ), then the memory complexity of the class-feature centroids can be much smaller than domain discriminators.
Require:
Labeled source samples ˆ S 1 , . .
, ˆ S T , Target samples
ˆ T
Ensure: Label distribution ratio ˆ α t and task relation simplex λ .
Feature Learner g , Classiﬁer h ,
Statistic critic function d 1 , . . .
, d T , class centroid for source C yt and target C y ( ∀ t =
[1 , T ] , y ∈
Y ).
1: (cid:46) (cid:46) (cid:46) DNN Parameter Training Stage (ﬁxed α t and λ ) (cid:47) (cid:47) (cid:47) 2: for mini-batch of samples ( x S 1 , y S 1 ) ∼ ˆ S 1 , . .
, ( x S T , y S T ) ∼ ˆ S T , ( x T ) ∼ ˆ T do 3:
Predict target pseudo-label ¯ y T =
argmax y h
( g ( x T ) , y ) 4:
Compute source confusion matrix for each batch (un-normalized)
C ˆ S t =
#[ argmax y (cid:48) h
( z, y (cid:48) )
=
y, Y =
k ] ( t = 1 , . .
, T ) 5:
Compute the batched class centroid for source C yt and target C y .
Moving Average for update source/target class centroid: (We set (cid:15) 1 = 0 . 7 ) 7:
Source class centroid update
C yt =
(cid:15) 1 × C yt
+ (1 − (cid:15) 1 ) × C yt 8:
Target class centroid update
C y =
(cid:15) 1 × C y
+ (1 − (cid:15) 1 ) ×
C y 9:
Updating g, h, d 1 , . .
, d T (SGD and Gradient Reversal), based on Eq.(6) 10: end for 11: (cid:46) (cid:46) (cid:46) Estimation
ˆ α t and λ (cid:47) (cid:47) (cid:47) 12: Compute the global(normalized) source confusion matrix
C ˆ S t =
[Equation]
S t [ argmax y (cid:48) h
( z, y (cid:48) )
=
y, Y =
k ] ( t = 1 , . .
. , T ) 13: Solve α t (denoted as { α (cid:48) t } Tt =1 ) by Equation (2) (Or Eq.(5))
in the partial scenario).
14: Update α t by moving average: α t = (cid:15) 1 × α t
+ (1 − (cid:15) 1 ) × α (cid:48)
t 15: Compute the weighted loss and weighted centroid distance, then solve λ (denoted as λ (cid:48) ) from
Sec. 2.3.
16: Updating λ by moving average: λ
8 × + 0 .
Require:
Labeled source samples ˆ S , ˆ S
, Target samples ˆ , Label shift ratio α
Ensure:
Task relation simplex λ .
Feature Learner g , Classiﬁer h , Statistic critic function do 3:
Compute the batched class for
Moving Average for update source centroid:
Source class centroid update
[Equation]
Target class centroid update
[Equation]
1 × C y + (1 − (cid:15) 1 ) ×
C y 7:
Updating g, h, d 1 , . .
, d T (SGD and Gradient Reversal), based on Eq.(6).
8: end for 9: (cid:46) (cid:46) (cid:46) Estimation λ (cid:47) (cid:47) (cid:47) 10: Solve λ by Sec.
(denoted as λ (cid:48) )
11: Updating λ by moving average: λ = (cid:15) 1 × λ
+ (1 − (cid:15) 1 ) × λ (cid:48)
M D ATASET D ESCRIPTION AND E XPERIMENTAL D ETAILS M. A MAZON R EVIEW D ATASET
We used the amazon review dataset (Blitzer et al., 2007).
It contains four domains (Books, DVD,
Electronics, and Kitchen) with positive (label ”1”) and negative product reviews (label ”0”).
The data size is 6465 (Books), 5586 (DVD), 7681 (Electronics), and 7945 (Kitchen).
We follow the common data pre-processing strategies Chen et al.
(2012): use the bag-of-words (BOW) features then extract the top-5000 frequent unigram and bigrams of all the reviews.
We also noticed the original data-set are label balanced D ( y = 0)
=
D ( y = 1) .
To enhance the beneﬁts of the proposed approach, we create a new dataset with label distribution drift.
Speciﬁcally, in the experimental settings, we randomly drop 50% data with label ”0” (negative reviews) for all the source data while keeping the target identical, showing in Fig (5).
We choose the MLP model with
• feature representation function g : [5000 , 1000] units • Task prediction and domain discriminator function [1000 , 500 , 100] units,
We choose the dropout rate as 0 .
7 in the hidden and input layers.
The hyper-parameters are chosen based on cross-validation.
The neural network is trained for 50 epochs and the mini-batch size isper domain.
The optimizer is Adadelta with a learning rate of 0.5.
Experimental Setting We use the amazon Review dataset for two transfer learning scenarios (limited target labels and unsupervised DA).
We ﬁrst randomly select 2K samples for each domain.
Then we create a drifted distribution of each source, making each source ≈ 1500 and target sample still 2K.
In the unsupervised DA, we use these labeled source tasks and unlabelled target task, which aims to predict the labels on the target domain.
In the conventional transfer learning, we random sample only 10% dataset ( ≈ 200 samples) as the target training set and the rest 90% samples as the target test set.
We select C 0 = 0 .
01 and C 1 = 1 for these two transfer scenarios.
In both practical settings, we set the maximum training epoch as 50.
[Figure]
M. D IGIT R ECOGNITION
We follow the same settings of Ganin et al.
and we use four-digit recognition datasets in the experiments MNIST, USPS, SVHN, and Synth.
MNIST and USPS are the standard digits recognition task.
Street View House Number (SVHN)
Ganin et al.
(2016) is the digit recognition dataset from house numbers in Google Street View Images.
Synthetic Digits (Synth)
Ganin et al.
(2016) is a synthetic dataset that by various transforming SVHN dataset.
We also visualize the label distribution in these four datasets.
The original datasets show an almost uniform label distribution on the MNIST as well as Synth, (showing in Fig.
7 (a)).
In our paper, we generate a label distribution drift on the source datasets for each multi-source transfer learning.
Concretely, we drop 50% of the data on digits 5-9 of all the sources while we keep the target label distribution unchanged.
(Fig.
7 (b) illustrated one example with sources: Mnist, USPS, SVHN, and
Target Synth.
We drop the labels only on the sources.)
MNIST and USPS images are resized to 32 × 32 and represented as 3-channel color images to match the shape of the other three datasets.
Each domain has its own given training and test sets when downloaded.
Their respective training sample sizes are 60000, 7219, 73257, 479400, and the respective test sample sizes are 10000, 2017, 26032, 9553.
The model structure is shown in Fig.
There is no dropout and the hyperparameters are chosen based on cross-validation.
It is trained for 60 epochs and the mini-batch size is 128 per domain.
The optimizer is Adadelta with a learning rate of 1.0.
We adopted γ = 0 .
5 for MDAN and γ = 0 .
1 for
DARN in the baseline (Wen et al., 2020).
Experimental Setting We use the Digits dataset for two transfer learning scenarios (limited target labels and unsupervised DA).
Notice the USPS data has only 7219 samples and the digits dataset is relatively simple.
We ﬁrst randomly select 7K samples for each domain.
We create a drifted distribution of each source, making each source ≈ 5300 , and the target sample still 7K.
In the unsupervised DA, we use these labeled source tasks and unlabelled target task, which aims to predict the labels on the target domain.
In the transfer learning with limited data, we random sample only 10% dataset ( ≈ 700 samples) as the target training set and the rest 90% samples as the target test set.
We select C 0 = 0 .
01 and C 1 as the maximum prediction loss C 1 = max t R α t ( h ) as the hyperparameters across these two scenarios.
The maximum training epoch is 60 .
Feature extractor: with 3 convolution layers. ’
layer1’: ’conv’:
[3, 3, 64], ’relu’:
[], ’maxpool’:
[2, 2, 0], ’layer2’: ’conv’:
[3, 3, 128], ’relu’:
[], ’maxpool’:
[2, 2, 0], ’layer3’: ’conv’:
[3, 3, 256], ’relu’:
[], ’maxpool’:
Task prediction: with 3 fully connected layers. ’
layer1’: ’fc’:
[*, 512], ’act fn’: ’relu’, ’layer2’: ’fc’:
[512, 100], ’act fn’
: ’relu’, ’layer3’: ’fc’:
Domain Discriminator: with 2 fully connected layers.
[Figure]
() ’layer1’: ’fc’:
[*, 256], ’act fn’: ’relu’, ’layer2’: ’fc’:
[Figure] [Figure]
We randomly drop 50% data on digits 5-9 in all sources while keeping target label distribution unchanged
[Figure]
M. O FFICE -H OME DATASET
To show the dataset in the complex scenarios
[Figure]
we use the challenging Ofﬁce-Home dataset (Venkateswara et al., 2017).
It contains images of 65 objects such as a spoon, sink, mug
[Figure]
and pen from four different domains: Art (paintings, sketches, and/or artistic depictions), Clipart (clipart images), Product (images without background), and Real-World (regular images captured with a camera).
One of the four datasets is chosen as an unlabelled target domain and the other three datasets are used as labeled source domains.
The dataset size is 2427 (Art), 4365 (Clipart), 4439 (Product), 4357 (Real-World).
We follow the same training/test procedure as (Wen et al., 2020).
We additionally visualize the label distribution
D ( y ) in four domains in Fig.9, which illustrated the inherent different label distributions.
We did not re-sample the source label distribution to uniform distribution in the data pre-processing step.
All the baselines are evaluated under the same setting.
We use the ResNet50 (He et al., 2016) pretrained from the ImageNet in PyTorch as the base network for feature learning and put an MLP with the network structure shown in Fig.
Experimental Settings We use the original Ofﬁce-Home dataset for two transfer learning scenarios (unsupervised DA and label-partial unsupervised DA).
We use SGD optimizer with learning rate 0.005, momentum 0.9 and weight decay value 1e-3.
It is trained for 100 epochs and the mini-batch size is 32 per domain.
As for the baselines, MDAN use γ = 1.0 while DARN use γ = 0.5.
We select
C 0 = 0 .
01 and C 1 as the maximum prediction loss C 1 = max t R α t ( h ) as the hyper-parameters across these two scenarios.
In the multi-source unsupervised partial DA, we randomly select 35 classes from the target (by repeating 3 samplings), then at each sampling we run 5 times.
The ﬁnal result is based on these 3 × 5 = 15 repetitions.
[Figure]
N A NALYSIS ON THE P SEUDO -L ABELS
Task
Domain Discriminator: with 3 fully connected layers
reverse gradient () ’layer1’: ’fc’:
, ’batch normalization’, ’act fn’: ’Leaky relu’, ’layer2’: ’fc’:
[256, 256], ’batch normalization’, ’act fn’: ’Leaky relu’, ’layer3’: ’fc’:
[Figure]
O A NALYSIS IN U NSUPERVISED DA O.
A BLATION S TUDY :
D IFFERENT
D ROPPING R ATE
To show the effectiveness of our proposed approach, we change the drop rate of the source domains, showing in Fig.(12).
We observe that in task Book, DVD, Electronic, and Kitchen, the results are signiﬁcantly better under a large label-shift.
In the initialization with almost no label shift, the state-of-the-art DARN illustrates a slightly better ( < 1% ) result.
O. A DDITIONAL A NALYSIS ON A MAZON D ATASET
We present two additional results to illustrate the working principles of WADN, showing in Fig.
(13) and (14).
We visualize the evolution of λ between DARN and WADN, which both used theoretical principled approach to estimate λ .
We observe that in the source shifted data, DARN shows an inconsistent estimator of λ Fig.
This is different from the observation of Wen et al.
We think it may in the conditional and label distribution shift problem, using ˆ R S
( h ( z ))
+ Discrepancy ( S ( z ) , T ( z )) to update λ is unstable.
In contrast, WADN illustrates a relative consistent estimator of λ under the source shifted data.
In addition, WARN gradually and correctly estimates the unbalanced source data and assign higher wights α t for label y =
(ﬁrst row of Fig.(14)).
These principles in WADN jointly promote signiﬁcantly better results.
[Figure] [Figure]
[Figure] [Figure]
[Figure]
O. A DDITIONAL A NALYSIS ON D IGITS D ATASET
We show the evolution of ˆ α t on WADN, which veriﬁes the correctness of our proposed principle.
Since we drop digits 5-9 in the source domains, the results in Fig.
(15) illustrate a higher ˆ α t on these digits.
P P ARTIAL MULTI - SOURCE U NSUPERVISED DA
From Fig.
(16), WADN is consistently better than other baselines, given different selected classes.
[Figure]
pha
[Figure]
l pha
D
[Figure]
pha
K
[Figure] [Figure]
Besides
[Figure]
when fewer classes are selected
[Figure]
the accuracy in DANN
[Figure]
PADA
[Figure]
and DARN is not drastically dropping but maintaining a relatively stable result.
We think the following possible reasons:
• The reported performances are based on the average of different selected sub-classes
rather than one
sub-
class
selection
From
the
statistical
perspective, if we take a close look at the variance , the results in DANN are much more unstable (higher std) inducing by the different samplings.
Therefore, the conventional domain adversarial training is improper for handling the partial transfer since it is not reliable and negative transfer still occurs.
• In multi-source DA, it is equally important to detect the non-overlapping classes and ﬁnd the most similar sources.
Comparing the baselines that only focus on one or two principles shows the importance of uniﬁed principles in multi-source partial DA.
• We also observe that in the Real-World dataset, the DANN improves the performance by a
relatively large value.
This is due to the inherent difﬁcultly of the learning task itself.
In fact, the Real-World domain illustrates a much higher performance compared with other domains.
According to the Fano lower bound, a task with smaller classes is generally easy to learn .
It is possible the vanilla approach showed improvement but still with a much higher variance.
Fig (17), (18) showed the estimated ˆ α t with different selected classes.
The results validate the correctness of WADN in estimating the label distribution ratio.
[Figure]
A l pha ( C )
A l pha ( P )
X X
X X X X
X
X X X
X
X X X
X X X
X
X X X X
X X
X
X X X X
X X X X X X
X X X X
X
X X X X X
X X X X
X
A l pha ( R ) (
[Figure]
Target:
Art
A l pha ( A )
A l pha ( P )
X X
X X X X
X
X X X
X
X X X
X X X
X
X X X X
X X
X
X X X X
X X X X X X
X X X X
X
X X X X X
X X X X
X
A l pha ( R )
[Figure]
Target
[Figure]
Clipart
A l pha ( A )
A l pha ( C ) x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x
A l pha ( R ) (
[Figure]
Product
A l pha ( A )
A l pha ( C )
X X
X X X X
X
X X X
X
X X X
X X X
X
X X X X
X X
X
X X X X
X X X X X X
X X X X
X
X X X X X
X X X X
X
A l pha ( P ) (
[Figure]
A l pha ( C )
A l pha ( P )
X X XX
X
X X X X X
X X X X
X
X X X X
X
X
X X X
X
X X X
X X
A l pha ( R ) (a) Target:
Art
A l pha ( A )
A l pha ( P )
X X XX
X
X X X X X
X X X X
X
X X X X
X
X
X X X
X
X X X
X X
A l pha ( R ) (b) Target: Clipart
A l pha ( A )
A l pha ( C )
X X XX
X
X X X X X
X X X X
X
X X X X
X
X
X X X
X
X X X
X X
A l pha ( R ) (c) Target: Product
A l pha ( A )
A l pha ( C )
X X XX
X
X X X X X
X X X X X
X X X X
X
X
X X X
X
X X X
X X
A l pha ( P )
[Figure]