D OMAIN A GGREGATION N ETWORKS FOR M ULTI -S OURCE D OMAIN A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In many real-world applications, we want to exploit multiple source datasets of similar tasks to learn a model for a different but related target dataset – e.g., recognizing characters of a new font using a set of different fonts.
While most recent research has considered ad-hoc combination rules to address this problem, we extend previous work on domain discrepancy minimization to develop a ﬁnite-sample generalization bound, and accordingly propose a theoretically justiﬁed optimization procedure.
The algorithm we develop, Domain AggRegation
Network (DARN), is able to effectively adjust the weight of each source domain during training to ensure relevant domains are given more importance for adaptation.
We evaluate the proposed method on real-world sentiment analysis, digit recognition and object recognition datasets and show that DARN can signiﬁcantly outperform the state-of-the-art alternatives.
I NTRODUCTION
Many machine learning algorithms assume the learned predictor will be tested on the data from the same distribution as the training data.
This assumption, although reasonable, is not necessarily true for many real-world applications.
For example, patients from one hospital may have a different distribution of gender, height and weight from another hospital.
Consequently, a diagnostic model constructed at one location may not be directly applicable to another location without proper adjustment.
The situation becomes even more challenging when we want to use data from multiple source domains to build a model for a target domain , as this requires deciding, e.g., how to rank the source domains and how to effectively aggregate these domains when training complex models like deep neural networks.
Including irrelevant or worse, adversarial, data from certain source domains can severely reduce the performance on the target domain, leading to undesired consequences.
To address this problem, researchers have been exploring methods of transfer learning (Pan & Yang, 2009) or domain adaptation (Mansour et al., 2009a; Cortes et al., 2019), where a model is trained based on labelled source data and unlabelled target data.
Most existing works have focused on single-source to single-target (“one-to-one”) domain adaptation, using different assumptions such as covariate shift (Shimodaira, 2000; Gretton et al., 2009; Sugiyama & Kawanabe, 2012) or concept drift (Jiang & Zhai, 2007;
Gama et al., 2014).
When dealing with multiple source domains, one may attempt to directly use these approaches by combining all source data into a large joint dataset and then apply one-to-one adaptation.
This na¨ıve aggregation method will often fail as not all source domains are equally important when transferring to a speciﬁc target domain.
There are some works on multi-source to single-target adaptation.
Although many of them are theoretically motivated with cross-domain generalization bounds, they either use ad-hoc aggregation rules when developing actual algorithms (Zhao et al., 2018; Li et al., 2018) or lack ﬁnite-sample analysis (Mansour et al., 2009b;c; Hoffman et al., 2018a).
This leaves a gap between the theory for multi-source adaptation and theoretically sound algorithm for domain aggregation.
This research has three contributions: First, we extend prior work on one-to-one adaptation using discrepancy (Cortes et al., 2019) to develop a ﬁnite-sample cross-domain generalization bound for multi-source adaptation.
We show that in order to improve performance on the speciﬁc target domain, there is a trade-off between utilizing all source domains to increase effective sample size, versus removing source domains that are underperforming or not similar to the target domain.
Second, motivated by our theory and domain adversarial method (Ganin & Lempitsky, 2015; Ganin et al., 2016), we propose Domain AggRegation Network (DARN), that can effectively aggregate multiple source domains dynamically during the course of training.
Unlike previous works, our aggregation scheme (Eq. (6)), which itself is of independent interest in some other contexts, is a direct optimization of our generalization upper bound (Eq.
(3)) without resorting to heuristics.
Third, our experiments on sentiment analysis, digit recognition and object recognition show that DARN can signiﬁcantly outperform state-of-the-art methods.
Section 2 introduces necessary background on one-to-one adaptation based on discrepancy.
Then
Section 3 elaborates our theoretical analysis and the corresponding algorithm deployment.
Sectiondiscusses about related approaches in more detail, highlighting the key differences to the approach developed here.
Section 5 empirically compares the performance of the proposed method to other alternatives.
Finally, Section 6 concludes this work.
B ACKGROUND ON C ROSS - DOMAIN G ENERALIZATION
This section provides necessary background from previous work on one-to-one domain adaptation (Cortes et al., 2019).
Let X be the input space,
Y ⊆
R be output space and H ⊆
{ h : X (cid:55)→ Y} be a hypothesis set.
A loss function L : Y × Y (cid:55)→ R + is µ -admissible 1 if
[Equation]
The discrepancy (Mansour et al., 2009a) between two distributions P, Q over X is deﬁned as
[Equation]
( x ))] .
This quantity can be computed or approximated empirically given samples from both distributions.
For classiﬁcation problems with 0-1 loss, it reduces to the well-known d A -distance (Kifer et al., 2004; Blitzer et al., 2008; Ben-David et al., 2010) and can be approximated using a domain-classiﬁer loss w.r.t.
H (Zhao et al., 2018; Ben-David et al., 2007, Sec.4), while for regression problems with
L 2 loss, it reduces to a maximum eigenvalue (Cortes & Mohri, 2011, Sec.5); see Section 3.2 for more details.
For two domains ( P, f P ) , ( Q, f Q ) where f P , f Q : X (cid:55)→ Y are the corresponding labeling functions, we have the following cross-domain generalization bound:
Theorem 1 (Proposition 5 & 8, Cortes et al.
Let R m ( H ) be the Rademacher complexity of H given sample size m , H Q
= { x (cid:55)→ L ( h ( x ) , f Q ( x ) ) : h ∈ H} be the set of functions mapping x to its loss w.r.t.
f Q and H ,
[Equation]
be a constant measuring how well H can ﬁt the true models where supp( (cid:98) P ) means the support of the empirical distribution (cid:98) P (using the µ from Eq.
(1) ), and M Q = sup x ∈X ,h ∈H L ( h ( x ) , f Q ( x ) ) be the upper bound on loss for Q .
Given (cid:98) Q with m points sampled iid from Q labelled according to f Q , for δ ∈ (0 , 1) , ∀ h ∈ H , w.p. at least 1 − δ ,
[Equation]
This theorem provides a way to generalize across domains when we have sample (cid:98) Q labelled according to f Q and an unlabelled sample (cid:98) P .
The ﬁrst term is the usual loss function for the sample (cid:98) Q , while the second term disc( P, Q ) can be estimated based on the unlabelled data (cid:98) Q, (cid:98) P .
η H measures how well the model family H can ﬁt the example from the datasets, and it is not controllable once
H is given.
The ﬁnal term, as a function of sample size m , determines the convergence speed.
D OMAIN A GGREGATION N ETWORKS 3. T HEORY
Suppose we are given k source domains { ( S i , f S i ) : i ∈ [ k ] def = { 1 , 2 , . .
. , k }} and a target domain ( T, f T ) where S i , T are distributions over X and f S i , f T : X (cid:55)→ Y are their respective labelling functions.
For simplicity, assume that each sample (cid:98) S i has m points, drawn iid from S i and labelled according to f S i .
We are also given m unlabelled points (cid:98) T drawn iid from T .
We want to leverage all source domains’ information to learn a model h ∈ H minimizing L T ( h, f T ) .
One na¨ıve approach could be to combine all the source domains into a large joint dataset and conduct one-to-one adaptation to the target domain using Theorem 1.
However, including data from irrelevant or even adversarial domains is likely to jeopardize the performance on the target domain, which is sometimes referred to as negative transfer (Pan & Yang, 2009).
Moreover, as certain source domains may be more similar or relevant to the target domain than the others, it makes more sense to adjust their importance according to their utilities.
We propose to ﬁnd domain weight α i ≥such that (cid:80) ki
=1 α i = 1 to achieve this.
Our main theorem below sheds some light on how we should combine the source domains (the proof is provided in Appendix A):
Theorem 2 Given k source domains datasets { ( x ( i ) j , y ( i ) j ) : i ∈
[ k ] , j ∈ [ m ] } with m iid examples each where (cid:98) S
i = {
x ( i ) j } and y ( i )
j = f S i ( x ( i ) j ) , for any α ∈ ∆ = { α : α i ≥ 0 , (cid:80) i
α i = 1 } , δ ∈ (0 , 1) , and ∀ h ∈ H , w.p.
at least 1 − δ
L T ( h, f T ) ≤
[Equation]
where H S i
=
{ x (cid:55)→ L ( h ( x ) , f S i ( x )) :
h ∈ H} is the set of functions mapping x to the corresponding loss, η H ,i is a constant similar to Eq.
(2) with (cid:98) Q = (cid:98) S i , (cid:98) P = (cid:98) T and M S = sup i ∈
[ k ] ,x ∈X ,h ∈H L ( h ( x ) , f S i ( x )) is the upper bound on loss on the source domains.
There are several observations.
In the last term of the bound, m/ (cid:107) α (cid:107) 22 serves as the effective sample size.
If α is uniform (i.e., [1 /k, . . .
, 1 /k ] (cid:62) ), then the effective sample size is km ; if α is onehot, then we effectively only have m points from exactly one domain.
(2) R m ( H S i ) determines how expressive the model family H is w.r.t.
the source data S i .
It can be estimated from samples (Bartlett & Mendelson, 2002, Theorem 11), but the computation is non-trivial for a model family like deep neural networks.
The η H ,i is uncontrollable once the hypothesis class H (e.g., neural network architecture) is given.
Using a richer H is not always beneﬁcial.
Richer H can reduce the η H ,i and also help us ﬁnd a better function h with smaller source losses L (cid:98) S i , but it will increase the R m ( H S i ) .
As removing these two terms does not seem to be empirically signiﬁcant, we ignore them below for simplicity.
(3) Let g h,i def =
L (cid:98) S i ( h, f S i ) + disc( T, S i ) .
Small g h,i indicates that we can achieve small loss on domain S i , and it is similar to the target domain (i.e., small disc( T, S i ) , estimated from (cid:98) T, (cid:98) S i ).
We may want to emphasize on S i by setting α i close to 1, but this will reduce the effective sample size.
Therefore, we have to trade-off between the terms in this bound by choosing a proper α .
(4) When S i and T are only partially overlapped, it might be difﬁcult to ﬁnd a suitable α i .
In such cases, we can artiﬁcially split the given source domains into smaller datasets (e.g., by using clustering) then apply our method on the ﬁner scale.
This strategy requires that the learning algorithm has low computational complexities w.r.t.
k , the number of source domains.
As we will see later in Section 3.3, this is indeed the case for our algorithm.
Before we proceed to develop an algorithm based on the theorem, let us compare Eq.
(3) to existing ﬁnite-sample bounds.
The bound of Blitzer et al.
(2008, Theorem 3) is informative when we have access to a small set of labelled target examples.
In such cases, we can improve our bound by using this small labelled target set as an additional source domain
S k
+1 .
We can also perform better model selection using such labelled set.
The bound of Zhao et al.
(2018, Theorem 2) is based on the d A -distance, which is a special case of disc in our bound.
Moreover, our bound (Eq.
(3)) use samplebased Rademacher complexity, which is generally tighter than other complexity measures such as
VC-dimension (Bartlett & Mendelson, 2002; Koltchinskii et al., 2002; Bousquet et al., 2003).
3. A LGORITHM
In this section, we illustrate how to develop a practical algorithm based on Theorem 2.
Ignoring the constants, we would like to minimize the upper bound of Eq.
[Equation]
where g h =
[ g h, 1 , . . .
, g h,k ] (cid:62) and τ > 0 is a hyper-parameter.
If we can solve the inner minimization exactly given h , then we can treat the optimal α ∗ ( h ) as a function of h and solve the outer minimization over h effectively.
In the following, we show how to achieve this.
Given g h , the inner minimization can be reformulated as a second-order cone programming problem, but it has no closed-form solution due to the τ (cid:107) α (cid:107) 2 term.
Consider the following problem ( z = − g h /τ recovers the inner minimization):
[Equation]
The Lagrangian for its dual problem is
[Equation]
Taking the derivative w.r.t.
α and setting it to zero gives
[Equation]
Notice that α (cid:54) =
so we have the constraint (cid:107) z − ν 1 + λ (cid:107) 2 = 1 .
Using this α ∗ in Λ gives the following dual problem
[Equation]
We would like to decrease ν as much as possible, and the best we can attain is the ν ∗ satisfying (cid:107) [
z − ν ∗ 1 ] + (cid:107) 2 = 1 where [ v ] + = max( 0 , v ) .
In this case, the optimal λ ∗ can be attained as λ ∗ i =
if z i − ν ∗ > 0 , otherwise λ ∗ i =
ν ∗ −
z
i ; see Fig.
Although we do not have a closed-form expression for the optimal ν ∗ , we can use binary search to ﬁnd it, starting from the interval [ z min − 1 , z max ] where z min , z max are the minimum and maximum of z respectively.
Then we can recover the primal solution as
[Equation]
Eq.
(6) gives rise to a new way to project any vector z to the probability simplex, which is of independent interest and may be used in some other contexts.
2 It resembles the standard projection onto the simplex based on squared Euclidean distance (Duchi et al., 2008, Eq.(3)).
One subtle but crucial difference is that our Eq.
(5) uses (cid:107) α (cid:107) 2 instead of (cid:107) α (cid:107) 22 .
Recall that z
= − g h /τ .
Here τ can be interpreted as a temperature parameter.
On one hand, if τ (cid:29) 0 , all z will have similar values and thus the optimal ν ∗ will be close to z max and α ∗ will be close to uniform.
On the other hand, as τ → 0 , z max will stand out from the rest z i and eventually ν ∗ =
z max − 1 .
This means the g h,i corresponding to the z max is small enough so we focus solely on this domain and ignore all other domains (even though this will reduce effective sample size as discussed in Section 3.1).
From a different perspective, if we deﬁne F ∗ ∆ ( α ) = (cid:107) α (cid:107) 2 , then Eq.
(5) equals − F ∆ ( z ) , where
F ∆ , F ∗ ∆ are convex conjugates of each other, where we use ∆ to emphasize their dependency on the simplex domain.
Then our objective Eq.
(4) can be expressed as
[Equation]
We can optimize our objective and train a neural network h using gradient-based optimizer.
The optimal α ∗ ( h ) is merely a function of h and we can backprop through it.
To facilitate the gradient computation and show that we can efﬁciently backprop through the projection of Eq.
(6), we derive the Jacobian J
= ∂
α /∂ z in Appendix B.
This ensures effective end-to-end training.
Now we elaborate more on how to compute g h,i .
The L (cid:98) S i ( h, f S i ) is the task loss.
The disc( T, S i ) depends on whether the task is classiﬁcation or regression.
For classiﬁcation, disc coincides with the d A -distance, so we use the domain classiﬁcation loss (Zhao et al., 2018; Ben-David et al., 2007):
[Equation]
where (cid:98) (cid:15) T,S i ( h d ) is the sample domain classiﬁcation loss of a domain classiﬁer h d : X (cid:55)→ { 0 , 1 } .
This minimization over h d will become maximization once we move it outside of the disc due to the minus sign.
Then our objective consists of min h and max h d , which resembles adversarial training (Goodfellow et al., 2014): learning a task classiﬁer h to minimize loss and a domain classiﬁer h d to maximize domain confusion.
More speciﬁcally, if we decompose the neural network h into a feature extractor h fea and a label predictor h y (i.e., h ( x )
=
h y ( h fea ( x )) ), we can learn a domainclassiﬁer h d on top of h fea to classify h fea ( x ) between S i and T as a binary classiﬁcation problem, where the domain itself is the label (see Fig. 2).
To achieve this, we use the logistic loss to approximate (cid:98) (cid:15) T,S i ( h d ) and apply the gradient reversal layer (Ganin & Lempitsky, 2015; Ganin et al., 2016) when optimizing disc through backpropagation.
If we are solving regression problems with L 2 loss, then disc( (cid:98) T, (cid:98) S i ) = (cid:107) M T − M S
i (cid:107) 2 is the largest eigenvalue (in magnitude) of the difference of two matrices M T = 1 m (cid:80) j h fea ( x ( T ) j ) h (cid:62)
fea
( x ( T ) j )
and
M S i = 1 m (cid:80)
j h fea ( x ( i ) j ) h (cid:62) fea
( x ( i ) j ) (Mansour et al., 2009a; Cortes & Mohri, 2011, Sec.5), which can be conveniently approximated using mini-batches and a few steps of power iteration.
3. C OMPLEXITY
Here we analyze the time and space complexities of the algorithm in each iteration.
Similar to
MDAN (Zhao et al., 2018), in each gradient step, we need to compute the task loss L (cid:98) S
i ( h, f S i ) and the disc( T, S i ) (or d A -distance) using mini-batches from each source domain i ∈ [ k ] .
The question is whether one can maintain the O ( k ) complexity given that we need to compute the weights using
Eq.
(6) and backprop through it.
For the forward computation of α ∗ , in order to compute the threshold ν ∗ to the (cid:15) > 0 relative precision, the binary search will cost O ( k log(1 /(cid:15) )) .
As for the backward pass of gradient computation, according to our calculation in Appendix B, the Jacobian
J = ∂
α /∂ z has a concise form, meaning that it is possible to compute the matrix-vector product
J v for a given vector v in O ( k ) time and space.
Therefore, our space complexity is the same as
MDAN but our time complexity is slightly slower by a factor of log(1 /(cid:15) ) .
In comparison, the time complexity for MDMN (Li et al., 2018) is O ( k 2 ) because it requires computing the pairwise weights within the k source domains.
When we have a lot of source domains, MDMN will be noticeably slower than MDAN and our DARN.
R ELATED W ORK
The idea of utilizing data from the source domain ( S, f S ) to train a model for a different but related target domain ( T, f T ) has been explored extensively for the last decade using different assumptions (Pan & Yang, 2009; Zhang et al., 2015).
For instance, the covariate shift scenario (Shimodaira, 2000; Gretton et al., 2009; Sugiyama & Kawanabe, 2012; Wen et al., 2014) assumes S (cid:54) =
T but f S =
f T , while concept drift (Jiang & Zhai, 2007; Gama et al., 2014) assumes S = T but f S (cid:54)
=
f T .
More speciﬁcally, Theorem 1 (Cortes et al., 2019) shows that both covariate shift (as measured by the discrepancy disc ) and model misspeciﬁcation (as controlled by η H ) contribute to the adaptation performance (Wen et al., 2014).
Finding a domain-invariant feature space by minimizing a distance measure is common practice in domain adaptation, especially for training neural networks.
Tzeng et al.
(2017) provided a comprehensive framework that subsumes several prior efforts on learning shared representations across domains (Tzeng et al., 2015; Ganin et al., 2016).
DARN uses adversarial domain classiﬁer and the gradient reversal trick from Ganin et al.
Instead of proposing a new loss for each pair of the source and target domains, our main contribution is the aggregation technique of computing the mixing coefﬁcients α , which is derived from theoretical guarantees.
When dealing with multiple source domains, our aggregation method can certainly be applied to other forms of discrepancies such as MMD (Gretton et al., 2012;
Long et al., 2015; 2016), and other model architectures such as Domain Separation Network (Bousmalis et al., 2016), cycle-consistent model (Hoffman et al., 2018b), class-dependent adversarial domain classiﬁer (Pei et al., 2018) and Known Unknown Discrimination (Schoenauer-Sebag et al., 2019).
Our work focuses on multi-source to single-target adaptation, which has been investigated in the literature.
Sun et al.
(2011) developed a generalization bound but resorted to heuristic algorithms to adjust distribution shifts.
Zhao et al.
(2018) proposed a certain ad-hoc scheme for the combination coefﬁcients α , which, unlike ours, are not theoretically justiﬁed.
Multiple Domain Matching
Network (MDMN) (Li et al., 2018) computes domain similarities not only between the source and target domains but also within the source domain themselves based on Wasserstein-like measure.
Calculating such pairwise weights can be computationally demanding when we have a lot of source domains.
Their bound requires additional smooth assumptions on the labelling functions f S i , f T , and is not a ﬁnite-sample bound, as opposed to ours.
As for the actual algorithm, they also use adhoc coefﬁcients α without theoretical justiﬁcation.
Mansour et al.
(2009b;c) consider multi-source adaptation where T = (cid:80) i β i
S
i is a convex mixture of source distributions with some weights β i .
Our analysis does not require this assumption.
Hoffman et al.
(2018a) provides similar guarantees with different assumptions, but unlike ours, their bounds are not ﬁnite-sample bounds.
E XPERIMENTS
In this section, we compare DARN with several other baselines and state-of-the-art methods for the popular tasks: sentiment analysis, digit recognition and object recognition.
The following methods are compared
(1) The SRC (for source) method uses only labelled source data to train the model.
It merges all available source examples to form a large dataset to perform training without adaptation.
(2) TAR (for target) is another baseline that uses only m labelled target data instances.
It serves as upper bound of the best we can achieve if we had
access to the true label of the target data.
(3) Domain Adversarial Neural Network ( DANN ) (Ganin et al., 2016) is similar to our method in that we both use adversarial training objectives.
It is not obvious how to adapt DANN to the multisource setting.
Here we follow previous protocol (Zhao et al., 2018) and merge all source data to form a large joint source dataset of km instances for DANN to perform adaptation. (4) Moment
Matching for Multi-Source Domain Adaptation ( M3SDA ) (Peng et al., 2019) is a recent state-ofthe-art method that combines moment matching and maximizing classiﬁer discrepancy (Saito et al., 2018).
We use their public code with a few necessary adjustments (change classiﬁcation head based on the number of classes etc.)
(5) Multisource Domain Adversarial Network ( MDAN )
(Zhao et al., 2018) resembles our method in that we both dynamically assign each source domain an importance weight during training.
However, unlike ours, their weights are not theoretically justiﬁed.
We use the soft version of MDAN since they reported that it performs better than the hard version, and we use their code.
(6) Multiple Domain Matching Network ( MDMN ) (Li et al., 2018) computes weights not only between source and target domains but also within source domain themselves.
We use their code of computing weights in our implementation.
All the methods are applied to the same neural network structure to ensure fair comparison.
Our PyTorch implementation will be available online after acceptance.
5. S ENTIMENT A NALYSIS
Setup .
We use the Amazon review dataset (Blitzer et al., 2007; Chen et al., 2012) that consists of positive and negative product reviews from four domains (Books, DVD, Electronics and Kitchen).
Each of them is used in turn as the target domain and the other three are used as source domains.
Their sample sizes are 6465, 5586, 7681, 7945 respectively.
We follow the common protocol (Chen et al., 2012; Zhao et al., 2018) of using the top-5000 frequent unigrams/bigrams of all reviews as bag-of-words features and train a fully connected model (MLP) with [1000 , 500 , 100] hidden units for classifying positive versus negative reviews.
The dropout drop rate is 0.7 for the input and hidden layers.
In each run, we randomly sample 2000 reviews from each domain as labelled source or unlabelled target training examples, while the remaining instances are used as test examples for evaluation.
The hyper-parameters are chosen based on cross-validation.
The model is trained forepochs and the mini-batch size is 20 per domain.
The optimizer is Adadelta with a learning rate of 1.0.
The soft version of MDAN has an additional parameter γ = 1 /τ which is the inverse of our temperature τ .
The chosen parameters are γ = 10 . 0 for MDAN and γ = 0 .
9 for our DARN, which are selected from a wide range of candidate values.
Results and Analysis .
Table 1 summarizes the classiﬁcation accuracies.
The last column is the average accuracy of four domains, and the standard errors are calculated based on 20 runs.
(1) Most of the time, DANN with joint source data performs worse than SRC which has no adaptation.
This may be because it does not adjust for the each source domain, and as a result, it fails to ignore irrelevant data to avoid negative transfer.
(2) Some domains are harder to adapt to than the others.
For example, the accuracies of SRC and TAR on the Electronics domain are very close to each other, indicating that this requires little to no adaptation.
Yet, our DARN is the closest to the TAR performance here.
The Books domain is more challenging as the improvement over the SRC method is small, even though there exists a large gap between SRC and TAR.
(3) DARN is always within the best performing methods and signiﬁcantly outperforms others in the Electronics domain.
Note that
MDMN additionally computes similarities within source domains in each iteration, which can be computationally expensive ( O ( k 2 ) per iteration) if the number of source domains is large.
Instead, our method focuses on the discrepancy between source and target domains ( O ( k ) per iteration) and can achieve similar performance on this problem.
[Table]
5. D IGIT R ECOGNITION
Setup .
Following the setting from previous work (Ganin et al., 2016; Zhao et al., 2018) [Table] we use the four digit recognition datasets in this experiment (MNIST, MNIST-M, SVHN and Synth).
MNIST is is a variant where the black and white pixels are masked with color patches.
Street View House
Number (SVHN) (Netzer et al., 2011) is a standard digit dataset taken from house numbers in Google
Street View images.
Synthetic Digits (Synth) (Ganin & Lempitsky, 2015) is a synthetic dataset that mimic SVHN using various transformations.
One of the four datasets is chosen as unlabelled target domain in turn and the other three are used as labelled source domains.
MNIST images are resized to 32 × 32 and represented as 3-channel color images in order to match the shape of the other three datasets.
Each domain has its own given training and test sets when downloaded.
Their respective training sample sizes are 60000, 59001, 73257, 479400, and the respective test sample sizes are 10000, 9001, 26032, 9553.
In each run, 20000 images are randomly sampled from each domain’s training set as actual labelled source or unlabelled target training examples, and 9000 images are randomly sampled from each domain’s test set as actual test examples for evaluation.
The model structure is shown in Appendix D. There is no dropout and the hyperparameters are chosen based on cross-validation.
It is trained for 50 epochs and the mini-batch size is 128 per domain.
The optimizer is Adadelta with a learning rate of 1.0.
Validation selected γ = 0 .for
MDAN and γ = 0 .
1 for DARN.
Results and Analysis .
Table 2 shows the classiﬁcation accuracy of each target dataset overruns.
(1) All methods except DANN can consistently outperform SRC.
Again, without proper adjustment for each source domain, DANN with joint source data can perform worse than SRC.
This suggests the importance of ignoring irrelevant data to avoid negative transfer.
(2) DARN signiﬁcantly outperforms MDAN and MDMN across all four domains, especially on the MNIST-M and
SVHN domains.
Notice that even though MDAN and MDMN have generalization guarantees, they both resort to ad-hoc aggregation rules to combine the source domains during training.
Instead, our aggregation (Eq.
(6)) is a direct optimization of the upper bound (Theorem 2) thus is theoretically justiﬁed and empirically superior for this problem.
[Table]
5. O BJECT R ECOGNITION : O FFICE -H OME
To showcase the applicability of our method to more complicated real-world tasks
[Table]
we use the challenging Ofﬁce-Home dataset (Venkateswara et al., 2017).
It contains images of 65 everyday objects such as spoon, sink, mug and pen from four different domains: Art, Clipart, Product and Real-World.
One of the four datasets is chosen as unlabelled target domain in turn and the other three are used as labelled source domains.
The respective sample sizes are 2427, 4365, 4439, 4357.
In each run, 2000 images are randomly sampled from each domain as labelled source or unlabelled target training examples, and the rest images are used as test images for evaluation.
{+We use the ResNet
+}
He et al.
(2016) pretrained from the ImageNet in PyTorch as the base network for feature learning and put an MLP with [1000, 500, 100, 65] units on top for classiﬁcation.
It is trained for 50 epochs and the mini-batch size is 32 per domain.
The optimizer is Adadelta with a learning rate of 1.0.
MDAN uses γ = 1 . 0 while DARN uses γ = 0 . 5 .
Table 3 shows the classiﬁcation accuracy of each target dataset over 20 runs.
Most existing works in the literature focused on single-source to single-target adaptation on this problem (e.g., see (Long et al., 2018)).
Compared to them, using multi-source methods can signiﬁcantly boost performance, revealing the importance of using multiple source domains when possible.
Even though this is a signiﬁcantly more challenging problem with more classes and much fewer images compared to the digits datasets, our DARN achieves state-of-the-art performance in this setting, excelling existing methods by a noticeable margin.
[Table]
5. V ISUALIZING D OMAIN I MPORTANCE
the Amazon data.
To demonstrate how DARN can effectively aggregate multiple source domains, we visualize the source domain weights (i.e., α in DARN) for the Amazon dataset.
We also compared to the weights produced by MDMN, using the original authors’ code.
Fig.
3 and Fig.
4 compare the evolution of source domain weights during training.
In each subﬁgure, every row corresponds to the weights of the source domains when learning for one target domain.
The white stripe indicates there is no target domain weight and darker stripe means less weight.
They are evaluated at the end of each epoch over 50 epochs.
To avoid noisy values due to small mini-batch size, the values are exponential moving averages with a decay rate of 0.95.
(1) For DARN, as Electronics and Kitchen are more related to each other than Books and DVD, their respective weights remain higher during training.
This is reasonable since we have overlapping products (e.g., blenders) in both domains.
The α is changing dynamically during training, showing the ﬂexibility of our method to adjust domain importance when needed.
In comparison, the domain weights produced by MDMN are not very stable.
We take a closer look at the MDMN weights and notice that they change drastically, especially towards the end of the training.
It can produce alternating one-hot vectors α , changing from one domain to a different domain and ignoring the rest.
This instability makes their domain weights hard to interpret.
Our work uses the discrepancy (Mansour et al., 2009a; Cortes et al., 2019) to derive a ﬁnite-sample generalization bound for multi-source to single-target adaptation.
Our theorem shows that, in order to achieve the best possible generalization upper bound for a target domain, we need to trade-off between including all source domains to increase effective sample size and removing source domains that are underperforming or not similar to the target domain.
Based on this observation, we derive an algorithm, Domain AggRegation Network (DARN), that can dynamically adjust the weight of each source domain during end-to-end training.
Experiments on sentiment analysis, digits recognition and object recognition show that DARN outperforms state-of-the-art alternatives.
Recent analysis (Zhao et al., 2019; Johansson et al., 2019) show that solely focusing on learning domain invariant features can be problematic when the marginal label distributions on Y between source and target domains are signiﬁcantly different.
Thus it makes sense to take η H into consideration when a small amount of labelled data is available for the target domain, which we will explore in the future.
A P ROOF OF T HEOREM
Theorem 2 Given k source domains datasets { ( x ( i ) j , y ( i ) j ) : i ∈
[ k ] , j ∈ [ m ] } with m iid examples each where (cid:98) S
i = {
x ( i ) j } and y ( i )
j = f S i ( x ( i ) j ) , for any α ∈ ∆ = { α : α i ≥ 0 , (cid:80) i
α i = 1 } , δ ∈ (0 , 1) , and ∀ h ∈ H , w.p.
at least 1 − δ
[Equation]
≤
[Equation]
where H S i
=
{ x (cid:55)→ L ( h ( x ) , f S i ( x )) :
h ∈ H} is the set of functions mapping x to the corresponding loss, η H ,i is a constant similar to Eq.
(2) with (cid:98) Q = (cid:98) S i , (cid:98) P = (cid:98) T and M S = sup i ∈
[ k ] ,x ∈X ,h ∈H L ( h ( x ) , f S i ( x )) is the upper bound on loss on the source domains.
Proof: The proof is similar to Cortes et al.
; Zhao et al.
Given α ∈ ∆ , the mixture (cid:98) S =
(cid:80)
i
α i (cid:98) S i can be considered as the joint source data with km points, where a point x ( i ) from (cid:98) S i has weight α
i /m .
Deﬁne Φ = sup h
[Equation]
L T ( h, f T ) − (cid:80)
[Equation]
α
[Equation]
L
[Equation] [Equation]
, f S i ) .
Changing a point x ( i ) from (cid:98) S i will change Φ at most M S α i m .
Using the McDiarmid’s inequality, we have Pr(Φ − E [Φ] > (cid:15) ) ≤ exp
[Equation]
−
[Equation]
(cid:15)
[Equation]
m M 2 S (cid:107) α
(cid:107) 22 (cid:17) .
As a result, for δ ∈ (0 , 1) , w.p. at least 1 − δ , the following holds for any h ∈ H
[Equation]
Now we bound E [Φ] .
Let H S
i = { x (cid:55)→ L ( h ( x ) , f S i ( x )) : h ∈ H} .
E [
[Equation] [Equation]
where ﬁrst and second inequalities are using the subadditivity of sup , followed by the equality using the independence between the domains { (cid:98) S i } , the second last inequality is due to the standard “ghost sample” argument in terms of the Rademacher complexity and the last inequality is due to
Cortes et al.
(2019, Proposition 8) for each individual S i .
B J ACOBIAN
Here we calculate the Jacobian J ij = ∂α
i /∂z j for Eq.
(6) in the main text:
[Equation]
In the following, we write α = α ∗ , ν = ν ∗ to simplify notations.
Let S
= { i : z i − ν > 0 } be the support of the probability vector α .
J ij =
if i / ∈ S or j / ∈ S since α i = 0 in the former case while z j does not contribute to the α in the latter case.
Now consider the case i, j ∈ S .
Let
K = (cid:107) [
z − ν 1 ] + (cid:107)
= (cid:80) j ∈ S ( z j − ν ) .
Then α i =
( z i − ν ) · 1 K and ∂α
i
[Equation]
where δ i =
j is the indicator or delta function.
Now we compute ∂ν∂z j and ∂K∂z j .
By the deﬁnition of ν , we know that
[Equation]
|
[Equation]
| where A =  (cid:88)
j
[Equation]
S
[Equation]
(cid:88)
[Equation]
=
⇒ ∂ν ∂z
[Equation]
where B j =
[Equation] [Equation]
The ﬁrst right-arrow is due to the quadratic formula and realizing that (cid:80)
j ∈ S z j / | S | is the mean of the supported z
j so ν must be smaller than it (i.e., we take − in the ± of the quadratic formula, otherwise some of the z j will not be in the support anymore).
And
[Equation]
Plugging Eq.
(8) and Eq.
(9) in Eq.
(7) gives
[Equation]
Note that
B j
[Equation]
Then
[Equation]
In matrix form,
[Equation]
where s =
[ s 1 , . . .
, s k ] (cid:62) is a vector indicating the support s i
= δ i ∈ S and ◦ is element-wise multiplication.
More often, we need to compute its multiplication with a vector v
[Equation]
Note that all quantities except A have been computed during the forward pass of calculating Eq.
A can be computed in O ( | S | ) time so the overall computation is still O ( k ) since | S | ≤ k . [Figure]
C R EGRESSION ON S YNTHETIC D ATA
Due to the lack of publicly available multi-source regression datasets, we demonstrate our method on a synthetic regression problem here. Setup . We use eight source domains,
where the i th
( i = { 0 , 1 , . . .
, 7 } ) domain data is generated by x i ∼ N ( π 4 i − 7 π 8 , 0 .
and the output is y = sin( x ) + (cid:15) where (cid:15) ∼ N (0 , 0 .
05 2 ) is random noise.
These eight domains evenly cover the sin function on [ − π, π ] (see Fig. 5a).
Next we construct four target domains, where the j th ( j = { 0 , 1 , 2 , 3 } ) domain is generated by x j ∼ N ( π 2 j − 3 π 4 , 0 . 4 2 ) .
Similar to the source domains, these four target domains evenly cover [ − π, π ] (see Fig.
5b).
Each source/target domain has 100 data points.
We use labelled source data and unlabelled target data for learning.
We take on one target domain at a time, and learn a linear model from all eight source domains with MSE loss.
The goal is to see whether our method can focus on the relevant source domains and learn a linear model that can perform well on the target domain.
Results and Analysis .
First, Fig.
5b shows the learned models.
The learned linear models can ﬁt the target data very well.
This shows that our DARN can learn a meaningful model for a speciﬁc target domain, using only labelled source data and unlabelled target data.
Second, Fig.
5c shows the source domain weights (the α ) for each target domain after training.
The weight colors correspond to the colors in Fig.
5a.
It is noticeable that DARN can focus well on the respective relevant source domains for each target domain and ignore the rest.
Note that these values are automatically learned during the training of the model.
[Figure]
M ODEL A RCHITECTURE
[Figure]
D IGIT
[Figure]
ECOGNITION