M ULTIPLE S OURCE D OMAIN A DAPTATION WITH A DVERSARIAL L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
While domain adaptation has been actively researched in recent years, most theoretical results and algorithms focus on the single-source-single-target adaptation setting.
Naive application of such algorithms on multiple source domain adaptation problem may lead to suboptimal solutions.
We propose a new generalization bound for domain adaptation when there are multiple source domains with labeled instances and one target domain with unlabeled instances.
Compared with existing bounds, the new bound does not require expert knowledge about the target distribution, nor the optimal combination rule for multisource domains.
Interestingly, our theory also leads to an efﬁcient learning strategy using adversarial neural networks: we show how to interpret it as learning feature representations that are invariant to the multiple domain shifts while still being discriminative for the learning task.
To this end, we propose two models, both of which we call multisource domain adversarial networks (MDANs): the ﬁrst model optimizes directly our bound, while the second model is a smoothed approximation of the ﬁrst one, leading to a more data-efﬁcient and task-adaptive model.
The optimization tasks of both models are minimax saddle point problems that can be optimized by adversarial training.
To demonstrate the effectiveness of MDANs, we conduct extensive experiments showing superior adaptation performance on three real-world datasets: sentiment analysis, digit classiﬁcation, and vehicle counting.
I NTRODUCTION
The success of machine learning algorithms has been partially attributed to rich datasets with abundant annotations (Krizhevsky et al., 2012; Hinton et al., 2012; Russakovsky et al., 2015).
Unfortunately, collecting and annotating such large-scale training data is prohibitively expensive and time-consuming.
To solve these limitations, different labeled datasets can be combined to build a larger one, or synthetic training data can be generated with explicit yet inexpensive annotations (Shrivastava et al., 2016).
However, due to the possible shift between training and test samples, learning algorithms based on these cheaper datasets still suffer from high generalization error.
Domain adaptation (DA) focuses on such problems by establishing knowledge transfer from a labeled source domain to an unlabeled target domain, and by exploring domain-invariant structures and representations to bridge the gap (Pan & Yang, 2010).
Both theoretical results (Ben-David et al., 2010; Mansour et al., 2009a;
Mansour & Schain, 2012; Xu & Mannor, 2012) and algorithms (Becker et al., 2013; Hoffman et al., 2012; Ajakan et al., 2014) for DA have been proposed.
Recently, DA algorithms based on deep neural networks produce breakthrough performance by learning more transferable features (Glorot et al., 2011;
Donahue et al., 2014; Yosinski et al., 2014; Bousmalis et al., 2016;
Long et al., 2015).
Most theoretical results and algorithms with respect to DA focus on the single-source-single-target adaptation setting (Ganin et al., 2016).
However, in many application scenarios, the labeled data available may come from multiple domains with different distributions.
As a result, naive application of the single-source-single-target DA algorithms may lead to suboptimal solutions.
Such problem calls for an efﬁcient technique for multiple source domain adaptation.
In this paper, we theoretically analyze the multiple source domain adaptation problem and propose an adversarial learning strategy based on our theoretical results.
Speciﬁcally, we prove a new generalization bound for domain adaptation when there are multiple source domains with labeled instances and one target domain with unlabeled instances.
Our theoretical results build on the seminal theoretical model for domain adaptation introduced by Ben-David et al.
(2010), where a divergence measure, known as the H -divergence, was proposed to measure the distance between two distributions based on a given hypothesis space H .
Our new result generalizes the bound (Ben-David et al., 2010, Thm. 2) to the case when there are multiple source domains.
The new bound has an interesting interpretation and reduces to (Ben-David et al., 2010, Thm. 2) when there is only one source domain.
Technically, we derive our bound by ﬁrst proposing a generalized H -divergence measure between two sets of distributions from multi-domains.
We then prove a PAC bound (Valiant, 1984) for the target risk by bounding it from empirical source risks, using tools from concentration inequalities and the VC theory (Vapnik, 1998).
Compared with existing bounds, the new bound does not require expert knowledge about the target domain distribution (Mansour et al., 2009b), nor the optimal combination rule for multiple source domains (Ben-David et al., 2010).
Our results also imply that it is not always beneﬁcial to naively incorporate more source domains into training, which we verify to be true in our experiments.
Interestingly, our bound also leads to an efﬁcient implementation using adversarial neural networks.
This implementation learns both domain invariant and task discriminative feature representations under multiple domains.
Speciﬁcally, we propose two models (both named MDANs) by using neural networks as rich function approximators to instantiate the generalization bound we derive (Fig. 1).
After proper transformations, both models can be viewed as computationally efﬁcient approximations of our generalization bound, so that the goal is to optimize the parameters of the networks in order to minimize the bound.
The ﬁrst model optimizes directly our generalization bound, while the second is a smoothed approximation of the ﬁrst, leading to a more data-efﬁcient and task-adaptive model.
The optimization problem for each model is a minimax saddle point problem, which can be interpreted as a zero-sum game with two participants competing against each other to learn invariant features.
Both models combine feature extraction, domain classiﬁcation, and task learning in one training process.
MDANs is generalization of the popular domain adversarial neural network (DANN)
(Ganin et al., 2016) and reduce to it when there is only one source domain.
We propose to use stochastic optimization with simultaneous updates to optimize the parameters in each iteration.
To demonstrate the effectiveness of MDANs as well as the relevance of our theoretical results, we conduct extensive experiments on real-world datasets, including both natural language and vision tasks.
We achieve superior adaptation performances on all the tasks, validating the effectiveness of our models.
P RELIMINARY
We ﬁrst introduce the notation used in this paper and review a theoretical model for domain adaptation when there is only one source and one target domain (Kifer et al., 2004; Ben-David et al., 2007;
Blitzer et al., 2008; Ben-David et al., 2010).
The key idea is the H -divergence to measure the discrepancy between two distributions.
Other theoretical models for DA exist (Cortes et al., 2008;
Mansour et al., 2009a;c; Cortes & Mohri, 2014); we choose to work with the above model because this distance measure has a particularly natural interpretation and can be well approximated using samples from both domains.
Notations We use domain to represent a distribution D on input space X and a labeling function f : X → [0 , 1] .
In the setting of one source one target domain adaptation, we use (cid:104)D S , f S (cid:105) and (cid:104)D T , f T (cid:105) to denote the source and target domain, respectively.
A hypothesis is a binary classiﬁcation function h : X → { 0 , 1 } .
The error of a hypothesis h w.r.t.
a labeling function f under distribution
D S is deﬁned as: ε S ( h, f ) :=
E x ∼D S [ | h ( x ) − f ( x ) | ] .
When f is also a hypothesis, then this deﬁnition reduces to the probability that h disagrees with h under D S :
E x ∼D S [ | h ( x ) − f ( x ) | ] =
E x ∼D S [ I ( f ( x ) (cid:54) =
h ( x ))]
=
Pr x ∼D S ( f ( x ) (cid:54) =
h ( x )) .
We deﬁne the risk of hypothesis h as the error of h w.r.t.
a true labeling function under domain D S , i.e., ε S ( h ) :
= ε S ( h, f S ) .
As common notation in computational learning theory, we use (cid:98) ε S ( h ) to denote the empirical risk of h on the source domain.
Similarly, we use ε T ( h ) and (cid:98) ε T ( h ) to mean the true risk and the empirical risk on the target domain.
H -divergence is deﬁned as follows:
Deﬁnition 2.1.
Let H be a hypothesis class for instance space X , and A H be the collection of subsets of X that are the support of some hypothesis in H , i.e., A H :
=
{ h − 1 ( { 1 } ) | h ∈ H} .
The distance between two distributions D and D (cid:48) based on H is:
[Equation]
When the hypothesis class H contains all the possible measurable functions over X , d H ( D , D (cid:48) ) reduces to the familiar total variation.
Given a hypothesis class H , we deﬁne its symmetric difference w.r.t.
itself as: H ∆ H
= { h ( x ) ⊕ h (cid:48)
( x ) | h, h (cid:48) ∈ H} , where ⊕ is the xor operation.
Let h ∗ be the optimal hypothesis that achieves the minimum combined risk on both the source and the target domains:
[Equation]
and use λ to denote the combined risk of the optimal hypothesis h ∗ :
[Equation]
Ben-David et al.
(2007) and Blitzer et al.
(2008) proved the following generalization bound on the target risk in terms of the source risk and the discrepancy between the source domain and the target domain:
Theorem 2.1 ((Blitzer et al., 2008)) .
Let H be a hypothesis space of V C -dimension d and U S , U T be unlabeled samples of size m each, drawn from D S and D T , respectively.
Let (cid:98) d
H ∆ H be the empirical distance on U S and U T ; then with probability at least 1 − δ over the choice of samples, for each h ∈ H ,
[Equation]
The generalization bound depends on λ , the optimal combined risk that can be achieved by hypothesis in H .
The intuition is that if λ is large, then we cannot hope for a successful domain adaptation.
One notable feature of this bound is that the empirical discrepancy distance between two samples U S and
U T can usually be approximated by a discriminator to distinguish instances from these two domains.
A N EW G ENERALIZATION B OUND FOR M ULTIPLE S OURCE D OMAIN A DAPTATION
In this section we ﬁrst generalize the deﬁnition of the discrepancy function d H ( · , · ) that is only appropriate when we have two domains.
We will then use the generalized discrepancy function to derive a generalization bound for multisource domain adaptation.
We conclude this section with a discussion and comparison of our bound and existing generalization bounds for multisource domain adaptation (Mansour et al., 2009c; Ben-David et al., 2010).
We refer readers to appendix for proof details and we mainly focus on discussing the interpretations and implications of the theorems.
Let {D S i } ki =1 and D T be k source domains and the target domain, respectively.
We deﬁne the discrepancy function d H ( D T ; {D S i } ki =1 ) induced by H to measure the distance between D T and a set of domains {D S i } ki =1 as follows:
Deﬁnition 3.1.
[Equation]
Again, let h ∗ be the optimal hypothesis that achieves the minimum combined risk:
[Equation]
and deﬁne
[Equation]
i.e., the minimum risk that is achieved by h ∗ .
The following lemma holds for ∀ h ∈ H :
Theorem
[Equation]
Remark .
Let us take a closer look at the generalization bound: to make it small, the discrepancy measure between the target domain and the multiple source domains need to be small.
Otherwise we cannot hope for successful adaptation by only using labeled instances from the source domains.
In this case there will be no hypothesis that performs well on both the source domains and the target domain.
It is worth pointing out here that the second term and the third term together introduce a tradeoff (regularization) on the complexity of our hypothesis class H .
Namely, if H is too restricted, then the second term λ can be large while the discrepancy term can be small.
On the other hand, if H is very rich, then we expect the optimal error, λ , to be small, while the discrepancy measure d H ∆ H ( D T ; {D S i } ki =1 ) to be large.
The ﬁrst term is a standard source risk term that usually appears in generalization bounds under the PAC-learning framework (Valiant, 1984; Vapnik, 1998).
Later we shall upper bound this term by its corresponding empirical risk.
The discrepancy distance d H ∆ H ( D T ; {D S i } ki =1 ) is usually unknown.
However, we can bound d H ∆ H ( D T ; {D S i } ki =1 ) from its empirical estimation using i.i.d.
samples from D T and {D S i } ki =1 :
Theorem 3.2.
Let D T and {D S i } ki =1 be the target distribution and k source distributions over X .
Let
H be a hypothesis class where V C dim ( H ) =
d .
If (cid:98) D T and { (cid:98) D S i } ki =1 are the empirical distributions of D T and {D S i } ki =1 generated with m i.i.d.
samples from each domain, then for (cid:15) > 0 , we have:
[Equation]
The main idea of the proof is to use VC theory (Vapnik, 1998) to reduce the inﬁnite hypothesis space to a ﬁnite space when acting on ﬁnite samples.
The theorem then follows from standard union bound and concentration inequalities.
Equivalently, the following corollary holds:
Corollary 3.1.
Let D T and {D S i } ki =1 be the target distribution and k source distributions over X .
Let
H be a hypothesis class where V C dim ( H ) =
d .
If (cid:98) D T and { (cid:98) D S i } ki =1 are the empirical distributions of D T and {D S i } ki =1 generated with m i.i.d.
samples from each domain, then, for 0 < δ < 1 , with probability at least 1 − δ (over the choice of samples), we have:
[Equation]
Note that multiple source domains do not increase the sample complexity too drastically: it is only the square root of a log term in Corollary.
3.1 where k appears.
Similarly, we do not usually have access to the true error max i ∈
[ k ] ε S i ( h ) on the source domains, but we can often have an estimate ( max i ∈ [ k ] (cid:98) ε S i ( h ) ) from training samples.
We now provide a probabilistic guarantee to bound the difference between max i ∈ [ k ] ε S i ( h ) and max i ∈
[ k ] (cid:98)
ε S i
( h ) uniformly for all h ∈ H :
Theorem 3.3.
Let {D S i } ki =1 be k source distributions over X .
Let H be a hypothesis class where
V C dim ( H ) =
d .
If { (cid:98) D S i } ki =1 are the empirical distributions of {D S i } ki =1 generated with m i.i.d.
samples from each domain, then, for (cid:15) > 0 , we have:
[Equation]
Again, Thm. 3.3 can be proved by a combination of concentration inequalities and a reduction from inﬁnite space to ﬁnite space, along with the subadditivity of the max function.
Equivalently, we have the following corollary hold:
Corollary 3.2.
Let {D S i } ki =1 be k source distributions over X .
Let H be a hypothesis class where
V C dim ( H ) =
d .
If { (cid:98) D S i } ki =1 are the empirical distributions of {D S i } ki =1 generated with m i.i.d.
samples from each domain, then, for 0 < δ < 1 , with probability at least 1 − δ (over the choice of samples), we have:
[Equation]
Combining Thm. 3.1 and Corollaries. 3.1, 3.2 and realizing that V C dim ( H ∆ H ) ≤ 2 V C dim ( H ) (Anthony & Bartlett, 2009), we have the following theorem:
Theorem 3.4.
Let D T and {D S i } ki =1 be the target distribution and k source distributions over X .
Let
H be a hypothesis class where V C dim ( H ) =
d .
If (cid:98) D T and { (cid:98) D S i } ki =1 are the empirical distributions of D T and {D S i } ki =1 generated with m i.i.d.
samples from each domain, then, for 0 < δ < 1 , with probability at least 1 − δ (over the choice of samples), we have:
[Equation]
Remark .
Thm.
3.4 has a nice interpretation for each term: the ﬁrst term measures the worst case accuracy of hypothesis h on the k source domains, and the second term measures the discrepancy between the target domain and the k source domains.
For domain adaptation to succeed in the multiple sources setting, we have to expect these two terms to be small: we pick our hypothesis h based on its source training errors, and it will generalize only if the discrepancy between sources and target is small.
The third term λ is the optimal error we can hope to achieve.
Hence, if λ is large, one should not hope the generalization error to be small by training on the source domains.
The last term bounds the additional error we may incur because of the possible bias from ﬁnite samples.
It is also worth pointing out that these four terms appearing in the generalization bound also capture the tradeoff between using a rich hypothesis class H and a limited one as we discussed above: when using a richer hypothesis class, the ﬁrst and the third terms in the bound will decrease, while the value of the second term will increase; on the other hand, choosing a limited hypothesis class can decrease the value of the second term, but we may incur additional source training errors and a large λ due to the simplicity of H .
One interesting prediction implied by Thm.
3.4 is that the performance on the target domain depends on the worst empirical error among multiple source domains, i.e., it is not always beneﬁcial to naively incorporate more source domains into training.
As we will see in the experiment, this is indeed the case in many real-world problems.
Comparison with Existing Bounds First, it is easy to see that, upto a multiplicative constant, our bound in (2) reduces to the one in Thm.
2.1 when there is only one source domain ( k = 1 ).
Hence
Thm.
3.4 can be treated as a generalization of Thm. 2.1.
Blitzer et al.
(2008) give a generalization bound for semi-supervised multisource domain adaptation where, besides labeled instances from multiple source domains, the algorithm also has access to a fraction of labeled instances from the target domain.
Although in general our bound and the one in (Blitzer et al., 2008, Thm. 3) are incomparable, it is instructive to see the connections and differences between them: on one hand, the multiplicative constants of the discrepancy measure and the optimal error in our bound are half of those in Blitzer et al.
(2008)’s bound, leading to a tighter bound; on the other hand, because of the access to labeled instances from the target domain, their bound is expressed relative to the optimal error rate on the target domain, while ours is in terms of the empirical error on the source domain.
Finally, thanks to our generalized deﬁnition of d H ( D T ; {D S i } ki =1 ) , we do not need to manually specify the optimal combination vector α in (Blitzer et al., 2008, Thm. 3), which is unknown in practice.
Mansour et al.
(2009b) also give a generalization bound for multisource domain adaptation under the assumption that the target distribution is a mixture of the k sources and the target hypothesis can be represented as a convex combination of the source hypotheses.
While the distance measure we use assumes 0-1 loss function, their generalized discrepancy measure can also be applied for other losses functions (Mansour et al., 2009a;c;b).
M ULTISOURCE D OMAIN A DAPTATION WITH A DVERSARIAL N EURAL N ETWORKS
In this section we shall describe a neural network based implementation to minimize the generalization bound we derive in Thm. 3.4.
The key idea is to reformulate the generalization bound by a minimax saddle point problem and optimize it via adversarial training.
Suppose we are given samples drawn from k source domains {D S i } , each of which contains m instance-label pairs.
Additionally, we also have access to unlabeled instances sampled from the target domain D T .
Once we ﬁx our hypothesis class H , the last two terms in the generalization bound (2) will be ﬁxed; hence we can only hope to minimize the bound by minimizing the ﬁrst two terms, i.e., the maximum source training error and the discrepancy between source domains
The idea is to train a neural network to learn a representation with the following two properties: 1).
indistinguishable between the k source domains and the target domain; 2).
informative enough for our desired task to succeed.
Note that both requirements are necessary: without the second property, a neural network can learn trivial random noise representations for all the domains, and such representations cannot be distinguished by any discriminator; without the ﬁrst property, the learned representation does not necessarily generalize to the unseen target domain.
Taking these two properties into consideration, we propose the following optimization problem: minimize
[Equation]
One key observation that leads to a practical approximation of d H ∆ H ( [Equation] D T ; { (cid:98) D S i } ki =1 ) from BenDavid et al.
(2007) is that computing the discrepancy measure is closely related to learning a classiﬁer that is able to disintuish samples from different domains:
[Equation]
   Let (cid:98) ε T,S i ( h ) be the empirical risk of hypothesis h in the domain discriminating task.
Ignoring the constant terms that do not affect the optimization formulation, moving the max operator out, we can reformulate (3) as: minimize
[Equation]
The two terms in (4) exactly correspond to the two criteria we just proposed: the ﬁrst term asks for an informative feature representation for our desired task to succeed, while the second term captures the notion of invariant feature representations between different domains.
Inspired by Ganin et al.
[Equation] we use the gradient reversal layer to effectively implement (4) by backpropagation.
The network architecture is shown in Figure.
The pseudo-code is listed in Alg.(the hard version).
One notable drawback of the hard version in Alg.
1 is that in each iteration the algorithm only updates its parameter based on the gradient from one of the k domains.
This is data inefﬁcient and can waste our computational resources in the forward process.
To improve this, we approximate the max function in (4) by the log-sum-exp function, which is a frequently used smooth approximation of the max function.
Deﬁne
[Equation]
[Table]
1: for t = 1 to ∞ do 2:
Sample { S ( t ) i } ki =1 and T ( t ) from { (cid:98) D S i } ki =1 and (cid:98) D T , each of size m 3: for i
= 1 to k do 4:
Compute (cid:98) ε ( t ) i := (cid:98) ε S ( t ) i ( h )
− min h (cid:48) ∈H ∆ H (cid:98) ε T ( t ) ,S ( t ) i ( h (cid:48) ) 5:
Compute w ( t ) i := exp( (cid:98) ε ( t ) i ) 6: end for 7: # Hard version 8:
Select i ( t ) :
= arg max i ∈
[ k ] (cid:98) ε ( t ) i 9:
Update parameters via backpropagating gradient of (cid:98) ε ( t ) i ( t ) 10: # Smoothed version 11: for i = 1 to k do 12:
Normalize w ( t ) i ← w ( t ) i / (cid:80) i (cid:48)
∈
[ k ] w ( t ) i (cid:48) 13: end for 14:
Update parameters via backpropagating gradient of (cid:80) i ∈
[ k ] w ( t ) i (cid:98)
ε ( t ) i 15: end for where γ > 0 is a parameter that controls the accuracy of this approximation.
As γ → ∞ ,γ log (cid:80) i ∈ [ k ] exp( γ (cid:98) ε i ( h ))
→ max i ∈ [ k ] (cid:98) ε i ( h ) .
Correspondingly, we can formulate a smoothed version of (4) as: minimize
[Equation]
During the optimization, (5) naturally provides an adaptive weighting scheme for the k source domains depending on their relative error.
Use θ to denote all the model parameters, then:
[Equation]
The approximation trick not only smooths the objective, but also provides a principled and adaptive way to combine all the gradients from the k source domains.
In words, (6) says that the gradient of
MDAN is a convex combination of the gradients from all the domains.
The larger the error from one domain, the larger the combination weight in the ensemble.
We summarize this algorithm in the smoothed version of Alg. 1.
Note that both algorithms, including the hard version and the smoothed version, reduce to the DANN algorithm (Ganin et al., 2016) when there is only one source domain.
E XPERIMENTS
We evaluate both hard and soft MDANs and compare them with state-of-the-art methods on three real-world datasets: the Amazon benchmark dataset (Chen et al., 2012) for sentiment analysis, a digit classiﬁcation task that includes 4 datasets: MNIST (LeCun et al., 1998), MNIST-M (Ganin et al., 2016), SVHN (Netzer et al., 2011), and SynthDigits (Ganin et al., 2016), and a public, largescale image dataset on vehicle counting from city cameras (Zhang et al., 2017).
Details about network architecture and training parameters of proposed and baseline methods, and detailed dataset description will be introduced in the appendix.
5. A MAZON R EVIEWS
Domains within the dataset consist of reviews on a speciﬁc kind of product (Books, DVDs, Electronics, and Kitchen appliances).
Reviews are encoded as 5000 dimensional feature vectors of unigrams and bigrams, with binary labels indicating sentiment.
We conduct 4 experiments: for each of them, we pick one product as target domain and the rest as source domains.
Each source domain haslabeled examples, and the target test set has 3000 to 6000 examples.
During training, we randomly sample the same number of unlabeled target examples as the source examples in each mini-batch.
We implement the Hard-Max and Soft-Max methods according to Alg. 1, and compare them with three baselines:
MLPNet, marginalized stacked denoising autoencoders (mSDA) (Chen et al., 2012), and
DANN (Ganin et al., 2016).
DANN cannot be directly applied in multiple source domains setting.
In order to make a comparison, we use two protocols.
The ﬁrst one is to combine all the source domains into a single one and train it using DANN, which we denote as (cDANN).
The second protocol is to train multiple DANNs separately, where each one corresponds to a source-target pair.
Among all the DANNs, we report the one achieving the best performance on the target domain.
We denote this experiment as (sDANN).
For fair comparison, all these models are built on the same basic network structure with one input layer ( 5000 units) and three hidden layers ( 1000 , 500 , 100 units).
Results and Analysis
We show the accuracy of different methods in Table 1.
Clearly, Soft-Max signiﬁcantly outperforms all other methods in most settings.
When Kitchen is the target domain, cDANN performs slightly better than Soft-Max, and all the methods perform close to each other.
Hard-Max is typically slightly worse than Soft-Max.
This is mainly due to the low data-efﬁciency of the Hard-Max model (Section 4, Eq. 4, Eq. 5).
We argue that with more training iterations, the performance of Hard-Max can be further improved.
These results verify the effectiveness of MDANs for multisource domain adaptation.
To validate the statistical signiﬁcance of the results, we run a non-parametric Wilcoxon signed-ranked test for each task to compare Soft-Max with the other competitors, as shown in Table 2.
Each cell corresponds to the p -value of a Wilcoxon test between
Soft-Max and one of the other methods, under the null hypothesis that the two paired samples have the same mean.
From these p-values, we see Soft-Max is convincingly better than other methods.
[Table] [Table]
5. D IGITS D ATASETS
Following the setting in (Ganin et al., 2016)
[Table]
we combine four popular digits datasets (MNIST,
MNIST-M, SVHN
[Table]
and SynthDigits) to build the multisource domain dataset.
We take each of
MNIST-M, SVHN
[Table]
and MNIST as target domain in turn, and the rest as sources.
Each source domain has 20 , 000 labeled images and the target test set has 9 , 000 examples.
We compare Hard-Max and
Soft-Max of MDANs with ﬁve baselines: i).
best-Single-Source .
A basic network trained on each source domain ( 20 , 000 images) without domain adaptation and tested on the target domain.
Among the three models, we report the one achieves the best performance on the test set.
The source-target pairs are: MNIST to MNIST-M; SVHN to MNIST, and MNIST-M to SVHN.
ii).
Combine-Source .
A basic network trained on a combination of three source domains ( 20 , 000 images for each) without domain adaptation and tested on the target domain. iii).
best-Single-DANN .
We train DANNs (Ganin et al., 2016) on each source-target domain pair ( 20 , 000 images) and test it on target.
Again, we report the best score among the three. iv).
Combine-DANN .
We train a single DANN on a combination of three source domains ( 20 , 000 images for each).
v).
Target-only .
It is the basic network trained and tested on the target data.
It serves as an upper bound of DA algorithms.
All the MDANs and baseline methods are built on the same basic network structure to put them on a equal footing.
Results and Analysis The classiﬁcation accuracy is shown in Table 3.
The results show that a naive combination of different training datasets can sometimes even decrease the performance.
Furthermore, we observe that adaptation to the SVHN dataset (the third experiment) is hard.
In this case, increasing the number of source domains does not help.
We conjecture this is due to the large dissimilarity between the SVHN data to the others.
For the combined sources, MDANs always perform better than the source-only baseline (MDANs vs. Combine-Source).
However, directly training DANN on a combination of multiple sources leads to worse performance when compared with our approach (Combine-DANN vs. MDANs).
In fact, this strategy may even lead to worse results than the source-only baseline (Combine-DANN vs. Combine-Source).
Surprisingly, using a single domain (best-Single DANN) can sometimes achieve the best result.
This means that in domain adaptation the quality of data (how close to the target data) is much more important than the quantity (how many source domains).
As a conclusion, this experiment further demonstrates the effectiveness of MDANs when there are multiple source domains available, where a naive combination of multiple sources using DANN may hurt generalization.
[Table]
S
T
MDANs
DANN
FCN
T
MDANs
DANN
FCN Hard-Max
Soft-Max
Hard-Max
Soft-Max
A 1.
B 2.
A 1.
B 1.
A 1.
B 1.
A 1.
B 1.
A 1.
B 1.
A 1.
B 1.
[Figure]
WebCamT is a public dataset for vehicle counting from large-scale city camera videos, which has low resolution ( 352 × 240 ), low frame rate (1 frame/second), and high occlusion.
It has 60 ,frames annotated with vehicle bounding box and count, divided into training and testing sets, with 42 , 200 and 17 , 800 frames, respectively.
Here we demonstrate the effectiveness of MDANs to count vehicles from an unlabeled target camera by adapting from multiple labeled source cameras: we select 8 cameras that each has more than 2 , 000 labeled images for our evaluations.
As shown in
Fig.
3, they are located in different intersections of the city with different scenes.
Among thesecameras, we randomly pick two cameras and take each camera as the target camera, with the othercameras as sources.
We compute the proxy
A -distance (PAD)
(Ben-David et al., 2007) between each source camera and the target camera to approximate the divergence between them.
We then rank the source cameras by the PAD from low to high and choose the ﬁrst k cameras to form the k source domains.
Thus the proposed methods and baselines can be evaluated on different numbers of sources (from 2 to 7 ).
We implement the Hard-Max and Soft-Max MDANs according to Alg. 1, based on the basic vehicle counting network FCN (Zhang et al., 2017).
We compare our method with two baselines:
FCN (Zhang et al., 2017), a basic network without domain adaptation, and DANN (Ganin et al., 2016), implemented on top of the same basic network.
We record mean absolute error (MAE) between true count and estimated count.
Results and Analysis
The counting error of different methods is compared in Table 4.
The HardMax version achieves lower error than DANN and FCN in most settings for both target cameras.
The Soft-Max approximation outperforms all the baselines and the Hard-Max in most settings, demonstrating the effectiveness of the smooth and adaptative approximation.
The lowest MAE achieved by Soft-Max is 1 . 1942 .
Such MAE means that there is only around one vehicle miscount for each frame (the average number of vehicles in one frame is around 20 ).
Fig.
2 shows the counting results of Soft-Max for the two target cameras under the 5 source cameras setting.
We can see that the proposed method accurately counts the vehicles of each target camera for long time sequences.
Does adding more source cameras always help improve the performance on the target camera?
To answer this question, we analyze the counting error when we vary the number of source cameras as shown in Fig. 4.
From the curves, we see the counting error goes down with more source cameras at the beginning, while it goes up when more sources are added at the end.
This phenomenon corresponds to the prediction implied by Thm.
3.4 (the last remark in Section 3): the performance on the target domain depends on the worst empirical error among multiple source domains, i.e., it is not always beneﬁcial to naively incorporate more source domains into training.
To illustrate this prediction better, we show the PAD of the newly added camera (when the source number increases by one) in Fig.
By observing the PAD and the counting error, we see the performance on the target can degrade when the newly added source camera has large divergence from the target camera.
C ONCLUSION
We derive a new generalization bound for DA under the setting of multiple source domains with labeled instances and one target domain with unlabeled instances.
The new bound has interesting interpretation and reduces to an existing bound when there is only one source domain.
Following our theoretical results, we propose MDANs to learn feature representations that are invariant under multiple domain shifts while at the same time being discriminative for the learning task.
Both hard and soft versions of MDANs are generalizations of the popular DANN to the case when multiple source domains are available.
Empirically, MDANs outperform the state-of-the-art DA methods on three real-world datasets, including a sentiment analysis task, a digit classiﬁcation task, and a visual vehicle counting task, demonstrating its effectiveness for multisource domain adaptation.
A O UTLINE
Organization of the appendix: 1).
For the convenience of exposition in showing our technical proofs, we ﬁrst introduce the technical tools that will be used during our proofs in Sec.
B. 2).
We provide detailed proofs for all the claims, lemmas and theorems presented in the main paper in Sec. C. 3).
We describe more experiment details in Sec.
D, including dataset description, network architecture and training parameters of the proposed and baseline methods, and more analysis of the experimental results. 4).
We introduce and discuss more related work about domain adaptation in Sec.
E.
B T ECHNICAL T OOLS
Deﬁnition B.1 (Growth function) .
The growth function Π H : N → N for a hypothesis class H is deﬁned by:
[Equation]
where X m = { x 1 , .
, x m } is a subset of X with size m .
Roughly, the growth function Π H ( m ) computes the maximum number of distinct ways in which m points can be classiﬁed using hypothesis in H .
A closely related concept is the Vapnik–Chervonenkis dimension (VC dimension) (Vapnik, 1998):
Deﬁnition B.2 (VC dimension) .
The VC-dimension of a hypothesis class H is deﬁned as:
[Equation]
A well-known result relating V C dim ( H ) and the growth function Π H ( m ) is the Sauer’s lemma:
Lemma B.1 (Sauer’s lemma) .
Let H be a hypothesis class with V C dim ( H ) =
d .
Then, for m ≥ d , the following inequality holds:
[Equation]
The following concentration inequality will be used:
Theorem B.1 (Hoeffding’s inequality) .
Let X 1 , . . .
, X n be independent random variables where each X i is bounded by the interval [ a i , b i ] .
Deﬁne the empirical mean of these random variables by ¯
X := 1 n (cid:80) ni
=1 X i , then ∀ ε > 0 :
[Equation]
The VC inequality allows us to give a uniform bound on the binary classiﬁcation error of a hypothesis class H using growth function:
Theorem B.2 (VC inequality) .
Let Π H be the growth function of hypothesis class H .
For h ∈ H , let ε ( h ) be the true risk of h w.r.t.
the generation distribution D and the true labeling function h ∗ .
Similarly, let ˆ ε n ( h ) be the empirical risk on a random i.i.d.
sample containing n instances from D , then, for ∀ ε > 0 , the following inequality hold:
[Equation]
Although the above theorem is stated for binary classiﬁcation error, we can extend it to any bounded error.
This will only change the multiplicative constant of the bound.
C P ROOFS
For all the proofs presented here, the following lemma shown by Blitzer et al.
(2008) will be repeatedly used:
Lemma C.
[Equation]
et al.,
[Equation] [Equation]
(cid:48) ∈ H ,
[Equation]
( D S
[Equation]
D T ) .
C. P ROOF OF T HM . 3.
One technical lemma we will frequently use to prove Thm.
3.1 is the triangular inequality w.r.t.
ε D ( h ) , ∀
h ∈ H :
Lemma C.2.
For any hypothesis class H and any distribution D on X , the following triangular inequality holds:
[Equation]
Proof.
[Equation]
Now we are ready to prove Thm. 3.1:
Theorem 3.1.
ε T ( h ) ≤ max i ∈
[ k ] ε S i ( h ) + λ + 12 d H ∆ H ( D T ; {D S i } ki =1 ) .
Proof.
∀ h ∈ H , deﬁne i h :=
arg max i ∈
[ k ] ε S i ( h, h ∗ ) : ε T ( h ) ≤ ε T ( h ∗ ) + ε T ( h, h ∗ ) =
ε T ( h ∗ ) + ε T ( h, h ∗ ) − max i ∈ [ k ] ε S i ( h, h ∗ )
+ max i ∈
[ k ] ε S i ( h, h ∗ ) ≤ ε T ( h ∗ ) + | ε T ( h, h ∗ ) − ε S
ih
( h, h ∗ ) | + ε S
ih
( h, h ∗ ) ≤
[Equation]
) ≤
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
S ih (
[Equation]
h ∗
[Equation]
The ﬁrst and the ﬁfth inequalities are due to the triangle inequality, and the third inequality is based on Lemma C.1.
The second holds due to the property of | · | and the others follow by the deﬁnition of
H -divergence.
[Equation]
C. P ROOF OF T HM . 3.
Theorem 3.2.
Let D T and {D S i } k i =1 be the target distribution and k source distributions over X .
Let
H be a hypothesis class where V C dim ( H ) =
d .
If (cid:98) D T and { (cid:98) D S i } ki =1 are the empirical distributions of D T and {D S i } ki =1 generated with m i.i.d.
samples from each domain, then for (cid:15) > 0 , we have:
[Equation]
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
The ﬁrst inequality holds due to the sub-additivity of the max function
[Equation]
and the second inequality is due to the union bound.
The third inequality holds because of the triangle inequality
[Equation]
and we use the averaging argument to establish the fourth inequality.
The ﬁfth inequality is an application of the
VC-inequality
[Equation]
and the sixth is by the Hoeffding’s inequality.
Finally, we use the Sauer’s lemma to prove the last inequality.
[Equation]
C. P ROOF OF T HM . 3.
We now show the detailed proof of Thm. 3.3.
Proof.
[Equation]
Again, the ﬁrst inequality is due to the subadditivity of the max function, and the second inequality holds due to the union bound.
We apply the VC-inequality to bound the third inequality, and
Hoeffding’s inequality to bound the fourth.
Again, the last one is due to Sauer’s lemma.
(cid:4)
C. D ERIVATION OF THE D ISCREPANCY D ISTANCE AS C LASSIFICATION E RROR
We show that the H -divergence is equivalent to a binary classiﬁcation accuracy in discriminating instances from different domains.
Suppose A H is symmetric,
i.e., A ∈ A H ⇔ X\ A ∈
A H , and we have samples { S i } ki =1 and T from {D S i } ki =1 and D T respectively, each of which is of size m , then: d H ∆ H ( ˆ D T ; { ˆ D S i } ki =1 ) =
[Equation]
i ∈ [ k ]
[Equation]
D D ETAILS ABOUT E XPERIMENTS
In this section, we describe more details about the datasets and the experimental settings.
We extensively evaluate the proposed methods on three datasets: 1).
We ﬁrst evaluate our methods on
Amazon Reviews dataset (Chen et al., 2012) for sentiment analysis. 2).
We evaluate the proposed methods on the digits classiﬁcation datasets including MNIST (LeCun et al., 1998), MNIST-M (Ganin et al., 2016), SVHN (Netzer et al., 2011), and SynthDigits (Ganin et al., 2016).
We further evaluate the proposed methods on the public dataset WebCamT (Zhang et al., 2017) for vehicle counting.
It contains 60,000 labeled images from 12 city cameras with different distributions.
Due to the substantial difference between these datasets and their corresponding learning tasks, we will introduce more detailed dataset description, network architecture, and training parameters for each dataset respectively in the following subsections.
D. D ETAILS ON A MAZON R EVIEWS EVALUATION
Amazon reviews dataset includes four domains, each one composed of reviews on a speciﬁc kind of product (Books, DVDs, Electronics, and Kitchen appliances).
Reviews are encoded asdimensional feature vectors of unigrams and bigrams.
The labels are binary:
0 if the product is ranked up to 3 stars, and 1 if the product is ranked 4 or 5 stars.
We take one product domain as target and the other three as source domains.
Each source domain has 2000 labeled examples and the target test set has 3000 to 6000 examples.
We implement the
Hard-Max and Soft-Max methods according to Alg. 1, based on a basic network with one input layer ( 5000 units) and three hidden layers ( 1000 , 500 , 100 units).
The network is trained for 50 epochs with dropout rate 0 .
We compare Hard-Max and Soft-Max with three baselines: Baseline 1: MLPNet .
It is the basic network of our methods (one input layer and three hidden layers), trained for 50 epochs with dropout rate 0 .
Baseline 2: Marginalized Stacked Denoising Autoencoders (mSDA)
(Chen et al., 2012).
It takes the unlabeled parts of both source and target samples to learn a feature map from input space to a new representation space.
As a denoising autoencoder algorithm, it ﬁnds a feature representation from which one can (approximately) reconstruct the original features of an example from its noisy counterpart.
Baseline 3: DANN .
We implement DANN based on the algorithm described in (Ganin et al., 2016) with the same basic network as our methods.
Hyper parameters of the proposed and baseline methods are selected by cross validation.
Table 5 summarizes the network architecture and some hyper parameters.
D. D ETAILS ON D IGIT D ATASETS EVALUATION
We evaluate the proposed methods on the digits classiﬁcation problem.
Following the experiments in (Ganin et al., 2016), we combine four popular digits datasets-MNIST, MNIST-M, SVHN, and
SynthDigits to build the multi-source domain dataset.
MNIST is a handwritten digits database with 60 , 000 training examples, and 10 , 000 testing examples.
The digits have been size-normalized and
[Table]
centered in a 28 × 28 image.
MNIST-M is generated by blending digits from the original MNIST set over patches randomly extracted from color photos from BSDS500 (Arbelaez et al., 2011; Ganin et al., 2016).
It has 59 , 001 training images and 9 , 001 testing images with 32 × 32 resolution.
An output sample is produced by taking a patch from a photo and inverting its pixels at positions corresponding to the pixels of a digit.
For DA problems, this domain is quite distinct from MNIST, for the background and the strokes are no longer constant.
SVHN is a real-world house number dataset with 73 , 257 training images and 26 , 032 testing images.
It can be seen as similar to MNIST, but comes from a signiﬁcantly harder, unsolved, real world problem.
SynthDigits consists of 500;digit images generated by Ganin et al.
from WindowsTM fonts by varying the text, positioning, orientation, background and stroke colors, and the amount of blur.
The degrees of variation were chosen to simulate SVHN, but the two datasets are still rather distinct, with the biggest difference being the structured clutter in the background of SVHN images.
We take MNIST-M, SVHN, and MNIST as target domain in turn, and the remaining three as sources.
We implement the Hard-Max and Soft-Max versions according to Alg. 1 based on a basic network, as shown in Fig.
The baseline methods are also built on the same basic network structure to put them on a equal footing.
The network structure and parameters of MDANs are illustrated in Fig.
The learning rate is initialized by 0 . 01 and adjusted by the ﬁrst and second order momentum in the training process.
The domain adaptation parameter of MDANs is selected by cross validation.
In each mini-batch of MDANs training process, we randomly sample the same number of unlabeled target images as the number of the source images.
D. D ETAILS ON W EB C AM T V EHICLE C OUNTING
WebCamT is a public dataset for large-scale city camera videos, which have low resolution ( 352 × 240 ), low frame rate (1 frame/second), and high occlusion.
WebCamT has 60 , 000 frames annotated with rich information: bounding box, vehicle type, vehicle orientation, vehicle count, vehicle reidentiﬁcation, and weather condition.
The dataset is divided into training and testing sets, with 42,200 and 17,800 frames, respectively, covering multiple cameras and different weather conditions.
WebCamT is an appropriate dataset to evaluate domain adaptation methods, for it covers multiple city cameras and each camera is located in different intersection of the city with different perspectives and scenes.
Thus, each camera data has different distribution from others.
The dataset is quite challenging and in high demand of domain adaptation solutions, as it has 6 , 000 , 000 unlabeled images from cameras with only 60 , 000 labeled images from 12 cameras.
The experiments on WebCamT provide an interesting application of our proposed MDANs: when dealing with spatially and temporally large-scale dataset with much variations, it is prohibitively expensive and time-consuming to label large amount of instances covering all the variations.
As a result, only a limited portion of the dataset can be annotated, which can not cover all the data domains in the dataset.
MDAN provide an effective solution for this kind of application by adapting the deep model from multiple source domains to the unlabeled target domain.
We evaluate the proposed methods on different numbers of source cameras.
Each source camera provides 2000 labeled images for training and the test set has 2000 images from the target camera.
In each mini-batch, we randomly sample the same number of unlabeled target images as the source images.
We implement the Hard-Max and Soft-Max version of MDANs according to Alg. 1, based on the basic vehicle counting network FCN described in (Zhang et al., 2017).
Please refer to (Zhang et al., 2017) for detailed network architecture and parameters.
The learning rate is initialized by 0 .and adjusted by the ﬁrst and second order momentum in the training process.
The domain adaptation parameter is selected by cross validation.
We compare our method with two baselines: Baseline 1:
FCN .
It is our basic network without domain adaptation as introduced in work (Zhang et al., 2017).
Baseline 2: DANN .
We implement DANN on top of the same basic network following the algorithm introduced in work (Ganin et al., 2016).
E M ORE R ELATED W ORK
A number of adaptation approaches have been studied in recent years.
From the theoretical aspect, several theoretical results have been derived in the form of upper bounds on the generalization target error by learning from the source data.
A keypoint of the theoretical frameworks is estimating the distribution shift between source and target.
Kifer et al.
(2004) proposed the H -divergence to measure the similarity between two domains and derived a generalization bound on the target domain using empirical error on the source domain and the H -divergence between the source and the target.
This idea has later been extended to multisource domain adaptation (Blitzer et al., 2008) and the corresponding generalization bound has been developed as well.
Ben-David et al.
(2010) provide a generalization bound for domain adaptation on the target risk which generalizes the standard bound on the source risk.
This work formalizes a natural intuition of DA: reducing the two distributions while ensuring a low error on the source domain and justiﬁes many DA algorithms.
Based on this work,
Mansour et al.
(2009a) introduce a new divergence measure: discrepancy distance, whose empirical estimate is based on the Rademacher complexity (Koltchinskii, 2001) (rather than the VC-dim).
Other theoretical works have also been studied such as (Mansour & Schain, 2012) that derives the generalization bounds on the target error by taking use of the robustness properties introduced in (Xu & Mannor, 2012).
See (Cortes et al., 2008; Mansour et al., 2009a;c) for more details.
Following the theoretical developments, many DA algorithms have been proposed, such as instancebased methods (Tsuboi et al., 2009); feature-based methods (Becker et al., 2013); and parameterbased methods (Evgeniou & Pontil, 2004).
The general approach for domain adaptation starts from algorithms that focus on linear hypothesis class (Blitzer et al., 2006; Germain et al., 2013;
Cortes & Mohri, 2014).
The linear assumption can be relaxed and extended to the non-linear setting using the kernel trick, leading to a reweighting scheme that can be efﬁciently solved via quadratic programming (Huang et al., 2006;
Gong et al., 2013).
Recently, due to the availability of rich data and powerful computational resources, non-linear representations and hypothesis classes have been increasingly explored (Glorot et al., 2011; Baktashmotlagh et al., 2013; Chen et al., 2012;
Ajakan et al., 2014; Ganin et al., 2016).
This line of work focuses on building common and robust feature representations among multiple domains using either supervised neural networks (Glorot et al., 2011), or unsupervised pretraining using denoising auto-encoders (Vincent et al., 2008; 2010).
Recent studies have shown that deep neural networks can learn more transferable features for
DA (Glorot et al., 2011; Donahue et al., 2014; Yosinski et al., 2014).
Bousmalis et al.
(2016) develop domain separation networks to extract image representations that are partitioned into two subspaces: domain private component and cross-domain shared component.
The partitioned representation is utilized to reconstruct the images from both domains, improving the DA performance.
Reference (Long et al., 2015) enables classiﬁer adaptation by learning the residual function with reference to the target classiﬁer.
The main-task of this work is limited to the classiﬁcation problem.
Ganin et al.
(2016) propose a domain-adversarial neural network to learn the domain indiscriminate but main-task discriminative features.
Although these works generally outperform non-deep learning based methods, they only focus on the single-source-single-target DA problem, and much work is rather empirical design without statistical guarantees.
Hoffman et al.
(2012) present a domain transform mixture model for multisource DA, which is based on non-deep architectures and is difﬁcult to scale up.
Adversarial training techniques that aim to build feature representations that are indistinguishable between source and target domains have been proposed in the last few years (Ajakan et al., 2014;
Ganin et al., 2016).
Speciﬁcally, one of the central ideas is to use neural networks, which are powerful function approximators, to approximate a distance measure known as the H -divergence between two domains (Kifer et al., 2004; Ben-David et al., 2007; 2010).
The overall algorithm can be viewed as a zero-sum two-player game: one network tries to learn feature representations that can fool the other network, whose goal is to distinguish representations generated from the source domain between those generated from the target domain.
The goal of the algorithm is to ﬁnd a Nash-equilibrium of the game, or the stationary point of the min-max saddle point problem.
Ideally, at such equilibrium state, feature representations from the source domain will share the same distributions as those from the target domain, and, as a result, better generalization on the target domain can be expected by training models using only labeled instances from the source domain.