A S IMPLE U NIFIED I NFORMATION R EGULARIZATION F RAMEWORK FOR M ULTI -S OURCE D OMAIN A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Adversarial learning strategy has demonstrated remarkable performance in dealing with single-source unsupervised Domain Adaptation (DA) problems, and it has recently been applied to multi-source DA problems.
Although most existing DA methods use multiple domain discriminators, the effect of using multiple discriminators on the quality of latent space representations has been poorly understood.
Here we provide theoretical insights into potential pitfalls of using multiple domain discriminators: First, domain-discriminative information is inevitably distributed across multiple discriminators.
Second, it is not scalable in terms of computational resources.
Third, the variance of stochastic gradients from multiple discriminators may increase, which signiﬁcantly undermines training stability.
To fully address these issues, we situate adversarial DA in the context of information regularization.
First, we present a uniﬁed information regularization framework for multi-source
DA.
It provides a theoretical justiﬁcation for using a single and uniﬁed domain discriminator to encourage the synergistic integration of the information gleaned from each domain.
Second, this motivates us to implement a novel neural architecture called a Multi-source Information-regularized Adaptation Networks (MIAN).
The proposed model signiﬁcantly reduces the variance of stochastic gradients and increases computational-efﬁciency.
Large-scale simulations on various multi-source
DA scenarios demonstrate that MIAN, despite its structural simplicity, reliably outperforms other state-of-the-art methods by a large margin especially for difﬁcult target domains.
I NTRODUCTION
Although a large number of studies have demonstrated the ability of deep neural networks to solve challenging tasks, the tasks solved by networks are mostly conﬁned to a similar type or a single domain.
One remaining challenge is the problem known as domain shift (Gretton et al.
(2009)), where a direct transfer of information gleaned from a single source domain to unseen target domains may lead to signiﬁcant performance impairment.
Domain adaptation (DA) approaches aim to mitigate this problem by learning to map data of both domains onto a common feature space.
Whereas several theoretical results (Ben-David et al.
; Blitzer et al.
; Zhao et al. (2019a)) and algorithms for DA (Long et al. (2015; 2017); Ganin et al.
(2016)) have focused on the case in which only a single-source domain dataset is given, we consider a more challenging and generalized problem of knowledge transfer, referred to as Multi-source unsupervised DA (MDA).
Following a seminal theoretical result on MDA (Blitzer et al.
; Ben-David et al.
, technical advances have been made, mainly on the adversarial methods.
(Xu et al.
; Zhao et al. (2019c)).
While most of adversarial MDA methods use multiple independent domain discriminators (Xu et al.
; Zhao et al.
; Li et al.
; Zhao et al.
(2019c;b)), the potential pitfalls of this setting have not been fully explored.
The existing works do not provide a theoretical guarantee that the unnecessary domain-speciﬁc information is fully ﬁltered out, because the domain-discriminative information is inevitably distributed across multiple discriminators.
For example, the multiple domain discriminators focus only on estimating the domain shift between source domains and the target, while the discrepancies between the source domains are neglected, making it hard to align all the given domains.
This necessitates garnering the domain-discriminative information with a uniﬁed discriminator.
Moreover, the multiple domain discriminator setting is not scalable in terms of computational resources especially when large number of source domains are given, e.g., medical reports from multiple patients.
Finally, it may undermine the stability of training, as earlier works solve multiple independent adversarial minimax problems.
To overcome such limitations, we propose a novel MDA method, called Multi-source Informationregularized Adaptation Networks ( MIAN ), that constrains the mutual information between latent representations and domain labels.
First, we show that such mutual information regularization is closely related to the explicit optimization of the H -divergence between the source and target domains.
This affords the theoretical insight that the conventional adversarial DA can be translated into an information-theoretic-regularization problem.
Second, based on our ﬁndings, we propose a new optimization problem for MDA: minimizing adversarial loss over multiple domains with a single domain discriminator.
We show that the domain shift between each source domain can be indirectly penalized, which is known to be beneﬁcial in MDA (Li et al.
; Peng et al.
(2019)), with a single domain discriminator.
Moreover, by analyzing existing studies in terms of information regularization, we found that the variance of the stochastic gradients increases when using multiple discriminators.
Despite its structural simplicity, we found that MIAN works efﬁciently across a wide variety of MDA scenarios, including the DIGITS-Five (Peng et al.
, Ofﬁce-31 (Saenko et al.
(2010)), and
Ofﬁce-Home datasets (Venkateswara et al.
Intriguingly, MIAN reliably and signiﬁcantly outperformed several state-of-the-art methods that either employ a domain discriminator separately for each source domain (Xu et al.
or align the moments of deep feature distribution for every pairwise domain (Peng et al.
R ELATED WORKS
Several DA methods have been used in attempt to learn domain-invariant representations.
Along with the increasing use of deep neural networks, contemporary work focuses on matching deep latent representations from the source domain with those from the target domain.
Several measures have been introduced to handle domain shift, such as maximum mean discrepancy (MMD)
(Long et al.
(2014; 2015)), correlation distance (Sun et al.
; Sun & Saenko (2016)), and Wasserstein distance (Courty et al.
Recently, adversarial DA methods (Ganin et al.
; Tzeng et al.
; Hoffman et al.
Saito et al.
(2018; 2017)) have become mainstream approaches owing to the development of generative adversarial networks (Goodfellow et al.
However, the abovementioned single-source DA approaches inevitably sacriﬁce performance for the sake of multi-source DA.
Some MDA studies (Blitzer et al.
; Ben-David et al.
; Mansour et al.
; Hoffman et al.
(2018)) have provided the theoretical background for algorithm-level solutions.
(Blitzer et al.
(2008); Ben-David et al.
(2010)) explore the extended upper bound of true risk on unlabeled samples from the target domain with respect to a weighted combination of multiple source domains.
Following these theoretical studies, MDA studies with shallow models (Duan et al.
(2012b;a);
Chattopadhyay et al.
(2012)) as well as with deep neural networks (Mancini et al.
; Peng et al.
; Li et al.
(2018)) have been proposed.
Recently, some adversarial MDA methods have also been proposed.
Xu et al.
(2018) implemented a k-way domain discriminator and classiﬁer to battle both domain and category shifts.
Zhao et al.
(2018) also used multiple discriminators to optimize the average case generalization bounds.
Zhao et al.
(2019c) chose relevant source training samples for the DA by minimizing the empirical Wasserstein distance between the source and target domains.
Instead of using separate encoders, domain discriminators or classiﬁers for each source domain as in earlier works, our approach uses uniﬁed networks, thereby improving resource-efﬁciency and scalability.
Several existing MDA works have proposed methods to estimate the source domain weights following (Blitzer et al.
; Ben-David et al.
Mansour et al.
(2009) assumed that the target hypothesis can be approximated by a convex combination of the source hypotheses.
(Peng et al.
; Zhao et al.
(2018)) suggested ad-hoc schemes for domain weights based on the empirical risk of each source domain.
Li et al.
(2018) computed a softmax-transformed weight vector using the empirical Wasserstein-like measure instead of the empirical risks.
Compared to the proposed methods without robust theoretical justiﬁcations, our analysis does not require any assumption or estimation for the domain coefﬁcients.
In our framework, the representations are distilled to be independent of the domain, thereby rendering the performance relatively insensitive to explicit weighting strategies.
T HEORETICAL INSIGHTS
We ﬁrst introduce the notations for the MDA problem in classiﬁcation.
A set of source domains and the target domain are denoted by { D S i } Ni =1 and D T , respectively.
Let X S
i
=
(cid:110)
x
jS i (cid:111) m j =1 and
Y S i
=
(cid:110)
y jS i (cid:111) m j =1 be a set of m i.i.d.
samples from D S i .
Let X T =
(cid:8) x T j (cid:9) m
j =1 ∼ ( D XT )
m be the set of m i.i.d.
samples generated from the marginal distribution D XT .
The domain label and its probability distribution are denoted by V and P V ( v ) , where v ∈ V and V is the set of domain labels.
In line with prior works (Hoffman et al.
; Gong et al.
; Mancini et al.
; Gong et al.
, domain label can be generally treated as a stochastic latent random variable in our framework.
However, for simplicity, we take the empirical version of the true distributions with given samples assuming that the domain labels for all samples are known.
The latent representation of the sample is given by Z , and the encoder is deﬁned as F : X → Z , with X and Z representing data space and latent space, respectively.
Accordingly, Z S i and Z T refer to the outputs of the encoder
F ( X S i ) and F ( X T ) , respectively.
For notational simplicity, we will omit the index i from D S
i , X S i and Z S i
when N = 1 .
A classiﬁer is deﬁned as C :
Z → Y where Y is the class label space.
3. P ROBLEM FORMULATION
For comparison with our formulation, we recast single-source DA as a constrained optimization problem.
The true risk (cid:15) T ( h ) on unlabeled samples from the target domain is bounded above the sum of three terms (Ben-David et al.
(2010)): (1) true risk (cid:15) S ( h ) of hypothesis h on the source domain; (2)
H -divergence d H ( D S , D T ) between a source and a target domain distribution; and (3) the optimal joint risk λ ∗ .
Theorem 1 (Ben-David et al.
Let hypothesis class H be a set of binary classiﬁers h : X → { 0 , 1 } .
Then for the given domain distributions D S and D T ,
[Equation]
where
[Equation]
(cid:2)
I ( h ( x ) = 1) (cid:3)(cid:12)(cid:12)(cid:12) and I ( a ) is an indicator function whose value is 1 if a is true, and 0 otherwise.
The empirical H -divergence ˆ d H ( X S , X T ) can be computed as follows (Ben-David et al.
Lemma 1.
[Equation]
Following Lemma 1, a domain classiﬁer h :
Z → V can be used to compute the empirical Hdivergence.
Suppose the optimal joint risk λ ∗ is sufﬁciently small as assumed in most adversarial DA studies (Saito et al.
; Chen et al.
Thus, one can obtain the ideal encoder and classiﬁer minimizing the upper bound of (cid:15) T ( h ) by solving the following min-max problem:
[Equation]
where L ( F, C ) is the loss function on samples from the source domain, β is a Lagrangian multiplier,
V = { 0 , 1 } such that each source instance and target instance are labeled as 1 and 0 , respectively, and h is the binary domain classiﬁer.
Note that the latter min–max problem is obtained by converting − min into max and removing the constant term from Lemma 1.
3. I NFORMATION - REGULARIZED MIN – MAX PROBLEM FOR MDA
Intuitively, it is not highly desirable to adapt the learned representation in the given domain to the other domains, particularly when the representation itself is not sufﬁciently domain-independent.
This motivates us to explore ways to learn representations independent of domains.
Inspired by a contemporary fair model training study (Roh et al.
the mutual information between the latent representation and the domain label I ( Z ; V ) can be expressed as follows:
Theorem 2.
Let P Z ( z ) be the distribution of Z where z ∈ Z .
Let h be a domain classiﬁer h : Z → V , where Z is the feature space and V is the set of domain labels.
Let h v ( z ) be a conditional probability of V where v ∈ V given Z = z , deﬁned by h .
Then the following holds:
[Equation]
The detailed proof is provided in the Roh et al.
(2020) and Supplementary Material.
As done in Roh et al.
(2020), we can derive the empirical version of Theorem 2 as follows:
[Equation]
where M is the number of total representation samples, i is the sample index, and v i is the corresponding domain label of the i th sample.
Using this equation, we combine our information-constrained objective function and the results of Lemma 1.
For binary classiﬁcation V = { 0 , 1 } with Z S and Z T of equal size M/ 2 , we propose the following information-regularized minimax problem:
[Equation]
where β is a Lagrangian multiplier, h ( z i ) (cid:44)
h v i =
( z i ) and 1 − h
( z i ) (cid:44)
h v i =
( z i ) , with h ( z i ) representing the probability that z i belongs to the source domain.
This setting automatically dismisses the condition (cid:80) v ∈V h v ( z ) = 1 , ∀ z .
Note that we have accommodated a simple situation in which the entropy H ( V ) remains constant.
3. A DVANTAGES OVER OTHER MDA METHODS
The relationship between (3) and (6) provides us a theoretical insights that the problem of minimizing mutual information between the latent representation and the domain label is closely related to minimizing the H -divergence using the adversarial learning scheme.
This relationship clearly underlines the signiﬁcance of information regularization for MDA.
Compared to the existing MDA approaches (Xu et al.
; Zhao et al.
(2018)), which inevitably distribute domain-discriminative knowledge over N different domain classiﬁers, the above objective function (6) enables us to seamlessly integrate such information with the single-domain classiﬁer h .
Using a single domain discriminator also helps reduce the variance of gradient.
Large variances in the stochastic gradients slow down the convergence, which leads to poor performance (Johnson &
Zhang (2013)).
Herein, we analyze the variances of the stochastic gradients of existing optimization constraints.
By excluding the weighted source combination strategy, we can simplify the optimization constraint of existing adversarial MDA methods as sum of the information constraints:
[Equation]
where U k is the k th domain label with U = { 0 , 1 } , P Z k | u =0 ( · )
=
P Z | v = N +1 ( · ) corresponding to the target domain, P Z k | u
=1 ( · ) =
P Z | v = k ( · ) corresponding to the k th source domain, and h k u ( z k ) being the conditional probability of u ∈ U given z k deﬁned by the k th discriminator indicating that the sample is generated from the k th source domain.
Again, we treat the entropy H ( U k ) as a constant.
Note that the interaction information cannot be measured with (7).
Given M = m ( N +1) samples with m representing the number of samples per domain, an empirical version of (7) is:
[Equation]
Let I k be a shorthand for the k th term inside the ﬁrst summation.
Without loss of generality we make simplifying assumptions that all V ar
[ I k ] are the same for all k
and so are Cov
[ I k , I j ] for all pairs.
Then the variance of (8) is given by:
[Equation]
As earlier works solve N adversarial minimax problems, the covariance term is additionally included and its contribution to the variance does not decrease with increasing N .
In other words, the covariance term may dominate the variance of the gradients as the number of domain increases.
In contrast, the variance of our constraint (5) is inversely proportional to ( N + 1) 2 .
Let I m be a shorthand for the maximization term except 1 M in (5).
Then the variance of (5) is given by:
[Equation]
It implies that our framework can signiﬁcantly improve the stability of stochastic gradient optimization compared to existing approaches, especially when the model is deemed to learn from many domains.
3. S ITUATING DOMAIN ADAPTATION IN CONTEXT OF INFORMATION BOTTLENECK THEORY
In this Section, we bridge the gap between the existing adversarial DA method and the information bottleneck (IB) theory (Tishby et al.
; Tishby & Zaslavsky (2015); Alemi et al.
Tishby et al.
(2000) examined the problem of learning an encoding Z such that it is maximally informative about the class Y while being minimally informative about the sample X :
[Equation]
where β is a Lagrangian multiplier.
Indeed, the role of the bottleneck term
I
( Z ; X ) matches our mutual information
I ( Z ; V ) between the latent representation and the domain label.
We foster close collaboration between two information bottleneck terms by incorporating those into I ( Z ; X, V ) .
Theorem 3.
Let P Z | x , v ( z ) be a conditional probabilistic distribution of Z where z ∈ Z , deﬁned by the encoder F , given a sample x ∈ X and the domain label v ∈ V .
Let R Z ( z ) denotes a prior marginal distribution of Z .
Then the following inequality holds:
[Equation]
The proof of Theorem 3 uses the chain rule: I ( Z ; X, V )
=
I ( Z ; V ) + I
( Z ; X | V ) .
The detailed proof is provided in the Supplementary Material.
Whereas the role of I ( Z ; X | V ) is to purify the latent representation generated from the given domain, I ( Z ; V ) serves as a proxy for regularization that aligns the puriﬁed representations across different domains.
Thus, the existing DA approaches (Luo et al.
; Song et al.
using variational information bottleneck (Alemi et al.
(2016)) can be reviewed as special cases for Theorem 3 with a single-source domain.
M ULTI - SOURCE I NFORMATION - REGULARIZED A DAPTATION N ETWORKS
In this Section, we provide the details of our proposed architecture, referred to as a multi-source information-regularized adaptation network ( MIAN ).
MIAN addresses the information-constrained min–max problem for MDA (Section 3.2) using the three subcomponents depicted in Figure 1: information regularization, source classiﬁcation, and Decaying Batch Spectral Penalization (DBSP).
Information regularization.
To estimate the empirical mutual information ˆ I ( Z ; V ) in (5), the domain classiﬁer h should be trained to minimize softmax cross enropy.
Let V = { 1 , 2 , ..., N + 1 } and denote h ( z ) as N + 1 dimensional vector of the conditional probability for each domain given the sample z .
Let 1 be a N + 1 dimensional vector of all ones, and 1 [ k = v ] be a N + 1 dimensional vector whose v th value is 1 and 0 otherwise.
Given M = m ( N + 1) samples, the objective is:
[Equation]
In this study, we slightly modify the objective (13).
Speciﬁcally, we explicitly minimized the conditional probability of the remaining domains excepting the v th domain.
Let 1 [ k (cid:54) = v ] be the ﬂipped version of 1 [ k = v ] .
Then the objective function for the domain discriminator is:
[Equation]
where the objective function for encoder training is to maximize (14).
Our objective function is also closely related to that of GAN (Goodfellow et al.
and we experimentally found that using the variant objective function of GAN (Mao et al.
(2017)) works slightly better.
The above objective is closely related to optimizing every pairwise domain discrepancy between the given domain and the mixture of the others.
Let each D v and D v c represent the v th domain and the mixture of the remaining N domains with the same mixture weight 1 N , relatively.
Then we can deﬁne H -divergence as d H ( D v , D v c ) , and an average of such H -divergence for every v as d H ( V ) .
Assume that the samples of size m , Z v and Z v c , are generated from each D v and D v c , where Z v c = (cid:83) v (cid:48) (cid:54) = v Z v (cid:48) with | Z v (cid:48)
| = m/N for all v (cid:48) ∈ V .
Thus the domain label v j (cid:54) = v for every j th sample in Z v c .
Then the average of empirical H -divergence ˆ d H ( V ) is deﬁned as follows:
[Equation]
where h v ( z ) represents the v th value of h ( z ) .
Note that h ( z ) corresponds to N + 1 dimensional one-hot classiﬁcation vector in (15), unlike in (14).
Then, let I [ h ( z )] := (cid:2)
I ( h v ( z ) = 1) (cid:3) v ∈V be the
N + 1 dimensional one-hot indicator vector.
Given the uniﬁed domain discriminator h in the inner minimization for every v in (15), we train h to approximate the lower bound of ˆ d H ( V ) as follows:
[Equation]
where the latter equality is obtained by rearranging the summation terms in the ﬁrst equality.
Based on the close relationship between (14) and (16), we can make the link between information regularization and H -divergence optimization given multi-source domain; minimizing ˆ d H ( V ) is closely related to implicit regularization of the mutual information between latent representations and domain labels.
Because the output vector h ( z ) in (15) often comes from the arg max operation, (15) is not differentiable w.r.t.
z .
However, our framework has a differentiable objective as in (14).
There are two beneﬁts of minimizing d H ( V ) .
First, it includes H -divergence between the target and a mixture of sources, which directly affects the upper bound of the empirical risk on target samples (Theorem 5 in Ben-David et al.
Second, d H ( V ) lower-bounds the average of every pairwise
H -divergence between domains.
The detailed proof is provided in the appendix (Lemma 2).
Note that unlike our single domain classiﬁer setting, existing methods (Li et al.
(2018)) require a number of about O ( N 2 ) domain classiﬁers to approximate all pairwise combinations of domain discrepancy.
Source classiﬁcation.
Along with learning domain-independent latent representations illustrated in the above, we train the classiﬁer with the labeled source domain datasets that can be directly applied to the target domain representations in practice.
To minimize the empirical risk on source domain, we use a generic softmax cross-entropy loss function with labeled source domain samples as L ( F, C ) .
Decaying batch spectral penalization.
Applying above information-theoretic insights, we further describe a potential side effect of existing adversarial DA methods.
Information regularization may lead to overriding implicit entropy minimization, particularly in the early stages of the training, impairing the richness of latent feature representations.
To prevent such a pathological phenomenon, we introduce a new technique called Decaying Batch Spectral Penalization (DBSP), which is intended to control the SVD entropy of the feature space.
Our version improves training efﬁciency compared to original Batch Spectral Penalization (Chen et al.
We refer to this version of our model as MIAN- γ .
As vanila MIAN is sufﬁcient to outperform other state-of-the-art methods (Section 5),
MIAN- γ is further discussed in the Supplementary Material.
E XPERIMENTS
[Table]
M 3 SDA- β 72.
MIAN 84.
To assess the performance of MIAN , we ran a large-scale simulation using the following benchmark datasets: Digits-Five, Ofﬁce-31 and Ofﬁce-Home.
For a fair comparison, we reproduced all the other baseline results using the same backbone architecture and optimizer settings as the proposed method.
For the source-only and single-source DA standards, we introduce two MDA approaches (Xu et al.
; Peng et al.
(2019)): ( 1 ) source-combined, i.e., all source-domains are incorporated into a single source domain; ( 2 ) single-best, i.e., the best adaptation performance on the target domain is reported.
Owing to limited space, details about simulation settings, used baseline models and datasets are presented in the Supplementary Material.
[Table] [Table]
5. S IMULATION RESULTS
The classiﬁcation accuracy for Digits-Five, Ofﬁce-31, and Ofﬁce-Home are summarized in Tables 1,
[Table]
and
[Table]
respectively.
We found that MIAN outperforms most of other state-of-the-art single-source and multi-source DA methods by a large margin.
Note that our method demonstrated a signiﬁcant improvement in challenging domains, such as MNIST-M, Amazon or Clipart.
5. Q UALITATIVE AND QUANTITATIVE ANALYSES
Design of domain discriminator.
To quantify the extent to which performance improvement is achieved by unifying the domain discriminators, we compared the performances of the four different versions of MIAN (Figure 2a, 2b).
No S-S align is the same as MIAN with the exception that only the target and each source domains are aligned.
No LS uses the objective function as in (14), and unlike (Mao et al.
Multi D employs as many discriminators as the number of source domains which is analogous to the existing approaches.
For a fair comparison, all the other experimental settings are ﬁxed.
The results illustrate that all the versions with the uniﬁed discriminator reliably outperform Multi D in terms of both accuracy and reliability.
This suggests that uniﬁcation of the domain discriminators can substantially improves the task performance.
Variance of stochastic gradients.
With respect to the above analysis, we compared the variance of the stochastic gradients computed with different available domain discriminators.
We trained MIAN and Multi D using mini-batches of samples.
The number of samples in a batch was ﬁxed as 128 per
[Figure] [Figure]
domain.
After the early stages of training, we computed the gradients for the weights and biases of both the top and bottom layers of the encoder on the full training set.
Figures 2c, 2d show that MIAN with the uniﬁed discriminator yields exponentially lower variance of the gradients compared to Multi
D .
Thus it is more feasible to use the uniﬁed discriminator when a large number of domains are given.
Proxy A -distance.
To analyze the performance improvement in depth, we measured Proxy ADistance (PAD) as an empirical approximation of domain discrepancy (Ganin et al.
Given the generalization error (cid:15) on discriminating between the target and source samples, PAD is deﬁned as ˆ d A =
2(1 − 2 (cid:15) ) .
Figure 3a shows that MIAN yields lower PAD between the source and target domain on average, potentially associated with optimizing ˆ d H ( V ) .
To test this conjecture, we conducted an ablation study on the objective of domain discriminator (Figure 3b, 3c).
All the other experimental settings were ﬁxed except for using the objective of the uniﬁed domain discriminator as (13), or (14).
While both cases help the adaptation, using (14) yields lower ˆ d H ( V ) and higher test accuracy.
Estimation of mutual information.
We measure the empirical mutual information ˆ I ( Z ; V ) with assuming H ( V ) as a constant.
For the measurement, we trained the domain discriminator to minimize the softmax cross entropy (13) with sufﬁcient iterations.
Figure 3d shows that MIAN yields the lowest ˆ
I ( Z ; V ) , guaranteeing that the obtained representation achieves low-level of domain dependence.
C ONCLUSION
In this paper, we have presented a uniﬁed information-regularization framework for MDA.
The proposed framework allows us to examine the existing adversarial DA methods and also motivates us to implement a novel neural architecture for MDA.
We provided both theoretical arguments and empirical evidence to fully justify three potential pitfalls of using multiple discriminators: dispersed domain-discriminative knowledge, lack of scalability and high variance in the objective.
Our framework also establishes a bridge between adversarial DA and Information Bottleneck theory.
The proposed model does not require complicated settings such as image generation, pretraining, multiple discriminators, multiple encoders or classiﬁers, which are often adopted in the existing MDA methods (Zhao et al.
(2019b;c); Wang et al.
A P ROOFS
In this Section, we present the detailed proofs for Theorems 2 and 3, explained in the main paper.
We also present Lemma 2, as mentioned in Section 4.
Following (Roh et al.
, we provide a proof of Theorem 2 below for the sake of completeness.
A. P ROOF OF T HEOREM
Theorem 2.
Let P Z ( z ) be the distribution of Z where z ∈ Z .
Let h be a domain classiﬁer h : Z → V , where Z is the feature space and V is the set of domain labels.
Let h v ( Z ) be a conditional probability of V where v ∈ V given Z = z , deﬁned by h .
Then the following holds:
[Equation]
Proof.
By deﬁnition,
[Equation]
Let us constrain the term inside the log by h v ( z )
=
P Z,V ( z , v ) P Z ( z ) where h v ( z ) represents the conditional probability of V = v for any v ∈ V given Z
= z .
Then we have: (cid:80) v ∈V h v ( z )
= 1 for all possible values of z according to the law of total probability.
Let h denote the collection of h v ( z ) for all possible values of v and z , and λ be the collection of λ z for all values of z .
Then, we can construct the Lagrangian function by incorporating the constraint (cid:80) v ∈V h v ( z ) = 1 as follows:
[Equation]
We can use the following KKT conditions:
[Equation]
Solving the two equations, we have 1 − (cid:80)
[Equation]
= 0 such that λ ∗ z
= P Z ( z ) for all z .
Then for all the possible values of z ,
[Equation]
where the given h ∗ v ( z ) is same as the term inside log in (18).
Thus, the optimal solution of concave Lagrangian function (19) obtained by h ∗ v ( z ) is equal to the mutual information in (18).
The substitution of h ∗ v ( z ) into (18) completes the proof.
Our framework can further be applied to segmentation problems because it provides a new perspective on pixel space (Sankaranarayanan et al.
(2018a;b); Murez et al.
and segmentation space (Tsai et al.
(2018)) adaptation.
The generator in pixel space and segmentation space adaptation learns to transform images or segmentation results from one domain to another.
In the context of information regularization, we can view these approaches as limiting information
I
( ˆ X ; V ) between the generated output ˆ X and the domain label V , which is accomplished by involving the encoder for pixel-level generation.
This alleviates the domain shift in a raw pixel level.
Note that one can choose between limiting the feature-level or pixel-level mutual information.
These different regularization terms may be complementary to each other depending on the given task.
A. P ROOF OF T HEOREM
Theorem 3.
Let P Z | x , v ( z ) be a conditional probabilistic distribution of Z where z ∈ Z , deﬁned by the encoder F
[Equation]
given a sample x ∈ X and the domain label v ∈ V .
Let R Z ( z ) denotes a prior marginal distribution of Z .
Then the following inequality holds:
[Equation]
Proof.
Based on the chain rule for mutual information,
[Equation]
where the latter equality is given by Theorem
[Equation]
Considering I ( Z ; X | V ) ,
[Equation]
The second equality is obtained by using P Z,X | v ( z , x ) =
P X | v ( x ) P Z | x , v ( z ) .
The inequality is obtained by using D KL [ P Z | v (cid:107) R Z ]
=
E z ∼ P Z | v (cid:2) log P Z | v ( z ) − log R Z ( z ) (cid:3) ≥ 0 , where R Z ( z ) is a variational approximation of the prior marginal distribution of Z .
The last equality is obtained from the deﬁnition of KL-divergence.
The substitution of (25) into (24) completes the proof.
The existing DA work on semantic segmentation tasks (Luo et al.
; Song et al.
(2019)) can be explained as the process of fostering close collaboration between the aforementioned information bottleneck terms.
The only difference between Theorem 3 for V = { 0 , 1 } and the objective function in (Luo et al.
(2019)) is that (Luo et al.
(2019)) employed the shared encoding P Z | x ( z ) instead of
P Z | x , v ( z ) , whereas some adversarial DA approaches use the unshared one (Tzeng et al.
A. P ROOF OF L EMMA
Lemma 2.
Let
d H ( V ) = 1 N +
[Equation]
D
[Equation]
c ) .
Let H be a hypothesis class.
Then,
[Equation]
Proof.
Let α = 1 N represents the uniform domain weight for the mixture of domain D v c .
Then,
[Equation]
[Table]
[Equation]
where the inequality follows from the triangluar inequality and jensen’s inequality.
B E XPERIMENTAL SETUP
In this Section, we describe the datasets, network architecture and hyperparameter conﬁguration.
B. D ATASETS
We validate the Multi-source Information-regularized Adaptation Networks ( MIAN ) with the following benchmark datasets: Digits-Five, Ofﬁce-31 and Ofﬁce-Home.
Every experiment is repeated four times and the average accuracy in target domain is reported.
Digits-Five (Peng et al.
dataset is a uniﬁed dataset including ﬁve different digit datasets:
MNIST (LeCun et al.
MNIST-M (Ganin & Lempitsky (2014)), Synthetic Digits (Ganin &
Lempitsky (2014)), SVHN, and USPS.
Following the standard protocols of unsupervised MDA (Xu et al.
; Peng et al.
, we used 25000 training images and 9000 test images sampled from a training and a testing subset for each of MNIST, MNIST-M, SVHN, and Synthetic Digits.
For
USPS, all the data is used owing to the small sample size.
All the images are bilinearly interpolated to 32 × 32 .
Ofﬁce-31 (Saenko et al.
(2010)) is a popular benchmark dataset including 31 categories of objects in an ofﬁce environment.
Note that it is a more difﬁcult problem than Digits-Five, which includes 4652 images in total from the three domains: Amazon, DSLR, and Webcam.
All the images are interpolated to 224 × 224 using bicubic ﬁlters.
Ofﬁce-Home (Venkateswara et al.
(2017)) is a challenging dataset that includes 65 categories of objects in ofﬁce and home environments.
It includes 15,500 images in total from the four domains: Artistic images (Art), Clip Art(Clipart
B. A RCHITECTURES
For the Digits-Five dataset, we use the same network architecture and optimizer setting as in (Peng et al.
For all the other experiments, the results are based on ResNet-50, which is pre-trained on ImageNet.
The domain discriminator is implemented as a three-layer neural network.
Detailed architecture is shown in Figure 5.
We compare our method with the following state-of-the-art domain adaptation methods: Deep
Adaptation Network ( DAN , Long et al.
Joint Adaptation Network ( JAN , Long et al.
Manifold Embedded Distribution Alignment ( MEDA , Wang et al.
Correlation Alignment ( CORAL , Sun et al.
Domain Adversarial Neural Network ( DANN , Ganin et al.
[Figure] [Figure]
[Figure]
Spectral Penalization ( BSP , Chen et al
Adversarial
Domain
( ADDA
Tzeng et al et al
[Table]
Hyperparameters Details of the experimental setup are summarized in Table 4.
Other state-of-theart adaptation models are trained based on the same setup except for these cases: DCTN show poor performance with the learning rate shown in Table 4 for both Ofﬁce-31 and Ofﬁce-Home datasets.
Following the suggestion of the original authors, 1 e − 5 is used as a learning rate with the Adam optimizer (Kingma & Ba (2014)); MCD show poor performance for the Ofﬁce-Home dataset with the learning rate shown in Table 4.
1 e − 4 is selected as a learning rate.
For both the proposed and other baseline models, the learning rate of the classiﬁer or domain discriminator trained from the scratch is set to be 10 times of those of ImageNet-pretrained weights, in Ofﬁce-31 and Ofﬁce-Home datasets.
More hyperparameter conﬁgurations are summarized in Table 5 (Section E)
C P SEUDOCODE
Due to the limited space, we provide the algorithm of MIAN in this Section.
Details about trainingdependent scaling of β t are in Section E.
Algorithm mini-batch size for each domain= m , Number of source domains= N , Training iteration T .
M= m ( N + 1) , Set of domain labels V = { 1 , . .
. , N + 1 } .
for t ← 1 to T do
X = { x i } Mi =1 is a union of samples { X S 1 , . . .
, X S N , X T }
Y =
{ y i } mNi =1 is a union of samples { Y S 1 , . . .
, Y S N }
Let z i
= F ( x i ) , and ˆy i
= C ( F ( x i ))
, ∀ x i ∈ X
L ( h ) =
− 1 M (cid:80) v ∈V (cid:80)
i : v i = v (cid:2) 1 T [ k = v i ] log h
( z i ) + 1 T [ k (cid:54) =
v i ] log( 1 − h ( z i )) (cid:3)
Backpropagate gradient of L ( h ) , or the variant (Mao et al.
(2017)), to h .
L ( F, C ) =
− 1 mN (cid:80) y ∈Y (cid:80) i : y i =
y (cid:2) 1 T [ k = y i ] log ˆy i (cid:3) β t = β 0 · 2 (cid:0) 1 − 1 1+ exp ( − σ · t/T ) (cid:1) //
See Appendix E
L ( F ) =
L ( F, C ) − β t L ( h )
Backpropagate gradient of L ( F ) to F .
Backpropagate gradient of L ( F, C ) to C .
D A DDITIONAL RESULTS
Visualization of learned latent representations.
We visualized domain-independent representations extracted by the input layer of the classiﬁer with t-SNE (Figure 6).
Before the adaptation process, the representations from the target domain were isolated from the representations from each source domain.
However, after adaptation, the representations were well-aligned with respect to the class of digits, as opposed to the domain.
Hyperparameter sensitivity.
We conducted the analysis on hyperparameter sensitivity with degree of regularization β .
The target domain is set as Amazon or Art, where the value β 0 changes from 0 .
1 to 0 .
The accuracy is high when β 0 is approximately between 0.1 and 0.3.
We thus choose β 0 = 0 .
2 for Ofﬁce-31, and β 0 = 0 .
3 for Ofﬁce-Home.
[Figure]
E D ECAYING B ATCH S PECTRAL P ENALIZATION
In this Section, we provides details on the Decaying Batch Spectral Penalization (DBSP) which expands MIAN into MIAN- γ .
E. B ACKGROUNDS
There is little motivation for models to control the complex mutual dependence to domains if reducing the entropy of representations is sufﬁcient to optimize the value of I ( Z ; V )
=
H ( Z ) − H
( Z |
V ) .
If so, such implicit entropy minimization substantially reduce the upper bound of I ( Z ; Y ) , leading to a increase in optimal joint risk λ ∗ .
In other words, the decrease in the entropy of representations may occur as the side effect of I ( Z ; V ) regularization.
Such unexpected side effect of information regularization is highly intertwined with the hidden deterioration of discriminability through adversarial training (Chen et al.
Liu et al.
Based on these insights, we employ the SVD-entropy H SV D ( Z ) (Alter et al.
(2000)) of a representation matrix Z to assess the richness of the latent representations during adaptation, since it is difﬁcult to compute H ( Z ) .
Note that while H SV D ( Z ) is not precisely equivalent to H ( Z ) , H SV D ( Z ) can be used as a proxy of the level of disorder of the given matrix (Newton & DeSalvo (2010)).
In future works, it would be interesting to evaluate the temporal change in entropy with other metrics.
We found that H SV D ( Z ) indeed decreases signiﬁcantly during adversarial adaptation, suggesting that some eigenfeatures (or eigensamples) become redundant and, thus, the inherent feature-richness diminishes.
(Figure 7a) To preclude such deterioration, we employ Batch Spectral Penalization (BSP)
(Chen et al.
(2019)), which imposes a constraint on the largest singular value to solicit the contribution of other eigenfeatures.
The overall objective function in the multi-domain setting is deﬁned as:
[Equation]
28) where β and γ are Lagrangian multipliers and s i,j is the j th singular value from the i th domain.
We found that SVD entropy of representations is severely deteriorated especially in the early stages of training, suggesting the possibility of over-regularization.
The noisy domain discriminative signals in the initial phase (Ganin et al.
(2016)) may distort and simplify the representations.
To circumvent the impaired discriminability in the early stages of the training, the discriminability should be prioritized ﬁrst with high γ and low β , followed by a gradual decaying and annealing in γ and β , respectively, so that a sufﬁcient level of domain transferability is guaranteed.
Based on our temporal analysis, we introduce the training-dependent scaling of β and γ by modifying the progressive training schedule (Ganin et al.
[Equation]
where β 0 and γ 0 are initial values, σ is a decaying parameter, and p is the training progress from 0 to 1 .
We refer to this version of our model as MIAN- γ .
Note that MIAN only includes annealing- β , excluding DBSP.
For the proposed method, β 0 is chosen from { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } for Ofﬁceand Ofﬁce-Home dataset, while β 0 = 1 . 0 is ﬁxed in Digits-Five.
γ 0 is ﬁxed to { 1 e − 4 } following
Chen et al.
Decaying batch spectral penalization is not adopted in
[Table]
N/A 10.
N/A
[Table] [Table]
[Table] [Table]
[Table] [Table]
N/A 10.
N/A
[Table] [Table]
[Table]
E. E XPERIMENTS
SVD-entropy.
We evaluated the degree of compromise of SVD-entropy owing to transfer learning.
For this, DSLR was ﬁxed as the source domain, and each Webcam and Amazon target domain was used to simulate low (DSLR → Webcam; DW ) and high domain (DSLR → Amazon; DA ) shift conditions, respectively.
SVD-entropy was applied to the representation matrix extracted from ResNet50 and MIAN (denoted as Adapt in Figure 7a) with constant β = 0 . 1 .
For accurate assessment, we avoided using spectral penalization.
As depicted in the Figure 7a, adversarial adaptation, or information regularization, signiﬁcantly decreases the SVD-entropy of both the source and target domain representations, especially in the early stages of training, indicating that the representations are simpliﬁed in terms of feature-richness.
Moreover, when comparing the Adapt_DA_source and
Adapt_DW_source conditions, we found that SVD-entropy decreases signiﬁcantly as the degree of domain shift increases.
We additionally conducted analyses on temporal changes of SVD entropy by comparing BSP and decaying BSP (Figure 7b).
SVD entropy gradually decreases as the degree of compensation decreases in DBSP which leads to improved transferability and accuracy.
Thus DBSP can control the trade-off between the richness of the feature representations and adversarial adaptation as the training proceeds.
Ablation study.
We performed an ablation study to assess the contribution of the decaying spectral penalization and annealing information regularization to DA performance (Table 6, 7).
We found that the prioritization of feature-richness in early stages (by controlling β and γ ) signiﬁcantly improves the performance.
We also found that the constant penalization schedule (Chen et al.
(2019)) is not reliable and sometimes impedes transferability in the low domain shift condition (Webcam, DSLR in
Table 6).
This implies that the conventional BSP may over-regularize the transferability when the degree of domain shift and SVD-entropy decline are relatively small.
[Figure]
information regularization (Ofﬁce-31).
For accurate assessment of extent to which performance improvement is caused by each strategies, γ is ﬁxed as 0 in Annealing- β , and β is ﬁxed as 0 .
1 in Decaying- γ .
Results from
Annealing- β are reported in main paper.
[Table] [Table]
[Table] [Table]
β 0 = 0 .
1 , σ =
[Table] [Table]
[Table] [Table]
[Table]
γ = 1 e −
[Table]
a constant
[Table] [Table]
[Table] [Table]
γ 0 = 1 e − 4 , σ =
[Table]
[Table] [Table]
[Table] [Table]
β
[Table]
[Table] [Table]
, σ
[Table] [Table]
[Table] [Table]
[Table] [Table]