D OMAIN A DAPTATION WITH A SYMMETRICALLYR ELAXED D ISTRIBUTION A LIGNMENT
Anonymous authors
Paper under double-blind review
A BSTRACT
Domain adaptation addresses the common problem when the target distribution generating our test data drifts from the source (training) distribution.
While absent assumptions, domain adaptation is impossible, strict conditions, e.g. covariate or label shift, enable principled algorithms.
Recently-proposed domain-adversarial approaches consist of aligning source and target encodings, often motivating this approach as minimizing two (of three) terms in a theoretical bound on target error.
Unfortunately, this minimization can cause arbitrary increases in the third term, e.g. they can break down under shifting label distributions.
We propose asymmetricallyrelaxed distribution alignment , a new approach that overcomes some limitations of standard domain-adversarial algorithms.
Moreover, we characterize precise assumptions under which our algorithm is theoretically principled and demonstrate empirical beneﬁts on both synthetic and real datasets.  I NTRODUCTION
Despite breakthroughs in supervised deep learning across a variety of challenging tasks, current techniques depend precariously on the i.i.d.
assumption.
Unfortunately, real-world settings often demand not just generalization to unseen examples but robustness under a variety of shocks to the data distribution.
Ideally, our models would leverage unlabeled test data, adapting in real time to produce improved predictions.
Unsupervised domain adaptation formalizes this problem as learning a classiﬁer from labeled source domain data and unlabeled data from a target domain , to maximize performance on the target distribution. Without further assumptions, guarantees of target-domain accuracy are impossible (Ben-David et al., 2010b). However, well-chosen assumptions can make possible algorithms with non-vacuous performance guarantees. For example, under the covariate shift assumption (Heckman, 1977;
Shimodaira, 2000), although the input marginals can vary between source and target ( p S ( x ) (cid:54) =
p T ( x ) ), the conditional distribution of the labels (given features) exhibits invariance across domains ( p S ( y | x )
= p T ( y | x ) ).
Traditional approaches to the covariate shift problem require the source distributions’ support to cover the target support, estimating adapted classiﬁers via importanceweighted risk minimization (Shimodaira, 2000;
Huang et al., 2007; Gretton et al., 2009; Yu &
Szepesv´ari, 2012; Lipton et al., 2018).
Problematically, assumptions of contained support are violated in practice.
A recent sequence of deep learning papers have proposed empirically-justiﬁed adversarial training schemes aimed at practical problems with non-overlapping supports (Ganin et al., 2016; Tzeng et al.).
Example problems include generalizing from gray-scale images to colored images or product images on white backgrounds to photos of products in natural settings.
While importance-weighting solutions are useless here (with non-overlapping support, weights are unbounded), domain-adversarial networks (Ganin et al., 2016) and subsequently-proposed variants report strong empirical results on a variety of image recognition challenges.
The key idea of domain-adversarial networks is to simultaneously minimize the source error and align the two distributions in representation space.
The scheme consists of an encoder, a label classiﬁer , and a domain classiﬁer .
During training, the domain classiﬁer is optimized to predict each image’s domain given its encoding.
The label classiﬁer is optimized to predict labels from encodings (for source images).
The encoder weights are optimized for the twin objectives of accurate label classiﬁcation (of source data) and fooling the domain classiﬁer (for all data).
[Figure]
Although Ganin et al.
(2016) motivate their idea via theoretical results due to Ben-David et al.
(2010a), the theory is insufﬁcient to justify their method.
Put simply, Ben-David et al.
(2010a) bound the test error by a sum of three terms.
The domain-adversarial objective minimizes two among these, but this minimization may cause the third term to increase.
This is guaranteed to happen when the label distribution shifts between source and target (Figure 1(a)).
In this paper, we propose asymmetrically-relaxed distribution alignment, a relaxed distance for aligning data across domains that can be minimized without requiring latent-space distributions to match exactly.
The new distance is minimized whenever the density ratios in representation space from target to source are upper bounded by a certain constant, such that the target representation support is contained in the source representation’s.
The relaxed distribution alignment need not lead to a poor classiﬁer on the target domain under label distribution mismatch (Figure 1(b)).
We demonstrate theoretically that the relaxed alignment is sufﬁcient for a good target domain performance under a concrete set of assumptions on the data distributions.
Further, we propose several practical ways to achieve the relaxed distribution alignment, translating the new distance into adversarial learning objectives.
Empirical results on synthetic and real datasets show that incorporating our relaxed distribution alignment loss into adversarial domain adaptation gives better classiﬁcation performance on the target domain.
Due to space constraints, we only brieﬂy state our results in the main text and append the full version of our paper after references.
B ACKGROUND AND M OTIVATION Unsupervised domain adaptation with representations
For simplicity, we address the binary classiﬁcation scenario.
Let X be the input space and f : X (cid:55)→ { 0 , 1 } be the (domain-invariant) ground truth labeling function.
Let p S and p T be the input distributions over X for source and target domain respectively.
Let Z be a latent space and Φ denote a class of mappings from X to Z .
Deﬁne
H to be a class of predictors over the latent space Z , i.e., each h ∈ H maps from Z to { 0 , 1 } .
Given a representation mapping φ ∈ Φ , classiﬁer h ∈ H , and input x ∈ X , our prediction is h ( φ ( x )) .
In the unsupervised domain adaptation setting, we have access to labeled source data ( x, f ( x )) for x ∼ p S and unlabeled target data x ∼ p T .
We are interested in bounding the classiﬁcation risk of a ( φ, h ) -pair on the target domain:
[Equation]
where r is the risk function in the latent space.
Domain-adversarial learning
Domain-adversarial approaches focus on minimizing the ﬁrst and third term in (1) jointly.
Informally, these approaches minimize the source domain classiﬁcation risk and the distance between the two distributions in the latent space:
[Equation]
[Equation]
where D is a distance metric between distributions and Ω is a regularization term.
Standard choices of D have the property that D ( p φS , p φT )
{+= 
+}
if p φS ≡ p φT and D ( p φS , p φT ) > 0 otherwise. This exact distribution matching, however, can lead to undesirable performance. More speciﬁcally, the following proposition says that the target error is lower bounded if label distribution shifts: Proposition 2.1.
Let ρ S and ρ T be the proportion of data with positive label.
{+If D ( p φS , p φT ) = 
+}
if and only if p φS ≡ p φT , E S ( φ, h )
=
D ( p φS , p φT ) = 0 indicates E T ( φ, h ) ≥ | ρ S − ρ T | .
This problem happens because although D ( p φS , p φT ) = 0 is a sufﬁcient condition for the third term of (1) to be zero, it is not a necessary condition .
We now examine the third term of (1): (cid:82) d
z (cid:16) p φT ( z ) − p φS
( z ) (cid:17) r S
( z ; φ, h ) ≤ (cid:16) sup
z ∈Z p φT ( z ) p φS ( z ) − 1 (cid:17) E S ( φ, h ) .
This expression shows that if the source error E S ( φ, h ) is zero then it is sufﬁcient to say the third term of (1) is zero when the density ratio p φT ( z ) /p φS ( z ) is upper bounded by some constant for all z , as shown in Figure 1(b).
Given this motivation, we propose relaxing from exact distribution matching to bounding the density ratio in the domain-adversarial learning objective (2).
We call this asymmetrically-relaxed distribution alignment .
More speciﬁcally, our proposed approach is to replace the typical distribution distance
D in the domain-adversarial objective (2) with a β -admissible distance D β so that minimizing the new objective does not necessarily lead to a failure under label distribution shift.
Deﬁnition 2.2 ( β -admissible distances) .
Given a family of distributions deﬁned on the same space
Z , a distance metric D β between distributions is called β -admissible if D β ( p, q ) = 0 when sup z ∈Z p ( z ) /q
( z ) ≤ 1 + β and D β ( p, q ) > 0 otherwise.
T HEORETICAL R ESULTS
We bound the target domain error under our proposed asymmetrically-relaxed distribution alignment.
Our theoretical result makes distinct contribution to the domain adaptation literature: We provide a risk bound that explains the behavior of domain-adversarial methods with model-independent assumptions on data distributions.
Existing theories without assumptions of contained support (BenDavid et al., 2007; 2010a; Ben-David & Urner, 2014; Mansour et al., 2009; Cortes & Mohri, 2011) do not exhibit this property.
Construction 3.1.
The following statements hold simultaneously: (1) ( Lipschitzness of representation mapping. )
φ is L -Lipschitz:
d Z ( φ ( x 1 ) , φ ( x 2 ))
≤ Ld X ( x 1 , x 2 ) for any x 1 , x 2 ∈ X .
(2) ( Imperfect asymmetrically-relaxed distribution alignment. )
For some β ≥ 0 , there exist a set B ⊂ Z such that p φT ( z ) p φS ( z ) ≤ 1+ β holds for all z ∈ B and p φT ( B ) ≥ 1 − δ 1 .
(3) ( Separation of source domain in the latent space. )
There exist two sets C 0 , C 1 ⊂ X that satisfy: (a) C 0 ∩ C 1 = ∅ (b) p S
( C 0 ∪ C 1 ) ≥ 1 − δ 2 .
(c) For i ∈ { 0 , 1 } , f ( x ) =
i for all x ∈ C i .
(d) inf z 0 ∈ φ ( C 0 ) ,z 1 ∈ φ ( C 1 )
d Z ( z 0 , z 1 ) ≥ ∆ > 0 .
Assumption 3.2.
( Connectedness from target domain to source domain. )
Given constants ( L, β, ∆ , δ 1 , δ 2 , δ 3 ) , assume that, for any B S , B T ⊂ X with p S ( B S ) ≥ 1 − δ 2 and p T ( B T ) ≥ 1 − δ 1 − (1+ β )
δ 2 , there exists C T ⊂ B T that satisﬁes the following conditions: (1) For any x ∈ C T , there exists x (cid:48) ∈ C T ∩ B S such that one
can ﬁnd a sequence of points x 0 , x 1 , ..., x m ∈ C T with x 0 = x , x m = x (cid:48) , f ( x ) =
f
( x (cid:48) ) and d X ( x i − 1 , x i ) <
∆ L for all i = 1 , ..., m .
(2) p T ( C T ) ≥ 1 − δ 3 .
Theorem 3.3.
Given a L -Lipschitz mapping φ ∈ Φ and a binary classiﬁer h ∈ H , if φ satisﬁes the properties in Construction 3.1 with constants ( L, β, ∆ , δ 1 , δ 2 ) , and Assumption 3.2 holds with the same set of constants plus δ 3 , then the target domain error can be bounded as
[Equation]
A SYMMETRICALLY - RELAXED DISTANCES
In this section, we derive several β -admissible distance metrics that can be practically minimized with adversarial training.
f
-divergence
We propose a general approach to make any f -divergence β -admissible by partially linearizing the function f .
Plugging in the corresponding f for JS-divergence gives
[Equation]
Wasserstein distance The idea behind modifying the Wasserstein distance is to model the optimal transport from p to the region where distributions have 1 + β maximal density ratio with respect to q .
Following the dual-form derivation for the original Wasserstein distance gives
[Equation]
Reweighting distance Given any distance metric D , a generic way to make it β -admissible is to allow reweighting for one of the distances within a β -dependent range:
Given a distribution q over Z and a reweighting function w : Z (cid:55)→ [0 , ∞ ) .
The reweighted distribution q w is deﬁned as q w ( z ) =
q ( z ) w ( z ) (cid:82) d zq ( z ) w ( z ) .
Deﬁne W β,q to be a set of β -qualiﬁed reweighting with respect to q :
W β,q = (cid:110)
w : Z (cid:55)→ [0 , 1] , (cid:82) d zq ( z ) w ( z ) = 1 1+ β (cid:111) .
Then the relaxed distance can be deﬁned as
D β ( p, q ) =
min w ∈W β,q D ( p, q w ) .
E XPERIMENTS
To evaluate our approach, we implement Domain Adversarial Neural Networks (DANN), (Ganin et al., 2016) replacing the JS-divergence with several β -admissible distances.
Table 5-2 summarize our experimental results.
Compared to the original DANN, our approaches fare signiﬁcantly better under label distribution shift while achieving comparable performance absent label distribution shift.
[Table]
[Table]
[Table]
A BSTRACT
Domain adaptation addresses the common problem when the target distribution generating our test data drifts from the source (training) distribution.
While absent assumptions, domain adaptation is impossible, strict conditions, e.g. covariate or label shift, enable principled algorithms.
Recently-proposed domain-adversarial approaches consist of aligning source and target encodings, often motivating this approach as minimizing two (of three) terms in a theoretical bound on target error.
Unfortunately, this minimization can cause arbitrary increases in the third term, e.g. they can break down under shifting label distributions.
We propose asymmetricallyrelaxed distribution alignment , a new approach that overcomes some limitations of standard domain-adversarial algorithms.
Moreover, we characterize precise assumptions under which our algorithm is theoretically principled and demonstrate empirical beneﬁts on both synthetic and real datasets.
I NTRODUCTION
Despite breakthroughs in supervised deep learning across a variety of challenging tasks, current techniques depend precariously on the i.i.d.
assumption.
Unfortunately, real-world settings often demand not just generalization to unseen examples but robustness under a variety of shocks to the data distribution.
Ideally, our models would leverage unlabeled test data, adapting in real time to produce improved predictions.
Unsupervised domain adaptation formalizes this problem as learning a classiﬁer from labeled source domain data and unlabeled data from a target domain , to maximize performance on the target distribution.
Without further assumptions, guarantees of target-domain accuracy are impossible (Ben-David et al., 2010b).
However, well-chosen assumptions can make possible algorithms with non-vacuous performance guarantees.
For example, under the covariate shift assumption (Heckman, 1977; Shimodaira, 2000), although the input marginals can vary between source and target ( p S ( x ) (cid:54) =
p T ( x ) ), the conditional distribution of the labels (given features) exhibits invariance across domains ( p S ( y | x )
= p T ( y | x ) ).
Some consider the reverse setting label shift (Saerens et al., 2002; Zhang et al., 2013; Lipton et al., 2018), where although the label distribution shifts ( p S ( y ) (cid:54) =
p T ( y ) ), the class-conditional input distribution is invariant ( p S ( x | y )
=
p T ( x | y ) ).
Traditional approaches to both problems require the source distributions’ support to cover the target support, estimating adapted classiﬁers via importance-weighted risk minimization (Shimodaira, 2000;
Huang et al., 2007; Gretton et al., 2009; Yu & Szepesv´ari, 2012; Lipton et al., 2018).
Problematically, assumptions of contained support are violated in practice.
Moreover, most theoretical analyses do not guaranteed target accuracy when the source distribution support does not cover that of the target.
A notable exception, Ben-David et al.
(2010a) leverages capacity constraints on the hypothesis class to enable generalization to out-of-support samples.
However, their results (i) do not hold for high-capacity hypothesis classes, e.g., neural networks; and (ii) do not provide intuitive interpretations on what is sufﬁcient to guarantee a good target domain performance.
A recent sequence of deep learning papers have proposed empirically-justiﬁed adversarial training schemes aimed at practical problems with non-overlapping supports (Ganin et al., 2016; Tzeng et al.).
Example problems include generalizing from gray-scale images to colored images or product images on white backgrounds to photos of products in natural settings.
While importance-weighting solutions are useless here (with non-overlapping support, weights are unbounded), domain-adversarial [Figure] networks (Ganin et al., 2016) and subsequently-proposed variants report strong empirical results on a variety of image recognition challenges.
The key idea of domain-adversarial networks is to simultaneously minimize the source error and align the two distributions in representation space.
The scheme consists of an encoder, a label classiﬁer , and a domain classiﬁer .
During training, the domain classiﬁer is optimized to predict each image’s domain given its encoding.
The label classiﬁer is optimized to predict labels from encodings (for source images).
The encoder weights are optimized for the twin objectives of accurate label classiﬁcation (of source data) and fooling the domain classiﬁer (for all data).
Although Ganin et al.
(2016) motivate their idea via theoretical results due to Ben-David et al.
(2010a), the theory is insufﬁcient to justify their method.
Put simply, Ben-David et al.
(2010a) bound the test error by a sum of three terms.
The domain-adversarial objective minimizes two among these, but this minimization may cause the third term to increase.
This is guaranteed to happen when the label distribution shifts between source and target.
Consider the case of cat-dog classiﬁcation with non-overlapping support.
Say that the source distribution contains 50% dogs and 50% cats, while the target distribution contains 25% dogs and 75% cats.
Successfully aligning these distributions in representation space requires the classiﬁer to predict the same fraction of dogs and cats on source and target.
If one achieves 100% accuracy on the source data, then target accuracy will be at most 75% (Figure 1(a)).
In this paper, we propose asymmetrically-relaxed distribution alignment, a relaxed distance for aligning data across domains that can be minimized without requiring latent-space distributions to match exactly.
The new distance is minimized whenever the density ratios in representation space from target to source are upper bounded by a certain constant, such that the target representation support is contained in the source representation’s.
The relaxed distribution alignment need not lead to a poor classiﬁer on the target domain under label distribution mismatch (Figure 1(b)).
We demonstrate theoretically that the relaxed alignment is sufﬁcient for a good target domain performance under a concrete set of assumptions on the data distributions.
Further, we propose several practical ways to achieve the relaxed distribution alignment, translating the new distance into adversarial learning objectives.
Empirical results on synthetic and real datasets show that incorporating our relaxed distribution alignment loss into adversarial domain adaptation gives better classiﬁcation performance on the target domain.
We make the following key contributions:
• We propose an asymmetrically relaxed distribution matching objective, overcoming the limitation of standard objectives under label distribution shift.
• We provide theoretical analysis demonstrating that under a clear set of assumptions, the asymmetrically relaxed distribution alignment can provide target-domain performance guarantees.
• We propose several distances that satisfy the desired properties and are optimizable by adversarial training.
• We empirically show that our asymmetrically relaxed distribution matching losses improve target performance when there is a label distribution shift in the target domain, and perform comparably otherwise.
P RELIMINARIES
We use subscripts S and T to distinguish between source and target domains, e.g., p S and p T , and employ the notation U for statements that are true for any domain U ∈ { S, T } .
For simplicity, we dispense with some rigorousness in notating probability measures.
For example, we use the terms measure and distribution interchangeably and assume that a density function exists when necessary without explicitly stating the base measure and required regularity conditions.
We use a single lowercase letter, e.g. p , to denote both the probability measure function and the probability density function: p ( x ) is a density when the input x is a single point while p ( C ) is a probability when the input C is a set.
We will use Supp( p ) to denote the support of distribution p , i.e., the set of points where the density is positive.
Similarly, for a function mapping φ , φ ( x ) denotes an output if x is a point and φ ( C ) denotes the image if C is a set.
The inverse mapping φ − 1 always outputs a set (the inverse image) regardless of whether its input is a point or a set.
We will also be less careful about the use of sup v.s. max , inf v.s. min and “everywhere” v.s. “almost everywhere”.
1 {·} is used as the indicator function for statements that output 1 if the statement is true and 0 otherwise.
For two functions f
and g we use f ≡ g to denote that f ( x )
=
g ( x ) for every input x .
For simplicity, we address the binary classiﬁcation scenario.
Let X be the input space and f : X (cid:55)→ { 0 , 1 } be the (domain-invariant) ground truth labeling function.
Let p S and p T be the input distributions over X for source and target domain respectively.
Let Z be a latent space and Φ denote a class of mappings from X to Z .
For a domain U , let p φU ( · ) be the induced probability distribution over Z such that p φU ( C ) =
p U ( φ − 1 ( C )) for any C ⊂ Z .
Given z ∈ Z let φ U ( ·| z ) be the conditional distribution induced by p U and φ such that (cid:82) d zp φU ( z ) φ U ( x | z ) =
p U ( x ) holds for all x ∈ X .
Deﬁne H to be a class of predictors over the latent space Z , i.e., each h ∈ H maps from Z to { 0 , 1 } .
Given a representation mapping φ ∈ Φ , classiﬁer h ∈ H , and input x ∈ X , our prediction is h ( φ ( x )) .
The risk for a single input x can be written as | h ( φ ( x ))
− f ( x ) | and the expected risk for a domain U is
[Equation]
(1) where we deﬁne a domain-dependent latent space labeling function f φU ( z )
= (cid:82) d xφ U ( x | z ) f ( x ) and the risk for a classiﬁer h as r U ( z ; φ, h ) =
(cid:12)(cid:12)(cid:12) h ( z ) − f φU
( z ) (cid:12)(cid:12)(cid:12) ∈ [0 , 1] .
We are interested in bounding the classiﬁcation risk of a ( φ, h ) -pair on the target domain:
[Equation]
The second term in (2) becomes zero if the latent space labeling function is domain-invariant.
To see this, we apply
[Equation] [Equation]
The third term in (2) is zero when p φT and p φS are the same.
In the unsupervised domain adaptation setting, we have access to labeled source data ( x, f ( x )) for x ∼ p S and unlabeled target data x ∼ p T , from which we can calculate 1 the ﬁrst and third term in (2).
For x ∈ Supp( p T ) \ Supp( p S ) , we have no information about its true label f
( x ) and thus f φT ( z ) becomes inaccessible
when z = φ ( x ) for such x .
So the second term in (2) is not directly controllable.
Domain-adversarial approaches focus on minimizing the ﬁrst and third term in (2) jointly.
Informally, these approaches minimize the source domain classiﬁcation risk and the distance between the two distributions in the latent space:
[Equation]
(4) where D is a distance metric between distributions and Ω is a regularization term.
Standard choices of D such as a domain classiﬁer (Jensen-Shannon (JS) divergence 2 ) (Ganin et al., 2016), Wasserstein distance (Shen et al., 2018) or Maximum Mean Discrepancy (Huang et al., 2007) have the property that D ( p φS , p φT )
=
if p φS ≡ p φT and D ( p φS , p φT ) > 0 otherwise.
In the next section, we will show that minimizing (4) with such D will lead to undesirable performance and propose an alternative objective to align p φS and p φT instead of driving them to be identically distributed.
A M OTIVATING S CENARIO
To motivate our approach, we formally show how exact distribution matching can lead to undesirable performance.
More speciﬁcally, we will lower bound E T ( φ, h ) when both E S ( φ, h ) and D ( p φS , p φT ) are zero with respect to the shift in the label distribution.
Let ρ S and ρ T be the proportion of data with positive label, i.e., ρ U = (cid:82)
d xp U ( x ) f ( x ) .
We formalize the result as follows.
Proposition 3.1.
If D ( p φS , p φT ) =
if and only if p φS ≡ p φT , E S ( φ, h )
=
D ( p φS , p φT ) = 0 indicates E T ( φ, h ) ≥ | ρ S − ρ T | .
The proof follows the intuition of Figure 1(a):
If ρ S < ρ T , the best we can do is to map ρ T − ρ S proportion of positive samples from the target inputs to regions of latent space corresponding to negative examples from the source domain while maintaining the label consistency for remaining ones.
Switching the term positive/negative gives a similar argument for ρ T < ρ S .
Proposition 3. says that if there is a label distribution mismatch ρ T (cid:54) = ρ S , minimizing the objective (4) to zero imposes a positive lower bound on the target error.
This is especially problematic in cases where a perfect pair φ, h may exist, achieving zero error on both source and target data (Figure 1(b)).
Asymmetrically-relaxed distribution alignment It may appear contradictory that minimizing the ﬁrst and third term of (2) to zero guarantees a positive E T ( φ, h ) and thus a positive second term when there exists a pair of φ, h such that E T ( φ, h )
=
(all three terms are zero).
However, this happens because although D ( p φS , p φT ) = 0 is a sufﬁcient condition for the third term of (2) to be zero, it is not a necessary condition .
We now examine the third term of (2):
[Equation]
This expression (5) shows that if the source error E S ( φ, h ) is zero then it is sufﬁcient to say the third term of (2) is zero when the density ratio p φT ( z ) /p φS ( z ) is upper bounded by some constant for all z .
Note that it is impossible to bound p φT ( z )
/p φS ( z ) by a constant that is smaller than 1 so we write this condition as sup z ∈Z p φT ( z ) /p φS ( z ) ≤ 1 + β for some β ≥ 0 .
Note that this is a relaxed condition compared with p φT ( z ) ≡ p φS ( z ) , which is a special case with β = 0 .
Relaxing the exact matching condition to the more forgiving bounded density ratio condition makes it possible to obtain a perfect target domain classiﬁer in many cases where the stricter condition does not, by requiring only that the (latent space) target domain support is contained in the source domain support, as shown in Figure 1(b).
The following proposition states that our relaxed matching condition does not suffer from the previously-described problems concerning shifting label distributions (Proposition 3.1), and provides intuition regarding just how large β may need to be to admit a perfect target domain classiﬁer.
Proposition 3.2.
For every ρ S , ρ T , there exists a construction of ( p S , p T , φ, h ) such that E S ( φ, h ) = 0 , E T ( φ, h ) = 0 and sup z ∈Z p φT ( z ) /p φS ( z [Equation] ≤ max (cid:110)
[Equation]
Given this motivation, we propose relaxing from exact distribution matching to bounding the density ratio in the domain-adversarial learning objective (4).
We call this asymmetrically-relaxed distribution alignment since we aim at upper bounding p φ T /p φ S (but not p φ S /p φ T ).
We now introduce a class of distances between distributions that can be minimized to achieve the relaxed alignment:
Deﬁnition 3.3 ( β -admissible distances) .
Given a family of distributions deﬁned on the same space Z , a distance metric D β between distributions is called β -admissible if D β ( p, q ) = 0 when sup z ∈Z p ( z ) /q
( z ) ≤ 1 + β and D β ( p, q ) > 0 otherwise.
Our proposed approach is to replace the typical distribution distance D in the domain-adversarial objective (4) with a β -admissible distance D β so that minimizing the new objective does not necessarily lead to a failure under label distribution shift.
However, it is still premature to claim the justiﬁcation of our approach due to the following issues: (i) We may not be able get a perfect source domain classiﬁer with E S ( φ, h ) = 0 .
This also indicates a trade-off in selecting β as (a) higher β will increase the upper bound ( β E S ( φ, h ) according to (5)) on the third term in (2) (b) lower β will make a good target classiﬁer impossible under label distribution shift.
(ii) Minimizing D
β
( p φT , p φS ) as part of an objective does not necessarily mean that we will obtain a solution with D β ( p φT , p φS )
= 0 .
There may still be some proportion of samples from the target domain lying outside the support of source domain in the latent space Z .
In this case, the density ratio p φT /p φS is unbounded and (5) becomes vacuous.
(iii)
Even when we are able optimize the objective perfectly, i.e., E S ( φ, h )
=
D β ( p φS , p φT ) = 0 , with a proper choice of β such that there exists φ, h such that E T ( φ, h ) = 0 holds simultaneously (e.g.
Figure 1(b), Proposition 3.2), it is still not guaranteed that such φ, h is learned (e.g. Figure 2(a)), as the second term of (2) is unbounded and changes with φ .
Put simply, the problem is that although there may exist alignments perfect for prediction, there also exist other alignments that satisfy the objective but predict poorly (on target data).
To our knowledge this problem effects all domain-adversarial methods proposed in the literature, and how to theoretically guarantee that the desired alignment is learned remains an open question.
Next, we theoretically study the target classiﬁcation error under asymmetrically-relaxed distribution alignment.
Our analysis resolves the above issues by (i) working with imperfect source domain classiﬁer and relaxed distribution alignment; and (ii) providing concrete assumptions under which a good target domain classiﬁer can be learned.
B OUNDING THE T ARGET D OMAIN E RROR
In a manner similar to (2), Ben-David et al.
(2007; 2010a) bound the target domain error by a sum of three terms: (i) the source domain error (ii) an H -divergence between p φS and p φT (iii) the best possible classiﬁcation error that can be achieved on the combination of p φS and p φT .
We motivate our analysis by explaining why their results are insufﬁcient to give a meaningful bound for domain-adversarial learning approaches.
From a theoretical upper bound, we may desire to make claims in the following pattern:
Let M A be a set of models that satisfy a set of properties
A (e.g. with low training error), and B be a set of assumptions on the data distributions ( p S , p T , f ) .
For any given model M ∈ M A , its performance can be bounded by a certain quantity, i.e. E T ( M ) ≤ (cid:15) A , B .
Ideally, A should be observable on available data information (i.e. without knowing target labels), and assumptions B should be model-independent (independent of which model M = ( φ, h ) is learned among M A ).
In the results of Ben-David et al.
(2007; 2010a), terms (i) and (ii) are observable so A can be set as achieving low quantities on these two terms.
Since term (iii) is unobservable we may want to make assumptions on it.
This term, however, is model-dependent when φ is learned jointly.
To make a model-independent assumption on term (iii), we need to take the supremum over all ( φ, h ) ∈ M A , i.e., all possible models that achieve low values on (i) and (ii).
This supremum can be vacuous without further assumptions as a cross-label mapping may also achieve low source error and distribution alignment (e.g. Figure 2(a)
v.s. Figure 1(b)).
Moreover, when H contains all possible binary classiﬁers, the H -divergence is minimized only if the two distributions are the same, thus suffering the same problem as Proposition 3.1 and is therefore not suitable for motivating a learning objective.
To overcome these limitations, we propose a new theoretical bound on the target domain error which (a) treats the difference between p φS and p φT asymmetrically and (b) bounds the label consistency (second term in 2) by exploiting the Lipschitz-ness of φ as well as the separation and connectedness of data distributions.
Our result can be interpreted as a combination of observable model properties and unobservable model-independent assumptions while being non-vacuous: it is able to guarantee correct classiﬁcation for (some fraction of) data points from the target domain even where the source domain has zero density.
4. A GENERAL BOUND
We introduce our result with the following construction:
Construction 4.1.
The following statements hold simultaneously:
( Lipschitzness of representation mapping. )
φ is L -Lipschitz: d Z ( φ ( x 1 ) , φ ( x 2 ))
≤ Ld X ( x 1 , x 2 ) for any x 1 , x 2 ∈ X . 2.
( Imperfect asymmetrically-relaxed distribution alignment. )
For some β ≥ 0 , there exist a set B ⊂ Z such that p φT ( z ) p φS ( z ) ≤ 1 + β holds for all z ∈ B and p φT ( B ) ≥ 1 − δ 1 . 3.
( Separation of source domain in the latent space. )
There exist two sets C 0 , C 1 ⊂ X that satisfy: (a) C [-
-]0 ∩ C 1 = ∅ (b) p S
( C 0 ∪ C 1 ) ≥ 1 − δ 2 .
(c) For i ∈ { 0 , 1 } , f ( x ) =
i for all x ∈ C i .
(d) inf z 0 ∈ φ ( C 0 ) ,z 1 ∈ φ ( C 1 )
d Z ( z 0 , z 1 ) ≥ ∆ > 0 .
Note that this construction does not require any information about target domain labels so the statements [1-3] can be viewed as observable properties of φ .
We now introduce our model-independent assumption:
Assumption 4.2.
( Connectedness from target domain to source domain. )
Given constants ( L, β, ∆ , δ 1 , δ 2 , δ 3 ) , assume that, for any B S , B T ⊂ X with p S ( B S ) ≥ 1 − δ 2 and p T ( B T ) ≥ 1 − δ 1 − (1 + β )
δ 2 , there exists C T ⊂ B T that satisﬁes the following conditions:
For any x ∈ C T , there exists x (cid:48) ∈ C T ∩ B S such that one
can ﬁnd a sequence of points x 0 , x 1 , ..., x m ∈ C T with x 0 = x , x m = x (cid:48) , f ( x ) =
f
( x (cid:48) ) and d X ( x i − 1 , x i ) <
∆ L for all i = 1 , ..., m .
p T ( C T ) ≥ 1 − δ 3 .
We are ready to present our main result:
[Figure]
Theorem 4.3.
Given a L -Lipschitz mapping φ ∈ Φ and a binary classiﬁer h ∈ H , if φ satisﬁes the properties in Construction 4.1 with constants ( L, β, ∆ , δ 1 , δ 2 ) , and Assumption 4.2 holds with the same set of constants plus δ 3 , then the target domain error can be bounded as
[Equation]
Notice that it is always possible to make Construction 4.1 by adjusting the constants L, β, ∆ , δ 1 , δ 2 .
Given these constants, Assumption 4.2 can always be satisﬁed by adjusting δ 3 .
So Theorem 4.3 is a general bound.
The key challenge in bounding E T ( φ, h ) is to bound the second term in (2) by identifying sufﬁcient conditions that prevent cross-label mapping (e.g. Figure 2(a)).
To resolve this challenge, we exploit the fact that if there exist a path from a target domain sample to a source domain sample in the input space X and all samples along the path are mapped into two separate regions in the latent space (due to distribution alignment), then these two connected samples cannot be mapped to different regions, as shown in Figure 2(b).
4. E XAMPLE OF A PERFECT TARGET DOMAIN CLASSIFIER
To interpret our result, we construct a simple situation where E T ( φ, h ) = 0 is guaranteed when the domain adversarial objective with relaxed distribution alignment is minimized to zero, exploiting pure data-dependent assumptions:
Assumption 4.4.
Assume the target support consists of disjoint clusters Supp( p T )
=
S T, 0 , 1 ∪ ... ∪ S T, 0 ,m 0 ∪ S T, 1 , 1 ∪ ... ∪ S T, 1 ,m 1 , where any cluster S T,i,j is connected and its labels are consistent: f ( x ) = i for all x ∈ S T,i,j .
Moreover, each of these cluster overlaps with source distribution.
That is, for any i ∈ { 0 , 1 } and j ∈ { 1 , ..., m i } , S T,i,j ∩ Supp( p S ) (cid:54) = ∅ .
Corollary 4.5.
If Assumption 4.4 holds and there exists a continuous mapping φ such that (i) sup z ∈Z p φT ( z ) /p φS ( z ) ≤ 1 + β for some β ≥ 0 ; (ii) for any pair x 0 , x 1 ∈ Supp( p S )
such that f ( x 0 ) =
and f ( x 1 ) = 1 , we have d Z ( φ ( x 0 ) , φ ( x 1 )) ≥ ∆ > 0 , then E S ( φ, h ) = 0 indicates
E T ( φ, h ) = 0 .
Proof follows directly by observing that a construction of δ 1 =
δ 2 = δ 3 = 0 exists in Theorem 4.3.
A simple example that satisﬁes Assumption 4.4 is Figure 2(b).
For a real world example, consider the cat-dog classiﬁcation problem.
Say that source domain contains small-to-medium cats and dogs while target domain contains medium-to-large cats and dogs.
The target domain consists of clusters (e.g. cats and dogs, or multiple sub-categories) and each of them overlaps with the source domain (the medium ones).
A SYMMETRICALLY - RELAXED DISTANCES
So far, we have motivated the use of asymmetrically-relaxed distribution alignment which aims at bounding p φT /p φS by a constant instead of driving towards p φS ≡ p φT .
More speciﬁcally, we propose to use a β -admissible (Deﬁnition 3.3) distance D β in objective (4) to align the source and target encodings rather than the standard distances corresponding an adversarial domain classiﬁer.
In this section, we derive several β -admissible distance metrics that can be practically minimized with adversarial training.
More speciﬁcally, we propose three types of distances (i) f-divergences; (ii) modiﬁed Wasserstein distance; (iii) reweighting distances; and demonstrate how to optimize them by adversarial training.
5. f - DIVERGENCE
Given a convex and continuous function f which satisﬁes f (1) = 0 , the f -divergence between two distributions p and q can be written as D f ( p, q ) =
(cid:82) d zp ( z ) f (cid:16) q ( z ) p ( z ) (cid:17) .
According to Jensen’s inequality D f ( p, q ) ≥ f (cid:16)(cid:82)
d zp ( z ) q ( z ) p ( z ) (cid:17)
= 0 .
Standard choices of f (see a list in Nowozin et al.
(2016)) are strictly convex thus D f ( p, q )
=
if and only if p ≡ q when f is strictly convex.
To derive a β -adimissible variation for each standard choice of f , we linearize f ( u ) where u ≥ 1 1+ β .
If and only if p ( z ) q ( z ) ≤ 1 + β for all z , f becomes a linear function with respect to all q ( z ) /p
( z ) and thus Jensen’s inequality holds with equality.
Given a convex, continuous function f : R + (cid:55)→ R with f (1) = 0 and some β ≥ 0 , we introduce the partially linearized ¯ f β as follows [Equation] u
[Equation]
where C f,β = − f ( 1 1+ β ) + f (cid:48) ( 1 1+ β ) 1 1+ β − f (cid:48) ( 1 1+ β ) .
It can be shown that ¯ f β is continuous, convex and ¯ f β (1) = 0 .
As we already explained, D ¯ f β ( p, q ) =
if and only if p ( z ) q ( z ) ≤ 1 + β for all z .
Hence is D ¯ f β is β -admissible.
Adversarial training According to Nowozin et al.
(2016), adversarial training (Goodfellow et al., 2014) can be viewed as minimizing the dual form of f -divergences [Equation] T
[Equation]
where f ∗ is the Fenchel Dual of f with f ∗ ( t )
= sup u ∈ dom( f ) { ut − f ( u ) } .
Applying the same derivation for ¯ f β we get
[Equation]
where dom( ¯ f ∗ β ) = dom( f ∗ ) ∩ (cid:0) − ∞ , f (cid:48) ( 1 1+ β ) (cid:3) .
Plugging in the corresponding f for JS-divergence gives
[Equation]
where g ( z ) can be parameterized by a neural network with sigmoid output as typically used in adversarial training.
5. W ASSERSTEIN DISTANCE
The idea behind modifying the Wasserstein distance is to model the optimal transport from p to the region where distributions have 1 + β maximal density ratio with respect to q .
We deﬁne the relaxed Wassertein distance as
[Equation]
where (cid:81) β
( p, q ) is deﬁned as the set of joint distributions γ over Z × Z
such that
[Equation]
W β is β -admissible since no transportation is needed if p already lies in the qualiﬁed region with respect to q .
Adversarial training Following the derivation for the original Wasserstein distance, the dual form becomes
[Equation]
Optimization with adversarial training can be done by parameterizing g as a non-negative function (e.g. with soft-plus output log(1 + e x ) or RELU output max(0 , x ) ) and following Arjovsky et al.
(2017); Gulrajani et al.
(2017) to enforce its Lipschitz continuity approximately.
5. R EWEIGHTING DISTANCE
Given any distance metric D , a generic way to make it β -admissible is to allow reweighting for one of the distances within a β -dependent range.
The relaxed distance is then deﬁned as the minimum achievable distance by such reweighting.
Given a distribution q over Z and a reweighting function w : Z (cid:55)→ [0 , ∞ ) .
The reweighted distribution q w is deﬁned as q w ( z ) =
q
[Equation]
Deﬁne W β,q to be a set of β -qualiﬁed reweighting with respect to q :
[Equation]
Then the relaxed distance can be deﬁned as
[Equation]
Such D β is β -admissible since the set { q w : w ∈ W β,q } is exactly the set of p such that sup
[Equation]
Adversarial training We propose an implicit-reweighting-by-sorting approach to optimize D β without parameterizing the function w when D can be optimized by adversarial training.
Adversarially trainable D shares a general form as
[Equation]
where f 1 and f 2 are monotonically increasing functions.
According to (9), the relaxed distance can be written as
[Equation] [Equation]
One step of alternating minimization on D β , could consist of ﬁxing p, q, g and optimizing w .
Then the problem becomes
[Equation]
[Equation] Observe that the optimal solution to (11) is to assign w ( z )
= 1 for the 1 1+ β fraction of z from distribution q , where f 2 ( g ( z )) take the largest values.
Based on this observation, we propose to do the following sub-steps when optimizing (11) as an alternating minimization step: (i) Sample a minibatch of z ∼ q ; (ii) Sort these z in descending order according to f 2 ( g ( z )) ; (iii) Assign w ( z )
= 1 to the ﬁrst 1 1+ β fraction of the list.
Note that this optimization procedure is not justiﬁed in principle with mini-batch adversarial training but we found it to work well in our experiments. [Figure]
E XPERIMENTS
To evaluate our approach, we implement Domain Adversarial Neural Networks (DANN), (Ganin et al., 2016) replacing the JS-divergence (domain classiﬁer) with our proposed β -admissible distances (Section 5).
Our experiments address the following questions: (i) Does DANN suffer the limitation as anticipated (Section 3) when faced with label distribution shift?
(ii)
If so, do our β -admissible distances overcome these limitations?
(iii)
Absent shifting label distributions, is our approach comparable to DANN?
We implement adversarial training with different β -admissible distances (Section 5) and compare their performance with vanilla DANN.
We name different implementations as follows.
(a) S OURCE : source-only training.
(b) DANN: JS-divergence (original DANN).
(c) WDANN: original Wasserstein distance.
(d) F DANN- β : β -admissible f -divergence, JS-version (7).
(e) S DANN-
β : reweighting JS-divergence (10), optimized by our proposed implicit-reweighting-by-sorting .
(f) WDANN1-
β : β -admissible Wasserstein distance (8) with soft-plus on critic output.
(g) WDANN2-
β : β -admissible Wasserstein distance (8) with RELU on critic output.
(h) S
WDANN- β : reweighting Wasserstein distance (10), optimized by implicit-reweighting-by-sorting .
Adversarial training on Wasserstein distances follows Gulrajani et al.
but uses one-sided gradient-penalty.
We always perform adversarial training with alternating minimization (see Appendix for details).
We create a mixture-of-Gaussians binary classiﬁcation dataset where each domain contains two Gaussian distributions, one per label.
For each label, the distributions in source and target domain have a small overlap, validating the assumptions in our analysis.
We create a label distribution shift with balanced source data (50% 0’s v.s. 50% 1’s) and imbalanced target data (10% 0’s v.s. 90% 1’s) as shown in Figure 3(a).
Table 1 shows the target domain accuracy for different approaches.
As expected, vanilla DANN fails under label distribution shift because a proportion of samples from the target inputs are mapped to regions of latent space corresponding to negative samples from the source domain (Figure 3(b)).
In contrast, with our β -admissible distances, domainadversarial networks are able to adapt successfully (Figure 3(c)), improving target accuracy from 89% (source-only) to 99% accuracy (with adaptation), except the cases where β is too small to admit a good target domain classiﬁer (in this case we need β ≥ 0 .
5 − 1 = 0 . 8 ).
We also experiment with label-balanced target data (no label distribution shift).
All approaches except source-only achieve an accuracy above 99%, so we do not present these results in a separate table. [Table]
Real datasets We experiment with the MNIST and USPS handwritten-digit datasets.
For both directions (MNIST → USPS and USPS → MNIST), we experiment both with and without label distribution shift.
The source domain is always class-balanced.
To simulate label distribution shift, we sample target data from only half of the digits, e.g. [0-4] or [5-9].
Tables 2 and 3 show the target domain accuracy for different approaches with/without label distribution shift.
As on synthetic datasets, we observe that DANN performs much worse than source-only training under label distribution shift.
Compared to the original DANN, our approaches fair signiﬁcantly better while achieving comparable performance absent label distribution shift [Table] [Table]
[Table]
R ELATED WORK
Our paper makes distinct theoretical and algorithmic contributions to the domain adaptation literature.
Concerning theory, we provide a risk bound that explains the behavior of domain-adversarial methods with model-independent assumptions on data distributions.
Existing theories without assumptions of contained support (Ben-David et al., 2007; 2010a; Ben-David & Urner, 2014; Mansour et al., 2009;
Cortes & Mohri, 2011) do not exhibit this property since (i) when applied to the input space, their results are not concerned with domain-adversarial learning as no latent space is introduced, (ii) when applied to the latent space, their unobservable constants/assumptions become φ -dependent, which is undesirable as explained in Section 4.
Concerning algorithms, several prior works demonstrate empirical success of domain-adversarial approaches, (Tzeng et al., 2014;
Ganin et al., 2016; Bousmalis et al., 2016; Tzeng et al.; Hoffman et al., 2017; Shu et al., 2018).
Among those, Cao et al.
(2018a;b) deal with the label distribution shift scenario through a heuristic reweighting scheme.
However, their re-weighting presumes that they have a good classiﬁer in the ﬁrst place, creating a cyclic dependency.
C ONCLUSIONS
We propose to use asymmetrically-relaxed distribution distances in domain-adversarial learning objectives, replacing standard ones which seek exact distribution matching in the latent space.
While overcoming some limitations of the standard objectives under label distribution mismatch, we provide a theoretical guarantee for target domain performance under assumptions on data distributions.
As our connectedness assumptions may not cover all cases where we expect domain adaptation to work in practice, (e.g. when the two domains are completely disjoint), providing analysis under other type of assumptions might be of future interest.
A P ROOFS Derivation of (1) .
[Equation]
where we use the following fact: For any ﬁxed z , h ( z ) ∈ { 0 , 1 } , if h ( z ) = 0 then | h ( z ) − f ( x ) |
= f ( x ) − h ( z ) for all x .
Similarly, when h ( z ) = 1 , we have | h ( z ) − f ( x ) |
= h ( z ) − f ( x ) for all x .
Thus we can move the integral over x inside the absolute operation.
Proof of Proposition 3.1.
First we have
[Equation]
When E S ( φ, h ) =
we have
[Equation]
thus (cid:82) d zp φS ( z ) h ( z ) =
ρ S .
Applying the fact that p φS ( z ) = p φT ( z ) for all z ∈ Z ,
[Equation]
which concludes the proof
[Equation]
Proof of Proposition 3.2.
Let p S be the uniform distribution over [0 , 1] and p T be the uniform distribution over [2 , 3] .
The labeling function f is set as f ( x )
=
iff x ∈ [0 , ρ S ] ∪ [2 , 2 + ρ T ] such that the deﬁnition of ρ S and ρ T is preserved.
We construct the following mapping φ : For x ∈ [0 , 1] φ ( x ) = x .
For x ∈ [2 , 2 + ρ T ] φ ( x ) =
( x − 2) ρ S /ρ T .
For x ∈ [2 + ρ T , 3] φ ( x ) = 1 − (3 − x )(1 − ρ S ) / (1 − ρ T ) .
φ maps both source and target data into [0 , 1] with p φS to be uniform over [0 , 1] and p φT ( z ) =
ρ T /ρ S when z ∈ [0 , ρ S ] and p φT ( z ) =
(1 − ρ T ) / (1 − ρ S ) when z ∈ [ ρ S , 1] .
Since p φS ( z ) = 1 for all z ∈ [0 , 1] we can conclude that sup z ∈Z p φT ( z ) /p φS ( z ) ≤ max (cid:110)
ρ T ρ S , 1 − ρ T 1 − ρ S (cid:111) .
Proof of Theorem 4.3.
Instead of working with Assumption 4.2 we ﬁrst extend Construction 4.1 with the following addition
Construction A.1.
( Connectedness from target domain to source domain. )
Let C T ⊂ X be a set of points in the raw data space that satisfy the following conditions:
φ ( C T ) ⊂
φ ( C 0 ∪ C 1 ) .
For any x ∈ C T , there exists x (cid:48) ∈ C T ∩ ( C 0 ∪ C 1 ) such that one
can ﬁnd a sequence of points x 0 , x 1 , ..., x m ∈ C T with x 0 = x , x m = x (cid:48) , f ( x ) =
f
( x (cid:48) ) and d X ( x i − 1 , x i ) <
∆ L for all i = 1 , ..., m .
p T ( C T ) ≥ 1 − δ 3 .
We now proceed to prove bound based on Constructions 4.1 and A.1.
Later on we will show that Assumption 4.2 indicates the existence of Construction A.1 so that the bound holds with a combination of Constructions 4.1 and Assumption 4. [Equation]
The third term of (2) can be written as
[Equation]
, h ) + p φT ( B c ) ≤ β E S ( φ, h )
+ δ 1 .
For the second term of (2), plugging in r U ( z ; φ, h ) =
(cid:12)(cid:12)(cid:12) h ( z ) − f φU ( z ) (cid:12)(cid:12)(cid:12) gives (cid:90) d zp φT ( z ) ( r T ( z ; φ, h ) − r S ( z [Equation] , h ))
= (cid:90) d zp φT [Equation] z ) [Equation]
h
[Equation]
z )
[Equation]
z )
[Equation]
z ) [Equation] f
[Equation]
z ) (cid:12)(cid:12)(cid:12)(cid:17) = (cid:90)
[Equation]
( z )
[Equation]
f
[Equation]
z )
[Equation]
f
[Equation]
( z ) (cid:12)(cid:12)(cid:12) = (cid:90) d zp φT ( z ) (cid:12)(cid:12)(cid:12) f φT ( z ) − f φS ( z ) (cid:12)(cid:12)(cid:12) ( 1 { z ∈ φ ( C 0 ) } + 1 { z ∈ φ ( C 1 ) } + 1 { z ∈ ( φ ( C 0 ) ∪ φ ( C 1 )) c } ) =
(cid:90)
d zp φT ( z ) (cid:12)(cid:12)(cid:12) f φT ( z ) − f
[Equation]
z ) (cid:12)(cid:12)(cid:12) [Equation] { z
[Equation]
φ ( C 0 ) } + (cid:90)
[Equation]
zp
[Equation]
z ) (cid:12)(cid:12)(cid:12) f φT ( z ) − f φS ( z ) (cid:12)(cid:12)(cid:12)
[Equation]
Applying (cid:12)(cid:12)(cid:12) f φT ( z ) − f φS ( z ) (cid:12)(cid:12)(cid:12) ≤ f φT ( z ) + f φS ( z ) to the ﬁrst part of (13) gives [Equation] = (cid:90) [Equation]
= (cid:90) d xf ( x ) (cid:90) d zp φT ( z ) φ T ( x
[Equation]
z ) 1 { z ∈ φ ( C 0 ) } + (cid:90) d zp [Equation] ( z ) f
[Equation]
( z ) 1 { z
[Equation]
( C 0 ) } =
[Equation]
d
[Equation]
( x )
[Equation]
C
[Equation]
[Equation]
+
[Equation]
d zp
[Equation]
( z
[Equation]
f
[Equation]
[Equation]
) 1 { z ∈ φ ( C 0 ) } = (cid:90)
[Equation] [Equation]
Similarly, applying
[Equation]
to the second part of (13) gives
[Equation]
[Equation]
[Equation] x [Equation] For i ∈ { 0 , 1 } if x ∈ C i then f ( x ) =
i and φ ( x ) ∈ C i .
So if f ( x ) = 1 , φ ( x ) ∈ φ ( C 0 ) or f ( x ) = 0 , φ ( x ) ∈ φ ( C 1 ) holds we must have x / ∈ C 0 ∪ C 1 .
Therefore, following (16) gives
[Equation]
Now looking at the ﬁrst part of (14) and the ﬁrst part of (15) (cid:90) d xp T
[Equation]
x ) 1 { f ( x
[Equation]
= 1 , φ
[Equation]
x
[Equation]
= (cid:90) d xp T ( x ) 1 { f ( x )
= 1 , φ ( x ) ∈ φ ( C 0 ) , x ∈ C T } + (cid:90) d xp T ( x ) 1 { f
[Equation]
x ) =
[Equation]
φ ( x ) ∈
φ ( C 0 ) , x / ∈ C T
[Equation]
+ (cid:90) d xp T ( x ) 1 { f ( x ) = 0 , φ ( x )
∈ φ ( C 1 ) , x ∈ C T } + (cid:90) d xp T ( x ) 1 { f ( x ) =
[Equation]
φ ( x ) ∈
φ ( C 1 )
[Equation]
x / ∈ C T } ≤ (cid:90) d xp T ( x ) ( 1 { f ( x ) = 1 , φ ( x ) ∈ φ ( C 0 ) , x ∈ C T } + 1 { f ( x ) = 0 , φ ( x ) ∈ φ ( C 1 ) , x ∈ C T } ) + p T ( C cT ) ≤ (cid:90) d xp T ( x ) 1 { x ∈ C T } 1 { f ( x )
= 1 , φ ( x ) ∈ φ ( C 0 ) ∨ f ( x ) = 0 , φ ( x ) ∈ φ ( C 1 ) }
+ δ 3 .
Next we show that the ﬁrst part of (18) is 0 .
Recall that φ ( C T ) ⊂
φ ( C 0 ∪ C 1 ) and if x ∈ C T there exists x (cid:48) ∈ C T ∩ ( C 0 ∪ C 1 ) with a sequence of points x 0 , x 1 , ..., x m ∈ C T such that x 0 = x , x m = x (cid:48) , f ( x ) =
f ( x (cid:48) ) and d X ( x i − 1 , x i ) <
∆ L for all i = 1 , ..., m .
So for x ∈ C T and f ( x ) =
i , we pick such x (cid:48) .
Since φ is L -Lipschitz and φ ( C T ) ⊂ φ ( C 0 ∪ C 1 ) we have φ ( x 0 ) , φ ( x 1 ) , ..., φ ( x m
) ∈ φ ( C 0 ∪ C 1 ) and d Z ( φ ( x i − 1 ) , φ ( x i ))
<
∆ for all i
= 1 , ..., m .
Applying the fact that inf z 0 ∈ φ ( C 0 ) ,z 1 ∈ φ ( C 1 )
d Z ( z 0 , z 1 ) ≥ ∆ >
we know that if φ ( x ) =
φ ( x 0 ) ∈
φ ( C j ) for some j ∈ { 0 , 1 } then φ ( x (cid:48) )
=
φ ( x m ) ∈
φ ( C j ) .
From x (cid:48) ∈ C 0 ∪ C 1 and f ( x (cid:48) )
=
f ( x ) =
i we have φ ( x (cid:48) )
∈ φ ( C i ) .
Since C 0 ∩ C 1 = ∅ we can conclude i = j and thus φ ( x ) ∈ φ ( C i ) if f ( x ) =
i for any x ∈ C T .
Therefore, if x ∈ C T , neither f ( x ) = 1 , φ ( x ) ∈ φ ( C 0 ) nor f ( x ) = 0 , φ ( x ) ∈ φ ( C 1 ) can hold.
Hence the ﬁrst part of (18) is 0 .
So far by combining (17) and (18) we have shown that the sum of (14) and (15) (which are the ﬁrst two parts of (13)) can be upper bounded by δ
+ (1 + β )
δ
+ δ 3 .
For the third part of (13) we have
[Equation]
Putting (19) into (13) gives
[Equation]
Plugging (12) and (20) into (2) gives the result of Theorem 4.3 under Constructions 4.1 and A.1.
It remains to show that Assumption 4.2 implies the existence of a Construction A.1.
To prove this, we ﬁrst write φ ( C T ) ⊂
φ ( C 0 ∪ C 1 ) as C T ⊂ φ − 1 ( φ ( C 0 ∪ C 1 )) .
By Construction 4.1 we have p S ( C 0 ∪ C 1 ) ≥ 1 − δ 2 .
From (19) we have
[Equation]
Setting B S
=
C 0 ∪ C 1 and B T = φ − 1 ( φ ( C 0 ∪ C 1 ) in Assumption 4.2 gives a construction of Construction A.1, thus concluding the proof.
Proof of Corollary 4.5.
Based on the statement of Corollary 4.5 it is obvious that Construction 4. can be made with δ 1 = 0 , δ 2 = 0 and a ﬁnitely large L .
(Here we implicitly assume that φ is bounded on X ).
It remains to show that Assumption 4.2 holds with δ 3 = 0 .
As δ 1 = δ 2 = 0 , any B S and B T will be supersets of Supp( p S ) and Supp( p T ) respectively.
So it sufﬁcies to consider B S =
Supp( p S ) and B T =
Supp( p T ) .
Now we verify that C T =
Supp( p T ) satisﬁes the requirements in Assumption 4.2.
According to Assumption 4.4, for any x ∈ Supp( p T ) , there must exist S T,i,j such that x ∈ S T,i,j , S T,i,j is connected, f ( x (cid:48) )
=
i for all x (cid:48)
∈ S T,i,j and S T,i,j ∩ Supp( p S ) (cid:54) = ∅ .
Pick x (cid:48) ∈ S T,i,j ∩ Supp( p S ) .
Such x (cid:48) satisﬁes x (cid:48) ∈ C T ∩ B S with our choice of C T and B S .
Since S T,i,j is connected we can ﬁnd a sequence of points x 0 , ..., x m ∈ S T,i,j with x 0 = 0 , x m = x (cid:48) and d X ( x i − 1 , x i ) < (cid:15) for any (cid:15) > 0 .
As S T,i,j is label consistent we have f ( x ) =
f
( x (cid:48) ) .
Picking (cid:15) =
∆ L concludes the fact that C T =
Supp( p T ) satisﬁes the requirements in Assumption 4.2.
Since p T (Supp( p T ))
=
1 we have δ 3 = 0 .
As a result, E T ( φ, h ) ≤ (1 + β ) E S ( φ, h ) holds according to Theorem 4.3, which concludes the proof of Corollary 4.5.
Derivation of (6) .
The Fenchel Dual of ¯ f β ( u ) can be written as
[Equation]
where C f,β = f ( 1 1+ β ) − f (cid:48) ( 1 1+ β ) 1 1+ β + f (cid:48) ( 1 1+ β ) .
Therefore, the modiﬁed ¯ f β -divergence can be written as
[Equation]
Derivation of (7) .
According to Nowozin et al.
, the GAN objecitve uses f ( u ) =
u log u − (1 + u ) log(1 + u ) .
Hence f ∗ ( t ) =
− log(1 − e t ) , f (cid:48) ( u ) = log uu +1 and f (cid:48) ( 1 1+ β )
= log 1 2+ β .
So we need to parameterize T : Z (cid:55)→ (cid:0) − ∞ , log 1 2+ β (cid:3) .
T ( z ) = log g ( z )
+ β with g ( z ) ∈ (0 , 1] satisﬁes the range constraint for T .
Plugging T ( z ) =
log g ( z ) 2+ β into (6) gives the result of (7).
B E XPERIMENT D ETAILS
Synthetic datasets For source distribution, we sample class 0 from N ([ − 1 , − 0 .
3] , diag (0 . 1 , 0 . 4)) and class 1 from N ([1 , 0 . 3] , diag (0 . 1 , 0 . 4)) .
For target distribution, we sample class 0 from N ([ − 0 .
3 , − 1] , diag (0 . 4 , 0 . 1)) and class 1 from N ([0 . 3 , 1] , diag (0 . 4 , 0 . 1)) .
For label classiﬁer, we use a fully-connect neural net with 3 hidden layers (50 , 50 , 2) and the latent space is set as the last hidden layer.
For domain classiﬁer (critic) we use a fully-connect neural net with 2 hidden layers (50 , 50) .
Image datasets For MNIST we subsample 2000 data points and for USPS we subsample 1800 data points.
The subsampling process depends on the given label distribution (e.g. shift or no-shift).
For label classiﬁer, we use LeNet and the latent space is set as the last hidden layer.
For domain classiﬁer (critic) we use a fully-connect neural net with 2 hidden layers (500 , 500) .
In all experiments, we use λ = 1 in the objective (4) and ADAM with learning rate 0.0001 and β 1 = 0 .
5 as the optimizer.
We also apply a l2-regularization on the weights of φ and h with coefﬁcient 0 .
More discussion on synthetic experiments.
The only unexcepted failure is WDANN1- 2 , which achieves only 20% accuracy in 2-out-of-5 runs.
Looking in to the low accuracy runs we found that the l2-norm of the encoder weights is clearly higher than the successful runs.
Large l2-norm of weights in φ likely results in a high Lipschitz constant L , which is undesirable according to our theory.
We only implemented l2-regularization to encourage Lipschitz continuity of the encoder φ , which might be insufﬁcient.
How to enforce Lipschitz continuity of a neural network is still an open question.
Trying more sophisticated approaches for Lipschitz continuity can a future direction.
Choice of β .
Since a good value of β may depend on the knowledge of target label distribution which is unknown, we experiment with different values of β .
Empirically we did not ﬁnd any clear pattern of correlation between value of β and performance as long as it is big enough to accommodate label distribution shift so we would leave it as an open question.
In practice we suggest to use a moderate value such as 2 or 4 , or estimate based on prior knowledge of target label distribution.