Information-Theoretic Analysis of Unsupervised Domain Adaptation
Anonymous Author(s)
AffiliationAddress email
Abstract
This paper uses information-theoretic tools to analyze the generalization error inunsupervised domain adaptation (UDA).
This study presents novel upper boundsfor two notions of generalization errors.
The first notion measures the gap betweenthe population risk in the target domain and that in the source domain, and thesecond measures the gap between the population risk in the target domain and theempirical risk in the source domain.
While our bounds for the first kind of errorare in line with the traditional analysis and give similar insights, our bounds onthe second kind of error are algorithm-dependent and also inspire insights intoalgorithm designs.
Specifically, we present two simple techniques for improvinggeneralization in UDA and validate them experimentally.
Introduction
This paper focuses on the unsupervised domain adaptation (UDA) task, where the learner is confrontedwith a source domain and a target domain and the algorithm is allowed to access to a labeled trainingsample from the source domain and an unlabeled training sample from the target domain.
The goal isto find a predictor that performs well on the target domain.
A main obstacle in such a task is the discrepancy between the two domains.
Some recent works have[1–9] proposed various measures to quantify such discrepancy, either for the UDA setting or for themore general domain generalization tasks, and many learning algorithms are proposed.
For example,most recently, Nguyen et al.
[9] uses a (reverse) KL divergence to measure the misalignment ofthe distributions of the two domains, and motivated by their generalization bound, they design analgorithm that penalizes the KL divergence between the marginal distributions of two domains in therepresentation space.
Despite that this “KL guided domain adaptation” algorithm is demonstratedto outperform many existing marginal alignment algorithms [10, 11, 6, 12], it is not clear whether
KL-based alignment of marginal distributions is adequate for UDA, and more fundamentally whatrole the unlabelled target-domain training sample should play to achieve cross-domain generalization.
Notably, most UDA algorithms are heuristically designed and intuitively justified and most existinggeneralization bounds are algorithm-independent.
Then there appears significant room for bothdeeper theoretical understanding and more principled algorithm design.
In this paper, we analyze the generalization ability of hypotheses and algorithms for UDA tasks usingan information-theoretic framework developed in [13, 14].
The foundation of our bounding techniqueis the Donsker-Varadhan representation of KL divergence (see Lemma A.1) with the application ofsub-gaussianity (see Assumption 2).
We present novel upper bounds for two notions of generalizationerrors.
The first notion (“PP generalization error”) measures the gap between the population riskin the target domain and that in the source domain for a hypothesis , and the second (“expected EPgeneralization error”) measures the gap between the population risk in the target domain and theempirical risk in the source domain for a learning algorithm .
The specific contributions of this work are as follows.
We show that the PP generalization error for all hypotheses are uniformly boundedby a quantity governed by the KL divergence between the two domain distributions, which, underbounded losses, recovers the the bound in [9].
We then show that such this KL term upper-boundssome other measures including Total-Variation distance [1], Wasserstein distance [6] and domaindisagreement [7].
Thus, minimizing KL-divergence forces the minimization of other discrepancymeasures as well.
This, together with the ease of minimizing KL [9], explains the effectivenessof the KL-guided alignment approach.
For expected EP generalization error, we develop severalalgorithm-dependent generalization bounds.
These algorithm-dependent bounds further inspire thedesign of two new and yet simple strategies that can further boost the performance of the KL guidedmarginal alignment algorithms.
Experiments are performed on standard benchmarks to verify theeffectiveness of these strategies.
Related Work
Domain Adaptation From a theoretical perspective, many domain adaptation generalization boundshave been developed [1, 2, 15, 3, 6, 5, 7, 8], and some discrepancy measures are designed to derivethese bounds including the reduction of the total variation [1, 2, 15, 3], Wasserstein distance [6],domain disagreement [7] and so on.
In particular, bounds based on H ∆ H in [2] are restricted toa binary classification setting and assume a deterministic labeling function.
Furthermore, [2] alsoassumes the loss is the L 1 distance between the predicted label and true label (which is bounded).
Our bounds work for the general supervised learning problems with any labelling mechanism (e.g.,stochastic labelling), and we do not require the specific choice of the loss (which could be unbounded).[16] proposed some generalization bounds based on Jensen-Shannon (JS) divergence, which arerelated to our Corollary 4.2.
Most existing works including [2, 16] that give upper bounds for Err ,while we give upper bounds for its absolute value, | Err | , which also serves as a lower bound forgeneralization, highlighting some fundamental difficulty of the UDA learning task (see Corollary 4.1).
For more details about the domain adaptation theory, we refer readers to [17] for a completedsurvey.
From the algorithmic perspective of the domain adaptation, the most common method is toalign the marginal distribution of representation between the source domain and the target domain,including using the adversarial training mechanism [10, 6, 8] and aligning the first two moments ofthe representation distribution [11].
There are numerous other domain adaptation algorithms, and werefer readers to [18–21] for recent advances.
Information-Theoretic Generalization Bounds Information-theoretic analysis are usually usedto analyze the expected generalization error of supervised learning, where the training and testingdata come from the same distribution [13, 22, 14, 23–27].
By exploiting the chain rule property ofmutual information, these bounds are successfully applied to characterize the generalization ability ofstochastic gradient based optimization algorithms [28, 24, 26, 29–31].
Recently, this framework hasalso been used in the multi task setting including meta-learning [32–35], semi-supervised learning[36, 37] and some other transfer learning problems [38, 32, 39–41].
In particular, [38, 39] consider adifferent transfer learning problem setup with ours.
Specifically, their expected generalization error isthe gap between the target population risk and the empirical weighted risk (or the convex combinationof the source empirical risk and the target empirical risk), while our “EP” error is the gap betweenthe target population risk and the source empirical risk.
That is to say, our work studies how to makeuse of the unlabelled target data to improve the generalization performance on target domain exceptfor minimizing the empirical risk of source domain, and their works assume the training objectivefunction for the target domain data, which could be labelled, has already been known.
In addition,bounds in [38, 39] fail to characterize the dependence between W and S ′ X ′ .
More precisely, thealgorithm-dependent term in their bounds is I ( W ; Z i ) or I ( W ; S ) , while our algorithm-dependentterm is I X
′ j
( W ; Z i ) that directly depends on the unlabelled target data (see Theorem C.1 for morediscussion in Appendix).
Preliminary
Unless otherwise noted, a random variable will be denoted by a capitalized letter, and its realizationdenoted by the corresponding lower-case letter.
Consider a prediction task with instance space
Z = X
× Y , where X and Y are the input space and the label (or output) space respectively.
Let Fbe the hypothesis space of interesting, in which each f ∈ F is a function or predictor mapping X to
Y .
We assume that each hypothesis f ∈ F is parameterized by some weight parameter w in somespace W and may write f as f w as needed.
Let µ and µ ′ be two distributions on Z , unknown to the learner.
Normally, µ and µ ′ are not thesame
and we consider µ characterizing the source domain and µ ′ characterizing the target domain.
For the ease of notation, we may also write µ as P Z or P XY and µ ′ as P Z ′ or P X ′ Y ′ , which alsodefines random variables Z =
( X, Y ) and Z ′ =
( X ′ , Y ′ ) .
Let S
=
{ Z i } ni =1 ∼ µ ⊗ n be a labeledsource-domain training sample and S
′ X
′ =
{ X ′ j } mj =1 ∼ P ⊗ mX ′ be an unlabelled target-domain trainingsample.
The objective of UDA is to design an algorithm A takes S and S ′ X ′
as the input and outputsa weight W ∈ W , giving rise to a predictor f W ∈ F that “works well” on the target domain.
Notethat the algorithm A is in general characterized by a conditional distribution P W | S,S ′ X ′ .
To be precise on the performance metric of UDA, let ℓ :
Y × Y → R +0 be a loss function.
Then foreach weight configuration w ∈ W , its population risk in the target domain is defined as
[Equation]
and a good UDA algorithm hopes to return a weight w that minimizes this risk.
Since µ ′ is unknown,this risk can not be measured or minimized.
On the other hand, one does have access to the empiricalrisk in the source domain, as is defined by
[Equation]
Then the notion generalization error in this setting measures how well the hypothesis returned fromthe algorithm generalize from the source-domain training sample to the target-domain unknowndistribution µ ′ .
Taking into account the stochastic nature of the algorithm A , a natural notion ofgeneralization error for UDA can be defined by
[Equation]
where the expectation in the first equation is taken over the joint distribution of ( W, S ) ∼ P W | S × µ ⊗ n , 109 and the expectation of the second equation is taken over the joint distribution of ( W, S, S ′ X ′ )
∼
P W | S,S ′ X ′
× µ ⊗ n × P ⊗ mX ′ .
Note that there is another notion of generalization error, more traditional in the domain adaptationliterature, namely, the gap between the population risk in the target domain and that in the sourcedomain, as us define by
[Equation]
where R µ ( w ) ≜ E Z [ ℓ ( f w ( X ) , Y )] .
It is apparent that (cid:103) Err( w ) and Err are related by the followingtriangle inequality:
[Equation]
where the second term on the right hand side is the standard generalization error in the source domain,which can be bounded by classical learning-theoretic tools, e.g., Rademacher complexity [42].
Thusbounding (cid:103) Err( w ) helps bounding Err .
This paper studies both notions of generalization error for UDA.
Specifically, starting from Section 5,we will mainly use information-theoretic tools to bound Err directly, without going through (cid:103) Err( w ) .
For the ease of reference, we refer to (cid:103) Err( w ) as the population-to-population (PP) generalizationerror for w and Err as the expected empirical-to-population (EP) generalization error for thealgorithm A .
Some definitions are prerequisite in this paper, we now present some uncommon notions and deferthe common notions to Appendix.
Definition 1 (Disintegrated Mutual Information) .
Let X , Y and Z be random variables and z bea realization of Z .
The disintegrated mutual information of X and Y given Z =
z is I z
( X ; Y ) ≜
D KL ( P X,Y | Z =
z || P X
| Z = z P Y | Z = z ) .
Note that the conditional mutual information I ( X ; Y | Z ) =
E Z I Z
( X ; Y ) .
Definition 2 (Lautum Information [43]) .
Define the lautum information between X and Y as
L ( X ; Y ) ≜ D KL ( P X P Y || P XY ) .
Upper Bounds for PP Generalization Error
In this section, we present some upper bounds for (cid:103) Err( w ) .
The key techniques used in developingthese bounds are the information-theoretic tools in the style of Lemma A.1.
All these bounds adoptcertain KL divergence as a key quantity measuring the discrepancy between the source and targetdomain.
Notably, some previously established bounds are recovered under a different assumption ofthe loss function.
Additionally we demonstrate that under certain conditions, the KL-based bound isan upper bound of many other discrepancy measures and hence minimizing the KL divergence forcesthe minimization of these other measures.
We first list some common assumptions on the loss function, which we consider in this paper.
Assumption 1 (Boundedness) .
ℓ ( · , · ) is bounded in [0 , M ] .
Assumption 2 (Subgaussianity) .
ℓ ( f w ( X ) , Y ) is R -subgaussian 1 under µ for any w ∈ W .
Remark 4.1.
Note that Assumption 1 implies Assumption 2, i.e., if ℓ ( f w ( X ) , Y ) is bounded in [0 , M ] ,then it is also M/ 2 -subgaussian.
Thus, Assumption 2 is weaker than Assumption 1.
Assumption 3 (Lipschitzness) .
ℓ ( f w ( X ) , Y ) is β -Lipschitz continues in Z for any w ∈ W , i.e.,| ℓ ( f w ( x 1 ) , y 1 ) − ℓ ( f w ( x 2 ) , y 2 ) | ≤ βd ( z 1 , z 2 ) .
Remark 4.2.
Note that Assumption 1 implies Assumption 3 when d is a discrete metric, i.e., ifℓ ( f w ( X ) , Y ) is bounded in [0 , M ] , then it is also M -Lipschitz under the discrete metric.
Assumption 4 (Triangle) .
ℓ ( · , · ) satisfies the following the triangle inequality: ℓ ( y 1 , y 2 ) ≤ ℓ ( y 1 , y 3 )
+ℓ ( y 3 , y 2 ) for any y 1 , y 2 , y 3 ∈ Y .
4.1 Generalization Bounds via the Subgaussian Condition
The following generalization bound is established by combining Lemma A.1 and Assumption 2, atechnique developed in [14] for information-theoretic analysis of generalization.
Theorem 4.1.
If Assumption 2 holds, then for any w ∈ W , (cid:12)(cid:12)(cid:12)(cid:103) Err
[Equation]
We note that this result on one hand can be turned into a generalization upper bound providingguidance to algorithm design, and on the other hand provides a lower bound of the generalizationerror, which highlights some fundamental difficulty of the learning task.
To illustrate this, we presentan corollary of Theorem 4.1, while noting that similar development can also be applied to otherbounds presented later in this paper.
To that end, suppose that each f w in the model family is expressed as the composition g ◦ h , where his a function mapping X to a representation space T and g is a function mapping T to Y .
For anygiven h : X → T , denote by µ h the distribution on T × Y obtained by pushing over µ via h , that is,µ
h
( t, y )
= (cid:82) δ ( t − h ( x ))
dµ
( x, y ) , where δ is the Dirac measure on T .
Similarly, let µ ′ h denote thedistribution on T × Y obtained by pushing over µ ′ via h .
Corollary 4.1.
Suppose that f w = g ◦ h and that Assumption 2 holds.
then for any w ∈ W ,
[Equation]
In this result, the lower bound of R µ ′ ( w ) indicates a fundamental difficulty in UDA learning in that,using the same predictor mapping f w , there is no way for the population risk in the target domain tobe lower than that of the source domain less a constant which depends only on the domain difference.
On the other hand, the upper bound suggests that it is possible to squeeze the gap between the twopopulation risks by choosing an appropriate representation map h - evidently such a map should beattempting to align µ ′ h with µ h or to align their respective proxies.
It is also remarkable that under Assumption 1 and due to Remark 4.1, Theorem 4.1 implies
[Equation]
Similarly applying this result in the representation space T , we see that Eq.
(3) recovers the bound in
Proposition 1 of [9].
Notice that unlike [9], Theorem 4.1 ( or Eq.
(3)) does not require the loss to bethe cross entropy loss.
Theorem 4.1 and [9] both use the KL divergence from source domain to target domain, D KL ( µ ′ || µ ) ,and in fact, (cid:12)(cid:12)(cid:12)(cid:103)
Err( w ) (cid:12)(cid:12)(cid:12) can also be upper bounded by D KL ( µ || µ ′ ) .
This can be done by invoking thesubgaussianality of ℓ ( f w ( X ′ ) , Y ′ ) (rather than ℓ ( f w ( X ) , Y ) ); for bounded loss, the subgaussianalityof ℓ ( f w ( X ′ ) , Y ′ ) is also satisfied.
Then we obtain the following corollary.
Corollary 4.2.
If Assumption 1 holds, (cid:12)(cid:12)(cid:12)(cid:103) Err( w ) (cid:12)(cid:12)(cid:12) ≤ M √ 2 (cid:112) min
{ D KL ( µ || µ ′ ) , D KL ( µ ′ || µ ) } ≤
M 2 (cid:112) D KL ( µ || µ ′ )
+ D KL ( µ ′ || µ ) .
Remark 4.3.
In the second inequality of Corollary 4.2, D KL ( µ || µ ′ )
+ D KL ( µ ′ || µ ) is usually calledthe symmetrized KL divergence (or Jeffrey’s divergence [44]), and the regularization term used in[9] is indeed the symmetrized KL divergence between the distributions of the source and targetrepresentations.
Notice that bounds in [16] are based on the JS divergence.
Since there is a sharpupper bound of the JS divergence based on Jeffrey’s divergence [45], minimizing Jeffrey’s divergence(in the representation space) will simultaneously penalize the JS divergence.
In UDA, since Y ′ is completely unavailable to the algorithm A , it is impossible to minimize themisalignment of conditional distributions, i.e. D KL ( P Y ′ | T ′
||
P Y | T ) , without any additional information.
A common method is to assign pseudo labels to target data.
However, it may also causesome additional issues.
For concreteness, suppose the trained model Q can well approximate thereal mapping between X and Y on source domain (i.e. Q Y | T =
P Y | T ), which is usually the trainingobjective.
Let ˆ Y ′ be the pseudo label of T ′ generated by the trained model, i.e., Q ˆ Y ′ | T ′ =
Q Y | T .
Let
Q T ′ , ˆ Y ′ =
P T ′ Q ˆ Y ′ | T ′ , then the following holds,
[Equation]
For a specific t ′ , if P ( Y ′ =
y ′ | T ′ =
t ′ )
̸ = 0 and P ( ˆ Y ′ =
y ′ | T ′ =
t ′ )
= 0 , then the second termin RHS of Eq.
(4), D KL ( P Y ′ | T ′ ||
Q ˆ Y ′ | T ′ )
→ ∞ .
In this case, even the marginal distributions areperfectly aligned, the overall value of the upper bound is large.
Thus, incorrect pseudo labels mayeven have negative impact on the target domain performance, and we hope two supports, Supp( P Y ′ )and Supp( P ˆ Y ′ ) , could largely overlap with each other for every target data.
Indeed, the misalignment of the conditional distributions appears to be the main difficulty of UDA[1, 8].
The next corollary suggests that this difficulty may be alleviated when the loss function satisfiesthe triangle property, namely, Assumption 4.
It can be verified that this assumption is satisfied by the0-1 loss and square error loss; this assumption has also been considered in previous works [3, 6].
Theorem 4.2.
If Assumption 4 holds and let ℓ ( f w ′ ( X ) , f w ( X )) be R -subgaussian for any w, w ′ ∈ W .
Then for any w , (cid:103) Err( w ) ≤ (cid:112) 2 R 2 D KL ( P X
′ ||
P X ) + λ ∗ , where λ ∗ =
min w ∈W R µ ′ ( w ) + R µ ( w ) .
In this theorem, λ ∗ measures the possibility of whether the domain adaptation algorithm will succeedunder the oracle knowledge of µ and µ ′ .
In particular, if the hypothesis space is large enough,the minimizer w ∗ for the “joint population risk” R µ ′ ( w ) + R µ ( w ) may give rise to R µ ′ ( w ∗ ) =
R µ ( w ∗ )
= 0 .
then we’re likely to generalize well on the target domain.
Then the KL divergence
D KL ( P X
||
P X ) between the two X -marginals alone bounds the PP generalization error uniformlyfor all w ∈ W .
This theorem motivates the strategy of penalizing D KL ( P T ′ || P T ) in the representation space toachieve better a generalization error.
The next theorem suggests that such an approach also penalizesother notions of domain discrepancy, for example, domain disagreement defined in [7, Definition 1.]and serving as a key quantity in the PAC-Bayes type of domain adaptation generalization bounds [7]:
[Equation]
Theorem 4.3.
If ℓ ( f w ′ ( X ) , f w ( X )) is R -subgaussian for any w, w ′ ∈ H , then dis( P X , P X ′ ) ≤ 217 (cid:112)
[Equation]
Note that unlike [7], here we do not require the loss function to be the 0-1 loss.
4.2 Generalization Bounds via the Lipschitz Condition
Wasserstein distance based generalization bound are often directly connected to, or even includedin, the information-theoretic bounds [46, 27].
We now present such a bound for UDA under the
Lipschitz continuity assumption of the loss function.
Theorem 4.4.
If Assumption 3 holds, then (cid:12)(cid:12)(cid:12)(cid:103) Err( w ) (cid:12)(cid:12)(cid:12) ≤ β W ( µ ′ , µ ) .
Note that Theorem 4.4 can be related to the KL divergence based bounds in the previous sectionwhen the Wasserstein distance is defined with respect to the discrete metric d .
In this case, if the lossfunction is bounded, it is also Liptschitz continuous, and hence Theorem 4.4 applies.
On the otherhand, Wasserstein distance is equivalent to the total variation distance [1, 2, 15, 3], while the latter isconnected to the KL divergence via Pinsker’s inequality [47, Theorem 6.5] and the Bretagnolle-Huberinequality [48, Lemma 2.1].
Thus we arrive at the following result.
Corollary 4.3.
If Assumption 1 holds holds and let d be the discrete metric, then
[Equation]
The bound in Corollary 4.3 can be immediately verified to be tighter than the bound in Eq.
Parallel to Theorem 4.2, if the loss function satisfies the triangle property, we may establish anotherbound below, which recovers a similar result in [6, Theorem 1.].
Theorem 4.5.
If Assumption 4 holds and ℓ ( f w ( X ) , f w ′ ( X )) is β -Lipschitz in X for any w, w ′ ∈ W ,then for any w ∈ W , (cid:103) Err( w )
≤ L W ( P X ′ , P X ) + λ ∗ , where λ ∗ =
min w ∈W R µ ′ ( w ) + R µ ( w ) .
Unlike the bound in [6], we do not require the classification tasks to be binary in Theorem 4.5, andthe loss does not need to be the L 1 distance.
This section may convey the following message.
Since the KL divergence based bounds upperbounds those based on other measures of domain differences, (e.g. total variation distance, domaindiscrepancy etc), if we penalize the KL divergence, we will also penalize those other measures.
This ispractically advantageous since it is usually easier and more stable to minimize the KL divergence[9].
Upper Bounds for Expected EP Generalization Error and Applications
There are two limitations in the bounds on the PP generalization error developed in the previoussection and in the traditional analysis of domain adaptation.
First, such bounds are independent of wand hence algorithm-independent.
Second, although these bounds may inspire strategies to exploit theunlabelled target sample, e.g., aligning its marginal distribution with that of the source sample in therepresentation space, they only provide very limited knowledge on the role that the unlabelled targetsample plays in the algorithm.
We now derive upper bounds for the EP generalization error, whichbetter utilize the dependence of the algorithm output on the unlabelled target data.
Applications ofthese bounds in designing the learning algorithms are also presented.
5.1 Bounds
Theorem 5.1.
Assume ℓ ( f w ( X ′ ) , Y ′ ) is R -subgaussian under µ ′ for any w ∈ W .
Then
[Equation]
Remark 5.1.
Note that the unlabelled target data plays a role in the first term of the bound.
Indeed,more source and target data will reduce the first term of the bound.
Specifically, moving theexpectation inside the square root function by Jensen’s inequality and since Z i ⊥⊥ X ′ j , the equations
I ( W ; Z i | X ′ j ) =
I ( W ; Z i | X ′ j )
+ I
( Z i ; X ′ j ) =
I ( W ; Z i ) + I
( X ′ j ; Z i | W ) hold by the chain rule.
Theterm I ( W ; Z i ) will vanish as n → ∞ and the term I ( X ′ j ; Z i | W ) will also vanish as n, m → ∞ .
It is also worth mentioning that, from a practical perspective, the number of samples may havedifferent impact on the different algorithms.
For example, the second term (KL divergence) inour Theorem 5.1 can not be computed in the original space and we can only estimate it in therepresentation space.
On the one hand, it seems that having more data will make the approximation(of KL between marginal distributions) more accurate.
While on the other hand, some domainadaptation algorithms involve the pseudo labelling process, and assigning incorrect pseudo labels tothe target data may even have negative impact on the target domain performance (as discussed in
Section 4).
In this case, having more target data will not improve the performance.
Corollary 5.1.
Let Assumption 1 hold.
Then
[Equation]
where L X ′ j ( · ; · ) is the disintegrated version of Lautum information.
Theorem 5.2.
Assume ℓ is Lipschitz for both w ∈ W and z ∈ Z , i.e., | ℓ ( f w ( x ) , y ) − ℓ ( f w ( x ′ ) , y ′ )
| ≤βd 1 ( z, z ′ ) for all z, z ′ ∈ Z and | ℓ ( f w ( x ) , y ) − ℓ ( f w ′ ( x ) , y ) | ≤ β ′ d 2 ( w, w ′ ) for all w, w ′ ∈ W , then|
[Equation]
This bound is tighter than the bound in Theorem 5.1, as can be indicated by the following corollary.
Corollary 5.2.
Let Assumption 1 hold.
Then
[Equation]
Notice that to recover Theorem 5.1 from Corollary 5.2, we can use Jensen’s inequality to move theexpectation over Z i inside the convex square root function.
5.2 Gradient Penalty as an Universal Regularizer
The algorithm-dependent bound in Theorem 5.1 tells us that one can reduce the expected generalization error by limiting the disintegrated mutual information I X
′ j
( W ; Z i ) .
In the stochastic gradientbased optimization algorithms, this term can be controlled by penalizing the gradient.
To see this, wenow consider a “noisy” iterative algorithm for updating W , e.g., SGLD.
At each time step t , let thelabelled mini-batch from the source domain be Z B t , let the unlabelled mini-batch from the targetdomain be X ′ B t , and let g ( W t − 1 , Z B t , X ′ B t ) be the gradient at time t .
Thus, the updating rule of Wis W t =
W t − 1 − η t g ( W t − 1 , Z B t , X ′ B t ) + N t where η t is the learning rate and N t ∼ N (0 , σ 2 I d ) isan isotropic Gaussian noise.
The next theorem is an application of Theorem 5.1 in this setting.
Theorem 5.3.
Let the total iteration number be T and let G t =
g ( W t − 1 , Z B t , X ′ B t ) , then
[Equation]
Remark 5.2.
Considering a noisy iterative algorithm here is to simplify analysis.
In fact it is alsopossible to analyze the original iterative gradient optimization method without noise injected.
Forexample, one can follow the same development in [30, 31] to analyze vanilla SGD.
In that case, therewill be some additional terms in the bound, which are related to flatness of the found minima.
Theorem 5.3 hints that to reduce the generalization error, one can restrict the gradient norm at eachstep.
This strategy will also restrict the distance between the final output W T and the initialization
W 0 , effectively shrinking the hypothesis space accessible by the algorithm.
Indeed, adding gradient penalty can be applied to any existing UDA algorithm
and it is simple buteffective in practice.
Later on we will show that even when the algorithm A does not access toany target data, in which case I ( W ; Z i | X ′ j ) reduces to I ( W ; Z i ) and g ( W t − 1 , Z B t , X ′ B t ) becomesg ( W t − 1 , Z B t ) , minimizing the empirical loss of source domain sample while penalizing gradientnorm will still improve the performance.
Notice that gradient penalty is also used in Wassersteindistance based adversarial training [49, 6], and their motivation is to stabilize the training to avoidgradient vanishing problem while here we use it to improve the generalization performance directly.
Notably the bound in Theorem 5.3 only depends on the size n of labelled source sample and doesnot explicitly depend on m , the size of unlabelled target sample.
With a more careful design, if we consider the mutual information as the expected KL divergence of a posterior and a prior, basedon I X ′ j
( W ; Z i ) in Theorem 5.1, it is possible to create a target data dependent prior and derive atighter bound based on some quantity similar to "gradient incoherence" in [24].
As this will introduceadditional complexity in practice, we leave this as a future study.
5.3 Controlling Label Information for KL Guided Marginal Alignment
Consider instances in the representation space,
Z =
( T, Y ) and Z ′ =
( T ′ , Y ) .
Theorem 5.1 alsoencourage us to align the distributions of two domains in the representation space, as argued earlier.
Then the KL guided marginal alignment algorithm proposed in [9] can be invoked here.
One maynotice that Theorem 5.1 uses D KL ( µ || µ ′ ) while [9] uses D KL ( µ ′ || µ ) .
As already discussed in
Section 4, this inconsistency can be ignored when loss is bounded (see Corollary 5.1).
Most domain adaptation algorithms aim to align the marginal distributions of two domains in therepresentation space.
However, without accessing to Y ′ , it remains unknown if an UDA algorithmwill work well since we cannot guarantee that discrepancy between conditional distribution P Y | Tand P Y ′ | T ′ won’t become too large when we align the marginals.
In [9], the authors show that
D KL ( P Y ′ | T ′
||
P Y | T ) can be upper-bounded by D KL ( P Y ′ |
X ′ ||
P Y | X ) , if I ( X ; Y )
=
I ( T ; Y ) .
Theauthors then argue that penalizing the KL divergence of the marginals distributions is safe.
We now argue that in practice the condition I ( X ; Y ) =
I ( T ; Y ) can be difficult to satisfy if thecross-entropy loss is used to define the source-domain empirical risk.
By data processing inequality on Y − X − T , we know that I ( X ; Y ) ≥ I ( T ; Y ) =
H ( Y ) − H
( Y | T ) .
Thus, to let I ( T ; Y ) reach its maximum, one must minimize H ( Y | T ) .
On the other hand, let Q Y | T,Wbe the predictive distribution of labels in the source domain generated by the classifier.
The expectedcross-entropy loss for each Z i in the representation space is then
[Equation]
which also decomposes as [50, 51]
[Equation]
Then minimizing the expected cross-entropy loss may not adequately reduce H ( Y i | T i ) but rathercause I ( W ; Y i | T i ) to significantly increase, particularly when the model capacity is large.
Thismay have two negative effects.
First, the condition I ( X ; Y )
=
I ( T ; Y ) is significantly violated,and D KL ( P Y ′ | T ′
||
P Y | T ) is no longer upper bounded by D KL ( P Y ′ | X ′
||
P Y | X ) .
As a consequence,aligning the two marginals alone may not be adequate.
Second, large I ( W ; Y i | T i ) indicates Wjust simply memorizes the label Y i , resulting a form of overfitting and hurting the generalizationperformance.
The key take-away from the above analysis is that when aligning the marginals in UDA, controlling thesource label information in the weights can be important to achieve good cross-domain generalization.
A similar message can also be deduced from Theorem 5.1, when it is viewed in the repsentation spaceand noting I T ′
[Equation] [Equation]
[Equation] [Equation]
+ I T ′
j ( W ; Y i | T
i
[Equation]
To control label information, [51] proposed an approach called LIMIT.
However this method is rathercomplicated and arguably hard to train in domain adaptation (see Appendix).
We now derive a simplealternative strategy for this purpose.
Notice that I T ′
j ( W ; Y i | T i ) ≤ inf Q E T i (cid:2) D KL ( P ( W | Y i , T i , T ′
j = t ′ j ) || Q
( W | T i , T ′
j = t ′ j ))
(cid:3) , which isa simple extension of variational representation of mutual information [47, Corollary 3.1.].
Here
Q could be any distribution.
By assuming P = N ( W, σ 2 I d | Y i , T i , T ′ j = t ′ j ) and taking Q =
N ( (cid:102) W, ˜ σ 2 I d | T i , T ′ j = t ′ j ) , we have
[Equation]
Thus, we may create an auxiliary classifier f (cid:101) w that is not allowed to access to the real source label 343 Y .
In each iteration, we use the pseudo labels of target data (and source data) assigned by f w totrain f (cid:101) w and adding || W − (cid:102) W || 2 as a regularizer in the training of W .
The algorithm is given inthe Appendix.
Remarkably the regularizer here resembles “Projection Norm” designed in [52] forout-of-distribution generalization.
[Table]
Experimental Results
We now perform experiments to verify the proposed techniques inspired by our theory in the previoussection.
The experimental setup follows that in [9].
Datasets We select two popular small datasets, RotatedMNIST and Digits, to compare the differentmethods.
In particular, RotatedMNIST is built based on the MNIST dataset [53] and consists of sixdomains with each domain containing 11 , 666 images.
These six domains are rotated MNIST imageswith rotation angle 0 ◦ , 15 ◦ , 30 ◦ , 45 ◦ , 60 ◦ and 75 ◦ , respectively.
We will take the original MNISTdataset ( 0 ◦ ) as the source domain and take other five domains as target domains.
Hence there are fivedomain adaptation tasks on RotatedMNIST.
Digits consists of three sub-datasets, namely MNIST,
USPS [54] and SVHN [55], and the corresponding domain adaptation tasks are MNIST → USPS( M → U ), USPS → MNIST ( U → M ), SVHN → MNIST ( S → M ).
Compared Methods Baseline methods are some popular marginal alignment UDA methodsincluding DANN [10], MMD [12], CORAL [11], WD [6] and KL [9].
We also choose ERM foranother baseline in which the algorithm can only access to the source domain sample during training.
To verify the strategies inspired by our theory, we first add the gradient penalty to the ERM algorithm
( ERM-GP ), and we then combine gradient penalty (GP) and controlling label information (CL)with the recent proposed KL guided marginal alignment method, which are denoted by KL-GP and
KL-CL , respectively.
Implementation Details Most part of the implementation is based on the famous DomainBedsuite [56].
Other settings are exactly the same with [9] and the results of baseline methods arereported directly from [9].
Specifically, each algorithm is run three times and we show the averageperformance with the error bar.
Every dataset has a validation set, and the model selection scheme isbased on the best performance achieved on the validation set of target domain during training (oracle).
The hype-parameter searching process is also built upon the implementation in the DomainBed suite.
Other details and additional experiments can be found in Appendix.
Results From Table 1, we first notice that gradient penalty is able to help ERM to be morecomparable with other marginal alignment methods.
For example, on RotatedMNIST, ERM-GPoutperforms CORAL and performs nearly the same with DANN .
On Digits, ERM-GP outperforms
WD .
When GP and CL combined with KL guided algorithm, we can see that the performance can befurther boosted.
This justifies the discussion in Section 5.2 and Section 5.3.
Conclusion
Despite that the numerous learning techniques have been developed for domain adaptation, significantroom exists for more in-depth theoretical understanding and more principled design of learning algorithms.
This paper presents the information-theoretic analysis for unsupervised domain adaptation,where we query two notions of the generalization errors in this context and present novel learningbounds.
Some of these bounds recover the previous KL-based bounds under different conditions andconfirm the insights in the learning algorithms that align the source and target distributions in therepresentation space.
Our other bounds are algorithm-dependent, better exploiting the unlabelledtarget data, which have inspired novel and yet simple schemes for the design of learning algorithms.
We demonstrate the effectiveness of these schemes on standard benchmark datasets.