I NFORMATION -T HEORETIC A NALYSIS OF U NSUPERVISED D OMAIN A DAPTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper uses information-theoretic tools to analyze the generalization error in unsupervised domain adaptation (UDA).
We present novel upper bounds for two notions of generalization errors.
The first notion measures the gap between the population risk in the target domain and that in the source domain and the second measures the gap between the population risk in the target domain and the empirical risk in the source domain.
While our bounds for the first kind of error are in line with the traditional analysis and give similar insights, our bounds on the second kind of error are algorithm-dependent which also provide insights into algorithm designs.
Specifically, we present two simple techniques for improving generalization in UDA and validate them experimentally
I NTRODUCTION
This paper focuses on the unsupervised domain adaptation (UDA) task, where the learner is confronted with a source domain and a target domain and the algorithm is allowed to access to a labeled training sample from the source domain and an unlabeled training sample from the target domain.
The goal is to find a predictor that performs well on the target domain.
A main obstacle in such a task is the discrepancy between the two domains.
Some recent works have (Ben-David et al., 2006; 2010;
Mansour et al., 2009; Zhao et al., 2019; Zhang et al., 2019;
Shen et al., 2018; Germain et al., 2020; Acuna et al., 2021; Nguyen et al., 2022) proposed various measures to quantify such discrepancy, either for the UDA setting or for the more general domain generalization tasks, and many learning algorithms are proposed.
For example, Nguyen et al.
(2022) uses a (reverse) KL divergence to measure the misalignment of the two domain distributions, and motivated by their generalization bound, they design an algorithm that penalizes the KL divergence between the marginal distributions of two domains in the representation space.
Despite that this “KL guided domain adaptation” algorithm is demonstrated to outperform many existing marginal alignment algorithms (Ganin et al., 2016; Sun & Saenko, 2016;
Shen et al., 2018; Li et al., 2018), it is not clear whether KL-based alignment of marginal distributions is adequate for UDA, and more fundamentally, what role the unlabelled target-domain sample should play in cross-domain generalization.
Notably, most UDA algorithms are heuristically designed and intuitively justified.
Moreover, most existing generalization bounds are algorithm-independent.
Then there appears significant room for both deeper theoretical understanding and more principled algorithm design.
In this paper, we analyze the generalization ability of hypotheses and learning algorithms for UDA tasks using an information-theoretic framework developed in (Russo & Zou, 2016; Xu & Raginsky, 2017).
The foundation of our technique is the Donsker-Varadhan representation of KL divergence (see Lemma A.1).
We present novel upper bounds for two notions of generalization errors.
The first notion (“population-to-population (PP) generalization error”) measures the gap between the population risk in the target domain and that in the source domain for a hypothesis , and the second (“expected empirical-to-population (EP) generalization error”) measures the gap between the population risk in the target domain and the empirical risk in the source domain for a learning algorithm .
We show that the PP generalization error for all hypotheses are uniformly bounded by a quantity governed by the KL divergence between the two domain distributions, which, under bounded losses, recovers the the bound in Nguyen et al.
We then show that this KL term upper-bounds some other measures including Total-Variation distance (Ben-David et al., 2006), Wasserstein distance (Shen et al., 2018) and domain disagreement (Germain et al., 2020).
Thus, minimizing KL- divergence forces the minimization of other discrepancy measures as well.
This, together with the ease of minimizing KL (Nguyen et al., 2022), explains the effectiveness of the KL-guided alignment approach.
For expected EP generalization error, we develop several algorithm-dependent generalization bounds.
These algorithm-dependent bounds further inspire the design of two new and yet simple strategies that can further boost the performance of the KL guided marginal alignment algorithms.
Experiments are performed to verify the effectiveness of these strategies.
R ELATED W ORK
Domain Adaptation Many domain adaptation generalization bounds have been developed (BenDavid et al., 2006; 2010; David et al., 2010;
Mansour et al., 2009; Shen et al., 2018; Zhang et al., 2019; Germain et al., 2020; Acuna et al., 2021), and various discrepancy measures are introduced to derive these bounds including total variation (Ben-David et al., 2006; 2010; David et al., 2010; Mansour et al., 2009), Wasserstein distance (Shen et al., 2018), domain disagreement (Germain et al., 2020) and so on.
In particular, bounds based on H ∆ H in Ben-David et al.
(2010) are restricted to a binary classification setting and assume a deterministic labeling function.
Furthermore, BenDavid et al.
(2010) also assumes the loss is the L 1 distance between the predicted label and true label (which is bounded).
Our bounds work for the general supervised learning problems with any labelling mechanism (e.g., stochastic labelling), and we do not require the specific choice of the loss (even unbounded).
Shui et al.
(2020) proposed some generalization bounds based on JensenShannon (JS) divergence, which are related to our Corollary 4.2.
There also exist some algorithmdependent bounds under other transfer learning setting, e.g., Wang et al.
Most existing works including (Ben-David et al., 2010; Shui et al., 2020) that give upper bounds for Err , while we give upper bounds for its absolute value, | Err | , which also serves as a lower bound for generalization, highlighting some fundamental difficulty of the UDA learning task (see Corollary 4.1).
For more details about the domain adaptation theory, we refer readers to Redko et al.
(2020) for a comprehensive survey.
The most common methods for domain adaptation involve aligning the marginal distributions of the representations between the source and target domains, for example, using an adversarial training mechanism (Ganin et al., 2016;
Shen et al., 2018; Acuna et al., 2021) to align the first two moments of the two distributions in the representation space (Sun & Saenko, 2016).
There are numerous other domain adaptation algorithms, and we refer readers to (Csurka, 2017; Wilson &
Cook, 2020; Zhou et al., 2021; Wang et al., 2021b) for recent advances.
Information-Theoretic Generalization Bounds Information-theoretic analysis are usually used to bound the expected generalization error of supervised learning, where the training and testing data come from the same distribution (Russo & Zou, 2016; 2019; Xu & Raginsky, 2017; Bu et al., 2019;
Negrea et al., 2019; Steinke & Zakynthinou, 2020; Haghifam et al., 2020; Rodr ´ ıguez G ´ alvez et al., 2021).
Exploiting the chain rule of mutual information, these bounds are successfully applied to characterize the generalization ability of stochastic gradient based optimization algorithms (Pensia et al., 2018; Negrea et al., 2019; Haghifam et al., 2020; Wang et al., 2021a; Neu et al., 2021; Wang & Mao, 2022).
Recently, this framework has also been used in multi-task settings including metalearning (Jose & Simeone, 2021a;
Jose et al., 2021; Rezazadeh et al., 2021; Chen et al., 2021), semisupervised learning (He et al., 2021; Aminian et al., 2022) and transfer learning (Wu et al., 2020;
Jose & Simeone, 2021a;b; Masiha et al., 2021; Bu et al., 2022).
In particular, (Wu et al., 2020;
Jose & Simeone, 2021b) consider a different transfer learning problem setup with ours.
Specifically, their expected generalization error is the gap between the target population risk and a weighted empirical risk combining both the source and the target empirical risks, while our “EP” error is the gap between the target population risk and the source empirical risk.
That is, we focus on the role of the unlabelled target data in cross-domain generalization when the source empirical risk is taken as a training objective, whereas their works assume the existence of labelled target data and study their role in domain adaptation.
P RELIMINARY
Unless otherwise noted, a random variable will be denoted by a capitalized letter, and its realization denoted by the corresponding lower-case letter.
Consider a prediction task with instance space
Z = X
× Y , where X and Y are the input space and the label (or output) space respectively.
Let F be the hypothesis space of interest, in which each f ∈ F is a function or predictor mapping X to
Y .
We assume that each hypothesis f ∈ F is parameterized by some weight parameter w in some space W and may write f as f w as needed.
Let µ and µ ′ be two distributions on Z , unknown to the learner, where µ characterizes the source domain and µ ′ the target domain.
We may also write µ as P Z or P XY and µ ′ as P Z ′ or P X ′ Y ′ , which defines random variables Z =
( X, Y ) and Z ′ =
( X ′ , Y ′ ) .
Let S
=
{ Z i } ni =1 ∼ µ ⊗ n be a labeled source-domain sample and S
′ X
′ =
{ X ′ j } mj =1 ∼ P ⊗ mX ′ be an unlabelled target-domain sample.
The objective of UDA is to design an algorithm A that takes S and S ′ X ′
as the input and outputs a weight
W ∈ W , giving rise to a predictor f W ∈ F that “works well” on the target domain.
Note that the algorithm A is characterized by a conditional distribution P W | S,S ′ X ′ .
Let ℓ :
Y × Y
→ R +0 be a loss function.
The population risk for each w ∈ W in the target domain is defined as
[Equation]
and a good UDA algorithm hopes to return a weight w that minimizes this risk.
Since µ ′ is unknown, one often use recourse to the empirical risk in the source domain, defined as
[Equation]
Generalization error in this setting measures how well the hypothesis returned from the algorithm generalize from the source-domain training sample to the target-domain unknown distribution µ ′ .
Taking into account the stochastic nature of the algorithm A , a natural notion of generalization error for UDA can be defined by
[Equation]
where the expectation in the first expression is taken over the joint distribution of ( W, S ) ∼
P W | S × µ ⊗ n , and the expectation of the second expression is taken over the joint distribution of
[Equation]
There is another notion of generalization error, more traditional in the domain adaptation literature, defined as the gap between the population risk in the target domain and that in the source domain:
[Equation]
where R µ ( w ) ≜ E Z [ ℓ ( f w ( X ) , Y )] .
It is apparent that (cid:103) Err( w ) and Err are related by the following triangle inequality:
[Equation]
where the second term on the right hand side is the standard generalization error in the source domain, which can be bounded by classical learning-theoretic tools, e.g., Rademacher complexity (Bartlett & Mendelson, 2002).
Thus bounding (cid:103) Err( w ) helps bounding Err .
This paper studies both notions of generalization error for UDA.
Specifically, starting from Section 5, we will mainly use information-theoretic tools to bound Err directly, without going through (cid:103) Err( w ) .
For the ease of reference, we refer to (cid:103) Err( w ) as the population-to-population (PP) generalization error for w and Err as the expected empirical-to-population (EP) generalization error.
The following definitions are useful.
Definition 3.1 (Disintegrated Mutual Information) .
Let X , Y and Z be random variables and z be a realization of Z .
The disintegrated mutual information of X and Y given Z =
z is I z
( X ; Y ) ≜
D KL ( P X,Y | Z =
z || P X
| Z = z P Y | Z = z ) .
Note that the conditional mutual information I ( X ; Y | Z ) =
E Z I Z
( X ; Y ) .
Definition 3.2 (Lautum Information (Palomar & Verd´u, 2008)) .
The lautum information between X and Y is defined as L ( X ; Y ) ≜
D KL
P X P Y || P XY ) .
U PPER B OUNDS FOR PP G ENERALIZATION E RROR
We now present some upper bounds for (cid:103) Err( w ) .
The key techniques used in developing these bounds are the information-theoretic tools in the style of Lemma A.1.
These bounds adopt certain
KL divergence to measure the discrepancy between the source and target domains.
Notably, some previously established bounds are recovered under weaker conditions.
Additionally we demonstrate that under certain conditions, the KL-based bound is an upper bound of several other discrepancy measures and hence minimizing the KL divergence forces the minimization of these other measures.
We first list some common assumptions on the loss function, which we consider in this paper.
Assumption 1 (Boundedness) .
ℓ ( · , · ) is bounded in [0 , M ] .
Assumption 2 (Subgaussianity) .
ℓ ( f w ( X ) , Y ) is R -subgaussian 1 under µ for any w ∈ W .
Remark 4.1.
Note that Assumption 1 implies Assumption 2, i.e., if ℓ ( f w ( X ) , Y ) is bounded in [0 , M ] , then it is also M/ 2 -subgaussian.
Thus, Assumption 2 is weaker than Assumption 1.
Assumption 3 (Lipschitzness) .
ℓ ( f w ( X ) , Y ) is β -Lipschitz continuous in Z with respect to a metric d on Z for any w ∈ W , i.e., | ℓ ( f w ( x 1 ) , y 1 ) − ℓ ( f w ( x 2 ) , y 2 )
| ≤ βd ( z 1 , z 2 ) for some metric d on Z .
Remark 4.2.
Note that Assumption 1 implies Assumption 3 when d is a discrete metric, i.e., if ℓ ( f w ( X ) , Y ) is bounded in [0 , M ] , then it is also M -Lipschitz under the discrete metric.
Assumption 4 (Triangle and Symmetric) .
ℓ ( · , · ) satisfies the following: ℓ ( y 1 , y 2 ) = ℓ ( y 2 , y 1 ) and ℓ ( y 1 , y 2 ) ≤ ℓ ( y 1 , y 3 ) + ℓ ( y 3 , y 2 ) for any y 1 , y 2 , y 3 ∈ Y .
4. G ENERALIZATION B OUNDS VIA THE S UBGAUSSIAN C ONDITION
The following generalization bound is established by combining Lemma A.1 and Assumption 2, and the corresponding sample complexity bound is discussed in Appendix B.8.
Theorem 4.1.
If Assumption 2 holds, then for any w ∈ W , (cid:12)(cid:12)(cid:12)(cid:103) Err( w ) (cid:12)(cid:12)(cid:12) ≤ (cid:112) 2 R 2 D KL ( µ ′ || µ ) .
Notably this result can be turned into a generalization upper bound providing guidance to algorithm design, and at the same time it provides a lower bound of the generalization error, highlighting some fundamental difficulty of the learning task.
To illustrate this, we present a corollary while noting that similar development can also be applied to other bounds presented later in this paper.
Consider that each f w is expressed as the composition g ◦ h , where h is a function mapping X to a representation space T and g is a function mapping T to Y .
For any given h : X → T , denote by µ h the distribution on T × Y obtained by pushing forward µ via h , that is, µ h ( t, y ) = (cid:82) δ ( t − h ( x )) dµ ( x, y ) , where δ is the Dirac measure on T .
Similarly, let µ ′ h denote the distribution on T × Y obtained by pushing forward µ ′ via h .
Corollary 4.1.
Suppose that f w = g ◦ h and that Assumption 2 holds.
then for any w ∈ W ,
[Equation]
In this result, the lower bound of R µ ′ ( w ) indicates a fundamental difficulty in UDA learning in that, using the same predictor mapping f w , there is no way for the population risk in the target domain to be lower than that of the source domain less a constant which depends only on the domain difference.
On the other hand, the upper bound suggests that it is possible to squeeze the gap between the two population risks by choosing an appropriate representation map h - evidently such a map should be attempting to align µ ′ h with µ h or to align their respective proxies.
It is also noteworthy that under Assumption 1 and due to Remark 4.1, Theorem 4.1 implies
[Equation]
Similarly applying this result in the representation space T , we see that Eq.
(3) recovers the bound in Proposition 1 of Nguyen et al.
Notice that unlike Nguyen et al.
Theorem 4.1 ( or
Eq.
(3)) does not require the loss to be the cross entropy loss.
Theorem 4.1 and Nguyen et al.
both use the KL divergence from source domain to target domain, D KL ( µ ′ || µ ) , and in fact, (cid:12)(cid:12)(cid:12)(cid:103)
Err( w ) (cid:12)(cid:12)(cid:12) can also be upper bounded by D KL ( µ || µ ′ ) .
This can be done by invoking the subgaussianality of ℓ ( f w ( X ′ ) , Y ′ ) (rather than ℓ ( f w ( X ) , Y ) ); for bounded loss, the subgaussianality of ℓ ( f w ( X ′ ) , Y ′ ) is also satisfied.
Then we obtain the following corollary.
Corollary 4.2.
If Assumption 1 holds,
[Equation]
Remark 4.3.
In the second inequality of Corollary 4.2, D KL ( µ || µ ′ )
+ D KL ( µ ′ || µ ) is known as the symmetrized KL divergence, or Jeffrey’s divergence (Jeffreys, 1946), and Nguyen et al.
(2022) indeed penalizes this measure between the source and target distributions in the representation space.
Notice that bounds in Shui et al.
(2020) are based on the JS divergence.
Since there is a sharp upper bound of the JS divergence based on Jeffrey’s divergence (Crooks, 2008), minimizing
Jeffrey’s divergence (in the representation space) will simultaneously penalize the JS divergence.
In UDA, since Y ′ is completely unavailable to the algorithm A , it is impossible to minimize the misalignment of conditional distributions, i.e. D KL ( P Y ′ | T ′
|| P Y | T )
where T and T ′ are representations of source domain and target domain, respectively.
A common method is to assign pseudo labels to target data based on a learned source classifier (Liang et al., 2020).
However, it may also cause additional issues (Shen et al., 2022).
For concreteness, suppose the trained model Q can well approximate the real mapping between X and Y on source domain (i.e. Q Y | T =
P Y | T ), which is usually the training objective.
Let ˆ Y ′ be the pseudo label of T ′ generated by the trained model, i.e.,
Q ˆ Y ′ | T ′ =
Q Y | T .
Let Q T ′ , ˆ Y ′
=
P T ′ Q ˆ Y ′ | T ′ , then the following holds,
[Equation]
For a specific t ′ , if P ( Y ′ =
y ′ | T ′ =
t ′ )
̸ = 0 and P ( ˆ Y ′ =
y ′ | T ′ =
t ′ )
= 0 , then the second term in
RHS of Eq.
(4), D KL ( P Y ′ | T ′ ||
Q ˆ Y ′ | T ′ )
→ ∞ .
In this case, even when the marginal distributions are perfectly aligned, the overall value of the upper bound is large.
Thus, incorrect pseudo labels may even have negative impact on the target domain performance.
Indeed, the misalignment of the conditional distributions appears to be the main difficulty of UDA (Ben-David et al., 2006; Acuna et al., 2021).
The next corollary suggests that this difficulty may be alleviated when the loss function satisfies the triangle property, namely, Assumption 4.
It can be verified that this assumption is satisfied by the 0-1 loss 2 ; this assumption has also been considered in previous works (Mansour et al., 2009;
Shen et al., 2018).
Theorem 4.2.
If Assumption 4 holds and let ℓ ( f w ′ ( X ) , f w ( X )) be R -subgaussian for any w, w ′ ∈
W .
Then for any w , (cid:103) Err( w ) ≤ (cid:112) 2 R 2 D KL ( P X
′ ||
P X )+ λ ∗ , where λ ∗ =
min w ∈W R µ ′ ( w )+ R µ ( w ) .
Here λ ∗ measures the possibility of whether the domain adaptation algorithm will succeed under the oracle knowledge of µ and µ ′ .
In particular, if the hypothesis space is large enough, the minimizer w ∗ for the “joint population risk” R µ ′ ( w ) + R µ ( w ) may give rise to R µ ′ ( w ∗ )
=
R µ ( w ∗ )
= 0 .
then we’re likely to generalize well on the target domain.
Then the KL divergence D KL ( P X ′ ||
P X ) between the two X -marginals alone bounds the PP generalization error uniformly for all w ∈ W .
This theorem motivates the strategy of penalizing D KL ( P T ′ || P T ) in the representation space for
UDA.
The next theorem suggests that such an approach also penalizes other notions of domain discrepancy, for example, the key quantity in the PAC-Bayes type of domain adaptation generalization bounds (Germain et al., 2020), that is defined as
[Equation]
Theorem 4.3.
If ℓ ( f w ′ ( X ) , f w ( X )) is R -subgaussian for any f w , f ′ w ∈ F , then dis( P X , P X ′ ) ≤ (cid:112)
[Equation]
Note that unlike Germain et al.
, here we do not require the loss function to be the 0-1 loss.
4. G ENERALIZATION B OUNDS VIA THE L IPSCHITZ C ONDITION
We now present such generalization bound for UDA under the Lipschitz continuity assumption of the loss function, where W ( · , · ) denotes the Wasserstein distance.
Theorem 4.4.
If Assumption 3 holds, then (cid:12)(cid:12)(cid:12)(cid:103) Err( w ) (cid:12)(cid:12)(cid:12) ≤ β W ( µ ′ , µ ) .
Theorem 4.4 can be related to the KL-based bounds in the previous section when the Wasserstein distance is defined with respect to the discrete metric d .
In this case and under bounded loss function, which is also Liptschitz continuous, Theorem 4.4 follows.
On the other hand, Wasserstein distance is equivalent to the total variation distance (Ben-David et al., 2006), while the latter is connected to the
KL divergence via Pinsker’s inequality (Polyanskiy & Wu, 2019, Theorem 6.5) and the BretagnolleHuber inequality (Bretagnolle & Huber, 1979, Lemma 2.1).
Thus we arrive at the following result.
Corollary 4.3.
If Assumption 1 holds holds and let d be the discrete metric, then
[Equation]
Note that results here are inspired by the work of Rodr´ıguez G´alvez et al.
The bound in Corollary 4.3 can be immediately verified to be tighter than the bound in Eq.
Parallel to Theorem 4.2, if the loss function satisfies the triangle property, we may establish the bound below, which recovers a similar result in (Shen et al., 2018, Theorem 1.)
but without restricting the task to be binary classification or requiring the loss to be the L 1 distance.
Theorem 4.5.
If Assumption 4 holds and ℓ ( f w ( X ) , f w ′ ( X )) is β -Lipschitz in X for any w, w ′ ∈ W , then for any w ∈ W , ( cid:103) Err( w ) ≤ β W ( P X ′ , P X ) + λ ∗ , where λ ∗ =
min w ∈W R µ ′ ( w ) + R µ ( w ) .
These results justify the strategy of minimizing domain discrepancy in the representation space.
Since the KL-based bounds upper-bound those based on other measures of domain differences, penalizing the KL divergence will also penalize those other measures.
This is practically advantageous since it is usually easier and more stable to minimize the KL divergence (Nguyen et al., 2022).
U PPER B OUNDS FOR EP G ENERALIZATION E RROR AND A PPLICATIONS
There are two limitations in the bounds on the PP generalization error developed so far and in the traditional analysis of UDA.
First, such bounds are independent of w and hence algorithm-independent.
Second, although these bounds may inspire strategies to exploit the unlabelled target sample, e.g., aligning the source and target distributions in the representation space, they only provide very limited knowledge on the role that the unlabelled target sample. With the similar spirit of Negrea et al.
(2019) and Rodr ´ ıguez-G ´ alvez et al.
(2021), we now derive upper bounds for the EP generalization error, which better utilize the dependence of the algorithm output on the unlabelled target data.
Applications of these bounds in designing the learning algorithms are also presented.
5. EP G ENERALIZATION B OUNDS
Theorem 5.1.
Assume ℓ ( f w ( X ′ ) , Y ′ ) is R -subgaussian under µ ′ for any w ∈ W .
Then
[Equation]
Remark 5.1.
Note that the unlabelled target data plays a role in the first term of the bound.
Indeed, more source and target data will reduce the first term of the bound.
Specifically, moving the expectation inside the square root function by Jensen’s inequality and since Z i ⊥⊥ X ′ j , the equations
I ( W ; Z i | X ′ j ) =
I ( W ; Z i | X ′ j )
+ I
( Z i ; X ′ j ) =
I ( W ; Z i ) + I
( X ′ j ; Z i | W ) hold by the chain rule.
The term I ( W ; Z i ) will vanish as n → ∞ and the term I ( X ′ j ; Z i | W ) will also vanish as n, m → ∞ .
The theorem can be turned into a version that is more practically relevant, in which the KL term is replaced with their representation-space counter-part (following a similar argument used for deriving Corollary 4.1).
Although larger sample sizes allows better estimation of that KL term, but estimation utilizing pseudo-labels may have a negative impact (as discussed in Section 4), which can get amplified by the larger sample size.
Corollary 5.1.
Let Assumption 1 hold.
Then
[Equation]
Theorem 5.2.
Assume ℓ is Lipschitz for both w ∈ W and z ∈ Z , i.e., | ℓ ( f w ( x ) , y ) − ℓ ( f w ( x ′ ) , y ′ )
| ≤ βd 1 ( z, z ′ ) for all z, z ′ ∈ Z and | ℓ ( f w ( x ) , y )
− ℓ ( f w ′ ( x ) , y ) | ≤ β ′ d 2 ( w, w ′ ) for all w, w ′ ∈ W , then
[Equation]
This bound is tighter than the bound in Theorem 5.1, as can be indicated by the following corollary.
Corollary 5.2.
Let Assumption 1 hold.
Then
[Equation]
Notice that to recover Theorem 5.1 from Corollary 5.2, we can use Jensen’s inequality to move the expectation over Z i to inside the square root function.
5. G RADIENT P ENALTY AS AN U NIVERSAL R EGULARIZER
The algorithm-dependent bound in Theorem 5.1 tells us that one can reduce the expected generalization error by limiting the disintegrated mutual information I X
′ j
( W ; Z i ) .
In the stochastic gradient based optimization algorithms, this term can be controlled by penalizing the gradient.
To see this, we now consider a “noisy” iterative algorithm for updating W , e.g., SGLD.
At each time step t , let the labelled mini-batch from the source domain be Z B t , let the unlabelled mini-batch from the target domain be X ′ B t , and let g ( W t − 1 , Z B t , X ′ B t ) be the gradient at time t .
Thus, the updating rule of W is W t =
W t − 1 − η t g ( W t − 1 , Z B t , X ′ B t ) + N t where η t is the learning rate and N t ∼ N (0 , σ 2 I d ) is an isotropic Gaussian noise. Inspired by Pensia et al.
(2018), the next theorem is an application of Theorem 5.1 in this setting.
Theorem 5.3.
Let the total iteration number be T and let G t =
g ( W t − 1 , Z B t , X ′ B t ) , then
[Equation]
Remark 5.2.
Considering a noisy iterative algorithm here is merely for simplifying analysis.
In fact it is also possible to analyze the original iterative gradient optimization method without noise injected.
For example, one can follow the same development in (Neu et al., 2021;
Wang & Mao, 2022) to analyze vanilla SGD.
In that case, there will be some additional terms in the bound, which are related to flatness of the found minima.
Theorem 5.3 hints that to reduce the generalization error, one can simply restrict the gradient norm at each step (so that || G t − E Z Bt [ G t ] || 2 is reduced).
This strategy will also restrict the distance between the final output W T and the initialization W 0 , effectively shrinking the hypothesis space accessible by the algorithm.
We also note that the importance of gradient penalty is also theoretically justified in the supervised learning setting (Negrea et al., 2019; Haghifam et al., 2020; Rodr´ıguez-G´alvez et al., 2021; Neu et al., 2021; Wang & Mao, 2022; Smith et al., 2021).
Indeed, adding gradient penalty can be applied to any existing UDA algorithm
and it is simple but effective in practice.
Later on we will show that even when the algorithm A does not access to any target data, in which case I ( W ; Z i | X ′ j ) reduces to I ( W ; Z i ) and g ( W t − 1 , Z B t , X ′ B t ) becomes g ( W t − 1 , Z B t ) , minimizing the empirical loss of source domain sample while penalizing gradient norm will still improve the performance.
Notice that gradient penalty has been used in standard supervised learning as a regularization technique (Geiping et al., 2022; Jastrzebski et al., 2021).
It is also used in Wasserstein distance based adversarial adaptation (Gulrajani et al., 2017; Shen et al., 2018), and their motivation is to stabilize the training to avoid gradient vanishing problem.
Here we suggest, with strong theoretical justification, that gradient penalty is a universal technique for improving the generalization performance in UDA for any gradient-based learning method.
Notably the bound in Theorem 5.3 only depends on the size n of labelled source sample and does not explicitly depend on m , the size of unlabelled target sample.
With a more careful design, if we consider the mutual information as the expected KL divergence of a posterior and a prior, based on
I X ′
j
( W ; Z i ) in Theorem 5.1, it is possible to create a target-data-dependent prior and derive a tighter bound based on some quantity similar to ”gradient incoherence” in Negrea et al.
5. C ONTROLLING L ABEL I NFORMATION FOR KL G UIDED M ARGINAL A LIGNMENT
Consider instances in the representation space,
Z =
( T, Y ) and Z ′ =
( T ′ , Y ) .
Theorem 5.1 also encourages us to align the distributions of two domains in the representation space, as argued earlier.
Then the KL guided marginal alignment algorithm proposed in Nguyen et al.
(2022) can be invoked here.
One may notice that Theorem 5.1 uses D KL ( µ || µ ′ ) while Nguyen et al.
(2022) uses
D KL ( µ ′ || µ ) .
As already discussed in Section 4, this inconsistency can be ignored when loss is bounded (see Corollary 5.1).
Most domain adaptation algorithms aim to align the marginal distributions of two domains in the representation space.
However, without accessing to Y ′ , it remains unknown if an UDA algorithm will work well since we cannot guarantee that discrepancy between conditional distribution P Y | T and P Y ′
| T ′ won’t become too large when we align the marginals.
In Nguyen et al.
, the authors show that D KL ( P Y ′ | T ′
||
P Y | T ) can be upper-bounded by D KL ( P Y ′ |
X ′ ||
P Y | X ) , if I ( X ; Y ) =
I ( T ; Y ) .
The authors then argue that penalizing the KL divergence of the marginals is safe.
We now argue that in practice the condition I ( X ; Y ) =
I ( T ; Y ) can be difficult to satisfy if the
By data processing inequality on Y − X − T , we know that I ( X ; Y ) ≥ I ( T ; Y ) = H ( Y ) − H
( Y | T ) .
Thus, to let I ( T ; Y ) reach its maximum, one must minimize H ( Y | T ) .
On the other hand, let
Q Y | T,W be the predictive distribution of labels in the source domain generated by the classifier.
The expected cross-entropy loss for each Z i in the representation space is then
[Equation]
which also decomposes as (Achille & Soatto, 2018;
Harutyunyan et al., 2020)
[Equation]
Then minimizing the expected cross-entropy loss may not adequately reduce H ( Y i | T i ) but rather cause I ( W ; Y i | T i ) to significantly increase, particularly when the model capacity is large.
This may have two negative effects.
First, the condition I ( X ; Y )
=
I ( T ; Y ) is significantly violated, and D KL ( P Y ′ | T ′
||
P Y | T ) is no longer upper bounded by D KL ( P Y ′ | X ′
||
P Y | X ) .
Hence, aligning the two marginals alone may not be adequate.
Second, large I ( W ; Y i | T i ) indicates W just simply memorizes the label Y i , resulting a form of overfitting and hurting the generalization performance.
The key take-away from the above analysis is that when aligning the marginals in UDA, controlling the source label information in the weights can be important to achieve good cross-domain generalization.
A similar message can also be deduced from Theorem 5.1, when it is viewed in the repsentation space and noting I T ′
j ( W ; Z i )
=
I T ′ j ( W ; T i )
+ I T ′
j ( W ; Y i | T i ) .
To control label information, Harutyunyan et al.
(2020) proposed an approach called LIMIT.
However this method is rather complicated and arguably hard to train in domain adaptation (see Appendix).
We now derive a simple alternative strategy for this purpose.
Notice that I T ′
j ( W ; Y i | T i ) ≤ inf Q E T i (cid:104) D KL ( P W | Y
i ,T i ,T ′ j = t ′ j || Q W | T i ,T ′
j = t ′ j ) (cid:105) , which is a simple extension of variational representation of mutual information (Polyanskiy & Wu, 2019, Corollary 3.1.).
Here Q could be any distribution.
By assuming P = N ( W, σ 2 I d | Y i , T i , T ′ j = t ′ j ) and taking Q = N ( (cid:102) W, ˜ σ 2 I d | T i , T ′ j = t ′ j ) , we have
[Equation]
Thus, we may create an auxiliary classifier f (cid:101) w that is not allowed to access to the real source label
Y .
In each iteration, we use the pseudo labels of target data (and source data) assigned by f w to train f (cid:101) w and adding || W − (cid:102) W || 2 as a regularizer in the training of W .
The algorithm is given in the Appendix.
Remarkably the regularizer here resembles “Projection Norm” designed in Yu et al.
[Table]
E XPERIMENTAL R ESULTS
We perform experiments to verify the proposed techniques inspired by our theory.
Datasets We select two popular small datasets, RotatedMNIST and Digits, to compare the different methods.
RotatedMNIST is built based on the MNIST dataset (LeCun et al., 2010) and consists of six domains each containing 11 , 666 images.
These six domains are rotated MNIST images with rotation angle 0 ◦ , 15 ◦ , 30 ◦ , 45 ◦ , 60 ◦ and 75 ◦ , respectively.
We will take the original MNIST dataset ( 0 ◦ ) as the source domain and take other five domains as target domains.
Hence there are five domain adaptation tasks on RotatedMNIST.
Digits consists of three sub-datasets, namely MNIST,
USPS (Hull, 1994) and SVHN (Netzer et al., 2011), and the corresponding domain adaptation tasks are MNIST → USPS ( M → U ), USPS → MNIST ( U → M ), SVHN → MNIST ( S → M ).
Compared Methods Baseline methods are some popular marginal alignment UDA methods including DANN (Ganin et al., 2016), MMD (Li et al., 2018), CORAL (Sun & Saenko, 2016), WD (Shen et al., 2018) and KL (Nguyen et al., 2022).
We also choose ERM as another baseline, in which only the source-domain sample is accessible during training.
To verify the strategies inspired by our theory, we first add the gradient penalty to the ERM algorithm
( ERM-GP ), and we then combine gradient penalty (GP) and controlling label information (CL) with the recent proposed KL guided marginal alignment method, which are denoted by KL-GP and KL-CL , respectively.
Implementation Details Most of our implementation is based on the DomainBed suite (Gulrajani & Lopez-Paz, 2021).
Other settings exactly follow Nguyen et al.
and the results of baseline methods are taken from Nguyen et al.
Specifically, each algorithm is run three times and we show the average performance with the error bar.
Every dataset has a validation set, and the model selection scheme is based on the best performance achieved on the validation set of target domain during training (oracle).
The hype-parameter searching process is also built upon the implementation in the DomainBed suite.
Other details and additional experiments can be found in Appendix.
Results From Table 1, we first notice that gradient penalty allows ERM to perform more comparably to other marginal alignment methods.
For example, on RotatedMNIST, ERM-GP outperforms
CORAL and performs nearly the same with DANN .
On Digits, ERM-GP outperforms WD .
When
GP and CL combined with KL guided algorithm, we can see that the performance can be further boosted.
This justifies the discussion in Section 5.2 and Section 5.3.
C ONCLUSION
Despite that the numerous learning techniques have been developed for domain adaptation, significant room exists for more in-depth theoretical understanding and more principled design of learning algorithms.
This paper presents the information-theoretic analysis for unsupervised domain adaptation, where we query two notions of the generalization errors in this context and present novel learning bounds.
Some of these bounds recover the previous KL-based bounds under different conditions and confirm the insights in the learning algorithms that align the source and target distributions in the representation space.
Our other bounds are algorithm-dependent, better exploiting the unlabelled target data, which have inspired novel and yet simple schemes for the design of learning algorithms.
We demonstrate the effectiveness of these schemes on standard benchmark datasets.
Appendix Table of Contents
A Some Prerequisite Definitions and Useful Lemmas
B Omitted Proofs and Additional Results in Section
B.1 Proof of Theorem 4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.2 Proof of Corollary 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.3 Proof of Theorem 4.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.4 Proof of Theorem 4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.5 Proof of Theorem 4.4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.6 Proof of Corollary 4.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.7 Proof of Theorem 4.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
B.8 Additional Results: Sample Complexity Bounds . . . . . . . . . . . . . . . . .
B.9 Additional Discussions on the Convergence of Empirical KL Divergence . . . .
B.10 Generalize to Approximate Triangle Inequality . . . . . . . . . . . . . . . . . .
C Omitted Proofs and Additional Discussions in Section
C.1 Additional Discussion on Theorem 5.1 . . . . . . . . . . . . . . . . . . . . . .
C.2 Proof of Theorem 5.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.
of Corollary
C.4 Proof of
Theorem 5.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.
5 Proof
Corollary 5.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.
Proof of Theorem 5.3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.
7 Derivation of Eq
C.
8 Additional Discussion on LIMIT
D Experiment Details
D
Controlling
D
Algorithm
Controlling Label Information
Additional
D.
Architectures and Hyperparameters
D.
Additional Experimental Results
D
Ablation Study on the Effect of Gradient Penalty Hyperparameter
D.
Visualization
D.
on VisDA
D.8 Dynamics of Jeffrey’s divergence
A S OME P REREQUISITE D EFINITIONS AND U SEFUL L EMMAS
Definition A.1 (Wasserstein Distance) .
Let d ( · , · ) be a metric and let P and Q be probability measures on X .
Denote Γ( P, Q ) as the set of all couplings of P and Q (i.e. the set of all joint distributions on X × X with two marginals being P and Q ), then the Wasserstein Distance of order one between P and Q is defined as W ( P, Q ) ≜ inf γ ∈ Γ( P,Q ) (cid:
) X×X
d ( x, x ′ ) dγ ( x, x ′ ) .
Remark A.1.
Similar to Rodr´ıguez G´alvez et al.
(2021), here we mainly focus on 1 -Wasserstein distance but all the upper bounds based on 1 -Wasserstein distance also holds for higher order Wasserstein distance by H¨older’s inequality.
Definition A.2 (Total Variation) .
The total variation between two probability measures P and Q is
TV( P, Q ) ≜ sup E | P ( E ) − Q ( E ) | , where the supremum is over all measurable set E . distortion)
d ( x, x ′ )
=
1 ( x ̸ = x ′ ) where 1 is the indicator function.
Definition A.3 (Lautum Information (Palomar & Verd´u, 2008)) .
Define the lautum information between X and Y as L ( X ; Y ) ≜
D KL
P X P Y || P XY ) .
The key quantity in most information-theoretic generalization bounds is the mutual information between algorithm’s input and output.
Specifically, the core technique behind these bounds is the well-known Donsker-Varadhan representation of KL divergence (Polyanskiy & Wu, 2019, Theorem 3.5).
Lemma A.1 (Donsker and Varadhan’s variational formula) .
Let Q , P be probability measures on Θ , for any bounded measurable function f : Θ → R , we have D KL ( Q || P ) =
sup f E θ ∼ Q [ f ( θ )] − log E θ ∼ P [exp f ( θ )] .
Remark A.2.
Motivated by the classic f -divergence, Acuna et al.
(2021) proposed a discrepancy measure called D H ϕ -discrepancy.
Since KL divergence belongs to the family of f -divergences (e.g., choosing x log x as the Fenchel conjugate function) and both Acuna et al.
and our work invoke the variational representation of the divergence, it seems our work (in Section 4) is related to theirs.
However, the variational characterization of f -divergence used in Acuna et al.
(2021) is based on the result of Nguyen et al.
(2010), and the Donsker-Varadhan representation of KL divergence (see Lemma A.1)
used in this paper cannot be directly recovered from their variational characterization (Jiao et al., 2017; Agrawal & Horel, 2020).
Indeed, simply choosing x log x as the conjugate function will lead to a weaker bound than Lemma A.1.
Thus, our results (in Section 4) cannot be directly recovered from the results in Acuna et al.
Similar to Xu & Raginsky (2017, Lemma 1.), we need the following lemma as a main tool.
Lemma A.2.
Let Q and P be probability measures on Θ .
Let θ ′ ∼ Q and θ ∼ P .
If g ( θ ) is
R -subgaussian, then,
[Equation]
Proof.
Let f = t · g for any t ∈ R , by Lemma A.1, we have
[Equation]
where the last inequality is by the subgaussianity of g ( θ ) .
Then consider the case of t > 0 and t < 0 ( t = 0 is trivial), by AM–GM inequality (i.e. the arithmetic mean is greater than or equal to the geometric mean), the following is straightforward,
[Equation]
This completes the proof.
The following lemma is the Kantorovich–Rubinstein duality of Wasserstein distance (C´edric, 2008).
Lemma A.3 (KR duality) .
For any two distributions P and Q , we have
[Equation]
where the supremum is taken over all 1 -Lipschitz functions in the metric d , i.e. | f ( x ) − f ( x ′ ) |
≤ d ( x, x ′ ) for any x, x ′ ∈ X .
To connect total variation with KL divergence , we will use Pinsker’s inequality (Polyanskiy & Wu, 2019, Theorem 6.5) and Bretagnolle-Huber inequality (Bretagnolle & Huber, 1979, Lemma 2.1) in this paper, for more discussion about these two inequalities, we refer readers to Canonne (2022). S ′ X ′ ↓ S → W ↓ ↙
F Lemma A.4 (Pinsker’s inequality) .
TV( P, Q ) ≤ (cid:113) 12 D KL ( P || Q ) .
Lemma A.5 (Bretagnolle-Huber inequality) .
TV( P, Q ) ≤ √ 1 − e − D KL ( P || Q ) .
Below is the variational formula, or golden formula of mutual information.
Lemma A.6 (Polyanskiy & Wu (2019, Corollary 3.1.)) .
For two random variables X and Y , we have
[Equation]
where the infimum is achieved at P = Q Y .
B O MITTED P ROOFS AND A DDITIONAL R ESULTS IN S ECTIONB. P ROOF OF T HEOREM 4.
Proof.
Let Q = µ ′ , P = µ and g = ℓ , then Theorem 4.1 comes directly from Lemma A.2.
B. P ROOF OF C OROLLARY 4.
Proof.
When the loss is bounded in [0 , M ] , similar to the proof of Theorem 4.1, let Q = µ , P = µ ′ and g = ℓ , then the following bound also holds by Lemma A.2,
[Equation]
Then, recall Theorem 4.1 and by min { A, B } ≤ 12 ( A + B ) , the remaining part is straightforward,
[Equation]
(cid:112) min { D KL ( µ || µ ′ ) , D KL ( µ ′ || µ ) } ≤ M 2 (cid:112) D KL ( µ || µ ′ )
+ D KL ( µ ′ || µ ) .
This completes the proof.
B. P ROOF OF T HEOREM 4.
Proof.
Let w ∗ = arg min w ∈W E Z ′
[ ℓ ( f w ( X ′ ) , Y ′ )]
+ E Z [ ℓ ( f w ( X ) , Y )] .
By Lemma A.1,
D KL ( P X
||
P X ) ≥ sup t ∈ R ,w ∈W E X ′
[Equation]
∗ [Equation] [Equation] Recall that ℓ ( f w ′ ( X ) , f w ( X )) is R -subgaussian, by using Lemma A.2 (let Q =
P X ′ , P = P X and g ( · ) = ℓ ( f w ′ ( · ) , f w ( · )) ), we have | E X ′ [ ℓ ( f
[Equation]
( X ′ )
[Equation]
f w ∗
[Equation]
X ′ ))]
[Equation]
X [
[Equation] [Equation]
( X
[Equation]
] |
[Equation]
(cid:112)
[Equation]
( P X ′ || P X ) . For any f w ∈ F , by the triangle property of the loss, we have
[Equation]
where Eq.
(8) is by Eq.
(7) and Eq.
(9) is again by the triangle property of the loss function. Thus,
[Equation] Err( w ) ≤ [Equation] 2 R 2 D KL (P X
|| P X ) + λ ∗ , which completes the proof.
B. P ROOF OF T HEOREM 4.
Proof.
By Lemma A.1,
[Equation]
where the last inequality is by applying Jensen’s inequality to the concavity of logarithm function.
By Lemma A.2,
[Equation]
This concludes the proof
[Equation]
B. P ROOF OF T HEOREM 4.
Proof.
From the definition [Equation] we have
[Equation]
where the last inequality is by the KR duality of Wasserstein distance (see Lemma A.3).
B. P ROOF OF C OROLLARY 4.
Proof.
When d is the discrete metric, Wasserstein distance is equal to the total variation, then by
Theorem 4.4,
[Equation]
The remaining part is by using Lemma A.4 and Lemma A.5:
[Equation]
Then, if ℓ is bounded by M , we replace β by M above, which completes the proof.
B. P ROOF OF T HEOREM 4.
Proof.
Let w ∗ =
arg min w ∈W E Z ′
[ ℓ ( f w ( X ′ ) , Y ′ )]
+ E Z [ ℓ ( f w ( X ) , Y )] .
If ℓ ( f w ( X ) , f w ′ ( X )) is β -Lipschitz in X for any w, w ′ ∈ W , then similar to Theorem 4.4, it’s easy to show that
[Equation]
For any f w ∈ F , by the triangle property of the loss, we have
[Equation]
where Eq.
(11) is by Eq.
(10) and the last inequality is again by the triangle property of the loss function.
This completes the proof.
B. A DDITIONAL R ESULTS : S AMPLE C OMPLEXITY B OUNDS
One of the main ingredients to derive our sample complexity bound is the following lemma, where a concentration bound for a class of unbounded functions is given.
Lemma B.1 (Cortes et al. (2019, Corollary 9)) . Let κ > 2 and G = { g : Z → R s.t. E µ (cid:2)
e g ( Z ) (cid:3) <
+ ∞} .
Assume E µ [ g ( Z ) κ ] < + ∞ for all g ∈ G .
Let ˆ µ be the empirical distributions consist of n data points sampled i.i.d.
from µ .
If G has the finite pseudo-dimension d .
Then for ∀ δ ∈ (0 , 1) , the following inequality holds with probability at least 1 − δ ,
[Equation]
where Λ
[Equation]
=
[Equation]
Below is another useful lemma for the bounded case, which comes from Mohri et al.
(2018, Theorem 11.8) with a slight modification.
Lemma B.2.
Let F = { f : Z → R + } .
Assume E µ [ f ( Z )] < M for all f ∈ F and for some constant M > 0 .
Let ˆ µ be the empirical distributions consist of n data points sampled i.i.d. from µ . If F has the finite pseudo-dimension d .
Then for ∀ δ ∈ (0 , 1) , the following inequality holds with probability at least 1 − δ ,
[Equation]
We are now in a position to state our sample complexity bound.
Theorem B.1.
Let ˆ µ and ˆ µ ′ be the empirical distributions consist of n source data and m target data sampled i.i.d.
from µ and µ ′ , respectively.
Let G =
{ g : Z → R s.t.
E µ (cid:2)
e g ( Z ) (cid:3) < ∞} with finite pseudo-dimension d 1 , and let the pseudo-dimension of { exp ◦ g | g ∈ G} be d 2 .
Let κ > 2 and assume that E µ [ g ( Z ) κ ] < + ∞ for all g ∈ G . Assume there exists a constant α ≤ min g ∈G { E ˆ µ (cid:2)
e g ( Z ) (cid:3) , E µ (cid:2) e g ( Z ) (cid:3) } .
Then for ∀ δ ∈ (0 , 1) the following holds with probability at least 1 − δ ,
[Equation]
where C 1 ( κ ) = (cid:0) 1 2 (cid:1) 2 − κκ (cid:16) κ κ − 2 (cid:17) κ − 1 κ sup g ∈G κ (cid:112) E µ [ g ( Z ) κ ] and C 2 ( α )
{+= 
+}
α sup g ∈G E µ (cid:2) e g
( Z ) (cid:3) .
Proof.
Recall Lemma A [Equation] 1, we have
[Equation]
and
[Equation]
Then, with the probability at least 1 − δ ,
[Equation]
where Eq.
(12) is derived below.
W.L.O.G. assume that E ˆ µ (cid:2) e g ( Z ) (cid:3) ≤ E µ (cid:2) e g ( Z ) (cid:3) (and Eq.
(12) still holds when E ˆ µ (cid:2) e g ( Z ) (cid:3) ≥ E µ (cid:2) e g ( Z ) (cid:3) ), then
[Equation]
log
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Eq.
(13) is by Lemma B [Equation] 1 and Lemma B [Equation]
This concludes the proof.
With Theorem B.1 and Theorem 4. [Equation] we immediately have the following corollary.
Corollary B.1.
Let the conditions in Theorem B.1 and Theorem 4.1 hold, then for any w ∈ W ,
[Equation]
Err
[Equation]
where C 1 ( κ ) and C 2 ( α ) are the same as in Theorem B.1.
B. A DDITIONAL D ISCUSSIONS ON THE C ONVERGENCE OF E MPIRICAL KL D IVERGENCE
Although characterizing the convergence of the empirical KL divergence to the real KL is not easy without relying several additional assumptions (as in Theorem B.1), the result of convergence rate of empirical distribution to the real distribution in the KL sense is already known in the discrete space.
The following theorem comes directly from the classic result in (Cover & Thomas, 2006,
Theorem 11.2.1),
Theorem B.2.
Let ˆ µ and ˆ µ ′ be defined as in Theorem B.1.
Assume the space of Z is finite (i.e.
|Z| ≤ ∞ ), then for ∀ δ ∈ (0 , 1) , with the probability at least 1 − δ ,
[Equation]
Thus, it suffices to ensure that the empirical KL converge to the real KL with the similar rate, although we do not know if there might exist a faster convergence rate. B. G ENERALIZE TO A PPROXIMATE T RIANGLE I NEQUALITY
In Section 4, some results require that the loss obeys the triangle inequality (i.e. Assumption 4), such as Theorem 4.2 and Theorem 4.5.
While 0 / 1 -loss satisfies Assumption 4, some other loss may not.
Thus, to generalize Theorem 4.2 and Theorem 4.5, we invoke an approximate triangle inequality, which is initially defined in Crammer et al.
Assumption 5 ( α -Triangle) . ℓ ( · , · ) is symmetric and satisfies the following the α -triangle inequality: ℓ ( y 1 , y 2 ) ≤ α ( ℓ ( y 1 , y 3 ) + ℓ ( y 3 , y 2 )) for any y 1 , y 2 , y 3 ∈ Y ., where α ≥ 1 is a constant that may depend on the hypothesis space W and the loss ℓ . Remark B.1.
We note that squared loss satisfied 2 -triangle inequality.
Thus, Theorem 4.2 can be easily generalized below.
Theorem B.3.
If Assumption 5 holds and let ℓ ( f w ′ ( X ) , f w ( X )) be R -subgaussian for any w, w ′ ∈
W .
Then for any w ,
[Equation]
2 λ ∗ , where λ ∗ =
min w ∈W R µ ′ ( w ) + R µ ( w ) .
Theorem 4.5 can be generalized in the similar way.
Although strictly speaking, Theorem B.3 is not a generalization bound, it has the same spirit with Theorem 4.2, that is, to make the target domain population risk small, we hope source domain and target domain are close to each other, and both
R µ and λ ∗ are small.
C O MITTED P ROOFS AND A DDITIONAL D ISCUSSIONS IN S ECTIONC. A DDITIONAL D ISCUSSION ON T HEOREM 5.
To derive the bound in Theorem 5.1, we need to make use of the second equality in Eq.
Indeed, by the definition of Err (the first equality in Eq.
(1)), the unlabelled sample S ′ X
j does not explicitly appear, so one can easily apply the similar information-theoretic analysis starting from the first equality in Eq.
(1), and obtain an upper bound that consists of I ( W ; Z i ) and D KL ( µ || µ ′ ) .
Precisely, the following bound holds,
Theorem C.1.
Assume ℓ ( f w ( X ′ ) , Y ′ ) is R -subgaussian for any w ∈ W .
Then
[Equation]
The proof of Theorem C.1 is nearly the same to the proof of (Wu et al., 2020, Corollary 2) and (Masiha et al., 2021, Corollary 1).
It’s important to note that although
[Equation]
the bound in Theorem 5.1 is incomparable to the bound based on I ( W ; Z i ) .
This is mainly due to the fact that we use the disintegrated version of mutual information, I X
′ j
( W ; Z i ) , and the expectation over X ′ j is outside of the square root, which is a convex function.
Using I X
′ j
( W ; Z i ) instead of
I ( W ; Z i ) allows us to figure out more details about the role of unlabelled target data in the algorithm.
Additionally, one can also prove a bound based on I ( W ; Z i | X ′ j ) (e.g., simply applying Jensen’s inequality to Theorem 5.1), which is close to an individual and UDA version of (Bu et al., 2022,
Theorem 3).
In essence, the first term in Theorem 5.1 characterize the expected generalization gap on the source domain (i.e. E W,S [ R µ ( W )
− R S
( W )] ), then the bound suggests us that it’s possible to invoke the unlabelled target data to further improve the performance on source domain, and the simplest case is the semi-supervised learning (when µ
= µ ′ ).
Compared with (Wu et al., 2020; Jose & Simeone, 2021b).
To conclude, bounds in (Wu et al., 2020; Jose & Simeone, 2021b) fail to characterize the dependence between W and S ′ X ′ .
More precisely, the algorithm-dependent term in their bounds is I ( W ; Z i ) or I ( W ; S ) , while our algorithmdependent term is I X
′ j
( W ; Z i ) that directly depends on the unlabelled target data.
More Discussion on the Vanishing of I ( X ′ j ; Z i | W ) in Remark 5.1.
Note that S depends on S ′ X
′ given W , so intuitively the dependence between each individual instance Z i and X
j is weaker when n and m become larger.
More precisely, W.L.O.G let i = j = 1 , and recall that W = A ( S ′ X ′ , S ) , when n, m → ∞ , taking S and S ′ X ′
as the input of the algorithm is nearly equivalent to computing
W based on the source distribution µ and the target distribution P X ′ , thus, W will only depend on the two distributions, without depending on the realizations Z 1 and X ′ 1 drawn respectively from the two distributions, that is, I ( Z 1 ; X ′ 1 | W )
=
I ( Z 1 ; X ′ 1 |A ( µ, P X ′ ))
=
I ( Z 1 ; X ′ 1 ) = 0 .
In addition, one may argue that what if W = constant that does not really depend on the input data.
In this case, I ( Z 1 ; X ′ 1 | W ) =
I ( Z 1 ; X ′ 1 ) = 0 will hold trivially.
In the other extreme, if n = 1 and m = 1 , then W = A ( X ′ 1 , Z 1 ) , and the quantity
I ( Z 1 ; X ′ 1 |A
( X ′ 1 , Z 1 )) should be large.
When n and m increase, it becomes I ( Z 1 ; X ′ 1 |A
( X ′ 1: m , Z 1: n )) .
Now we want to guess Z 1 from X ′ 1 , this should be easier when having the knowledge of A ( X ′ 1 ; Z 1 ) compared with when having the knowledge of
A ( X ′ 1: m ; Z 1: n ) .
C. P ROOF OF T HEOREM 5.
Proof.
By Lemma A.1,
[Equation] [Equation]
where Eq.
(14) is by the independence between algorithm output W and unseen target domain data
Z ′ , Eq.
(15) is by Jensen’s inequality for the exponential function and the last inequality is by the subgaussian assumption [Equation]
Thus
[Equation]
[Table]
[Equation]
where the last two inequalities are by the Jensen’s inequality for the absolute function.
Since
[Equation]
Recall Eq.
(16), we have
[Equation]
This completes the proof
[Equation]
C. P ROOF OF C OROLLARY 5.
Proof.
We now modify the proof in Theorem 5.1.
Recall that
[Equation] [Equation]
where the last inequality is by Corollary 4.
[Equation]
Then for the first term in RHS, notice that
[Equation]
where the last inequality is due to the fact that ℓ is bounded by M and ℓ ( f W ( X i ) , Y i ) is M/ 2subgaussian
[Equation]
Thus
[Equation]
Plugging this inequality with the decomposition into the inequality at the beginning of the proof, we have
[Equation]
Similar development also holds for D KL (cid:16) P W,
Z i | X
[Equation]
j = x ′ j || P W | X ′
j = x ′ j P Z (cid:17) , thus
[Equation]
This completes the proof
[Equation]
C. P ROOF OF T HEOREM 5.
Proof.
Similar to the proof of Theorem 5.1, we exploit the fact that | Err | [Equation] [Table] [Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
which concludes the proof
[Table]
C. P ROOF OF C OROLLARY 5.
Proof.
Similar to the proof of Corollary 4.3, replacing Wasserstein distance by the total variation and replacing β and β ′ by M
[Equation]
will give us the first inequality,
[Equation]
The second inequality is by Lemma A.4,
[Equation]
Again
[Equation]
one can also apply Lemma A
[Equation]
5 here
[Equation]
This concludes the proof.
C. P ROOF OF T HEOREM 5.
Proof.
Recall Theorem
[Equation]
1 and by Jensen’s inequality we have
[Equation]
First notice that,
[Table]
[Equation]
Then, since S ⊥⊥ S ′ X ′ and Z i ⊥⊥ Z 1: i − 1 for any i ∈ [ n ] , by chain rule of the mutual information, we have
[Equation]
Thus
[Equation]
the generalization error bound becomes,
[Equation]
Recall the updating rule of W and notice that W 0 is independent of S and S ′ X ′ , the following process is by using chain rule of mutual information and data processing inequality recurrently,
[Equation]
For each t ∈ [ T ] , denote g ( W t − 1 , Z B t , X ′ B t ) as G t , then
[Equation]
where the inequality is by Lemma A.6 and the last equality is by the KL divergence between two
Gaussian distributions.
Finally, putting everything together,
[Equation]
which concludes the proof.
C. D ERIVATION OF E Q . (6)
Recall the expected cross-entropy loss, we have
[Equation]
C. A DDITIONAL D ISCUSSION ON LIMIT
As mentioned in Section
[Equation]
Harutyunyan et al.
(2020) proposed an approach called LIMIT, refers to limiting label information memorization in training, to control label information.
Roughly speaking, to update the parameters of the classifier, they construct an auxiliary network to predict the gradient instead of using the real gradient
[Equation]
in which case the true label is not directly used for training the classifier.
To provide accurate gradients, they also need to train the auxiliary network by using the true labels.
We find that the training of LIMIT is unstable and hard to tune the hyperparameters under the UDA setting.
Thus, we choose to use the pseudo label strategy proposed in Sectioninstead of pseudo gradient strategy.
D E XPERIMENT D ETAILS
The implementation in this paper is on PyTorch (Paszke et al., 2019), and all the experiments are carried out on NVIDIA Tesla V100 GPUs (32 GB).
D. O BJECTIVE F UNCTIONS OF G RADIENT P ENALTY AND C ONTROLLING L ABEL I NFORMATION
For every iteration, the objective function after adding the gradient penalty becomes
[Equation]
where ˆ L ( W, Z B t , X ′ B t ) is some loss function for the source and target domain data in the current mini-batch and λ 1 is the trade-off coefficient.
For example, if we combine ERM with gradient penalty then ˆ L ( W, Z B t , X ′ B t ) = 1 | B t | (cid:80) k ∈ B t ℓ ( f W ( X k ) , Y k ) and ℓ could be the cross-entropy loss.
Moreover, if we combine KL guided marginal alignment algorithm
(Nguyen et al., 2022) with gradient penalty then the objective function is min
[Equation]
,θ
[Equation]
where θ is the parameters of the representation network and the gradient is
[Equation]
In Nguyen et al.
(2022), the representation distribution is modelled as an Gaussian distribution, i.e.,
T ∼ N ( µ θ , σ 2 θ I d | X ) and T ′ ∼ N ( µ θ , σ 2 θ I d | X ′ ) .
Additionally, let the batch size be b = | B t | , the
[Table]
empirical KL divergence is estimated by the mini-batch data, as given in Nguyen et al. (2022), β 1 D KL [Equation] P
[Equation]
|| [Equation] + [Equation]
P
[Equation]
|
[Equation] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] 1 b (cid:88) k
[Equation]
B t (cid:34) logb (cid:88) k
[Equation]
t
P
[Equation]
log
[Equation]
(cid:88) k ∈ B t
P T ′ k
| X ′ k (cid:35) , where P T k | X k = N ( µ θ , σ 2 θ I d | X k ) and P T ′ k
| X ′ k
= N ( µ θ , σ 2 θ I d | X ′ k ) .
To be more precise, µ θ and σ θ are the outputs of the representation network.
Since the forward pass requires the sampling of T and
T ′ , we need to use the reparameterization trick (Kingma & Welling, 2013) for the backward pass.
When we train the model with controlling label information, the objective function becomes
[Equation]
where (cid:102) W is the auxiliary classifier and λ 2 is the trade-off hyperparameter.
Similarly, when we combine KL guided marginal alignment algorithm with controlling label information, then the objective function in every iteration is min W,θ
[Equation]
In addition, the training objective for the auxiliary classifier is
[Equation]
In practice, removing the second term would not affect the performance.
Note that we need to disenable the automatic differentiation of T , T ′ and W when executing the backward pass for the auxiliary classifier.
The detailed algorithm of controlling label information is given in the next section.
D. A LGORITHM OF C ONTROLLING L ABEL I NFORMATION AND A DDITIONAL R ESULTS OF ERM-CL
If we only provide the pseudo labels for the target domain data to the auxiliary classifier, i.e. removing the second term in Eq (17), the Algorithm 1 is the algorithm for combining any marginal alignment algorithm with controlling label information.
Even without incorporating with the marginal alignment algorithm, e.g., ERM, in which case L r is removed, Algorithm 1 still boosts the performance in practice.
Table 2 shows that ERM-CL can overall outperform the basic ERM and is close to the performance of ERM-GP .
D. A RCHITECTURES AND H YPERPARAMETERS
The network architecture in this work is the same as in Gulrajani & Lopez-Paz (2021) and Nguyen et al.
(2022), where a simple CNN is used.
Require: Source domain labelled dataset S , Target domain unlabelled dataset S ′ X ′ , Batch size b ,
Classification loss function ℓ c , Marginal alignment loss function ℓ r , Initial classifier parameter w 0 = (cid:101) w 0 , Initial representation network parameter θ 0 , Learning rate η , Lagrange multiplier λwhile w t not converged do 2:
Update iteration: t ← t +
Sample Z B = { z i } bi =1 from source domain training set S 4:
Sample X ′B = { x ′ i } bi =1 from target domain training set S ′ X ′
Compute distance from the auxiliary classifier dis ← || w t − (cid:101) w t ||6:
Compute marginal alignment loss L r ← 1 b (cid:80) bi =1 ℓ r ( θ t , z i , x ′ i )
Compute classification loss L c ← 1 b (cid:80) bi =1 ℓ c ( w t , θ t , z i , x ′ i ) 8:
Compute gradient: g B ← ∇ ( L c + L r + λ 2 dis )
Update parameter: w t +1 ← w t − η
g B , θ t +1 ← θ t − η · g B 10:
Obtain the pseudo labels Y ′B ← f w t ( g θ t ( X ′B ))
Compute auxiliary classification loss L a ← 1 b (cid:80) b
i =1 ℓ c ( (cid:101) w t , θ t , x ′ i , y ′ i ) 12:
Compute auxiliary classifier gradient: (cid:101) g B ← ∇ L a Update auxiliary classifier parameter: (cid:101) w t +1 ← (cid:101) w t − η · (cid:101) g B 14: end while [Table] 93.6±1. 93.8±1.
93.1±1. Other settings are also the same as Gulrajani & Lopez-Paz (2021) and Nguyen et al.
(2022), for example, each algorithm is trained for 100 epochs.
To select the hyperparameters ( λ 1 and λ 2 ) for
ERM-GP , ERM-KL , KL-GP and KL-CL , we perform random search.
Specifically, λ 1 is searched between [0 . 1 , 0 . 9] and λ 2 is searched between [10 − 6 , 0 . 8] .
Other hyperparameters searching range could be found in the source code.
D. A DDITIONAL E XPERIMENTAL R ESULTS D. A BLATION S TUDY ON THE E FFECT OF G RADIENT P ENALTY H YPERPARAMETER
We provide an ablation study on the value of λ 1 when using KL-GP . In the RotatedMNIST experiment, we take 0 ◦ as the source domain and take 60 ◦ as the target domain. In the Digits experiment, we take SVHN as the source and take MNIST as the target.
The results are shown in Table 3. Notice that when λ 1 = 0 , KL-GP becomes KL , and the results again justify the effeteness of adding gradient penalty term. D. V ISUALIZATION R ESULTS We visualize the representations of models trained with KL , KL-GP and KL-CL by using t-SNE (Van der Maaten & Hinton, 2008), and the visualization results are shown in Figure 2.
Here we take SVHN as the source domain and take MNIST as the target domain.
We notice that with the help of additional regularizers, the representation alignment could be slightly improved.
It is still important to note that the regularization terms are mainly designed for improving the performance of the classifier network instead of the representation network.
8060202060sourcetarget (a) Visualization of KL 80 60 40 2020 40 60 8060202060sourcetarget (b) Visualization of KL-GP       8060202060sourcetarget
(c) Visualization of KL-CL
[Table]
D. R ESULTS ON V IS DA
We also conduct experiments on the VisDA17 dataset (Peng et al., 2017), which is a real-world classification task with 280 K images from 12 classes.
Particularly, the source domain contains synthetic images and the target domain contains real images.
The results are given in Table 4.
Notice that our regularization techniques (i.e. KL-GP and KL-CL ) are still able to boost the performance of the KL guided marginal alignment algorithm.
[Figure] D. D YNAMICS OF J EFFREY ’ S DIVERGENCE The representation version of Corollary 4.2 hints that small Jeffrey’s divergence will make the testing error small. In Figure 3a, we show that the dynamic of Jeffrey’s divergence (computed in representation space) can well characterize the evolution of the testing error during the training phase.
In
Figure 3b, we show that the number of target data has some impact on the testing performance on the target data.
When we use less than half of the available unlabelled target data, the performance increases as the number of data increases.
When we use more than half of the unlabelled target data, the improvement on the performance is very small.