D OMAIN A DVERSARIAL T RAINING A G AME P ERSPECTIVE
Anonymous authors
Paper under double-blind review
A BSTRACT
The dominant line of work in domain adaptation has focused on learning invariant representations using domain-adversarial training.
In this paper, we interpret this approach from a game theoretical perspective .
Deﬁning optimal solutions in domain-adversarial training as local Nash equilibria, we show that gradient descent in domain-adversarial training can violate the asymptotic convergence guarantees of the optimizer, oftentimes hindering the transfer performance.
Our analysis leads us to replace gradient descent with high-order ODE solvers (i.e., Runge–Kutta), for which we derive asymptotic convergence guarantees.
This family of optimizers is signiﬁcantly more stable and allows more aggressive learning rates, leading to high performance gains when used as a drop-in replacement over standard optimizers.
Our experiments show that in conjunction with state-of-the-art domain-adversarial methods, we achieve up to 3.5% improvement with less than of half training iterations.
Our optimizers are easy to implement, free of additional parameters, and can be plugged into any domain-adversarial framework.
I NTRODUCTION
Unsupervised domain adaptation (UDA) deals with the lack of labeled data in a target domain by transferring knowledge from a labeled source domain (i.e., a related dataset with different distribution where abundant labeled data already exists).
The paramount importance of this paradigm has led to remarkable advances in the ﬁeld in terms of both theory and algorithms (Ben-David et al., 2007; 2010a;b; Mansour et al., 2009).
Several state-of-the-art algorithms tackle UDA by learning domaininvariant representations in an adversarial fashion (Shu et al., 2018;
Long et al., 2018;
Saito et al., 2018; Hoffman et al., 2018; Zhang et al., 2019; Acuna et al., 2021).
Their goal is to fool an auxiliary classiﬁer that operates in a representation space and aims to classify whether the datapoint belongs to either the source or the target domain.
This idea, called Domain-Adversarial Learning (DAL), was introduced by Ganin et al.
and can be more formally understood as minimizing the discrepancy between source and target domain in a representation space (Acuna et al., 2021).
Despite DAL being a dominant approach for UDA, alternative solutions have been sought as DAL is noticeably unstable and difﬁcult to train in practice (Sener et al., 2016;
Sun et al., 2019; Chang et al., 2019).
One major cause of instability is the adversarial nature of the learning algorithm which results from the introduction of the Gradient Reversal Layer (GRL) (Ganin et al., 2016)
(Figure 1).
GRL ﬂips the sign of the gradient during the backward pass, which has profound implications on the training dynamics and asymptotic behavior of the learning algorithm.
Indeed, GRL transforms gradient descent into a competitive gradient-based algorithm which may converge to periodic orbits and other non-trivial limiting behaviors that arise for instance in chaotic systems (Mazumdar et al., 2020).
Surprisingly, little attention has been paid to this fact, and speciﬁcally to the adversarial component and interaction among the three different networks in the algorithm.
In particular, three fundamental questions have not been answered from an algorithmic point of view, 1) what is optimality in DAL?
2) What makes DAL difﬁcult to train and 3)
How can
we
mitigate this problem
In this work, we aim to answer these questions by interpreting the DAL framework through the lens of game theory.
Speciﬁcally, we use tools developed by the game theoretical community in
Ba¸sar & Olsder (1998); Letcher et al.
; Mazumdar et al.
and draw inspiration from the existing two-player zero-sum game interpretations of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014).
We emphasize that in DAL, however, we have three rather than two networks interacting with each other, with partial cooperation and competition.
We propose a natural three-player game interpretation for DAL, which is not necessarily equivalent to two-player zero-sum game interpretations (see Example 1), which we coin as the Domain-Adversarial Game.
We also propose to interpret and characterize optimal solutions in DAL as local Nash Equilibria (see Section 3).
This characterization introduces a proper mathematical deﬁnition of algorithmic optimality for DAL.
It also provides sufﬁcient conditions for optimality that drives the algorithmic analysis.
With our proposed game perspective in mind, a simple optimization solution would be to use Gradient
Descent (GD) algorithms, which is the de-facto solution but known to be unstable.
Alternatively, we could also use other popular gradient based optimizers proposed in the context of differentiable games e.g. (Korpelevich, 1976; Mescheder et al., 2017).
However, we notice that these do not outperform GD in practice (see Section 6).
To understand why, we analyze the asymptotic behavior of gradient-based algorithms in the proposed domain-adversarial game (Section 4).
The main result of Section 4.2 (Theorem 2) shows that GD with GRL (i.e., the existing solution for DAL) violates the asymptotic convergence guarantees to local NE unless an upper bound is placed on the learning rate, which may explain its training instability and sensitivity to optimizer parameters.
In Section 4.
and appendices B.2 and E, we also provide a similar analysis for the popular game optimization algorithms mentioned above.
We emphasize however that while some of our results may be of independent interest for learning in general games, our focus is DAL .
Sections 4.3 and 6 show both theoretically and experimentally that the limitations mentioned above disappear if standard optimizers are replaced with ODE solvers of at least second order.
These are straightforward to implement as drop-in replacements to existing optimizers.
They also lead to more stable algorithms, allow for more aggressive learning rates and provide notable performance gains.
P RELIMINARIES
[Equation]
[Table]
/lC0/
RiVVCFMXKPmlQrv6eyLDQeiICmxTYjPSiNxP/83qpia78jMkkNVSS+UdRypGJ0ex6FDJFieETSzBRzO6KyAgrTIztqGJL8BZPXibts7p3UXfvzmuN66KOMhzBMZyCB5fQgFtoQgsICHiGV3hzlPPivDsf82jJKWYO4Q+czx+ZdZBG</latexit> R  (GRL)
We focus on the UDA scenario and follow the formulation from Acuna et al.
This makes our analysis general and applicable to most state-of-the-art DAL algorithms (e.g., Ganin et al.
Saito et al.
; Zhang et al.
We assume that the learner has access to a source dataset (S) with labeled examples and a target dataset (T) with unlabeled examples, where the source inputs x si are sampled i.i.d.
from a (source) distribution P s and the target inputs x ti are sampled i.i.d.
from a (target) distribution P t , both over
X .
We have Y = { 0 , 1 } for binary classiﬁcation, Y = { 1 , ..., k } in the multiclass case.
The risk of a hypothesis h :
X → Y w.r.t.
the labeling function f , using a loss function (cid:96) :
Y ×Y → R + under dis- tribution
D is deﬁned as: R (cid:96) D ( h, f ) :=
E D [ (cid:96) ( h ( x ) , f ( x ))] .
For simplicity, we deﬁne R (cid:96)S ( h ) :
= R (cid:96)P s ( h, f s ) and R (cid:96)T ( h ) :
= R (cid:96)P t ( h, f t ) .
The hypothesis class is denoted by H such that h ∈ H .
UDA aims to minimize the risk in the target domain while only having access to labeled data in the source domain.
This risk is upper bounded in terms of the risk of the source domain, the discrepancy between the two distributions and the joint hypothesis error λ ∗ :
Theorem 1.
(Acuna et al.
Let us note (cid:96) :
Y × Y → [0 , 1] , λ ∗ := min h ∈H R (cid:96)S ( h )
+ R (cid:96)T ( h ) , and D φh,
H
( P s || P t ) :
= sup h (cid:48) ∈H | E x ∼ P s [ (cid:96) ( h ( x ) , h (cid:48) ( x ))]
− E x ∼ P t [ φ ∗ ( (cid:96) ( h ( x ) , h (cid:48)
( x ))) | .
We have:
[Equation]
The function φ : R + → R deﬁnes a particular f -divergence and φ ∗ is its (Fenchel) conjugate.
As is typical in UDA, we assume that the hypothesis class is complex enough and both f s and f t are similar in such a way that the non-estimable term ( λ ∗ ) is negligible and can be ignored.
Domain-Adversarial Training (Figure 1) aims to ﬁnd a hypothesis h ∈ H that jointly minimizes the ﬁrst two terms of Theorem 1.
To this end, the hypothesis h is interpreted as the composition of h
= ˆ h ◦ g with g : X → Z , and ˆ h : Z → Y .
Another function class ˆ H is then deﬁned to formulate
H :=
{ ˆ h ◦ g : ˆ h ∈ ˆ H , g ∈ G} .
The algorithm tries to ﬁnd the function g ∈ G such that ˆ h ◦ g minimizes the risk of the source domain (i.e. ﬁrst term in Theorem 1), and its composition with ˆ h and ˆ h (cid:48) minimizes the divergence of the two distributions (i.e. second term in Theorem 1).
Algorithmically, the computation of the divergence function in Theorem 1 is estimated by a so called domain classiﬁer ˆ h (cid:48)
∈
ˆ H whose role is to detect whether the datapoint g ( x i )
∈
Z belongs to the source or to the target domain.
When there does not exist a function ˆ h (cid:48) ∈
ˆ H that can properly distinguish between g ( x si ) and g ( x ti ) , g is said to be invariant to the domains.
Learning is performed using GD and the GRL (denoted by R λ ) on the following objective:
[Equation]
where d s,t (ˆ h, ˆ h (cid:48) , g ) :=
E x ∼ p s [ˆ (cid:96) (ˆ h (cid:48) ◦ g, ˆ h ◦ g )]
− E x ∼ p t
[( φ ∗ ◦ ˆ (cid:96) )
(ˆ h (cid:48) ◦ g, ˆ h ◦ g )] .
Mathematically, the GRL R λ is treated as a “pseudo-function” deﬁned by two (incompatible) equations describing its forward and back-propagation behavior (Ganin & Lempitsky, 2015; Ganin et al., 2016).
Speciﬁcally,
[Equation]
and
[Equation]
where λ and α are hyper-parameters that control the tradeoff between achieving small source error and learning an invariant representation.
The surrogate loss (cid:96) :
Y × Y → R (e.g., cross-entropy) is used to minimize the empirical risk in the source domain.
The choice of function ˆ (cid:96) :
Y × Y → R and of conjugate φ ∗ of the f -divergence deﬁnes the particular algorithm (Ganin et al., 2016;
Saito et al., 2018; Zhang et al., 2019; Acuna et al., 2021).
From Equation (2), we can notice that GRL introduces a competitive scheme.
We next propose to interpret Equation (2) as a three-player game where the players are ˆ h, ˆ h (cid:48) and g .
We also study its continuous gradient dynamics.
A G AME P ERSPECTIVE ON DAL
We now interpret DAL from a game-theoretical perspective.
In Section 3.1, we rewrite the DAL objective as a three-player game.
In this view, each neural network (feature extractor and two classiﬁers) is a player that has a clear objective.
This allows us to deﬁne optimality in terms of local Nash Equilibrium (see Deﬁnition 2 in Appendix).
In Section 3.2, we introduce the vector ﬁeld, the game Hessian and the tools that allow us to characterize local NE for smooth players.
This characterization drives the algorithmic analysis of the continuous dynamics presented in Section 4.
3. D OMAIN -A DVERSARIAL G AME
We now rewrite and analyze the DAL problem in Equation (2) as a three-player game.
Let ˆ H , ˆ H (cid:48) and G be classes of functions deﬁned by ﬁxed parametric functions (i.e neural networks with ﬁxed architecture), and deﬁne ω 1 ⊆ Ω 1 and ω
3 ⊆ Ω 3 as a vector composed of the parameters of the classiﬁer and domain classiﬁer networks ˆ h ∈
ˆ H and ˆ h (cid:48)
∈
ˆ H , respectively.
Similarly, let ω 2 ⊆
Ω 2 be the parameters of the feature extractor network g ∈ G .
Their joint domain is denoted by Ω
=
(Ω 1 × Ω 2 × Ω 3 ) and the joint parameter set is ω
= ( ω 1 , ω 2 , ω 3 ) .
Let each neural network be a player and its parameters ( R d i ) its individual action strategy set (here continuous).
The goal of each player is then to selﬁshly minimize its own cost function J i : Ω → R .
We denote subscript − i to refer to all parameters/players but i .
With the notation introduced, we can now formally deﬁne the Domain-Adversarial Game as the three-player game G ( I , Ω i , J i ) where I := { 1 , 2 , 3 } , dim(Ω) =
(cid:80) 3 i =
dim(Ω i ) =
d , Ω i ⊆
R d i and:
[Equation]
We use the shorthand notation (cid:96) ( ω 1 , ω 2 ) to refer to E x,y ∼ p s [ (cid:96) ( ω 1 ◦ ω 2 ( x ) , y )] , and ω i
(cid:55)→ ζ i ( x, ω i ) with ζ i being the parametric function (i.e the feature extractor g and classiﬁers: ˆ h and ˆ h (cid:48) ).
Similar notation follows for d s,t .
Here, we assume that each J i is smooth in each of its arguments w
i ∈
Ω i as is typically the case when players are neural networks.
The gradient ﬁeld of Equation (2) and the game’s vector ﬁeld (see Section 3.2) are equivalent, making the original interpretation of DAL and our three-player formulation equivalent.
However, it is worth noting that our intepretation does not explicitly require the use of R λ in d s,t in Equation (4).
We can write optimality conditions of the above problem through the concept of Nash Equilibrium:
Deﬁnition 1.
(Nash Equilibrium (NE))
A point ω ∗ ∈ Ω is said to be a Nash Equilibrium of the
Domain-Adversarial Game if we have:
∀ i ∈ { 1 , 2 ,
[Equation]
[Equation]
ω
[Equation]
In our scenario, the losses are not convex/concave.
NE then does not necessarily exist and, in general, ﬁnding NE is analogous to, but much harder than, ﬁnding global minima in neural networks – which is unrealistic using gradient-based methods Letcher et al.
Thus, we focus on local-NE which relaxes the NE to a local neighborhood B ( w ∗ , δ ) :
= {|| w − w ∗
|| < δ } with δ > 0 (see Deﬁnition 2).
Intuitively, a NE means that no player has the incentive to change its own strategy (here parameters of the neural network) because it will not generate any additional pay off (here it will not minimize its cost function).
We emphasize that each player only has access to its own strategy set.
In other words, the player J 1 cannot change the parameters ω 2 , ω 3 .
It only has access to ω 1 ∈ Ω 1 .
While the motivation of the three-player game follows naturally from the original formulation of DAL where three networks interact with each other (see Figure 1), the optimization problem (2) could be also interpreted as the minimax objective of a two-player zero-sum game.
Thus, a natural question arises: can we interpret the domain-adversarial game as a two player zero-sum game? .
This can be done for example by letting ω ∗ 12 :=
( ω ∗ 1 , ω ∗ 2 ) , and considering the cost of the two players ( ω 12 , ω 3 ) as J 12 = J and J 3 = − J where J ( ω 12 , ω 3 ) :
= E p s [ (cid:96) ( ω 1 , ω 2 )]
+ d st ( ω ) .
In general, however, the solution of the two-player game i.e ( ω ∗ 12 , ω ∗ 3 ) is not equal to the NE solution of the three-player game ( ω ∗ 1 , ω ∗ 2 , ω ∗ 3 ) .
This is because the team optimization solution ω ∗ 12 (cid:54) =
( ω ∗ 1 , ω ∗ 2 ) in general.
We illustrate this in the following counterexample (see Ba¸sar & Olsder (1998) for more details):
Example 1.
Let J ( ω ) :
= 12 (cid:0) ω 21 + 4 ω 1 ω 2 + ω 22 − ω 23 (cid:1) .
(a) Suppose the three-player game ω
= ( ω 1 , ω 2 , ω 3 ) with J 1 = J 2 = J and J 3 = − J .
Each J
i is strictly convex in ω i .
The NE solution of the game ω ∗ = (0 , 0 , 0) is unique.
(b) Suppose the two-player game ω
=
( ω 12 , ω 3 ) with J 12 =
J and J 3 = − J .
The solution ω ∗ from (a) is not a NE solution .
To see this, let ˆ ω :
= ( − 1 , 1 , 0) .
We have that J 12 (ˆ ω )
=
− 1 < J 12 ( ω ∗ )
= 0 .
This contradicts Deﬁnition 1.
One can verify that there is no NE in this two-player scenario.
3. C HARACTERIZATION OF THE D OMAIN -A DVERSARIAL G AME
We now introduce the game’s vector ﬁeld (also called pseudo-gradient ) and the pseudo-gradient’s
Jacobian.
We also provide a characterization of local NE based on them (see Proposition 3).
These are the core mathematical concepts used in our analysis (Section 4).
We ﬁrst deﬁne the game’s vector ﬁeld v ( w ) , and its Jacobian H ( ω ) (also called game Hessian (Letcher et al., 2019)):
[Equation]
It is worth noting that v ( w ) can be different from the gradient of a single smooth (potential) function.
The game’s pseudo-gradient v ( w ) and the three-player formulation naturally capture the behavior introduced by the GRL in the original formulation.
Speciﬁcally, v ( ω ) is identical to the gradient with respect to the parameters of the original DAL objective (Equation (2)). Therefore, in both cases the behavior of GD is identical. Assuming the same initial conditions, they will reach the same solution. This shows the equivalence between our perspective and the original DAL formulation.
We emphasize that by equivalent, we mean same dynamics, and same intermediate and ﬁnal solutions.
Another fact worth emphasizing is that H ( ω ) is asymmetric.
This is in sharp contrast with the Hessian in standard supervised learning.
Before proceeding with a characterization of local NE in terms of v ( w ) and H ( ω ) , we ﬁrst deﬁne sufﬁcient and necessary conditions to deﬁne stationary points as local NEs.
Proposition 1.
(Necessary Condition) .
Suppose each J i is twice continuously differentiable in each ω
i , any local NE ω ∗ satisﬁes: i) ∇ ω
i J i ( ω ∗ ) = 0 and ii) ∀ i ∈ { 1 , 2 , 3 } , ∇ 2 ω i ,ω
i J i ( ω ∗ ) (cid:23) 0 .
Proposition 2.
(Sufﬁcient Condition) .
Suppose each J i is twice continuously differentiable in each ω i .
ω ∗ i is a local NE if i) ∇ ω
i J i ( ω ∗ ) = 0 and ii) ∀ i, ∇ 2 ω i ,ω
i J i ( ω ∗ ) (cid:31) 0 .
The necessary and sufﬁcient conditions from Propositions 1 and 2 are reminiscent of conditions for local optimality in continuous optimization Nocedal & Wright (2006).
Similar conditions were also proposed in Ratliff et al.
(2016) where the sufﬁcient condition is called differential Nash equilibrium .
We can now characterize local NEs in terms of v ( w ) and H ( ω ) :
Proposition 3.
(Strict Local NE) w is a strict local NE if v ( w ) =
and H ( ω ) + H ( ω ) (cid:62) (cid:31) 0 .
The sufﬁcient condition implies that the NE is structurally stable (Ratliff et al., 2016).
Structural stability is important as it implies that slightly biased estimators of the gradient (e.g., due to sampling noise) will not have vastly different behaviors in neighborhoods of equilibria (Mazumdar et al., 2020).
In the following, we focus on strict local NEs (i.e., ω ∗ for which Proposition 3 is satisﬁed).
L EARNING A LGORITHMS
We have deﬁned optimality as local NE and provided sufﬁcient conditions in terms of the pseudogradient and its Jacobian.
In this section, we assume the existence of strict local NE (Proposition 3) in the neighborhood of the current point (e.g., initialization), and analyze the continuous gradient dynamics of the Domain-Adversarial Game (Equations (4) and (5)).
We show that given the sufﬁcient conditions from Proposition 3, asymptotic convergence to a local NE is guaranteed through an application of Hurwitz condition (Khalil, 2002).
Most importantly, we show that using GD with the GRL could violate those guarantees unless its learning rate is upper bounded (see Theoremand corollary 1).
This is in sharp contrast with known results from standard supervised learning where the implicit regularization introduced by GD has been shown to be desirable (Barrett & Dherin, 2021).
We also analyze the use of higher-order ODE solvers for DAL and show that the above restrictions are not required if GD is replaced with them.
Finally, we compare our resulting optimizers with recently algorithms in the context of games.
Our algorithmic analysis is based on the continuous gradient-play dynamics and the derivation of the modiﬁed or high-resolution ODE of popular integrators (e.g., GD/Euler Method and Runge-Kutta).
This type of analysis is also known in the numerical integration community as backward error analysis (Hairer et al., 2006) and has recently been used to understand the implicit regularization effect of GD in supervised learning (Barrett & Dherin, 2021).
High resolution ODEs have also been used in Shi et al.
(2018) to understand the acceleration effect of optimization algorithms, and more recently in
Lu (2020).
As in Barrett & Dherin (2021); Lu (2020); Shi et al.
, our derivation of the high resolution ODEs is in the full-batch setting.
The derivation of the stochastic dynamics of stochastic discrete time algorithms is signiﬁcantly more complicated and it is beyond the scope of this work.
We experimentally demonstrate that our results are also valid when there is stochasticity due to sampling noise in the mini-batch.
We emphasize that our analysis does not put any constraint or structure on the players’ cost functions as opposed to Azizian et al.
; Zhang & Yu (2020).
In our problem, the game is neither bilinear nor necessarily strongly monotone.
See proofs in Appendices.
4. C ONTINUOUS G RADIENT D YNAMICS
Given v ( ω ) the continuous gradient dynamics can be written as:
[Equation]
For reasons that will become obvious later and to distinguish between these and the gradient ﬂow, we will refer to these as the gradient-play dynamics as in Ba¸sar & Olsder (1998); Mazumdar et al.
These dynamics are well studied and understood when the game is either a potential or a purely adversarial game (deﬁnitions in suppl. material).
We remark, however, that while Equation (2) may look like a single objective, the introduction of the GRL ( R λ ), makes a fundamental difference between our case and the dynamics that are analyzed in the single-objective gradient-based learning and optimization literature.
We summarize this in the following simple proposition.
Proposition 4.
The domain-adversarial game is neither a potential nor necessarily a purely adversarial game.
Moreover, its gradient dynamics are not equivalent to the gradient ﬂow.
Fortunately, we can directly apply Hurwitz condition (Khalil, 2002)
(also known as condition for asymptotic stability, see Appendix A.1) to derive sufﬁcient conditions for which the continuous dynamics of the gradient play would converge.
Lemma 1.
(Hurwitz condition)
Let ∇ v ( w ∗ ) be the Jacobian of the vector ﬁeld at a stationary point w ∗ where v ( w ∗ )
= 0 .
If the real part of every eigenvalue λ of ∇ v ( w ∗ ) (i.e. in the spectrum
Sp ( ∇ v ( w ∗ )) ) is positive then the continuous gradient dynamics are asymptotically stable.
In this work, we assume the algorithms are initialized in a neighborhood of a strict local NE ω ∗ .
Therefore, Lemma 1 provides sufﬁcient conditions for the asymptotic convergence of the gradientplay dynamics to local NE.
In practice this assumption may not hold, and it is computationally hard to verify.
In our experiments, we perform an analysis and show noticeable performance gains in several tasks, benchmarks and network architectures (see Section 6).
4. A NALYSIS OF GD WITH THE GRL
We showed above that given the existence of a strict local NE, the gradient-play dynamics are attracted to the strict local NE.
A natural question then arises: If under this assumption local asymptotic convergence is guaranteed, what could make DAL notoriously hard to train and unstable?
In practice, we do not have access to an explicit solution of the ODE.
Thus, we rely on integration algorithms to approximate the solution.
One simple approach is to use the Euler method which results in the following update:
[Equation]
This is commonly known as GD.
The equivalence between v ( w ) (game’s vector ﬁeld) and the gradient of Equation (2) (original DAL formulation) follows from the use of the GRL ( R λ ).
We remind the reader that the GRL is a “pseudo-function” deﬁned by two (incompatible) equations describing its forward and back-propagation behavior, i.e., a ﬂip in the gradient’s sign for the backward pass (see
Figure 1, Section 2 and Ganin et al.
Equation (7) is then the default algorithm used in DAL.
Now, to provide an answer to the motivating question of this section, we propose to analyze the high-resolution ODE of this numerical integrator (i.e., Euler) and in turn its asymptotic behavior.
This is similar to deriving the modiﬁed continuous dynamics for which the integrator produces the exact solution (Hairer et al., 2006) and applying Hurwitz condition on the high-resolution ODE.
Theorem 2.
The high resolution ODE of GD with the GRL up to O ( η ) is:
[Equation]
Moreover, this is asymptotically stable (see Appendix A.1) at a stationary point w ∗ (Proposition 3) iff for all eigenvalue written as λ =
a + ib ∈ Sp ( −∇ v ( w ∗ ))
, we have 0 > η ( a 2 − b 2 ) / 2 > a.
A striking difference between Equation (6) and Equation (8) is made clear (additional term marked in red).
This additional term is a result of the discretization of the gradient-play dynamics using Euler’s method (i.e. GD) and leads to a different Jacobian of the dynamics.
This term was recently shown to be beneﬁcial for standard supervised learning (Barrett & Dherin, 2021), where ∇ v ( ω ∗ ) is symmetric and thus only has real eigenvalues.
In our scenario, this term is undesirable.
In fact, this additional term puts an upper bound on the learning rate η .
The following corollary formalizes this:
Corollary 1.
The high resolution ODE of GD with GRL in Equation (8) is asymptotically stable only if the learning rate η is in the interval: 0 < η < − 2 a b 2 − a 2 , for all λ
= a + ib ∈ Sp ( −∇ v ( w ∗ )) with large imaginary part (i.e. such that | a | < | b | ).
To have good convergence properties, the imaginary part of the eigenvalues of −∇ v ( w ∗ ) must be small enough.
Therefore, if some eigenvalue λ = a + ib satisﬁes a < 0 and b 2 (cid:29) a 2 (cid:29) − 2 a , the learning rates should be chosen to be very small.
This is veriﬁed in Section 6 and in Example 2. Example 2.
Consider the three-player game where (cid:96) ( w 1 , w 2 )
=
w 21 + 2 w 1 w 2 + w 22 , λ
=
and d s,t ( w 2 , w 3 ) = w 22 + 99 w 2 w 3 − w 23 . Then ˙ w = − v ( w ) becomes: ˙ w = Aw = (cid:16) − 2 − 2 0 − 2 − 4 − 99 0−(cid:17) .
The eigenvalues of A are − 2 and − 3 ± 2 i √ 2449 .
From Corollary 1, η should be 0 < η < 6 . 2 · 10 − 3 .
4. H IGHER ORDER ODE S OLVERS
The limitation described above exists because GD with the GRL can be understood as a discretization of the gradient-play dynamics using Euler’s Method.
Thus, it only approximates the continuous dynamics up to O ( η ) .
To obtain a better approximation, we consider Runge-Kutta (RK) methods of order two and beyond (Butcher, 1996).
For example, take the improved Euler’s method (a particular
RK method of second order) that can be written as:
[Equation]
Comparing Equation (9) (update rule of RK2) with Equation (7) (update rule of GD), one can see that the RK2 method is straightforward to implement in standard deep-learning frameworks.
Moreover, it does not introduce additional hyper-parameters.
More importantly, such discrete dynamics approximate the continuous ODE of Equation (6) to a higher precision.
In Appendix C, we provide asymptotic guarantees for the high resolution ODE of general RK methods , their generalized expression and the algorithm pseudo-code.
See also PyTorch pseudo-code in Appendix E.
Limitation.
A disadvantage of using high order solvers is that they require additional extra steps.
Speciﬁcally, one extra step in the case of RK2 (computation of the additional second term in
Equation (9)).
In our implementation, however, this was less than 2x slower in wall-clock time (see Appendix E.5 for more details and wall-clock comparison).
Also, if not initialized in the neighborhood of a local NE, as with gradient-based methods, high order solvers might also converge to a non-Nash equilibrium as described in Mazumdar et al.
That said, this is likely a rare case.
Comparison vs other game optimization algorithms.
DAL has not been previously interpreted from a game perspective.
Our interpretation allows us to bring recently proposed algorithms in the context of differentiable games (Zhang & Yu, 2020; Azizian et al., 2020) to DAL.
Classic examples are the Extra-Gradient (EG) method (Korpelevich, 1976) and Consensus Optimization (CO) (Mescheder et al., 2017).
In Appendix B.2 we analyze the continuous dynamics of the EG method, and show that we cannot take the learning rate of EG to be large either.
Thus, we obtain a similar conclusion as Corollary 1.
Then, in practice for DAL, stability for EG comes at the price of slow convergence due to the use of small learning rates.
We experimentally show this in Figure 3.
With respect to
CO, we show in Appendix C that this algorithm can be interpreted in the limit as an approximation of the RK2 solver.
In practice, if its additional hyper-parameter ( γ ) is tuned thoroughly, CO may approximate the continuous dynamics better than GD and EG.
We believe this may be the reason why
CO slightly outperforms GD and EG (additional discussion on this in Appendix E.4).
In all cases,
RK solvers outperform GD, EG and CO.
This is in line with our theoretical analysis since they better approximate the continuous dynamics (Hairer et al., 2006).
R ELATED W ORK
To the best of our knowledge, DAL has not been previously analyzed from a game perspective.
Moreover, the stability of the optimizer and the implications of introducing the GRL has not been analyzed either.
Here, we compare our results with the general literature.
Gradient-Based Learning in Games.
Ratliff et al.
(2016) proposed a characterization of local
Nash-Equilibrium providing sufﬁcient and necessary conditions for its existence.
Mazumdar et al.
(2020) proposed a general framework to analyze the limiting behavior of the gradient-play algorithms in games using tools from dynamical systems.
Our work builds on top of this characterization but specializes them to the domain-adversarial problem.
We propose a stabler learning algorithm that better approximates the gradient-play dynamics.
Our resulting algorithm does not introduce explicit adjustments or modify the learning dynamics nor does it require the computation of the several Hessian vector products or new hyperparameters.
This is in contrast with general algorithms previously analyzed in the context of differentiable games (Azizian et al., 2020; Letcher et al., 2019).
Integration Methods and ML.
Scieur et al.
(2017) showed that accelerated optimization methods can be interpreted as integration schemes of the gradient ﬂow equation.
Zhang et al.
(2018) showed that the use of high order RK integrators can achieve acceleration in convex functions.
In the context of two-players game (i.e GANs), Gemp & Mahadevan (2018) consider using a second order ODE integrator.
More recently, Qin et al.
(2020) proposed to combine RK solvers with regularization on the generators’ gradient norm.
Chen et al.
(2018) interpreted the residual connection in modern networks as the Euler’s integration of a continuous systems.
In our case, we notice that the combination of
GD with the GRL can be interpreted as the Euler’s discretization of the continuous gradient play dynamics, which could prevent asymptotic convergence guarantees.
We then study the discretization step of popular ODE solvers and provide simple guarantees for stability.
Moreover, our analysis is based on a novel three-players game interpretation of the domain-adaptation problem.
This is also different from a single potential function or two-player games (i.e GANs).
Two-Player Zero-Sum Games have recently received signiﬁcant attention in the machine learning literature due to the popularity of generative adversarial networks (GANs) (Goodfellow et al., 2014).
For example, several algorithms have been proposed and analyzed (Mescheder et al., 2017;
Mertikopoulos et al., 2019; Gidel et al., 2019a;b; Zhang & Yu, 2020; Hsieh et al., 2020), in both deterministic and stochastic scenarios.
In our problem, we have a general three-player games resulting of a novel game interpretation of the domain-adversarial problem.
E XPERIMENTAL R ESULTS
We conduct an extensive experimental analysis.
We compare with default optimizers used in domainadversarial training such as GD, GD with Nesterov Momentum (GD-NM) (as in Sutskever et al.
and Adam (Kingma & Ba, 2014).
We also compare against recently proposed optimizers in the context of differentiable games such as EG (Korpelevich, 1976) and CO (Mescheder et al., 2017).
We focus our experimental analysis on the original domain-adversarial framework of Ganin et al.
(2016) (DANN).
However, in section 6.2, we also show the versatility and efﬁcacy of our approach improving the performance of recently proposed SoTA DAL framework (e.g., f -DAL (Acuna et al., 2021) combined with Implicit Alignment (Jiang et al., 2020)).
6. E XPERIMENTAL A NALYSIS ON D IGITS
[Figure]
Implementation Details.
Our ﬁrst experimental analysis is based on the digits benchmark with models trained from scratch (i.e., with random initialization).
This benchmark constitutes of two digits datasets MNIST (CC BY-SA 3.0) and USPS (LeCun et al., 1998; Long et al., 2018) with two transfer tasks (M → U and U → M).
We adopt the splits and evaluation protocol from Long et al.
and follow their standard implementation details.
For GD-NM, we use the default momentum value (0.9).
We follow the same approach for the additional hyperparameters of Adam.
Hyper-parameters such as learning rate, learning schedule and adaptation coefﬁcient ( λ ) are determined for all optimizers by running a dense grid search and selecting the best hyper-parameters on the transfer task
M → U. As usual in UDA, the best criteria are determined based on best transfer accuracy.
The same parameters are then used for the other task (i.e., U → M).
We use the same seed and identically initialize the network weights for all optimizers.
This analysis is conducted on Jax (Bradbury et al., 2018)
(see Appendix D for details and setup).
Comparison vs optimizers used in DAL.
Figure 2 (top) illustrates the training dynamics for the loss in the target domain and the performance transfer.
As expected, our optimizer converges faster and achieves noticeable performance gains.
A core idea of DAL is to learn domain-invariant representations, thus we plot in Figure 2 (bottom) t-SNE (Van der Maaten & Hinton, 2008) visualizations of the last layer features of the network.
We show this over a sequence of epochs for GD with GRL vs RK2.
A different color is used for the source and target datasets.
In the comparison vs Adam, we emphasize that Adam computes adaptive learning rates which our method does not.
That said, Figure 2 shows that our two methods RK2 and RK4 outperform all baselines in terms of both convergence and transfer performance.
In Figure 7, we show how unstable these standard optimizers are when more aggressive step sizes are used.
This is in line with our theoretical analysis.
Experimentally, it can be seen that in DAL, GD is more stable than GD-NM and Adam, with the latter being the most unstable.
This may shade lights into why well tuned GD-NM is often preferred over Adam in DAL.
[Figure]
[Figure]
DANN).
Comparison vs game optimization algorithms.
We now compare RK solvers vs other recently proposed game optimization algorithms.
Speciﬁcally, we compare vs the EG method (Korpelevich, 1976) and CO (Mescheder et al., 2017).
In every case, we perform a dense grid under the same budget for all the optimizer and report the best selection (see Appendix E for details).
In line with our theoretical analysis of the continuous dynamics of the EG, we notice that the EG method is not able to train with learning rates bigger than 0.006, as a result it performs signﬁcantly worse than any other optimizer (including simple GD).
Also inline with our theoretical analysis, CO performs better than EG and all other popular gradient algorithms used in DAL.
This is because CO can be seen as an approximation of Heun’s Method (RK2).
More details on this in supplementary.
Robustness to hyper-parameters.
Figure 4 shows transfer performance of our method for different choices of hyper-parameters while highlighting (green line) the best score of the best performing GD hyperparameters on the same dataset.
Our method is robust to a wide variety of hyperparameters.
6. C OMPARISON IN COMPLEX ADAPTATION TASKS
[Figure]
LR const poly
LR Schedule
RK RK
Method 0.
adapt
[Figure] [Figure]
[Figure] [Figure] [Figure]
[Table]
We evaluate the performance of our algorithm with Resnet-50 (He et al., 2016) on more challenging adaptation benchmarks.
Specifically, this analysis is conducted on Visda-2017 benchmark (Peng et al., 2017).
This is a simulation-to-real dataset with two different domains: (S) synthetic renderings of 3D models and (R) real images.
For this experiment, we use PyTorch (Paszke et al., 2019), our evaluation protocol follows Zhang et al.
and uses Resnetas the backbone network.
For the optimizer parameters, we tune thoroughly GD-NM, which is the optimizer used in this setting (Long et al., 2018; Zhang et al., 2019;
Jiang et al., 2020; Acuna et al., 2021).
For ours, we keep the hyper-parameters, but increase the learning rate (0.2), and the batch size to 128.
In this task, our approach corresponds to the improved
Euler’s method (RK2).
Table 1 shows the comparison.
Figure 6 compares the training dynamics of our method vs GD-NM.
In Figure 5, we evaluate the sensitivity of our method (in terms of transfer performance) to sampling noise as controlled by the batch size (average over 3 random seeds).
[Table]
Improving SoTA DAL frameworks.
We use this complex visual adaptation task to showcase the applicability of our method to SoTA DAL frameworks.
Speciﬁcally, we let the DA method being f -DAL Pearson as in Acuna et al.
(2021) with Implicit Alignment Jiang et al.
We use the tuned parameters and optimizer from Acuna et al.
; Jiang et al.
(2020) as the baseline.
In our case, we only increase the learning rate (0.2).
Table 2 shows that our method achieves peak results ( +3.5% ) in 10.5K iterations (vs 29.5K iterations for GD-NM).
Natural Language Processing Tasks.
We also evaluate our approach on natural language processing tasks on the Amazon product reviews dataset (Blitzer et al., 2006).
We show noticeable gains by replacing the GD with either RK2 or RK4.
Results and details can be found in Appendix E.1.
C ONCLUSIONS
We have analyzed DAL from a game-theoretical perspective where optimality is deﬁned as obtaining the local NE.
From this view, we showed that standard optimizers in DAL can violate the asymptotic guarantees of the gradient-play dynamics, thus requiring careful tuning and small learning rates.
Based on our analysis, we proposed to replace existing optimizers with higher-order ODE solvers.
We showed both theoretically and experimentally that these are more stable and allow for higher learning rates, leading to noticeable improvements in terms of the transfer performance and the number of training iterations.
We show that this family of optimizers does not require additional hyperparameters, can be used as a drop-in replacement and outperforms strong baselines.
1 Introduction
2 Preliminaries 3 A Game Perspective on DAL 3.1 Domain-Adversarial Game . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Characterization of the Domain-Adversarial Game . . . . . . . . . . . . . . . . . . . . . . .
4 Learning Algorithms 4.1 Continuous Gradient Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Analysis of GD with the GRL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Higher order ODE Solvers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
5 Related Work 6 Experimental Results 6.1 Experimental Analysis on Digits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
6.2 Comparison in complex adaptation tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . .
7 Conclusions
A Concepts in Game Theory
A.1 Deﬁnitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.2 Games Characterizations . . . .
A.3 Case of Study in from
B Derivation of high-resolution ODEs
B.1 High-resolution ODE of second-order Runge–Kutta method
B.2 Continuous dynamics of Extra-Gradient (EG)
B.3 High-resolution ODE of classic fourth-order Runge–Kutta method (RK4)
C Proofs and additional theoretical results
C.
Proposed Learning Algorithm
C.2 CO approximates RK2 (Heun’s Method)
D Experimental Setup Additional Details
E Additional Experiments
E.1 Natural Language Processing Tasks
E.2 Sensitivity to Sampling Noise
E.3 Additional Comparison vs Game Optimization Algorithms
E.4 CO vs Gradient Descent and Extra-Gradient Algorithms
E.5 Wall-Clock Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A C ONCEPTS IN G AME T HEORY A. D EFINITIONS
Deﬁnition 2.
(Local Nash Equilibrium) :
A point ( w ∗ i , w ∗− i ) ∈
Ω is said to be a local Nash Equilibrium of the domain-adversarial game if there exists some δ > 0 such that:
[Equation]
Intuitively, this is restricting the concept of NE to a local neighborhood B ( x ∗ , δ ) :
= {|| x − x ∗ || 2 < δ } with δ > 0 .
A more practical characterization of the NE can be given in terms of the Best Response Map of each player which we now deﬁne.
Deﬁnition 3.
(Best Response Map (BR))
The best response map BR i : Ω − i ⇒ Ω i of player i is deﬁned as:
[Equation]
here the symbol ⇒ emphasizes that the best response map is generally a set map and not a singleton, thus it is not a function in general.
In other words, there may be a subset of element in Ω i for which J i ( ., ω − i ) is a minimum.
The notion of NE can be deﬁned in terms of the generalized BR : Ω ⇒ Ω map.
This can be thought as an stacked vector where the i -th element of BR is BR i ( ω − i ) .
Proposition 5.
A point w ∗ i ∈ Ω is said to be a NE of the game if it is a ﬁxed point of the generalized BR : Ω ⇒ Ω map.
That is,
[Equation]
Proof.
This follows from the deﬁnitions of BR map and NE.
Deﬁnition 4.
( Asymptotically Stable )
A point ω is said to be a locally asymptotically stable point of the continuous dynamics ˙
ω = f ( ω ) if Re ( λ ) < 0 for all λ ∈ Sp ( ∇ f ( ω )) , where Sp ( ∇ f ( ω )) is the spectrum of ∇ f ( ω ) .
Deﬁnition 4 is also known as the Hurwitz condition Khalil (2002).
Deﬁnition 5.
A stationary point x of a C 2 function φ : R n → R is said to be a strict saddle point if:
• λ min ( ∇ 2 xx φ ( x ∗ ))
< 0 and, • λ ( ∇ 2 xx φ ( x ∗ )) > 0 , for any other λ ∈ sp ( ∇ 2 xx φ ( x ∗ ))
A. G AMES C HARACTERIZATIONS
Potential Games.
Potential Games were introduced in Monderer & Shapley (1996) and can be deﬁned as a type of game for which there exists an implicit potential function φ : Ω → R such that ∇ φ ( ω ) = v ( ω ) .
Consequently, a necessary and sufﬁcient condition for the game to be potential is the Jacobian of the vector ﬁeld ∇ v ( ω ) being symmetric (see 3.3 in Mazumdar et al.
and
Monderer & Shapley (1996)).
Purely Adversarial Games.
This particular type of game refers to the other extreme in which H ( ω ) is a non-symmetric matrix with purely imaginary eigenvalues.
If the game Hessian is skew-symmetric these have also been called Hamiltonian Games Letcher et al.
A. C ASE OF S TUDY IN DANN. O RIGINAL F ORMULATION FROM G ANIN ET AL . (2016)
As mentioned in the main text (Section 2), our analysis is compatible with both the original and more recent formulation of domain-adversarial training such as Zhang et al.
; Acuna et al.
In
In order to obtain the original formulation of DANN, let us deﬁne ˆ (cid:96) ( _ , b )
= log( σ ( b )) and φ ∗ ( t ) =
− log(1 − e t ) in Equation (2).
This corresponds to the Jensen-Shannon divergence (JS) (up to a constant shift that does not affect optimization).
We can then rewrite d s,t as:
[Equation]
where σ ( x ) := 1 1+ e − x .
To simplify the notation, we write H :
= σ ◦ H .
We can now re-deﬁne the pseudo-gradient v ( w ) of the game as the gradient of each player loss with respect to its parameters.
Letting α = 1 , we get from Equation (4).
[Equation]
The following propositions characterize local NE in terms of the pseudo-gradient v ( w ) and its Jacobian H ( ω ) .
Proposition 6.
(Local NE) Suppose v ( w ) =
and
[Equation]
then w is an isolated local NE.
The proof is simple and follows from Propositions 1 and 2, the deﬁnition of the vector ﬁeld v ( ω ) and the condition H + H (cid:62) (cid:31) 0 .
Cooperation with Competition .
By examining the matrix H ( ω ) , one can see that, in our scenario, the game is neither a potential game nor a purely adversarial game.
However, we can write the vector ﬁeld in the following form:
[Equation]
where the ﬁrst part corresponds to the gradient of the potential function φ ( ω )
= (cid:96) ( ω 1 , ω 2 ) .
The second part, on the other hand, corresponds to a function ˆ v ( w ) whose Jacobian is a non-symmetric matrix.
Analyzing them separately leads to either a potential or an adversarial game respectively.
We deﬁne this particular type of game as cooperation (i.e., in the potential term) with competition (i.e., the adversarial term).
It is worth noting that, while the spectrum of the game Hessian for the ﬁrst term has only real eigenvalues, the second term can have complex eigenvalues with a large imaginary component.
Indeed, it can be shown that this second term approximates the one obtained for a GAN using the non-saturating loss proposed by Goodfellow et al.
(e.g. λ = 1 ).
In other words, the second term can be written as the pseudo-gradient of the two player zero-sum game min ω 2 max ω 3 d st .
Building on this key observation and the work of Mescheder et al.
; Berard et al.
(2020) (Figure 4), where it was experimentally shown that the eigenvalues of the game
Hessian for GANs have indeed a large imaginary component around stationary points, we can assume that the spectrum of the game Hessian in our case also have eigenvalues with a large imaginary component around the stationary points.
This observation can also be used with Corollary 1 to further motivate the use of higher-order ODE solvers instead of GD with the GRL.
Example 3.
Consider the three-player game Equation (16) where (cid:96) ( w 1 , w 2 )
=
w 21 + 2 w 1 w 2 + w 22 , λ
=
and d s,t ( w 2 , w 3 )
= w 22 + 99 w 2 w 3 − w 23 .
The gradient play dynamics ˙ w = − v ( w ) becomes:
[Equation]
The eigenvalues of A are − 2 and − 3 ± 2 i √ 2449 .
From Corollary 1, η should be 0 < η < 6 . 2 · 10 − 3 .
Is the three-player game formulation desired?
In domain adaptation, optimization is a means to an end .
The ﬁnal goal is to minimize the upper bound from Theorem 1 to ensure better performance in the target domain.
One might then wonder whether interpreting optimality in terms of NE is desirable.
In our context, NE means ﬁnding the optimal g ∗ , ˆ h ∗ and ˆ h (cid:48) ∗ of the cost functions deﬁned in Equation (4).
This in turns leads to minimizing the upper bound in Theorem 1.
Remark on sequential games : Recently, Jin et al.
(2020) introduced a notion of local min-max optimality for two-player’s game exploiting the sequential nature of some problems in adversarial ML (i.e GANs).
In domain-adversarial learning, updates are usually performed in a simultaneous manner using the GRL.
Thus, we focus here on the general case where the order of the players is not known.
B D ERIVATION OF HIGH - RESOLUTION ODE S
Lemma 2.
The high resolution ODE of resulting of the GD algorithm with the GRL is:
[Equation]
[Equation]
Proof.
This follows from Corollary 1 of Lu (2020).
B. H IGH - RESOLUTION ODE OF SECOND - ORDER R UNGE –K UTTA METHOD
The high-resolution ODE was discussed in Shi et al.
; Lu (2020).
For discrete algorithms with the following update:
[Equation]
18) we can think of the trajectory as a discretization of the continuous dynamics w : [0 , + ∞ )
→ R d , and in
Equation (18), we have w = w ( t ) , w + = w ( t + η ) .
Here, with slight abuse of notation we also use w for the continuous function of dynamics.
We derive high-resolution ODE of the second-order Runge–Kutta method:
[Equation]
where 0 < α ≤ 1 and α is a constant.
If α = 1 / 2 , we obtain Heun’s method; if α = 1 , we obtain the midpoint method; if α = 2 / 3 , we obtain the Ralston’s method.
Combining the two equations, we have:
[Equation]
Using the Taylor expansion:
[Equation]
Plugging it back into Equation (19) and using the Taylor expansion w k {++
+}
= w k + η
[Equation]
w + η 2 ¨ w/ 2 , we have: ˙
[Equation]
Now we make the assumption that we have the high-resolution ODE that:
[Equation]
Taking the derivative over t we have:
[Equation]
Combining Equation (20), Equation (21) and Equation (22), we obtain that: [Equation] [Equation]
[Equation]
23) i.e., the high resolution ODE of second-order Runge–Kutta method is:
[Equation]
B. C ONTINUOUS DYNAMICS OF E XTRA -G RADIENT (EG)
The continuous dynamics of Gradient Descent Ascent (GDA), Extra-Gradient (EG) and Heun’s method can be summarized as follows:
[Equation]
For GDA, we have α = − η/ 2 ; for EG, we have α = η/ 2 (Lu, 2020); for Heun’s method, ˙ w = v ( w ) + O ( η 2 ) .
The Jacobian of the dynamics at the stationary point is ∇ v ( w ) + α ∇ v ( w ) 2 .
Take λ = a + ib ∈ Sp ( ∇ v ( w )) .
The eigenvalue of the Jacobian of the dynamics is:
[Equation]
We want the real part to be negative, i.e.:
[Equation]
(26) and thus:
[Equation]
(27) for EG, α = η/ 2 and the dynamics diverges if a (1+( η/ 2) a ) ≥ ηb
B. H IGH - RESOLUTION ODE OF CLASSIC FOURTH - ORDER R UNGE –K UTTA METHOD (RK4)
In this subsection, we derive the high-resolution ODE of the classic fourth-order Runge–Kutta method.
We prove the following result:
Theorem 3.
The high-resolution ODE of the classic fourth-order Runge–Kutta method (RK4):
[Equation]
where
[Equation]
is
[Equation]
Proof.
We use the following Taylor expansion:
[Equation]
where ∇ 2 v ( w ) : R d × R d → R d is a symmetric bilinear form
[Equation]
and ∇ 3 v ( w ) : R d × R d × R d → R d is a symmetric trilinear form.
With the formula we have:
[Equation]
Putting them together we have:
[Equation]
Bringing Equation (37) into Equation (36), we obtain:
[Equation]
Putting Equation (38) and Equation (37) together, we have:
[Equation]
Moreover, we have
[Equation]
and similarly
[Equation]
Bringing Equation (39) and Equation (40) into Equation (35) results in v
[Equation]
Let us now derive the high-resolution ODE.
From Equation (28)
[Equation]
we have:
[Equation]
Let us assume that w + = w ( t + η ) and w = w ( t ) .
Expanding the left we have:
[Equation]
Let us assume that the high-resolution ODE up to O ( η 4 ) has the form:
[Equation]
Taking derivatives on both sides
[Equation]
we have:
[Equation]
O
[Equation] [Equation]
Comparing the order O (1) on both sides of Equation (43) we have:
[Equation]
which gives f 0 ( w ) = − v ( w ) and f 1 ( w ) = 0 .
Bringing it back to Equation (46) we obtain:
[Equation]
We take the derivatives on both sides of Equation (48) to get:
[Equation]
Let us now compute ∇ ( ∇ v ( w ) v ( w )) v ( w ) .
We note that ∇ ( ∇ v ( w ) v ( w )) is a linear form and
[Equation]
Therefore, we have
[Equation]
With Equation (51) we can compare O ( η 2 ) on both sides on Equation (43) and obtain
[Equation]
Finally, we take the derivative of Equation (51).
Since ∇
[Equation]
w
[Equation]
we have:
[Equation]
Similarly, since
[Equation]
[Equation]
Combining Equation (51), Equation (54) and Equation (56) gives:
[Equation]
Combining Equation (45), Equation (46), Equation (51) and Equation (57) and comparing the O ( η 3 ) term of
Equation (43), we obtain
[Equation]
Therefore, the high-resolution ODE is:
[Equation]
C P ROOFS AND ADDITIONAL THEORETICAL RESULTS
Several proofs in this section are based on the derivations of the high resolution ODEs that were proven in Appendix B.
Theorem 4.
Suppose φ : R n → R and φ is C 2 .
Suppose also any stationary point x ∗ of φ s.t. ∇ φ ( x ∗ ) = 0 is either:
a strict local minimum,
i
e. ∇ 2 xx φ ( x ∗ ) (cid:31) 0 ; 2.
a strict saddle.
(Deﬁnition 5)
Then, the gradient ﬂow is attracted towards a local minimum of φ .
Proof.
[Equation] [Equation]
Deﬁne g ( x ) :
= −∇ φ ( x ) .
a) Let us ﬁrst show that strict saddles are non-asymptotically stable ﬁxed points.
Thus, gradient descent is not attracted to them.
[Equation]
( x − x ∗ ) since ∇ φ ( x ∗ ) =(61) g ( z )
≈ − Λ( z − z ∗ )
where z := U T x (62)
This implies g ( z ) i
≈ λ min ( ∇ 2 xx φ ( x ∗ ))
( z − z ∗ ) i > 0 .
Thus, we just showed that strict saddles are nonasymptotically stable, as desired.
Let us now show that all the local minima are asymptotically stable, and gradient descent is then attracted to them.
b) Deﬁne V ( x ) :
= φ ( x ) − φ ( x ∗ ) to be a Lyapunov function.
We have x ∗ is asymptotically stable if:
[Equation]
From which the result follows.
(e.g ∇ V ( x ) T g ( x ) = −|| g ( x ) || 22 ).
Similar results with a different proof can be derived from Lee et al.
Theorem 2.
The high resolution ODE of GD with the GRL up to O ( η ) is:
[Equation]
Moreover, this is asymptotically stable (see Appendix A.1) at a stationary point w ∗ (Proposition 3) iff for all eigenvalue written as λ =
a + ib ∈ Sp ( −∇ v ( w ∗ ))
, we have 0 > η ( a 2 − b 2 ) / 2 > a.
Proof.
This theorem can be obtained by computing the Jacobian of the ODE dynamics and using the Hurwitz condition Lemma 1.
Speciﬁcally, we have the Jacobian of the dynamics at the stationary point is:
[Equation]
Take λ =
a + ib ∈ Sp ( −∇ v ( ω ∗ )) , we must have:
[Equation]
[ a + ib − η 2( − a − ib ) 2 ] < 0 , where (cid:60) stands for the real part.
[Equation]
66) then,
[Equation]
67) as desired.
Corollary 1.
For the high resolution ODE of GD with GRL (i.e., Equation (64)) to be asymptotically stable, the learning rate η should be upper bounded by:
[Equation]
for all λ = a + ib ∈ Sp ( −∇ v ( w ∗ )) with large imaginary part (i.e. such that | a | < | b | ).
Proof.
With | a | < | b | where b is the imaginary part of λ ∈ C , an algebraic manipulation of Theorem 2.
i.e. η ( b 2 − a 2 ) <
− 2 a leads to the desired result.
Theorem 3.
The high resolution ODE of any RK2 method up to O ( η ) , ˙ w = − v ( w ) , is asymptotically stable if for all eigenvalues λ
= a + ib ∈ Sp ( −∇ v ( w ∗ )) , we have a < 0 .
Proof.
This theorem can be obtained by computing the Jacobian of the ODE dynamics (see Appendix B for the derivation) and using the Hurwitz condition Lemma 1.
For details, see the proof of Theorem 2.
Unlike the high resolution ODE of GD with GRL in Corollary 1, up to O ( η ) , there is no upper bound constraint on the learning rate.
Therefore, at the cost of an additional extra-gradient computation, we are allowed to take a more aggressive step.
We observe in practice that this leads to both faster convergence and better transfer performance.
Lemma 3.
Suppose the game G ( I , Ω i , J i ) is a potential game with potential function φ .
Suppose the minimizers of φ are either a local minimum or a strict saddle (see Deﬁnition 5).
The gradient play dynamics converges to a local Nash Equilibrium.
Proof.
A potential game can be analyzed in terms of the implicit function φ ( w ) since ∇ φ ( w ) = v ( w ) .
From
Theorem 4, we know the gradient ﬂow in φ ( w ) converges to a point w ∗ that is a local minimum.
Thus, this follows from Proposition 6.
Proposition 4.
The domain-adversarial game is neither a potential nor necessarily a purely adversarial game.
Moreover, its gradient dynamics are not equivalent to the gradient ﬂow.
Proof.
From appendix A.2, a sufﬁcient and necessary condition for a game to be potential is ∇ v ( ω ) being symmetric.
Computing ∇ v ( ω ) using Equation (4) leads to ∇ v ( ω ) being asymmetric, from which the ﬁrst part of the result follows.
Another way to see this, it is to notice a there is no potential function φ satisfying ∇ φ ( ω )
= v ( ω ) (because of the ﬂip in the sign in d s,t ).
Since the game is not a potential game then the vector ﬁeld is not equal to the gradient ﬂow (see also for more details Mazumdar et al.
To see the game is not necessarily a purely adversarial game, it sufﬁces to show an example for which this does not happen.
See Appendix A.3 and example therein.
C. P ROPOSED L EARNING A LGORITHM Algorithm
-Code of the
Learning Algorithm
Input: source data D s , target data D t , player losses J i , network weights { ω i }
i =1 , learning rate η for t = 1 to T − 1 do x s , y s ∼ D s , x t ∼ D t v =
[gradient ( x s , x t , y s , J i ) for i in 1..3 ].
ω =
Gradient Descent ( v, w, η ) ω =
Runge-KuttaSolver ( v, w, η, steps=1 ) end for
[Equation]
We have the update rule of CO Mescheder et al.
[Equation]
69) where 12 ∇|| v ( ω ) ||
= ∇ v ( w ) (cid:62) v ( w )
From RK2 (Heun‘s Method), we have:
[Equation]
If −∇ v ( w ) = ∇ v ( w ) (cid:62) =
⇒ ∇ v ( w ) skew-symmetric, we can see that Equation (71) approximates the CO optimization method from Mescheder et al.
This assumption is however not necessarily true in our case which may explain why RK2 was better in experiments.
Experimentally, for CO, the best transfer performance was obtained for γ = 0 . 0001 .
We also observed that CO is very sensitive to the choice of γ .
D E XPERIMENTAL S ETUP A DDITIONAL D ETAILS
Our algorithm is implemented in Jax Bradbury et al.
(Digits, NLP benchmark) and PyTorch (Visual Task).
Experiments are conducted on a NVIDIA Titan V and V100 GPU Cards.
The digits benchmark constitutes of two digits datasets MNIST and USPS LeCun et al.
Long et al.
(2018) with two transfer tasks (M → U and U → M).
We adopt the splits and evaluation protocol from Long et al.
(2018) which uses 60,000 and 7,291 training images and the standard test set of 10,000 and 2,007 test images for MNIST and USPS, respectively.
We follow the standard implementation details of Long et al.
(2018) for this benchmark.
Therefore, we use LeNet LeCun et al.
(1998) as the backbone architecture, dropout (0.5), ﬁx the batch size to 32, the number of iterations per epoch to 937 and use weight decay (0.005).
For GD-NM, we use the default momentum value (0.9).
We follow the same approach for the additional hyper-parameters of Adam.
Grid Search Details for Comparison vs Standard Optimizers.
For the learning rate, learning schedule and adaptation coefﬁcient ( λ ), we run the following grid search and select the best hyper-parameters for each optimizer.
• learning rate: 0 . 0001 , 0 . 001 , 0 . 01 , 0 .
• learning scheduler: none, polynomial.
• adaptation coefﬁcient ( λ ) : 0 . 1 , 0 . 5 , 1 .
The grid is run using Ray Tune Liaw et al.
(2018) and on the transfer task M → U. To avoid expending resources in bad runs we use a HyperBandScheduler Li et al.
The same hyper-parameters are used for the other task (i.e U → M).
The best criteria correspond to the transfer accuracy (as it is typically done in UDA).
Grid Search Details for Comparison vs Game Optimization Algorithms.
Similar to the previous experiment, for the comparison vs game optimization algorithms, we run the following grid search in order to select their best parameters
for
the comparison.
In the case of CO, we additionally add γ extra-hypermeter to the grid as we experimentally found this optimizer to be sensitive to this parameter.
• learning rate: 0 . 0001 , 0 . 001 , 0 . 006 , 0 . 01 , 0 .
• learning scheduler: none, polynomial.
• Adaptation coefﬁcient ( λ ) : 0 . 1 , 0 . 5 , 1 . • CO Only.
Additional ( γ ): 10 , 1 . 0 , 0 . 1 , 0 . 01 , 0 . 001 , 0 . 001 , 0 .
We show the best results of this grid search in Table 6.
For EG, we observe training diverges for learning rates greater than 0.006.
Experiment on Visda 2017.
For the visual tasks our implementation is built on PyTorch(1.5.1) and on top of the public available implementations of Long et al.
; Zhang et al.
Junguang Jiang (2020).
We did not run grid search here as it would be very computationally expensive.
For the optimizer parameters, we tune thoroughly GD with Nesterov Momentum following the recommendation and implementation from Long et al.
; Zhang et al.
For our method, we keep the exact same hyper-parameters of GD with NM, but increase the learning rate.
We do not experiment with RK4 since the performance was similar to RK2 in the other benchmarks (see Tables 3 and 4), and it is more computationally expensive.
Experiment on Visda 2017 with SoTA algorithms.
For the comparison using SoTA algorithms, we use the well-tuned hyper-parameters from Jiang et al.
and set the discrepancy measure to be f -DAL Pearson as in Acuna et al.
(2021) since this empirically was shown to be better than both JS and γ -JS/MDD.
We use Implicit Alignment Jiang et al.
(2020) as this method also allows to account for the dissimilarity between the labels marginals.
We did not run grid search here as it would be very computationally expensive.
In our case, we only increase the learning rate to 0.2.
Experiment on Natural Language.
In this experiment, we use the Amazon product reviews dataset Blitzer et al.
(2006) that contains online reviews of products collected on the Amazon website.
We follow the splits and evaluation protocol from Courty et al.
; Dhouib et al.
and choose 4 of its subsets corresponding to different product categories, namely: books (B), dvd (D), electronics (E) and kitchen (K).
As in Courty et al.
; Dhouib et al.
(2020), from where we took the baseline result, the network architecture is a two layer MLP with sigmoid function.
In our case, hyper-parameters are kept the same but the learning rate is increased by a factor of 10.
Implementation on different frameworks.
We use both Jax and Pytorch in our paper for simplicity.
Since the proposed method only requires changing some lines of code, it can be easily implemented in different frameworks.
Prototyping in Jax small scale experiments such as the one in the Digits datasets is very easy.
We then use PyTorch in order to integrate our optimizers with large-scale SoTA UDA methods (their codebase was implemented in PyTorch).
This shows the versatility and simplicity of implementation of high-order ODE solvers.
We did not see any issue while working with either Jax or Pytorch.
Our choice of framework was simply based on modifying existing source code to evaluate our optimizer.
Remark on error bars.
For experimental analysis involving a grid-search, we use the same seed and identically initialize the networks.
That is, we give every optimizer the same fair opportunity to win as reported above.
Note that reporting avg and std from a grid-search signiﬁcantly increases the computational demand so we did not do this.
That said, we report avg and std for our main experiments using the best parameters found on the grid-search.
E A DDITIONAL E XPERIMENTS E. N ATURAL L ANGUAGE P ROCESSING T ASKS .
We also evaluate our approach on natural language processing tasks.
We use the Amazon product reviews dataset (Blitzer et al., 2006).
We follow the splits and evaluation protocol from Courty et al.
; Dhouib et al.
and choose 4 of its subsets corresponding to different product categories, namely: books (B), dvd (D), electronics (E) and kitchen (K).
In our case, hyper-parameters are the same provided by authors but the learning rate is increased by a factor of 10.
Table 4 shows noticeable gains by replacing the optimizer with either RK2 or RK4.
[Table]
Reviews Sentiment Analysis Dataset (NLP)
E. S ENSITIVITY TO S AMPLING N OISE
[Table]
[Table]
E. A DDITIONAL C OMPARISON VS G AME O PTIMIZATION A LGORITHMS
In Table 5, we show the sensitivity of our method to sampling Noise controlled by the mini-batch size.
Speciﬁ- cally, we show results for 64 , 128 , 160 ( with 160 being the bigger size we could ﬁt in GPU memory).
We also add results for GD.
We observed that our method performs slightly better when the batch size was bigger.
Table 6 shows the results of the grid-search for the comparison vs game optimization algorithms.
For each optimizer, these results correspond to their best (in terms of transfer performance) determined from the gridsearch explained in Appendix E. We also add GD for comparison.
Our method notably outperforms the baselines.
E. CO VS G RADIENT D ESCENT AND E XTRA -G RADIENT A LGORITHMS
In Table 6 we can observe that RK solvers outperform the baselines.
Interestingly, we can also observe that CO outperforms both EG and GD-NM.
In this section, we provide some intuition about why this may be the case in practice and conduct an additional experimental on this direction.
In the analysis presented in Section 4, we show that a better discretization implies better convergence to a local optimality (assuming ω ∗ exists, and that it is a strict local NE).
Speciﬁcally, under those assumptions, we show that in both GD and EG methods we should put an upper bound to their learning rate (see Section 4 and appendix B.2).
In Appendix C.2, on the other hand, we show CO can be interpreted in the limit as an approximation of the RK2 solver.
Based on this, we believe in practice CO may approximate the continuous dynamics better than GD and EG.
Particularly, if we have an additional hyperparameter ( γ ) to tune thoroughly.
We believe this might also be the reason of why we found CO to be very sensitive to the γ parameter.
In Table 7, we compare the performance before and after removing the best performing γ from the grid search.
We can see for values others than the best γ , CO does not outperform GD-NM either.
[Table]
E. W ALL -C LOCK C OMPARISON
[Table]
In Table 8, we show experimental results of a wall-clock comparison between RK2 and GD-NM.
Speciﬁcally, we compute the average over 100 runs on a NVIDIA GPU: TITAN V, CudaVersion 10.1 and PyTorch version: 1.5.1.
As we can see, the average increase in time is less than 2x slower in wall-clock time.
This is in line with the claims made in our submission.
We additionally emphasize that this time can be improved with more efﬁcient implementations (e.g. Cuda).
Moreover, many more efﬁcient high-order ODE solvers exist and could be used in the future.
Our work also inspires future research in this direction.
F P Y T ORCH P SEUDO C ODE OF RK2 S OLVER
Below, we show how to implement the RK2 solver in PyTorch by simply modifying the existing implementation of SGD.
# Based on https://github.com/pytorch/pytorch/blob/release/1.5/torch/optim/sgd.py class
[Table]
(torch.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
that example: training loop def
[Table]
_fn ): y= loss= return ,y fwd_ _fn()optimizer.update
We want to compute the update equation: # w - n/2* v(w) - n/2 * v(w-n*v(w))
temp_step = {} with torch.no_grad(): for jj_, group in enumerate(self.param_groups):
[Table]
for ii_, p in enumerate(group['params']): if p.grad is None : continue
[Table]
[Table]
[Table]
[Table]
[Table] torch.no_ [Table] [Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table]
[Table]