GDA-AM: O N THE EFFECTIVENESS OF SOLVING MINIMAX OPTIMIZATION VIA A NDERSON M IXING
Anonymous authors
Paper under double-blind review
A BSTRACT
Many modern machine learning algorithms such as generative adversarial networks (GANs) and adversarial training can be formulated as minimax optimization.
Gradient descent ascent (GDA) is the most commonly used algorithm due to its simplicity.
However, GDA can converge to non-optimal minimax points.
We propose a new minimax optimization framework, GDA-AM , that views the GDA dynamics as a ﬁxed-point iteration and solves it using Anderson Mixing to converge to the local minimax.
It addresses the diverging issue of simultaneous GDA and accelerates the convergence of alternating GDA.
We show theoretically that the algorithm can achieve global convergence for bilinear problems under mild conditions.
We also empirically show that GDA-AM solves a variety of minimax problems and improves GAN training on several datasets.
I NTRODUCTION
Minimax optimization has received a surge of interest due to its wide range of applications in modern machine learning, such as generative adversarial networks (GAN), adversarial training and multiagent reinforcement learning (Goodfellow et al., 2014; Madry et al., 2018; Li et al., 2019).
Formally, given a bivariate function f ( x , y ) , the objective is to ﬁnd a stable solution where the players cannot improve their objective, i.e., to ﬁnd the Nash equilibrium of the underlying game (von Neumann &
Morgenstern, 1944):
[Equation]
It is commonplace to use simple algorithms such as gradient descent ascent (GDA) to solve such problems, where both players take a gradient update simultaneously or alternatively.
Despite its simplicity, GDA is known to suffer from a generic issue for minimax optimization: it may cycle around a stable point, exhibit divergent behavior, or converge very slowly since it requires very small learning rates (Gidel et al., 2019a; Mertikopoulos et al., 2019).
Given the widespread usage of gradient-based methods for solving machine learning problems, ﬁrst-order optimization algorithms to solve minimax problems have gained considerable popularity in the last few years.
Algorithms such as optimistic Gradient Descent Ascent (OG) (Daskalakis et al., 2018; Mertikopoulos et al., 2019) and extra-gradient (EG) (Gidel et al., 2019a) can alleviate the issue of GDA for some problems.
Yet, it has been shown that these methods can still diverge or cycle around a stable point (Adolphs et al.;
Mazumdar et al., 2019; Parker-Holder et al., 2020).
For example, these algorithms even fail to ﬁnd a local minimax (the set of local minimax is a superset of local Nash (Jin et al., 2020; Wang et al., 2020)) as shown in Figure 1.
This leads to the following question: Can we design better algorithms for minimax problems?
We answer this in the afﬁrmative, by introducing GDA-AM .
We cast the GDA dynamics as a ﬁxed-point iteration problem and compute the iterates effectively using an advanced nonlinear extrapolation method.
We show that indeed our algorithm has theoretical and empirical guarantees across a broad range of minimax problems, including GANs.
Our contributions: In this paper, we propose a different approach to solve minimax optimization.
Our starting point is to cast the GDA dynamics as a ﬁxed-point iteration.
We then highlight that the ﬁxed-point iteration can be solved effectively by using advanced non-linear extrapolation methods such as Anderson Mixing (Anderson, 1965), which we name as GDA-AM .
Although ﬁrst mentioned in Azizian et al.
(2020), to our best knowledge, this is still the ﬁrst work to investigate and improve the GDA dynamics by tapping into advanced ﬁxed-point algorithms.
[Figure]
We demonstrate that GDA dynamics can beneﬁt from Anderson Mixing.
In particular, we study bilinear games and give a systematic analysis of GDA-AM for both simultaneous and alternating versions of GDA.
We theoretically show that GDA-AM can achieve global convergence guarantees under mild conditions.
We complement our theoretical results with numerical simulations across a variety of minimax problems.
We show that for some convex-concave and non-convex-concave functions, GDA-AM can converge to the optimal point with little hyper-parameter tuning whereas existing ﬁrst-order methods are prone to divergence and cycling behaviors.
We also provide empirical results for GAN training across two different datasets, CIFAR10 and
CelebA.
Given the limited computational overhead of our method, the results suggest that an extrapolation add-on to GDA can lead to signiﬁcant performance gains.
Moreover, the convergence behavior across a variety of problems and the ease-of-use demonstrate the potential of GDA-AM to become the minimax optimization workhorse.
P RELIMINARIES AND BACKGROUND 2.
M
INIMAX OPTIMIZATION
Deﬁnition 1.
Point ( x ∗ , y ∗ ) is a local Nash equilibrium of f if there exists δ > 0 such that for any ( x , y ) satisfying (cid:107) x − x ∗ (cid:107) ≤ δ and (cid:107) y − y ∗ (cid:107) ≤
δ we have: f ( x ∗ , y ) ≤ f ( x ∗ , y ∗ ) ≤ f ( x , y ∗ ) .
To ﬁnd the Nash equilibria, common algorithms including GDA, EG and OG, can be formulated as follows.
For the two variants of GDA, simultaneous GDA (SimGDA) and alternating GDA (AltGDA), the updates have the following forms:
[Equation]
The EG update has the following form:
[Equation]
The OG update has the following form:
[Equation]
2. F IXED -P OINT I TERATION AND A NDERSON M IXING (AM)
Deﬁnition 2.
w (cid:63) is a ﬁxed point of the mapping g
if w (cid:63) = g ( w (cid:63) ) .
Consider the simple ﬁxed-point iteration w t +
= g ( w t ) which produces a sequence of iterates { w 0 , w 1 , · · · , w N } .
In most cases, this converges to the ﬁxed-point, w ∗ =
g ( w ∗ ) .
Take gradient descent as an example, it can be viewed as iteratively applying the operation: w t +1 = g ( w t ) (cid:44) w t − α t ∇ f ( w t ) , where the limit is the ﬁxed-point w (cid:63) =
g ( w (cid:63) )
(i.e. ∇ f ( w t = 0) .
SimGDA updates can be deﬁned as the repeated application of a nonlinear operator:
[Equation]
Similarly, we can write AltGDA updates as w t +1 =
G (alt
[Equation]
η ( w t ) .
An issue with ﬁxed-point iteration is that it does not always converge, and even in the cases where it does converge, it might do so very slowly.
GDA is one example that it could result in the possibility of the operator converging to a limit cycle instead of a single point for the GDA dynamic.
A way of dealing with these problems is to use acceleration methods, which can potentially speed up the convergence process and in some cases even decrease the likelihood for divergence.
There are many different acceleration methods, but we will put our focus on an algorithm which we refer to as Anderson Mixing (or Anderson Acceleration).
In short, Anderson Mixing (AM) shares the same idea as Nesterov’s acceleration.
Given a ﬁxed-point iteration w t = g ( w t − 1 ) , Anderson Mixing argues that a good approximation to the ﬁnal solution w ∗ can be obtained as a linear combination of the previous p iterates w t
+1 =
(cid:80) pi =
β
i g ( w t − p t
+ i ) .
Since obtaining the proper coefﬁcients β i is a nonlinear procedure, Anderson Mixing is also known as a nonlinear extrapolation method.
The general form of Anderson Mixing is shown in Algorithm 1.
For efﬁciency, we prefer a ‘restarted’ version with a small table size p that cleans up the table F every p iterations because it avoids solving a linear system of increasing size.
Input: Initial point w 0 , Anderson restart dimension p , ﬁxed-point mapping g : R n → R n .
Output: w t +1 for t = 0, 1, ... do
Set p t = min { t, p } .
Set F t =
[ f t − p t , . . .
, f t ], where f i = g ( w i ) − w i for each i ∈ [ t − p t . .
t ] .
Determine weights β = ( β 0 , . .
β p t ) T that solves min β (cid:107) F t β (cid:107) 2 , s. t. (cid:80)
p t i =
β i = 1 .
Set w t +
=
(cid:80) p t i =
β
i g ( w t − p t
+ i ) .
end
2. AM AND G ENERALIZED M INIMAL R ESIDUAL (GMRES)
Developed by Saad & Schultz (1986), Generalized Minimal Residual method (GMRES) is a Krylov subspace method for solving linear system equations.
The method approximates the solution by the vector in a Krylov subspace with minimal residual, which is described below.
Deﬁnition 3.
Assume we have the linear system of equations
Ax = b with A ∈ R n × n , b ∈ R n and an initial guess x 0 .
Then we denote the initial residual by r 0 =
b − Ax 0 and deﬁne the t th Krylov subspace as K t = span { r 0 , Ar 0 , · · · , A t − 1 r 0 } .
The t th iterate x t of GMRES minimizes the norm of the residual r t = b −
Ax t in K t , that is, x t solves min
[Equation]
The following formulation is equivalent to GMRES minimization problem and more convenient for implementation.
It computes (cid:98) x
t such that
[Equation]
Using a larger Krylov dimension will improve the convergence of the method, but will require more memory.
For this reason, a smaller Krylov subspace dimension t and ‘restarted’ versions of the method are used in practice Saad (2003).
The convergence of GMRES can be studied through the magnitude of the residual polynomial.
Theorem 2.1 (Lemma 6.31 of Saad (2003)) .
Let (cid:98) x
t be the approximate solution obtained at the t-th iteration of GMRES being applied to solve Ax = b , and denote the residual as r t = b − A (cid:98) x t .
Then, r t is of the form
[Equation]
where
[Equation]
where P p is the family of polynomials with degree p such that f p (0) = 1 , ∀ f p ∈ P p , which are usually called residual polynomials.
Although GMRES is applied to a system of linear equations not a ﬁxed-point problem, there is a strong connection between Anderson Mixing and GMRES.
In AM we are looking for a ﬁxed-point x such that Gx − b − x = 0 and by rearranging this equation we get
[Equation]
Theorem 2.2 shows that if GMRES is applied to the system ( I − G ) x
= b and AM is applied to g ( x ) =
Gx + b with the same initial guess
and I − G is non-singular, then these are equivalent in the sense that the iterates of each algorithm can be obtained directly from the iterates of the other algorithm.
Theorem 2.2 (Equivalence between AM with restart and GMRES (Walker & Ni, 2011a)) .
Consider the ﬁxed point iteration x = g ( x ) where g ( x ) =
Gx + b for G ∈ R n × n and b ∈ R n .
If I − G is non-singular, Algorithm 1 produces exactly the same iterates as GMRES being applied to solve ( I − G ) x
= b when both algorithms start with the same initial guess.
Theorem 2.2 can also be generalized to the restart version of AM an GMRES as well.
[Equation]
GDA WITH A NDERSON M IXING
We propose a novel minimax optimizer, called GDA-AM , that is inspired by recent advances in parameter (or weight) averaging (Wu et al., 2020; Yazici et al., 2019).
We argue that a nonlinear adaptive average (combination) is a more appropriate choice for minimax optimization.
3. GDA WITH N A ¨ IVE A NDERSON M IXING
We propose to exploit the dynamic information present in the GDA iterates to “smartly” combine the past iterates.
This is in contrast to the classical averaging methods (moving averaging and exponential moving averaging)
(Yang et al., 2019) that “blindly” combine past iterates.
A na¨ıve adoption of
Anderson Mixing using the past p GDA iterates for both simGDA and altGDA has the following form:
[Equation]
Since Zhang et al.
; Gidel et al.
(2019b) show the AltGDA is superior to SimGDA in many aspects, we brieﬂy summarized both Simultaneous and Alternating GDA-AM in Algorithms 2 andwith the truncated Anderson Mixing Algorithm 1 using a table size p .
Input: x 0 , y 0 , stepsize η , Anderson table size p
Output: x t , y t
Set w 0 =
[ x 0 , y 0 ] , sx = length ( x 0 ) for t = 0, 1, ... do x t , y t
= w t [0 : sx − 1] , w t [ sx : end ] x t +1 = x
t − η ∇ x f ( x t , y t )
y t +1 =
y t − η ∇ y f
( x t , y t ) w t +1 =
(cid:20) x t + y t
+ (cid:21)
Use Anderson Mixing with table size p to extrapolate w t +1 end x t , y t =
w t +1 [0 : sx − 1] , w t +1 [ sx : end ] return x t , y t
Algorithm
Input: x 0 , y 0 , stepsize η , Anderson table size p
Output: x t , y t
Set w 0 =
[ x 0 , y 0 ] , sx = length ( x 0 ) for t = 0, 1, ... do x t , y t
= w t [0 : sx − 1] , w t [ sx : end ] x t +1 = x
t − η ∇ x f ( x t , y t )
y t +1 =
y t − η ∇ y f ( x t +1 , y t )
w t +1 =
(cid:20) x t + y t
+ (cid:21)
Use Anderson Mixing with table size p to extrapolate w t +1 end x t , y t =
w t +1 [0 : sx − 1] , w t +1 [ sx : end ] return x t , y t
3. I MPLEMENTATION DETAILS
It is important to note that the Anderson Mixing form shown in Algorithm 1 is for illustrative purpose and not computationally efﬁcient.
For example, only one column of F t needs to be updated at each iteration.
In addition, the solution of the least-square problem in Algorithm 1 can also be solved by a quick QR update scheme which costs (2 n + 1) p 2 (Walker & Ni, 2011a).
Thus, from Algorithms 2 and 3, we can see that the major cost of GDA-AM arises from solving the additional linear least squares problem compared to regular GDA at each iteration.
Additional implementation details are provided in the Appendix.
C ONVERGENCE RESULTS FOR GDA-AM
In this section, we show that both simultaneous and alternating version GDA-AM converge to the equilibrium for bilinear problems.
First, we do not require the learning rate to be sufﬁciently small.
Second, we explicitly provide a linear convergence rate that is faster than EG and OG.
4. B ILINEAR G AMES
Bilinear games are often regarded as an important simple example for theoretically analyzing and understanding new algorithms and techniques for solving general minimax problems (Gidel et al., 2019a; Mertikopoulos et al., 2019; Schaefer & Anandkumar, 2019).
In this section, we analyze the convergence property of simultaneous GDA-AM and alternating GDA-AM schemes on the following zero-sum bilinear games:
[Equation]
The Nash equilibrium to the above problem is given by ( x ∗ , y ∗ ) =
( − A − T c , − A − 1 b ) .
We also investigate bilinear-quadratic games from a spectrum analysis perspective.
In addition, we show that analysis based on the numerical range (Bollapragada et al., 2018) can be also extended to such games, although it can not help derive a convergent bound for equation 7.
Detailed discussion can be found in Appendix B.1 and B.4.1.
4. S IMULTANEOUS GDA-AM
Suppose x 0 and y 0 are the initial guesses for x ∗ and y ∗ , respectively.
Then each iteration of simultaneous GDA can be written in the following matrix form:
[Equation]
I
[Equation]
η
[Equation]
[Equation] It has been shown that the iteration in equation 8 often cycles and fails to converge for the bilinear problem due to the poor spectrum/numerical range of the ﬁxed point operator G ( Sim ) (Gidel et al., 2019a; Azizian et al., 2020; Mokhtari et al., 2020a).
Next we show that the convergence can be improved with Algorithm 2.
Theorem 4.1.
[Global convergence for simultaneous GDA-AM on bilinear problem]
Denote the distance between the stationary point w ∗ and current iterate w ( k +1) p of Algorithm 2 with table size p as N [Equation] [Equation]
=
[Equation] − [Equation]
k [Equation] p [Equation]
Then we have the following bound for N t
[Equation]
where ρ ( A ) =
[Equation]
Here, T p is the Chebyshev polynomial of ﬁrst kind of degree p and
[Equation]
1 +
[Equation]
It is worthy emphasizing that the convergence rate of Algorithm 2 is independent of learning rate η while the convergence results of other methods like EG and OG depend on the learning rate.
Remark 4.1.1.
Both EG and OG have the following form of convergence rate (Mokhtari et al., 2020a) for bilinear problem
[Equation]
where c is a positive constant independent of the problem parameters.
[Figure]
4. A LTERNATING GDA-AM
The underlying ﬁxed point iteration in Algorithm 3 can be written in the following matrix form:
[Equation]
According to the equivalence between truncated Anderson acceleration and GMRES with restart
[Equation]
we can analyze the convergence of Algorithm 3 through the convergence analysis of applying GMRES to solve linear systems associated with G
=
I − G ( Alt ) :
[Equation]
Theorem 4.2.
[Global convergence for alternating GDA-AM on bilinear problem]
Denote the distance between the stationary point w ∗ and current iterate w ( k +1) p of Algorithm 3 with table size p as
N ( k +1) p
= (cid:107) w ∗ − w
k +1) p (cid:107) .
Assume A is normalized such that its largest singular value is equal to 1 .
Then when the learning rate η is less than 2 , we have the following bound for N t
[Equation]
where c and r are the center and radius of a disk D ( c, r ) which includes all the eigenvalues of G .
Especially, rc < 1 .
Theorem 4.2 shows that when p > log
[Figure]
, alternating GDA-AM will converge globally.
R ELATED WORK
There is a rich literature on different strategies to alleviate the issue of minimax optimization.
A useful add-on technique, Momentum, has been shown to be effective for bilinear games and strongly-convexstrongly-concave settings (Zhang & Wang, 2021;
Gidel et al., 2019b; Azizian et al., 2020).
Several second-order methods (Adolphs et al.; Mescheder et al., 2017; Mazumdar et al., 2019; Parker-Holder et al., 2020) show that their stable ﬁxed points are exactly either Nash equilibria or local minimax by incorporating second-order information.
However, such methods are computationally expensive and thus unsuitable for large applications such as image generation.
Focusing on variants of GDA,
EG and OG are two widely studied algorithms on improving the GDA dynamics.
EG proposed to apply extra-gradient to overcome the cycling behaviour of GDA.
OG, originally proposed in Popov (1980) and rediscovered in Daskalakis et al.
; Mertikopoulos et al.
(2019), is more efﬁcient by storing and re-using the extrapolated gradient for the extrapolation step.
Without projection, OG is equivalent to extrapolation from past.
Mokhtari et al.
(2020b) shows that both of these algorithms can be interpreted as approximations of the classical proximal point method and did a uniﬁed analysis for bilinear games.
These approaches mentioned the GDA dynamics can be viewed as a ﬁxed-point iteration, but none of them further provides a solution to improve it.
In this work, we ﬁll this gap by proposing the application of the extrapolation method directly on the entire GDA dynamics.
Unlike OG, EG and their variants (Hsieh et al., 2019; Lei et al., 2021; Thekumparampil et al., 2019; Yang et al., 2019), which regard minimax problems as variational inequality problems (Bruck, 1977; Nemirovski, 2004), our work is from a new perspective and thus orthogonal to these previous approaches.
In addition, several recent works consider nonconvex-concave minimax problems.
Zhang et al.
(2020) introduced a “smoothing” scheme combined with GDA to stabilize the dynamic of GDA.
Luo et al.
(2020) proposed a method called Stochastic Recursive gradiEnt Descent Ascent (SREDA) for stochastic nonconvex-strongly-concave minimax problems, by estimating gradients recursively and reducing its variance.
Lin et al.
(2020) showed that the two-timescale GDA can ﬁnd a stationary point of nonconvex-concave minimax problems effectively.
Ostrovskii et al.
(2021) proposed a variant of
Nesterov’s accelerated algorithm to ﬁnd (cid:15) -ﬁrst-order Nash equilibrium that is a stronger criterion than the commonly used proximal gradient norm.
Nouiehed et al.
(2019) proposed a iterative method that ﬁnds (cid:15) -ﬁrst-order Nash equilibrium in O ( (cid:15) − 2 ) iterations under Polyak-Lojasiewicz (PL) condition.
Focusing on nonconvex minimax problems, they studied an interesting and difﬁcult problem.
Since our work cast insight on the effectiveness of solving minimax optimization via Anderson Mixing, we expect the extension of this algorithm to general nonconvex problems can be further investigated in the future.
E XPERIMENTS
In this section, we conduct experiments to see whether GDA-AM improves GDA for minimax optimization from simple to practical problems.
We ﬁrst investigate performance of GDA-AM on bilinear games.
In addition, we evaluate the efﬁcacy of our approach on GANs.
6. B ILINEAR P ROBLEMS
In this section, we answer following questions: Q1:
How is GDA-AM perform in terms of iteration number and running time?
Q2:
How is the scalability of GDA-AM ?
Q3:
How is the performance of
GDA-AM using different table size p ?
Q4: Does GDA-AM converge for large step size η ?
We compare the performance with SimGDA, AltGDA, EG, and OG, and EG with Negative Momentum(Azizian et al.
on bilinear minimax games shown in equation 7 without any constraint.
A , b , c , and initial points are generated using normally distributed random number.
We set the maximum iteration number as 1 × 10 6 , stopping criteria 1 × 10 − 5 and depict convergence by use of the norm of distance to optima, which is deﬁned as (cid:107) w ∗ − w t (cid:107) .
Similar to Azizian et al.
Wei et al.
(2021a), the step size is set as 1 after rescaling A to have 2-norm 1.
We present results of different settings in Figures 4, 5, and 6.
We ﬁrst generate different problem size ( n = 100 , 1000 , 5000 ) and present results of convergence in terms of iteration number in Figure 4.
It can be observed that GDA-AM converges in much fewer iterations for different problem sizes.
Note that EG, EG-NM, and OG converge in the end but requires many iterations, thus we plot only a portion for illustrative purposes.
Figure 5 depicts the convergence for all methods in terms of time.
It can be observed that the running time of GDA-AM is faster than EG.
Although slower than OG, we can observe GDA-AM converges in much less time for all problems.
Figure 4 and Figure 5 answer Q1 and Q2; although there is additional computation for
GDA-AM , it does not hinder the beneﬁts of adopting Anderson Mixing.
Even for a large problem size, GDA-AM still converges in much less time than the baselines.
Next, we run GDA-AM using different table size p and show the results in Figure 6a and Figure 6b.
Figure 6a indicates an increasing of table size results in faster convergence in terms of iteration number, which also veriﬁes our claim in Theorem 4.1.
However, we also observe an increased running time when using a larger table size in Figure 6b.
Further, we can see that p = 50 converges in a comparable time and iterations to p = 100 .
Similar results are found in repeated experiments as well.
As a result, our answer to Q3 is that although a larger p means less iterations, a medium p is sufﬁcient and a small p still outperforms the baselines.
The optimal choice of p is related to the condition number and step size, which is another interesting topic in the Anderson Mixing community.
Next, we answer Q4 on convergence under different step sizes.
Although GDA-AM usually converges with suitable step size, our theorem suggests it requires a larger table size when combined with a extremely aggressive step size.
Figure 6c shows the convergence under such circumstance.
We can observe that although a very large step size goes the wrong way in the beginning, Anderson Mixing can still make it back on track except when η > 1 .
It answers the question and conﬁrms our claim that GDA-AM can achieve global convergence for bilinear problems for a large step size η > 0 .
[Figure]
6. GAN E XPERIMENTS : I MAGE G ENERATION
We apply our method to the CIFAR10 dataset (Krizhevsky, 2009) and use the ResNet architecture with
WGAN-GP (Gulrajani et al., 2017) and SNGAN (Miyato et al., 2018) objective.
We also compared
[Figure] [Figure]
the performance of GDA-AM using cropped CelebA (64 × 64 ) (Liu et al., 2015) on WGAN-GP.
We compare with Adam and extra-gradient with Adam (EG) as it offers signiﬁcant improvement over
OG.
Models are evaluated using the inception score (IS) (Salimans et al., 2016) and FID (Heusel et al., 2017) computed on 50,000 samples.
For fair comparison, we ﬁxed the same hyperparamters of Adam for all methods after an extensive search.
Experiments were run with 5 random seeds.
We show results in Table 1.
Table 1 reports the best IS and FID (averaged over 5 runs) achieved on these datasets by each method.
We see that GDA-AM yields improvements over the baselines in terms of generation quality.
[Table]
C ONCLUSION
We prove the convergence property of GDA-AM and obtain a faster convergence rate than EG and
OG on the bilinear problem.
Empirically, we verify our claim for such a problem and show the efﬁcacy of GDA-AM in a deep learning setting as well.
We believe our work is different from previous approaches and takes an important step towards understanding and improving minimax optimization by exploiting the GDA dynamic and reforming it with numerical techniques.
A A NDERSON M IXING I MPLEMENTATION D ETAILS
In this section, we discuss the efﬁcient implementation of Anderson Mixing.
We start with generic
Anderson Mixing prototype (Algorithm 4) and then present the idea of Quick QR-update Anderson
Mixing implementation as described in Walker & Ni (2011b), which is commonly used in practice.
For each iteration t ≥ 0 , AM prototype solves a least squares problem with a normalization constraint.
The intuition is to minimize the norm of the weighted residuals of the previous m iterates.
Input: Initial point w 0 , Anderson restart dimension p , ﬁxed-point mapping g : R n → R n .
Output: w t +1 for t = 0, 1, ... do
Set p t = min { t, p } .
Set F t =
[ f t − p t , . . .
, f t ], where f i = g ( w i ) − w i for each i ∈ [ t − p t . .
t ] .
Determine weights β = ( β 0 , . .
β p t ) T that solves min β (cid:107) F t β (cid:107) 2 , s. t. (cid:80)
p t i =
β i = 1 .
Set w t +
=
(cid:80) p t i =
β
i g ( w t − p t
+ i ) .
end
The constrained linear least-squares problem in Algorithm AA can be solved in a number of ways.
Our preference is to recast it in an unconstrained form suggested in Fang & Saad (2009);
Walker &
Ni (2011b) that is straightforward to solve and convenient for implementing efﬁcient updating of QR.
Deﬁne f i = g ( w i ) − w i , (cid:52) f
i = f
i +1 − f i for each i and set F t =
[ f t − p t , . . .
, f t ], F t =
[ (cid:52) f t − p t , . . .
, (cid:52) f t ] .
Then solving the least-squares problem ( min β (cid:107)
F t β
[Equation]
is equivalent to
[Equation]
where α and γ are related by α 0 = γ 0 , α i
= γ
i − γ
i − 1 for 1 ≤ i ≤ p t − 1 , and α p t = 1 − γ p t − 1 .
Now the inner minimization subproblem can be efﬁciently solved as an unconstrained least squares problem by a simple variable elimination.
This unconstrained least-squares problem leads to a modiﬁed form of Anderson Mixing
[Equation]
where G t =
[ (cid:52) g t − p t , . . .
, (cid:52) g t − 1 ] with (cid:52) g
i =
g ( w i +1 )
− g ( w i ) for each i .
To obtain γ ( t ) = (cid:16) γ ( t ) 0 , . .
. , γ ( t ) p t − 1 (cid:17) T by solving equation 10 efﬁciently, we show how the successive least-squares problems can be solved efﬁciently by updating the factors in the QR decomposition
F t = Q t R t as the algorithm proceeds.
We assume a think QR decomposition, for which the solution of the least-squares problem is obtained by solving the p t × p t linear system Rγ = Q (cid:48) ∗ f t .
Each F t is n × p t and is obtained from F t − 1 by adding a column on the right and, if the resulting number of columns is greater than p , also cleaning up (re-initialize) the table.
That is,we never need to delete the left column because cleaning up the table stands for a restarted version of AM.
As a result, we only need to handle two cases; 1 the table is empty(cleaned).
2 the table is not full.
When the table is empty, we initialize F 1 = Q 1 R 1 with Q 1 = (cid:52) f 0 / (cid:107)(cid:52) f
(cid:107) 2 and R = (cid:107)(cid:52) f 0 (cid:107) 2 .
If the table size is smaller than p , we add a column on the right of F t − 1 .
Have F t − 1 =
QR , we update Q and R so that
F t =
[ F t − 1 , ∆ f t − 1 ] = QR .
It is a single modiﬁed Gram–Schmidt sweep that is described as follows:
Algorithm for i = 1 , . .
. , p t − 1 do
Set R ( i, p t ) =
Q (: , i ) (cid:48) ∗ (cid:52)
f t − 1 .
Update (cid:52) f
t − 1 ← ∆ f t − 1 − R
( i, p t ) ∗ Q (: , i ) end
Set Q (: , p t ) = (cid:52) f
t − 1 / (cid:107)(cid:52) f t − 1 (cid:107) 2 and R ( p t , p t ) = (cid:107) ∆ f t − 1 (cid:107)
Note that we do not explicitly conduct QR decomposition in each iteration, instead we update the factors ( O ( p 2 n ) ) and then solve a linear system using back substitution which has a complexity of
O ( p 2 ) .
Based on this complexity analysis, we can ﬁnd Anderson
Mixing with QR-updating scheme has limited computational overhead than GDA (or OG).
This explains why GDA-AM is faster than
EG but slower than OG in terms of running time of each iteration.
B T HEORETICAL R ESULTS B. D IFFICULTY OF ANALYSIS ON GDA WITH A NDERSON M IXING
In the analysis, we study the inherent structures of the dynamics of the ﬁxed point iteration and provide the convergence analysis for both simultaneous and alternating schemes.
We want to emphasize that the direct application of existing convergence results of GMRES can not lead to convergent results.
A recent paper Bollapragada et al.
(2018) study the convergence acceleration schemes for multi-step optimization algorithms using Regularized Nonlinear Acceleration.
We also want to point out that a na¨ıve application of Crouzeix’s bound to the minimax optimization problem can not be used to derive the convergent result.
Theorem B.1 (Fischer & Freund (1991)) .
Let n ≥ 5 be an integer, r > 1 , and c ∈ R .
Consider the following constrained polynomial minmax problem
[Equation]
11) where
[Equation]
(12) and c ∈ C \ E r .
Then this problem can be solved uniquely by
[Equation] [Equation]
) where
[Equation]
14) if
[Equation]
a
[Equation]
b
[Equation]
, where a r := 12 (cid:0) r + 1 r (cid:1) .
This is because the point 0 where all the residual polynomials take the ﬁxed value of 1 is included in the numerical range of the iteration matrix, which violates the assumption of Theorem B.1.
As a result, it can not be used to prove that the residual norm is decreasing based on this approach.
Instead, we show that although the coefﬁcient matrix is non-normal, it is diagonalizable.
We then give the convergence results based on the eigenvalues instead of the numerical range.
More speciﬁcally,
Anderson mixing is equivalent to GMRES being applied to solve the following linear system:
[Equation]
Writing this linear system in the block form:
[Equation]
The residual norm bound for GMRES reads:
[Equation]
Notice that the matrix ( I − G ( Alt ) ) is non-normal.
If we apply Crouzeix’s bound in Crouzeix &
Palencia (2017) to our problem as Bollapragada et al.
(2018) did, then we have the following bound
[Equation] [Equation]
where W ( I − G ( Alt ) )
= { z ∗ ( I − G ( Alt ) )
z , ∀ z ∈ C 2 n \ { 0 } , (cid:107) z (cid:107) = 1 } is the numerical range for
I − G ( Alt ) .
In order to simplify the upper bound in the previous theorem, we study the numerical range of I − G ( Alt ) similar to Bollapragada et al.
Writing z
=
(cid:20) z
1 z 2 (cid:21) and computing the numerical range of I − G ( Alt ) explicitly yields:
[Equation]
For a general matrix A , there is no special structure about the numerical range of I − G ( Alt ) .
However, when A is symmetric, we can decompose A as A = (cid:80) ni
=
λ i v i v Ti where { λ i } ni =1 are eigenvalues of A in decreasing order and
{ v i } ni =1 are associated eigenvectors, and write A T A
= (cid:80) ni
=
λ 2 i v i
v Ti .
Then we can compute the numerical range of G ( Alt ) as follows:
[Equation]
Following the techniques proposed in Bollapragada et al.
(2018) to analyze the numerical range of general 2 × 2 matrices, we can show that the numerical range of I − G ( Alt ) is equal to the convex hull of the union of the numerical range of
[Equation]
And the boundary of numerical range of G i is an ellipse whose axes are the line segments joining the points x to y and w to z, respectively, with
[Equation]
[Equation]
Thus, the numerical range of I − G ( Alt ) can be spanned by convex hull of the union of the numerical range of a set of 2-by-2 matrices and the numerical range of each such a 2-by-2 matrix is an ellipse.
We can compute the center o and focal distance d of the ellipse generated by numerical range of
I − G ( Alt ) explicitly.
Then a linear transformation enables us to use Theorem B.1 to show that the near-best polynomial for the minimax problem on the numerical range of I − G ( Alt ) is given by t n ( z ; c ) :
= T n ( z − od ) T n ( c − od ) if 0 is excluded from the numerical range of I − G ( Alt ) .
However, according to equation 22 the numerical range includes the point 0 where the residual polynomial takes value 1, thus the analysis based on numerical range can not help derive the convergent result as the upper bound is not guaranteed to be less than 1.
B. P ROOFS OF THEOREM
We ﬁrst provide proof of Theorem 4.1.
Theorem B.2 (Global convergence for simultaneous GDA-AM on bilinear problem) .
Denote the distance between the stationary point w ∗ and current iterate w ( k +1) p of Algorithm 2 with Anderson restart dimension p as N ( k +1) p = dist ( w ∗ , w ( k +1) p ) .
Then we have the following bound for N t
Algorithm 2 is unconditionally convergent
[Equation]
where T p is the Chebyshev polynomial of ﬁrst kind of degree p and 1 T p (1+ 2 κ ( A T A ) − 1 ) < 1 since 1 + 2 κ ( A T A ) − 1 > 1 .
Proof of Theorem 4.1 .
Note that I − G ( Sim ) is a normal matrix which will be denoted as G for notational simplicity.
Thus it admits the following eigendecomposition:
[Equation] [Equation]
Based on the equivalence between GMRES and Anderson Mixing, we know that the convergence rate of simultaneous GDA-AM can be estimated by the spectrum of G .
Especially, it holds that
[Equation]
where P p is the family of residual polynomials with degree p such that f p (0) = 1 , ∀ f p ∈ P p .
According to Lemma 2.1, we have the following estimation
[Equation]
Due to the block structure of G , the eigenvalues of G can be computed explicitly as
[Equation]
where σ i is the i th largest singular value of matrix A .
This shows that the eigenvalues of G are n pairs of purely imaginary numbers excluding 0 since A has full rank.
Since the eigenvalues of G are distributed in two intervals excluding the origin
[Equation]
it can be shown that the following p-th degree polynomial with value 1 at the origin that has the minimal maximum deviation from 0 on I is given by:
[Equation] [Equation]
where l =
[ p 2 ] and T l is the Chebyshev polynomial of ﬁrst kind of degree l .
The function q ( √− 1 z ) maps I to [ − 1 , 1] .
Thus the numerator of the polynomial f p is bounded by 1 on I.
The size of denominator can be determined by the method discussed in Chapter 3 of Greenbaum (1997).
Assume q (0) =
12 ( y + y − 1 ) , then T l ( q (0))
=
12 ( y l + y − l ) .
Then y can be determined by solving
[Equation]
The solutions to this equation are
[Equation]
Then plugging the value of q (0) into the polynomial f p yields
[Equation]
Note that N t and r t is related through G ( w t − w ∗ ) =
r t .
Therefore,
[Equation]
Actually a tighter bound can be proved after noting that the problem is essentially equivalent to polynomial minmax problem on the interval:
[Equation]
Then it is well known that,
[Equation]
where T p Chebyshev polynomial of degree p of the ﬁrst kind and
[Equation]
Explicitly
[Equation]
Next, we give the proof of Theorem 4.2.
Theorem B.3 (Global convergence for alternating GDA-AM on bilinear problem) .
Denote the distance between the stationary point w ∗ and current iterate w ( k +1) p of Algorithm 3 with Anderson restart dimension p as N ( k +1) p = dist ( w ∗ , w ( k +1) p ) .
Assume A is normalized such that its largest singular value is equal to 1 .
Then when the learning rate η is less than 2 , we have the following bound for N t
[Equation]
where c and r are the center and radius of a disk D ( c, r ) which includes all the eigenvalues of G in equation 4.3.
Especially, rc < 1 .
Proof.
Since the residual r p of AA at p-th iteration has the form of r p
[Equation]
I
[Equation]
and AA minimizes the residual, we have
[Equation]
where P p is the family of polynomials with degree p such that
f p (0) = 1 , ∀ f p ∈ P p .
It’s easy to see that G is unitarily similar to a block diagonal matrix Λ with 2 × 2 blocks as follows:
[Equation]
Thus the eigenvalues of G can be easily identiﬁed as
[Equation]
where σ 1 ≥ σ 2 ≥ · · · ≥ σ n are the singular values of A .
Furthermore, the eigenvector and eigenvalue associated with each 2 × 2 diagonal block are
[Equation]
Thus G is diagonalizable and denote the matrix with the columns of eigenvectors of G by X.
The real part of the eigenvalues of G are at least
R
[Equation]
And since | ησ i | ≥ | (cid:112) ( ησ i ) 2 − 4) | , all the eigenvalues will be included in a disk D ( c, r ) which is included in the right half plane.
Moreover, both c and r being greater than zero indicates that rc < 1 .
Start from the following inequality:
[Equation]
We will use the eigendeomposition of G and the special polynomial ( c − t c ) p to derive the inequality in Theorem 3.
Now we know rc < 1 .
If we choose g p ( t ) =
( c − t c ) p , we can obtain
[Equation]
which implies
[Equation]
Since G is diagonalizable (which has been shown above), we assume the eigendecomposition of G is
G = X Λ X − 1 .
Then
[Equation]
where κ G is the condition number of X .
The last inequality comes from Lemma 6.26 and Proposition 6.32 in Saad (2003)..
Since G and Λ are unitarily similar, κ G is equal to the condition number of the eigenvector matrix of Λ .
The eigenvector matrix of Λ is a block diagonal matrix with the i th block as
[Equation]
(cid:21) .
Thus the singluar values of the eigenvector matrix of Λ is equal to the union of the singular values of these 2-by-2 blocks.
Under the assumption that the largest singular value of A are equal to 1 and the learning rate is less than 2 , it is easy to ﬁnd the singular values of the eigenvector matrix of Λ are √ 2 ± ησ i .
Thus, κ G = √ 2+ ησ max √ 2 − ησ max = √ 2+ η √ 2 − η = (cid:113) 1 + 2 η 2 − η .
B. D ISCUSSION OF OBTAINED RATES
We would like to ﬁrst explain on why taking Chebyshev polynomial of degree p at the point
+ 2 κ − 1 .
We evaluate the Chebyshev polynomial at this speciﬁc point because the reciprocal of this value gives the minimal value of inﬁnite norm of the all polynomials of degree p deﬁned on the interval ˜
I =
[ η 2 σ 2 min ( A ) , η 2 σ 2 max ( A )] based on Theorem 6.25 (page 209) (Saad, 2003).
In other words, taking the function value at this point leads to the tight bound.
When comparing between existing bounds, we would like to point our our derived bounds are hard to compare directly.
Alternatively, we can derive another bound for comparison with existing bounds for simultaneous GDA-AM .
If we use the inequality that T p ( t ) ≥ 12 (( t + √ t 2 − 1) p ) , we can obtain the bound ρ ( A ) = 4( √
[Equation] [Equation]
√ κ ( A
[Equation]
A )+1 ) 2 = 4(1 − O ( 1 √ κ ( A T A ) )) , which is in a form that is comparable with EG and can compete with EG + positive momentum.
The numerical experiments in ﬁgure 2b numerically verify that our bound is smaller than EG.
We wanted to numerically compare our rate with EG with positive momentum.
However the bound of EG with positive momentum is asymptotic.
Moreover, it does not specify the constants so we can not numerically compare them.
We do provide empirical comparison between GDA-AM and EG with positive momentum for bilinear problems in
Appendix C.1.
It shows GDA-AM outperforms EG with positive momentum.
Regarding alternating
GDA-AM , we would like to note that the bound in Theorem 4.2 depends on the eigenvalue distribution of the matrix G .
Condition number is not directly related to the distribution of eigenvalues of a nonsymmetric matrix G .
Thus, the condition number is not a precise metric to characterize the convergence.
If these eigenvalues are clustered, then our bound can be small.
On the other hand, if these eigenvalues are evenly distributed in the complex plane, then the bound can very close to 1 .
More importantly, we would like to stress several technical contributions.
Our obtained Theorem 4.1 and 4.2 provide nonasymptotic guarantees, while most other work are asymptotic.
For example, EG with positive momentum can achieve a asymptotic rate of 1 − O (1 / √ κ ) under strong assumptions (Azizian et al., 2020).
Our contribution is not just about ﬁx the convergence issue of GDA by applying Anderson Mixing; another contribution is that we arrive at a convergent and tight bound on the original work and not just adopting existing analyses.
We developed Theorem 4.1 and 4.2 from a new perspective because applying existing theoretical results fail to give us neither convergent nor tight bounds.
Theorem 4.1 and 4.2 only requires mild conditions and reﬂects how the table size p controls the convergence rate.
Theorem 4.1 is independent of the learning rate η .
However, the convergence results of other methods like EG and OG depend on the learning rate, which may yield less than desirable results for ill-speciﬁed learning rates.
B. C ONVEX - CONCAVE AND GENERAL CASE
Given the widespread usage of minimax problems in applications of machine learning, it is natural to ask about its properties when being applied to general nonconvex-nonconcave settings.
If f is a nonconvex-nonconcave function, the problem of ﬁnding global Nash equilibrium is NP-hard in general.
Recently, Jin et al.
(2020) show that local or global Nash equilibrium may not exist in nonconvex-nonconcave settings and propose a new notation local minimax as deﬁned below:
Deﬁnition 4.
A point ( x (cid:63) , y (cid:63) ) is said to be a local minimax point of f , if there exists δ 0 > 0 and a function h satisfying h ( δ ) → 0 as δ → 0 , such that for any δ ∈ (0 , δ 0 ] , and any ( x , y ) satisfying (cid:107) x − x (cid:63) (cid:107) ≤ δ and (cid:107) y − y (cid:63) (cid:107) ≤ δ , we have
[Equation]
Jin et al.
(2020) also establishes the following ﬁrst- and second-order conditions to characterize local minimax:
Proposition 1 (First-order Condition) .
Any local minimax point ( x ∗ , y ∗ ) satisﬁes ∇ f ( x ∗ , y ∗ )
= 0 .
Proposition 2 (Second-order Necessary Condition) .
Any local minimax point ( x ∗ , y ∗ ) satisﬁes ∇ yy f ( x ∗ , y ∗ ) (cid:52) 0 and ∇ xx f ( x ∗ , y ∗ ) − ∇ xy f ( x ∗ , y ∗ )( ∇ yy f ( x ∗ , y ∗ )) − 1 ∇ yx f ( x ∗ , y ∗ ) (cid:60) 0 .
Proposition 3 (Second-order Sufﬁcient Condition) .
Any stationary point ( x ∗ , y ∗ ) satisﬁes ∇ yy f ( x ∗ , y ∗ ) ≺ 0 and ∇ xx f ( x ∗ , y ∗ ) − ∇ xy f ( x ∗ , y ∗ )( ∇ yy f ( x ∗ , y ∗ )) − 1 ∇ yx f ( x ∗ , y ∗ ) (cid:31) 0 is a local minimax point.
Given the second-order conditions of local minimax, it turns out that above question is extremely challenging— GDA-AM is a ﬁrst-order method.
But we can prove the following result for GDA-AM :
Theorem B.4 (Local minimax as subset of limiting points of GDA-AM ) .
Consider a general objective function f ( x , y ) .
The set of limiting points of GDA-AM for minimax problem
[Equation]
includes the local minimax points of this function.
The deﬁnition of local minimax is stronger than that of ﬁrst order (cid:15) point.
The convergence analysis for complexity of ﬁnding (cid:15) stationary point is included in the next section.
The proof of Theorem B. needs the result from the following theorem.
Theorem B.5 (Calvetti et al.
Let δ satisfy 0 < δ ≤ δ 0 for some constant δ 0 > 0 (refer to
Calvetti et al.
for details), and let b δ ∈ X satisfy (cid:13)(cid:13)
b − b δ (cid:13)(cid:13) ≤ δ .
Let k ≤ (cid:96) and let x δk denote the kth iterate determined by the GMRES method applied to equation
Ax = b δ , with initial guess x δ 0 = 0 .
Similarly, let x k denote the kth iterate determined by the GMRES method applied to equation
Ax = b with initial guess x 0 = 0 .
Then, there are constants σ k independent of δ , such that
[Equation]
Then, we give the proof of Theorem B.4.
Proof of Theorem B.4 .
For notational simplicity, we will denote ∇ xx f ( x ∗ , y ∗ ) , ∇ xy f ( x ∗ , y ∗ ) and ∇ yy f ( x ∗ , y ∗ ) by H x ∗ x ∗ , H x ∗ y ∗ and H y ∗ y ∗ , respectively.
Simultaneous GDA can be written as
[Equation]
Since the function is differentiable, Taylor expansion holds for ∇ x f ( x t , y t ) and ∇ y f ( x t , y t ) at a local minimx point w ∗ =
( x ∗ , y ∗ ) ,
[Equation]
Use the fact that ∇ f ( x ∗ , y ∗ ) = 0 to simplify the above equations and obtain
[Equation]
Inserting the above formulas into the iteration scheme
[Equation]
it yields
[Equation]
where (cid:15) denotes the higher order error
o ( (cid:107) w t − w ∗ (cid:107) 2 ) .
According to Theorem 2.2, we know that simultaneous GDA-AM is equivalent to applying GMRES to solve the following linear system
[Equation]
where b =
[Equation]
We now know that GDA-AM is equivalent to GMRES being applied to solve the following linear system
[Equation]
The symmetric part of the coefﬁcient matrix of the above linear system is
[Equation]
According to Proposition 2, α I − η H
y ∗ y ∗ is positive deﬁnite since H y ∗ y ∗ (cid:52) 0 .
If H x ∗ x ∗ is positive semideﬁnite, then α I + η H x ∗ x ∗ is positive deﬁnite and we’re done.
Otherwise, assume λ min ( H x ∗ x ∗ ) < 0 .
Then for ﬁxed α , when η < − α λ min ( H x ∗ x ∗ ) , α I + η H x ∗ x ∗ will be positive deﬁnite.
Then according to Theorem 2.2, we know GDA-AM indeed converges.
Let’s create a new companion linear system as follows
[Equation]
Note that ˆ w = w ∗ and GMRES on this companion linear system is convergent under suitable choice of learning rate η .
Let the iterates of GMRES for ˜ w , ˆ w , w be denoted by ˜ w t , ˆ w t , w t .
Then (cid:107) ˜
w t − ˆ w t (cid:107) ≤ (cid:107) ˜ w t − w t (cid:107) + (cid:107) ˆ w t − w t (cid:107) .
According to Theorem B.5, we also have (cid:107) ˜ w t − w t (cid:107) ≤ σ k (cid:15), 1 ≤ k ≤ t .
Further more, again according to Theorem B.5, we know (cid:107) ˆ w t − ˆ w t (cid:107) ≤ σ k ( α w ∗ + (cid:15) ) .
Starting from an initial point very close to w ∗ and let t → ∞ and α, (cid:15) → 0 , ˆ w t will converge to w ∗ =
( x ∗ , y ∗ ) , which means the local minimax w ∗ =
( x ∗ , y ∗ ) is a limiting point of GDA-RAM.
Theorem B.6.
For strongly-convex-strongly-concave function f ( x , y ) , GDA-AM will converge to the Nash equilibrium of this function.
Proof: Since strongly-convex-strongly-concave function f ( x , y ) has unique Nash equilibrium which is also the unique minimax point, this minimax point must be the limiting point of GDA-AM according to Theorem B.4. B.4. B ILINEAR - QUADRATIC GAMES Moreover, we can further show that the GDA-AM converges on bilinear-quadratic games. Consider a quadratic problem as follows,
[Equation]
where A is full rank, B and C are both positive deﬁnite. Theorem B.7. [Global convergence for simultaneous GDA-AM on bilinear-quadratic problem] Let r ( Sim ) t be the residual of Algorithm 2 being applied to problem equation 35. For some constant ρ < 1 ,
[Equation]
where J = (cid:20) η B η A − η A T η C (cid:21) and λ min and λ max denote the smallest and largest eigenvalue, respectively.
The convergence property of GMRES has been studied in the next theorem.
We use this theorem to show the convergence rate of GDA-AM for bilinear-quadratic games.
Theorem B.8 (Elman (1982)) .
Consider solving a linear system Ex = b using GMRES.
Let r t =
b −
Ex t be the residual at t th iteration.
If the Hermitian part of E is positive deﬁnite, then for some positive constant ρ < 1 , it holds that
[Equation]
Proof of Theorem B.7 .
Applying simultaneous GDA-AM to solve the above problem is equivalent to applying Anderson Mixing on the following ﬁxed point iteration:
[Equation]
We know that we need to study the convergence properties of GMRES for solving the following linear system
[Equation]
For notational simplicity, the superscripts has been dropped.
Denote the coefﬁcient matrix (cid:20) η B η A − η A T η C (cid:21) by J .
The symmetric part of J is
[Equation]
which is positive deﬁnite.
Then immediately by Theorem B.8, the following convergence rate holds
For some constant 0 < ρ < 1 ,
[Equation]
Note that the convergence of GDA-AM for bilinear-quadratic games can also be analyzed by numerical range as shown in (Bollapragada et al., 2018).
Although we previously show that analysis based on the numerical range can not help us derive a convergent bound for bilinear games, we show analysis in
Bollapragada et al.
(2018) can be extended to bilinear-quadratic games.
When B and C are positive deﬁnite, 1 is outside of the numerical range of matrix G ( Quad − sim ) as shown in 7a.
When B or C is not positive deﬁnite, 1 can be included in the numerical range of matrix G ( Quad − sim ) as shown in 7b.
That is saying analysis based on the numerical range (Crouzeix & Palencia, 2017; Bollapragada et al., 2018) to the bilinear-quadratic problem can lead to a convergent result when B and C are positive deﬁnite.
And analysis based on the numerical range can not help us derive convergent results when
B or C is not positive deﬁnite.
[Figure]
[Equation]
I −
[Equation] C (cid:21) B. S TOCHASTIC CONVEX - NONCONVACE CASE In this section, we study the convergence of GDA-AM for convex-noncovace problem in the stochastic setting with the same assumptions in Wei et al.
(2021b);
Xu et al.
The recent work Wei et al.
(2021b) proves the convergence of the stochastic gradient descent with Anderson Mixing for min optimization.
The convergence of GDA-AM for minimax optimization builds on top of it with several modiﬁcations.
The minimax problem is equivalent to minimizing a function Φ( · ) = max y ∈Y f ( · , y )
(Lin et al., 2020).
And we are interested in complexity of a pair of (cid:15) -stationary point ( x, y ) instead of analysis of a point x .
Deﬁnition 5.
(Lin et al., 2020)
A pair of points ( x , y ) is an (cid:15) -stationary point ( (cid:15) ≥ 0) of a differentiable function Φ if
[Equation]
Assumption 1.
f : R d (cid:55)→ R is continuously differentiable.
f ( x ) ≥ f low >
−∞ for any x ∈ R d .
∇ f is globally L-Lipschitz continuous; namely (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) 2 ≤ L (cid:107) x − y (cid:107) 2 for any x, y ∈ R d .
Assumption 2.
For any iteration k , the stochastic gradient ∇ f ξ k ( x k ) satisﬁes E ξ k [ ∇ f ξ k ( x k )]
= ∇ f ( x k ) , E ξ k (cid:104) (cid:107)∇ f ξ k ( x k ) − ∇ f ( x k ) (cid:107) 22 (cid:105) ≤ σ 2 , where σ > 0 , and ξ k , k = 0 , 1 , . .
, are independent samples that are independent of { x i } k
Theorem B.9.
For a general convex-nonconcave function f , suppose that Assumptions 1 andhold.
Batch size n t = n for t = 0 , . .
, N − 1 .
C > 0 is a constant.
β t = µ 4 L ( [Equation] C − 1 ) .δ
t ≥
Cβ − 2 t , 0 ≤ α t ≤ min (cid:110) 1 , β 12 t (cid:111) and α t is chosen to make sure the positive deﬁniteness of H t .
Let
R be a random variable following P R ( t ) def =
Prob { R = t } = 1 /N , and ¯ N be the total number of stochastic GDA-AM calls needed to calculate stochastic gradients ˜ ∇ f S t ( w t ) in our algorithm.
To ensure E (cid:104)(cid:13)(cid:13)(cid:13)
˜ ∇ f ( w R ) (cid:13)(cid:13)(cid:13) 2 (cid:105) ≤ (cid:15) , total number of stochastic GDA-AM calls needed to calculate stochastic gradients ˜ ∇ f S t ( w t ) is O ( (cid:15) − 4 ) .
Recall that we can recast GDA scheme as the following ﬁxed point iteration.
[Equation]
Ignoring the stepsize η and let W t and R t record the ﬁrst and second order diffrence of recent m iterates:
[Equation]
Similarly as Wei et al.
(2021b),the Anderson mixing can be decoupled into
[Equation]
where β t is the mixing parameter, and ¯ V t =
V t − W t Γ t and Γ t is solved by
[Equation]
We want to argue that similar arguments in Wei et al.
(2021b) can be applied to the problem here.
To see why Anderson mixing works for minimax optimization, we assume function f is smooth.
Then the hessian matrix for G (sim) η is
[Equation]
Notice that in a small neighborhood of w t +1 , we have
[Equation]
Thus (cid:107) V t − R t Γ (cid:107) 2 ≈ (cid:107) V t + H W t Γ (cid:107) 2 , which is equivalent to solving for a vector p t
such that
Hp k = V t .
This is exactly the second order method for the ﬁxed point iteration problem.
Also at each step the AM is minimizing the residual, the reason that AM is equivalent to GMRES for linear problem is that this quadratic approximation is exact.
Finally, we rewrite AM as the quasi-newton framework as Wei et al.
(2021b) did.
w t +1 =
w t
+ H t V t where
[Equation]
Finally, with damping parameter, Anderson mixing has the following form
[Equation]
(41) we can also apply the very similar arguments to prove key results in lemma 1, lemma 2 in Wei et al.
(2021b).
There is also a key difference with Wei et al.
(2021b).
Here we are considering minimax optimization problem.
Thus our gradient is actually V ( w ) =
[Equation]
rather than ∇ f ( w ) = (cid:20) ∇ x f ( x , y ) ∇ y f ( x , y ) (cid:21)
This will introduce some difﬁculty to the dynamics of the ﬁxed pointe iteration.
However, noticing that (cid:107) V (cid:107) =
(cid:107)∇ f ( w ) (cid:107) and
[Equation]
where
[Equation]
we call this the ascent-descent gradient (ADG) which is the gradient for minimax optimization problem
[Equation]
To see why ∇ f ( w t ) T ( w t +1 − w t ) ≤ ˜ ∇ f ( w t ) T ( w t +1 − w t ) , we consider their difference
[Equation]
For ﬁxed y t , f ( x t , y t +1 ) has the Talyor expansion:
[Equation]
Assuming f is convex w.r.t x and apply safeguard to ensure f ( x t +1 , y t ) ≤ f ( x t , y t ) can guarantee ( ˜ ∇ − ∇ )
f ( w t ) T ( w t +1 − w t ) ≥ 0 .
Now applying lemmas in Wei et al.
(2021b), we can derive the convergence of our method for general convex-nonconcave function similarly.
C A DDITIONAL E XPERIMENTS C. C OMPARISON WITH EG WITH P OSITIVE M OMENTUM
In this section, we include additional comparison between GDA-AM and EG with positive momentum.
GDA-AM has two big theoretical advantages over EG with positive momentum.
First, convergence of GDA-AM does not require strong assumptions on choices of hyperparamters.
Second, 4.1 and 4.
provide nonasymptotic guarantees while convergence of EG with positive mometum is asymptotic.
Experimental results are shown in 8.
It indicates GDA-AM outperforms EG with positive momentum.
Finding a good choice of the inner and outer step size of EG and momentum term is hard.
For EG with positive momentum, we set the step size of extrapolation step as 1, the step size of update as 0.5, and the positive momentum term as 0.3 after grid search as shown in 8b and 8c.
On the other hand,
GDA-AM converges fast for different step size without hyperparameter tuning.
[Figure]
1 D M INIMAX FUNCTIONS
We begin with investigating the empirical performance of GDA-AM for 6 non-trivial 1d bivariate functions.
We set initial points as (3 , 3) and m as 20 or 5 for all functions.
We use optimal learning rates for all methods on each problem.
Results are shown in Figure 9, 10, 11, 12, 13 and 14.
We observe GDA-AM consistently outperforms all baselines and improves convergence.
It is worthwhile to mention that the difference between GDA-AM and traditional averaging is twofold.
First, traditional averaging does not involve an adaptive averaging scheme and thus blindly converge to (0 , 0) for all 1d bivariate functions.
In contrast, GDA-AM obtains optimal weights by solving a small linear system on past iterates.
Using different weights for each iteration, GDA-AM is able to minimize the residual of past iterates and thus ﬁnd the solution of a ﬁxed-point iteration.
More importantly, averaging does not change the GDA dynamic because averaging generates a new sequence of parameters based on GDA iterates.
This means averaging is independent with base training algorithm (GDA here).
However, GDA-AM changes the dynamic directly by overwriting the latest iterate.
It means Anderson
Mixing interacts with GDA, which is another major difference from averaging.
C. D ENSITY ESTIMATION
To test our proposed method we evaluate our method on two low-dimension density estimation problems, mixture of 25 Gaussians and Swiss roll.
For both generator and discriminator, we use fully connected neural networks with 3 hidden layers and 128 hidden units in each layer.
Except for the output layer of discriminator that uses a sigmoid activation, we use tanh-activation for all other layers.
We run Adam and GDA-AM for 50000 steps.
The learning rate is set as 2 × 10 − 4 and β 1 = 0 , β 2 = 0 .after
an extensive grid search, which is close to the maximal possible stepsize under which the methods rarely diverge.
Figure 15 and 16 show the output after { 1 K, 10 K, 30 K, 50 K } iterations.
It can be seen that our method converges faster to the target distribution offers a improvement over Adam.
In addition, we can observe that the generated samples using our method gather around the circle and are less connected with other circles.
[Figure] [Figure]
C. I MAGE G ENERATION
In this section, we provide additional experimental results that are not given in Section 6.
Figure 18a and 18b show the Inception Score for CIFAR10 using WGAN-GP and SNGAN.
It can be observed that our method consistently performs better than Adam and EG during training.
Further, on
CIFAR-10 using WGAN-GP and SNGAN, GDA-AM is slightly slower than Adam (about 110-115 % computational time), but signiﬁcantly faster than EG (about 65-75 % computational time).
[Figure] [Figure]
[Figure]
C. D ETAILS ON THE EXPERIMENTS
For our experiments, we used the PyTorch 1 deep learning framework.
Experiments were run one
NVIDIA V100 GPU.
The residual network architecture for generator and discriminator are summarized in Table 2 and 3.
We use a WGAN-GP loss, with gradient penalty λ
= 10 .
When using the gradient penalty (WGAN-GP), we remove the batch normalization layers in the discriminator.
When using SNGAN, we replace the batch normalization layers with spectral normalization.
Hyperparamters of Adam are selected after grid search.
We use a learning rate of 2 × 10 − 4 and batch size of 64 .
For table size of GDA-AM , we set it as 120 for CIFAR10 and 150 for CelebA.
We set β 1 = 0 .and
β 2 = 0 .
9 as we ﬁnd it gives us better models than default settings.
[Table] [Table]