Original: Momentum Extragradient is Optimal for Games with Cross-Shaped Spectrum
Revised: Momentum Extragradient is Optimal for Games with Cross-Shaped Spectrum

Original: author names withheld

Original: Under Review for OPT

Original: Abstract
Revised: Abstract

Original: The extragradient method has recently gained a lot of attention, due to its convergence behavior on smooth games.
Revised: The extragradient method has recently gained a lot of attention, due to its convergence behavior on smooth games.

Original: In games, the eigenvalues of the Jacobian of the vector field are distributed on the complex plane, exhibiting more convoluted dynamics compared to minimization.
Revised: In games, the eigenvalues of the Jacobian of the vector field are distributed on the complex plane, exhibiting more convoluted dynamics compared to minimization.

Original: In this work, we take a polynomial-based analysis of the extragradient with momentum for optimizing games with cross-shaped spectrum on the complex plane.
Revised: In this work, we take a polynomial-based analysis of the extragradient with momentum for optimizing games with cross-shaped spectrum on the complex plane.

Original: We show two results: first, the extragradient with momentum exhibits three different modes of convergence based on the hyperparameter setup: when the eigenvalues are distributed ( i ) on the real line, ( ii ) both on the real line along with complex conjugates, and ( iii ) only as complex conjugates.
Revised: We show two results: first, the extragradient with momentum exhibits three different modes of convergence based on the hyperparameter setup: when the eigenvalues are distributed ( i ) on the real line, ( ii ) both on the real line along with complex conjugates, and ( iii ) only as complex conjugates.

Original: Then, we focus on the case ( ii ) , i e., when the spectrum of the Jacobian has cross-shaped structure, as observed in training generative adversarial networks.
Revised: Then, we focus on the case ( ii ) , i.e., when the spectrum of the Jacobian has cross-shaped structure, as observed in training generative adversarial networks.

Original: For this problem class, we derive the optimal parameters and show that the extragradient with momentum achieves accelerated convergence rate.
Revised: For this problem class, we derive the optimal hyperparameters and show that the extragradient with momentum achieves accelerated convergence rate.

Original: 1. Introduction
Revised: 1. Introduction

Original: Background.
Revised: Background.

Original: Following [4, 17], we define the n -player differentiable game as a family of twice continuously differentiable losses ℓ
Revised: Following [4, 17], we define the n -player differentiable game as a family of twice continuously differentiable losses ℓ

Original: i : R d i → R , for i = 1 , . . .
Revised: i : R d i → R , for i = 1 , . . .

Original: , n. The player i controls the parameter w ( i ) ∈
Revised: , n. The player i controls the parameter w ( i ) ∈

Original: R d i .
Revised: R d i .

Original: We denote the concatenated parameters by w =
Revised: We denote the concatenated parameters by w =

Original: [ w (1) , . .
Revised: [ w (1) , . .

Original: . , w ( n ) ] ∈ R d , where d = (cid:80) ni =1 d i .
Revised: . , w ( n ) ] ∈ R d , where d = (cid:80) ni =1 d i .

Original: For this problem, a Nash equilibrium satisfies w ( i ) ,⋆ ∈ arg min w ( i ) ∈ R di ℓ i (cid:0) w ( i ) , w ( ¬ i ) ,⋆ (cid:1) , where the notation · ( ¬ i ) denotes all indices except for i .
Revised: For this problem, a Nash equilibrium satisfies w ( i ) ,⋆ ∈ arg min w ( i ) ∈ R di ℓ i (cid:0) w ( i ) , w ( ¬ i ) ,⋆ (cid:1) , where the notation · ( ¬ i ) denotes all indices except for i .

Original: We also define the vector field of the game v as the concatenation of the individual gradients, and its associated Jacobian ∇ v respectively as: 
Revised: We also define the vector field of the game v as the concatenation of the individual gradients, and its associated Jacobian ∇ v respectively as: v ( w ) =

Original: [Equation] =   ∇ 2 w (1) ℓ 1 ( w ) . . .
Revised: [ ∇ w (1) ℓ 1 ( w ) , . . . , ∇ w ( n ) ℓ n ( w )] ⊤ , and ∇ v ( w ) =   ∇ 2 w (1) ℓ 1 ( w ) . . .

Original: ∇ w ( n ) ∇ w (1) ℓ 1 ( w ) .
Revised: ∇ w ( n ) ∇ w (1) ℓ 1 ( w ) .

Original: ∇ w (1) ∇ w ( n ) ℓ n ( w ) . . .
Revised: ∇ w (1) ∇ w ( n ) ℓ n ( w ) . . .

Original: ∇ 2 w ( n ) ℓ 1 ( w ) 
Revised: ∇ 2 w ( n ) ℓ 1 ( w ) 

Original:  .
Revised:  .

Original: Unfortunately, finding Nash equilibria for general games is impractical to solve [17, 28].
Revised: Unfortunately, finding Nash equilibria for general games is impractical to solve [17, 28].

Original: Instead, we focus on finding a stationary point of the vector field v , since a Nash equilibrium is always a stationary point of the gradient dynamics.
Revised: Instead, we focus on finding a stationary point of the vector field v , since a Nash equilibrium is always a stationary point of the gradient dynamics.

Original: That is, we want to solve:
Revised: That is, we want to solve:

Original: Find
Revised: Find

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: Related work.
Revised: Related work.

Original: The above game formulation is strictly more general than minimization , and includes many problems such as saddle-point problems [25], bilinear games [18, 20], and variational inequality problems [9, 29].
Revised: The above game formulation is strictly more general than minimization, and includes many problems such as saddle-point problems [25], bilinear games [18, 20], and variational inequality problems [9, 29].

Original: For minimization, the vector field v and the Jacobian ∇ v are, respectively, the gradient and the Hessian of the objective function.
Revised: For minimization, the vector field v and the Jacobian ∇ v are, respectively, the gradient and the Hessian of the objective function.

Original: The Hessian is a symmetric matrix, so its eigenvalues lie on the real line; the ratio of the largest and the smallest eigenvalues is the condition number.
Revised: The Hessian is a symmetric matrix, so its eigenvalues lie on the real line; the ratio of the largest and the smallest eigenvalues is the condition number.

Original: It is now a common wisdom in numerical linear algebra and optimization theory how the notion of condition number characterizes the difficulty of a (strongly convex) minimization problem [23].
Revised: It is now a common wisdom in numerical linear algebra and optimization theory how the notion of condition number characterizes the difficulty of a (strongly convex) minimization problem [23].

Original: In conjunction with provable lower bounds within various function classes [22], an algorithm, whose attainable upper bound on the convergence rate matches the lower bound of its function class, is considered “opimal” for that function class [23, 24].
Revised: In conjunction with provable lower bounds within various function classes [22], an algorithm, whose attainable upper bound on the convergence rate matches the lower bound of its function class, is considered “opimal” for that function class [23, 24].

Original: Differentiable games, on the other hand, have much more convoluted dynamics, as the eigenvalues of the Jacobian at the solution are distributed over the complex plane [4, 19].
Revised: Differentiable games, on the other hand, have much more convoluted dynamics, as the eigenvalues of the Jacobian at the solution are distributed over the complex plane [4, 19].

Original: To address this, recent line of work proposes new algorithms with better upper bounds on the convergence rate [8, 10, 11, 19–21, 26].
Revised: To address this, recent line of work proposes new algorithms with better upper bounds on the convergence rate [8, 10, 11, 19–21, 26].

Original: Vast majority of the aforementioned literature proves convergence based on two approaches: either ( i ) bounding the real part of the eigenvalues of the (submatrices of) Jacobian, or ( ii ) the magnitude.
Revised: Vast majority of the aforementioned literature proves convergence based on two approaches: either ( i ) bounding the real part of the eigenvalues of the (submatrices of) Jacobian, or ( ii ) the magnitude.

Original: However, such approaches do not fully exploit the distribution of eigenvalues of the Jacobian, resulting in loose bounds [3].
Revised: However, such approaches do not fully exploit the distribution of eigenvalues of the Jacobian, resulting in loose bounds [3].

Original: Simply put, the notion of lower bounds, and subsequently that of optimal algorithms, is not well established for optimizing games.
Revised: Simply put, the notion of lower bounds, and subsequently that of optimal algorithms, is not well established for optimizing games.

Original: To understand the difficulty of a differentiable game, [3] proposed a geometric interpretation of the conditioning of a game via the notion of spectral shape , defined as the set containing all eigenvalues of the Jacobian.
Revised: To understand the difficulty of a differentiable game, [3] proposed a geometric interpretation of the conditioning of a game via the notion of spectral shape , defined as the set containing all eigenvalues of the Jacobian.

Original: Specifically, let M K be the set of matrices A whose spectrum, denoted by Sp ( A ) , belong to a set K on the complex plane with positive real part:
Revised: Specifically, let M K be the set of matrices A whose spectrum, denoted by Sp ( A ) , belong to a set K on the complex plane with positive real part:

Original: [Equation]
Revised: [Equation]

Original: When K is “simple” (e.g., a real line segment or a disc on the complex plane), [3] characterizes the lower bound and the optimality of some first-order methods [1].
Revised: When K is “simple” (e.g., a real line segment or a disc on the complex plane), [3] characterizes the lower bound and the optimality of some first-order methods [1].

Original: This work.
Revised: This work.

Original: We consider a more specific class of games, where the distribution of the eigenvalues of the Jacobian can be modeled via (shifted) cross shape on the complex plane.
Revised: We consider a more specific class of games, where the distribution of the eigenvalues of the Jacobian can be modeled via (shifted) cross-shape on the complex plane.

Original: That is, we consider the following spectrum model:
Revised: That is, we consider the following spectrum model:

Original: [Equation]
Revised: [Equation]

Original: In (2), the first set can be thought of as a segment on the real line, similarly to the case of minimizing µ -strongly convex and L -smooth functions.
Revised: In (2), the first set can be thought of as a segment on the real line, similarly to the case of minimizing µ -strongly convex and L -smooth functions.

Original: The second set has a fixed real component ( c ′ > 0 ), along with imaginary components symmetric across the real line, as the Jacobian is real.
Revised: The second set has a fixed real component ( c ′ > 0 ), along with imaginary components symmetric across the real line, as the Jacobian is real.

Original: This is a strict generalization of the purely imaginary interval [ − bi, bi ] commonly considered in the bilinear games literature [3, 18, 20].
Revised: This is a strict generalization of the purely imaginary interval [ − bi, bi ] commonly considered in the bilinear games literature [3, 18, 20].

Original: While many recent papers on bilinear games cite generative adversarial networks (GANs) [12] as a motivation, the work in [5, Figure 4] empirically shows that the spectrum of GANs is not contained in the imaginary axis.
Revised: While many recent papers on bilinear games cite generative adversarial networks (GANs) [12] as a motivation, the work in [5, Figure 4] empirically shows that the spectrum of GANs is not contained in the imaginary axis.

Original: Instead, we note that the cross shape spectrum assumption above would include some of the observed GAN spectra.
Revised: Instead, we note that the cross shape spectrum assumption above would include some of the observed GAN spectra.

Original: In this work, we focus on solving (1) for games with Jacobian eigenvalue structure in (2).
Revised: In this work, we focus on solving (1) for games with Jacobian eigenvalue structure in (2).

Original: We use the extragradient with momentum, which we expound in the next section.
Revised: We use the extragradient with momentum, which we expound in the next section.

Original: Notation.
Revised: Notation.

Original: We denote the spectrum of a matrix A by Sp ( A ) , and its spectral radius by ρ ( A ) :
Revised: We denote the spectrum of a matrix A by Sp ( A ) , and its spectral radius by ρ ( A ) :

Original: = max {| λ | : λ ∈ Sp ( A ) } .
Revised: = max {| λ | : λ ∈ Sp ( A ) } .

Original: R ( z ) and I ( z ) respectively denote the real and the imaginary part of a complex number z.
Revised: R ( z ) and I ( z ) respectively denote the real and the imaginary part of a complex number z.

Original: 2. Algorithm
Revised: 2. Algorithm

Original: Extragradient with momentum via Chebyshev polynomials.
Revised: Extragradient with momentum via Chebyshev polynomials.

Original: To solve (1) for games with Jacobian eigenvalue structure in (2), we focus on the the extragradient method with momentum (EGM):
Revised: To solve (1) for games with Jacobian eigenvalue structure in (2), we focus on the the extragradient method with momentum (EGM):

Original: [Equation]
Revised: [Equation]

Original: where h is the step size, γ is the extrapolation step size, and m is the momentum parameter.
Revised: where h is the step size, γ is the extrapolation step size, and m is the momentum parameter.

Original: The extragradient (EG) method (without momentum) was first proposed in [16] for saddle point problems, and recently got popularized due to its convergence behavior for some class of differentiable games such as bilinear games, for which the standard gradient method diverges [2, 3, 11].
Revised: The extragradient (EG) method (without momentum) was first proposed in [16] for saddle point problems, and recently got popularized due to its convergence behavior for some class of differentiable games such as bilinear games, for which the standard gradient method diverges [2, 3, 11].

Original: For completeness, we remind the gradient method (GD) and the gradient method with momentum (GDM): 
Revised: For completeness, we remind the gradient method (GD) and the gradient method with momentum (GDM): [Equation] and

Original: [Equation]
Revised: [Equation]

Original: We take a polynomial based approach to analyze EGM.
Revised: We take a polynomial based approach to analyze EGM.

Original: Such analysis was used in [3, 14, 15, 27].
Revised: Such analysis was used in [3, 14, 15, 27].

Original: On that end, we use the following lemma [3, 6], which connects first-order methods with [-polynomials: Lemma 
 If-] v ( w ) =
Revised: On that end, we use the following lemma [3, 6], which connects first-order methods with [-polynomials: Lemma 
 If-]polynomials, when the vector field satisfies v ( w ) =

Original: Aw + b , then the following holds:
Revised: Aw + b : Lemma 1 There exists a real polynomial p t of degree at most t satisfying

Original: [Equation]
Revised: [Equation]

Original: where p tis a real polynomial of degree at most t with p t (0) = 1 , and  w ⋆ satisfies
Revised: where p t (0) = 1 , and v ( w ⋆ ) =

Original: Aw ⋆ + b = 0 .
Revised: Aw ⋆ + b = 0 .

Original: Therefore, for v satisfying v ( w ) = Aw + b, (5) further implies ∥ w t − w ⋆ ∥ 2 = ∥
Revised: By taking norms, (5) further implies ∥ w t − w ⋆ ∥ 2 = ∥

Original: p t ( A )( w 0 − w ⋆ ) ∥ 2 , from which  convergence rate can be obtained.
Revised: p t ( A )( w 0 − w ⋆ ) ∥ 2 , from which a convergence rate can be obtained.

Original: As EGM in (3) is a first-order method [1, 3], we can study the associated residual polynomial.
Revised: As EGM in (3) is a first-order method [1, 3], we can study the associated residual polynomial.

Original: We summarize this in the following theorem:
Revised: We summarize this in the following theorem:

Original: Theorem 2 (Residual polynomials of EGM) Consider the extragradient with momentum in (3) .
Revised: Theorem 2 (Residual polynomials of EGM) Consider the extragradient with momentum in (3) .

Original: If v ( w ) =

Original: Aw + b, the associated residual polynomial is:˜ P [Equation] , and ˜ P t +1 ( λ ) = (1 + m − hλ (1 − γλ
Revised: Its associated residual polynomial is:

Original: [Equation]
Revised: [Equation]

Original: ˜ P t ( λ ) − m ˜ P t − 1 ( λ ) .

Original: Further, let T t ( · ) and U t ( · ) be the Chebyshev polynomials of the first and the second kind respectively.
Revised: Further, let T t ( · ) and U t ( · ) be the Chebyshev polynomials of the first and the second kind respectively.

Original: Then, the above expression simplifies to:
Revised: Then, the above expression simplifies to:

Original: [Equation]
Revised: [Equation]

Original: where  σ ( λ ) is what we call the link function.
Revised: where we refer to the term σ ( λ ) as the link function.

Original: As can be seen in (6), the link function for EGM is quadratic with respect to λ , which is different from the link function of GDM.
Revised: As can be seen in (6), the link function for EGM is quadratic with respect to λ , which is different from the link function of GDM.

Original: We write below the residual polynomial for GDM, expressed in terms of the Chebyshev polynomials:
Revised: We write below the residual polynomial for GDM, expressed in terms of the Chebyshev polynomials:

Original: [Equation]
Revised: [Equation]

Original: Notice that the Chebyshev polynomial-based expressions of EGD in (6) and that of GDM in (7) are identical except for the link functions σ ( λ ) and ξ ( λ ) , which enter T t ( · ) and U t ( · ) as arguments.
Revised: Notice that the Chebyshev polynomial-based expressions of EGD in (6) and that of GDM in (7) are identical except for the link functions σ ( λ ) and ξ ( λ ) , which enter T t ( · ) and U t ( · ) as arguments.

Original: This difference in link functions is crucial, as the Chebyshev polynomials behave very differently based on the domain, as shown in the following lemma:
Revised: This difference in link functions is crucial, as the Chebyshev polynomials behave very differently based on the domain, as shown in the following lemma:

Original: Lemma 3 ([13]) Let z be a complex number.
Revised: Lemma 3 ([13]) Let z be a complex number.

Original: The sequence(cid:16)(cid:12)(cid:12)(cid:12) 2 m 1+ m T t ( z ) + 1 − m 1+ m U t ( z ) (cid:12)(cid:12)(cid:12)(cid:17) t ⩾ 0 grows exponentially in t for z / ∈ [ − 1 , 1] , while in that interval they are bounded as the following:
Revised: The sequence

Original: [Equation]
Revised: [Equation]

Original:  t [Equation] | ⩽and
Revised: grows exponentially in t for z / ∈ [ − 1 , 1] , while in that interval they are bounded as the following:

Original: [Equation]
Revised: [Equation]

Original: Therefore, we are interested in the case where the set of step sizes and momentum parameters lead to | σ ( λ ) | ⩽ 1 , so that we can use the bound in (8).
Revised: Therefore, we are interested in the case where the set of step sizes and momentum parameters lead to | σ ( λ ) | ⩽ 1 , so that we can use the bound in (8).

Original: We call the set of hyperparameters such that the image of the link function is in [ − 1 , 1] as the robust region .
Revised: We call the set of hyperparameters such that the image of the link function is in [ − 1 , 1] as the robust region .

Original: Three modes of the extragradient with momentum.
Revised: Three modes of the extragradient with momentum.

Original: Within the robust region, we can compute the worst-case convergence rate as follows:
Revised: Within the robust region, we can compute the worst-case convergence rate as follows:

Original: [Equation]
Revised: [Equation]

Original: where the last inequality uses 0 < m < 1 .
Revised: where the last inequality uses 0 < m < 1 .

Original: Note that by definition r t denotes the worst-case convergence rate.
Revised: Note that by definition r t denotes the worst-case convergence rate.

Original: To get the asymptotic rate, we take the limit supremum of the  t -th root  of r t :lim sup t →∞ t
Revised: To get the asymptotic rate, we take the limit supremum of the 2 t -th root 2 of r t :

Original: [Equation]
Revised: [Equation]

Revised: sup

Original: [Equation]
Revised: [Equation]

Original: Recall that the Chebyshev polynomial expressions of EGD in (6) and that of GD in (7) are identical except for the link function.
Revised: Recall that the Chebyshev polynomial expressions of EGD in (6) and that of GDM 3 are identical except for the link functions.

Original: Hence, theasymptotic convergence rate√ m in (9) applies to both EGM and GDM, as long as the link functions | σ ( λ ) | and | ξ ( λ ) | are bounded by 1 .
Revised: Hence, the convergence rate in (9) applies to both EGM and GDM, as long as the link functions | σ ( λ ) | and | ξ ( λ ) | are bounded by 1 .

Original: Due to the difference in link functions, the robust region of EGM and GDM are drastically different.
Revised: Yet, due to the difference in link functions, the robust region of EGM and GDM are drastically different.

Original: Specifically, due to the quadratic link function of EGM in (6), the robust region becomes more complicated; in particular, this link function canbe used when λ ∈ C , and still satisfy the upper bounds (8) of Lemma 3, resulting in the asymptotic rate of (9).
Revised: Specifically, due to the quadratic link function of EGM in (6), the robust region becomes more flexible; in particular, this link function can still satisfy the upper bounds (8) in Lemma 3 for a subset of complex values, resulting in the convergence rate of (9).

Original: Hence, compared to GDM whose link function is linear in λ as in (7), EGM enjoys the √ m asymptotic rate  for a much wider set of λ values.
Revised: Hence, compared to GDM whose link function is linear in λ and so can only satisfy (8) for a subset of real values, EGM enjoys the convergence rate in (9) for a much wider set of λ values.

Original: The robust region of EGM can be described with the four extreme points below:
Revised: The robust region of EGM can be described with the four extreme points below:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Depending on the choice of hyperparameters, the above four extreme points can be distributed differently on three different modes, which we summarize in the following theorem:
Revised: Depending on the choice of hyperparameters, the above four extreme points can be distributed differently on three different modes, which we summarize in the following theorem:

Original: Theorem 4 Consider the extragradient with momentum in (3) , expressed with the Chebyshev polynomials as in (6) .
Revised: Theorem 4 Consider the extragradient with momentum in (3) , expressed with the Chebyshev polynomials as in (6) .

Original: Then, the robust region summarized in (10) have the following three modes:
Revised: Then, the robust region summarized in (11) have the following three modes:

Original: If h 4 γ ⩾ (1 + √ m ) 2 , then the four points in (10) are all real numbers; 2.
Revised: If h 4 γ ⩾ (1 + √ m ) 2 , then σ − 1 ( − 1) and σ − 1 (1) are all real numbers; 2.

Original: If (1 − √ m ) 2 ⩽ h 4 γ < (1 + √ m ) 2 , then σ − 1 ( − 1)of (10) are complex, and σ − 1 (1) are real; 3.
Revised: If (1 − √ m ) 2 ⩽ h 4 γ < (1 + √ m ) 2 , then σ − 1 ( − 1) are complex, and σ − 1 (1) are real; 3.

Original: If (1 − √ m ) 2 > h 4 γ , then the four points in (10) are all complex numbers.
Revised: If (1 − √ m ) 2 > h 4 γ , then σ − 1 ( − 1) and σ − 1 (1) are all complex numbers.

Original: We empirically verify Theorem 4 in Figure 1.
Revised: We empirically verify Theorem 4 in Figure 1.

Original: Note that the hyperparameters are set up so that each case in Theorem 4 is covered.
Revised: Note that the hyperparameters are set up so that each case in Theorem 4 is covered.

Original: 3. Optimal parameters and convergence rates
Revised: 3. Optimal parameters and convergence rates

Original: Optimal parameters for cross-shaped game problem.
Revised: Optimal parameters for cross-shaped game problem.

Original: We focus on solving (1) for games with cross-shaped Jacobian eigenvalue structure as in (2).
Revised: We focus on solving (1) for games with cross-shaped Jacobian eigenvalue structure as in (2).

Original: Therefore, in the remainder of the paper, we focus on the second case of Theorem 4, which is illustrated in the middle pannel of Figure 1.
Revised: Therefore, in the remainder of the paper, we focus on the second case of Theorem 4, which is illustrated in the middle pannel of Figure 1.

Original: We now connect the cross-shaped spectrum model in (2) and the robust region of EGM described with four extra points in (10).
Revised: We now connect the cross-shaped spectrum model in (2) and the robust region of EGM described with four extra points in (11).

Original: As we focus on the second case of Theorem 4, the hyperparameters satisfy the condition (1 − √ m )
Revised: As we focus on the second case of Theorem 4, the hyperparameters satisfy the condition (1 − √ m )

Original: 2 ⩽ h 4 γ < (1 + √ m ) 2 .
Revised: 2 ⩽ h 4 γ < (1 + √ m ) 2 .

Original: In this case, we can write the robust region of EGM as follows:
Revised: In this case, we can write the robust region of EGM as follows:

Original: [Equation]
Revised: [Equation]

Original: Here, the first interval lies on R , as the square root term is real; conversely, in the second interval, the square root term is imaginary, with fixed real component: 12 γ .
Revised: Here, the first interval lies on R , as the square root term is real; conversely, in the second interval, the square root term is imaginary, with fixed real component: 12 γ .

Original: The optimal parameters of EGM in terms of the worst-case convergence rate for this problem occurs when the robust region in (11) and the spectrum model in (2) coincide.
Revised: The optimal parameters of EGM in terms of the worst-case convergence rate for this problem occurs when the robust region in (12) and the spectrum model in (2) coincide.

Original: This condition can be summarized to the following three equalities:
Revised: This condition can be summarized to the following three equalities:

Original: [Equation]
Revised: [Equation]

Original: Based on (12) [Equation] we can compute the optimal hyperparameters, summarized in the next theorem:
Revised: Based on (13), we can compute the optimal hyperparameters, summarized in the next theorem:

Original: Theorem 5 (Optimal hyperparameters for EGM) Consider solving (1) for games where the Jacobian has cross-shaped spectrum as in (2) .
Revised: Theorem 5 (Optimal hyperparameters for EGM) Consider solving (1) for games where the Jacobian has cross-shaped spectrum as in (2) .

Original: For this problem, the optimal hyperparameters for the extragradient with momentum in (3) can be set as follows:
Revised: For this problem, the optimal hyperparameters for the extragradient with momentum in (3) can be set as follows:

Original: [Equation]
Revised: [Equation]

Original: Recalling (9) [Equation] we immediately get the asymptotic convergence rate from Theorem 5.
Revised: Recalling (10), we immediately get the asymptotic convergence rate from Theorem 5.

Original: Further [Equation] we consider the ill-conditioned regime, wherethe inverse condition number τ := µL approaches 0:
Revised: Further, this formula can be simplified the ill-conditioned regime, where τ := µL approaches 0:

Original: [Equation]
Revised: [Equation]

Original: We see that EGM achieves accelerated convergence rate: 1 − O ( (cid:112) µ/L ) as long as c
Revised: We see that EGM achieves accelerated convergence rate: 1 − O ( (cid:112) µ/L ) as long as c

Original: = O ( L ) .
Revised: = O ( L ) .

Original: An interesting special case of the spectrum model in (2) is when the length of the first set and the second equals; 
Revised: A special case of the spectrum model in (2) is when the length of the first set and the second equals; i [Equation]

Original: i.e., when c =
Revised: e., when c =

Original: L − µ 2 .
Revised: L − µ 2 .

Original: In this setting, the optimal hyperparameters of EGM in Theorem 5 further simplify, as we summarize in the next corollary:
Revised: In this setting, the optimal hyperparameters of EGM in Theoremfurther simplify, as we summarize in the next corollary:

Original: Corollary 6 Consider the same setting as Theorem 5.
Revised: Corollary 6 Consider the same setting as Theorem 5.

Original: Further, assume that c =
Revised: Further, assume that c =

Original: L − µ 2 .
Revised: L − µ 2 .

Original: Then, the optimal hyperparameters for the extragradient with momentum in (3) can be set as follows:
Revised: Then, the optimal hyperparameters for the extragradient with momentum in (3) can be set as follows:

Original: [Equation]
Revised: [Equation]

Original: and [Equation] Again, from Corollary 6, we can immediately get the asymptotic convergence rate via (9), which can be interpreted further in the ill-conditioned regime: τ → 0 :
Revised: Again, from Corollary 6, we can immediately get the asymptotic convergence rate via (10), which can be interpreted further in the ill-conditioned regime where τ → 0 :

Original: [Equation]
Revised: [Equation]

Original: Comparison with other methods.
Revised: Comparison with other methods.

Original: In [2], GD and EG 2 are interpreted as the fixed-point iterations of operators F GD h and F EG h respectively representing each method, and the local convergence rate is obtained by bounding the spectral radius of the Jacobian of the operators under certain assumptions.
Revised: In [2], GD and EG 4 are interpreted as the fixed-point iterations of operators F GD h and F EG h respectively representing each method, and the local convergence rate is obtained by bounding the spectral radius of the Jacobian of the operators under certain assumptions.

Original: We make the comparison for the simplified setting when c
Revised: We make the comparison for the simplified setting when c

Original: =
Revised: =

Original: L − µin (2); we leave the general case with any c for future work.
Revised: L − µin (2); we leave the general case with any c for future work.

Original: We summarize the relevant theorems from [2, 11] below:
Revised: We summarize the relevant theorems from [2, 11] below:

Original: Theorem 7 ([2, 11]) Let w ⋆ be a stationary point of v .
Revised: Theorem 7 ([2, 11]) Let w ⋆ be a stationary point of v .

Original: Denote by σ ⋆ the spectrum of ∇ v ( w ⋆ ) for notational brevity.
Revised: Denote by σ ⋆ the spectrum of ∇ v ( w ⋆ ) for notational brevity.

Original: Further, assume the eigenvalues of ∇ v ( w ⋆ ) all have positive real parts.
Revised: Further, assume the eigenvalues of ∇ v ( w ⋆ ) all have positive real parts.

Original: Then,1.
Revised: Then,

Original: For the gradient method with step size h = min λ ∈ σ ⋆ ,
Revised: For the gradient method with step size h = min λ ∈ σ ⋆ ,

Original: [Equation]
Revised: [Equation]

Original: For the extragradient method with step size h
Revised: For the extragradient method with step size h

Original: =
Revised: =

Original: (4 max λ ∈ σ ⋆ | λ | ) − 1 , it satisfies:
Revised: (4 max λ ∈ σ ⋆ | λ | ) − 1 , it satisfies:

Original: [Equation]
Revised: [Equation]

Original: Since our spectrum model in (2) satisfies the condition that the eigenvalues of ∇ v ( w ⋆ ) all have positive real parts, we can obtain the convergence rate of GD and EG, based on Theorem 7.
Revised: Since our spectrum model in (2) satisfies the condition that the eigenvalues of ∇ v ( w ⋆ ) all have positive real parts, we can obtain the convergence rate of GD and EG, based on Theorem 7.

Original: With our (simplified) spectrum model with c =
Revised: With our (simplified) spectrum model with c =

Original: L − µ 2 , it follows that min λ ∈ σ ⋆ R ( λ ) = µ , and min λ ∈ σ ⋆ R (1 /λ ) =
Revised: L − µ 2 , it follows that min λ ∈ σ ⋆ R ( λ ) = µ , and min λ ∈ σ ⋆ R (1 /λ ) =

Original: 1 /L .
Revised: 1 /L .

Original: Similarly, for EG, it follows that min λ ∈ σ ⋆ R ( λ ) = µ, max λ ∈ σ ⋆ | λ | =
Revised: Similarly, for EG, it follows that min λ ∈ σ ⋆ R ( λ ) = µ, max λ ∈ σ ⋆ | λ | =

Original: L, and max λ ∈ σ ⋆ | λ | 2 =
Revised: L, and max λ ∈ σ ⋆ | λ | 2 =

Original: L 2 .
Revised: L 2 .

Original: Furthermore, min λ ∈ σ ⋆ | λ | 2 = µ 2 if L ⩾ ( √ 2+1) µ , and min λ ∈ σ ⋆ | λ | 

Revised: Furthermore, min λ ∈ σ ⋆ | λ | 2 = µ 2 if L ⩾ ( √ 2+1) µ , and min λ ∈ σ ⋆ | λ |

Original: =
Revised: =

Original: ( L − µ ) 2 / 2 otherwise.
Revised: ( L − µ ) 2 / 2 otherwise.

Original: Thus, the convergence rates of GD and EG in(15) and (16)  respectively satisfy: ρ ( ∇ F GD h ( w ⋆ ))
Revised: Thus, the convergence rates of GD and EG in (16) and (17) respectively satisfy: ρ ( ∇ F GD h ( w ⋆ ))

Original: 2 ⩽ 1 − µL , and ρ ( ∇ F EG h ( w ⋆ ))
Revised: 2 ⩽ 1 − µL , and ρ ( ∇ F EG h ( w ⋆ ))

Original: 2 ⩽
Revised: 2 ⩽

Original: [Equation]

Original: otherwise

Original: [Equation]
Revised: [Equation]

Original: We see that both GD and EG have non-accelerated convergence rate, unlike (13) and (14).
Revised: We see that both GD and EG have non-accelerated convergence rate, unlike (14) and (15).

Original: 4. Conclusion
Revised: 4. Conclusion

Original: In this work, we focused on finding a stationary point of games.
Revised: In this work, we focused on finding a stationary point of games.

Original: Via polynomial-based analysis, we showed the extragradient with momentum converges in three different modes of spectrum structure of the game Jacobian, depending on the hyperparameters.
Revised: Via polynomial-based analysis, we showed the extragradient with momentum converges in three different modes of eigenvalue structure of the game Jacobian, depending on the hyperparameters.

Original: Then, we focused on a more specific problem class, where the eigenvalues of the game Jacobian is distributed in the complex plane with  cross-shape.
Revised: Then, we focused on a more specific problem class, where the eigenvalues of the game Jacobian are distributed in the complex plane with a cross-shape.

Original: For this problem class, we obtained the optimal hyperparameters for the  extragradient descent with momentum, which shows accelerated linear asymptotic convergence rate.
Revised: For this problem class, we obtained the optimal hyperparameters for the momentum extragradient method, which enjoys an accelerated linear asymptotic convergence rate.

Revised: Acknowledgements

Revised: This work is supported by NSF FET no. 1907936, NSF/Intel MLWiNS no. 2003137, NSF CMMI no. 2037545, and NSF CAREER no. 2145629.

Original: Appendix A. Proofs A.1. Proof for TheoremD ERIVATION OF THE FIRST PART
Revised: Appendix A. Proofs A.1. Proof for TheoremD ERIVATION OF THE FIRST PART

Original: Proof We want to find the residual polynomial ˜ P t ( A ) of the extragradient with momentum (EGM) in (3).
Revised: Proof We want to find the residual polynomial ˜ P t ( A ) of the extragradient with momentum (EGM) in (3).

Original: That is, we want to find
Revised: That is, we want to find

Original: [Equation]
Revised: [Equation]

Original: [Equation] where { w t } t =0 is the iterates generated by EGM, which is possible by Lemma 1, as EGM is a first-order method [3].
Revised: ) where { w t } t =0 is the iterates generated by EGM, which is possible by Lemma 1, as EGM is a first-order method [3].

Original: We now prove this is by induction.
Revised: We now prove this is by induction.

Original: Base case.
Revised: Base case.

Original: For t = 0 , ˜ P 

Revised: For t = 0 , ˜ P

Original: ( A ) is a degree-zero polynomial, and hence equals I , which denotes the identity matrix.
Revised: ( A ) is a degree-zero polynomial, and hence equals I , which denotes the identity matrix.

Original: Thus, w 0 − w ⋆
Revised: Thus, w 0 − w ⋆

Original: =
Revised: =

Original: I ( w 0 − w ⋆ ) holds true.
Revised: I ( w 0 − w ⋆ ) holds true.

Original: Induction step.
Revised: Induction step.

Original: As the induction hypothesis, assume ˜ P t satisfies (17).
Revised: As the induction hypothesis, assume ˜ P t satisfies (18).

Original: Also, since we are looking for a stationary point, it holds that v ( w ⋆ ) = 0 .
Revised: Also, since we are looking for a stationary point, it holds that v ( w ⋆ ) = 0 .

Original: Further, as v is linear by the assumption of Lemma 1, it holds that v ( w ) =
Revised: Further, as v is linear by the assumption of Lemma 1, it holds that v ( w ) =

Original: A ( w − w ⋆ ) .
Revised: A ( w − w ⋆ ) .

Original: Applying this to EGM, we have:
Revised: Applying this to EGM, we have:

Original: [Equation]
Revised: [Equation]

Original: Subtracting w ⋆ on both sides, we have:
Revised: Subtracting w ⋆ on both sides, we have:

Original: [Equation]
Revised: [Equation]

Original: where in the second to last inequality, we used the induction hypothesis in (17).
Revised: where in the third equality, we used the induction hypothesis in (18).

Original: D ERIVATION OF THE SECOND PART IN (6)
Revised: D ERIVATION OF THE SECOND PART IN (6)

Original: Proof We show P t =
Revised: Proof We show P t =

Original: ˜
Revised: ˜

Original: P t for all t via induction.
Revised: P t for all t via induction.

Original: Base case.
Revised: Base case.

Original: [Equation]
Revised: [Equation]

Original: Induction step.
Revised: Induction step.

Original: As the induction hypothesis, assume that P t =
Revised: As the induction hypothesis, assume that P t =

Original: ˜
Revised: ˜

Original: P t for t .
Revised: P t for t .

Original: We want to show this holds for t +
Revised: We want to show this holds for t +

Original: [Equation]
Revised: [Equation]

Original: where in the second to last equality we use the induction hypothesis.
Revised: where in the second to last equality we use the induction hypothesis.

Original: A.2. Proof of Lemma
Revised: A.2. Proof of Lemma

Original: Proof for Lemma 3 can be found in [13].
Revised: Proof for Lemma 3 can be found in [13].

Original: A.3. Proof of Theorem
Revised: A.3. Proof of Theorem

Original: Proof We analyze each case separately.
Revised: Proof We analyze each case separately.

Original: Case 1 there are two square roots: (cid:113) 14 γ 2 − (1 −√ m ) 2 hγ
Revised: Case 1: There are two square roots: (cid:113) 14 γ 2 − (1 −√ m ) 2 hγ

Original: and (cid:113) 14 γ 2 − (1+ √ m )
Revised: and (cid:113) 14 γ 2 − (1+ √ m )

Original: 2 hγ .
Revised: 2 hγ .

Original: The second one is real if:
Revised: The second one is real if:

Original: [Equation]
Revised: [Equation]

Original: which implies the first is real, as (1 + √ m ) 2 ⩾ (1 − √ m ) 2 .
Revised: which implies the first is real, as (1 + √ m ) 2 ⩾ (1 − √ m ) 2 .

Original: Case 3 There are two square roots: (cid:113) 14 γ 2 − (1 −√ m ) 2 hγ
Revised: Case 3: There are two square roots: (cid:113) 14 γ 2 − (1 −√ m ) 2 hγ

Original: and (cid:113) 14 γ 2 − (1+ √ m )
Revised: and (cid:113) 14 γ 2 − (1+ √ m )

Original: 2 hγ .
Revised: 2 hγ .

Original: The first one is complex if:
Revised: The first one is complex if:

Original: [Equation]
Revised: [Equation]

Original: which implies the second is complex, as (1 + √ m ) 2 ⩾ (1 − √ m ) 
Revised: which implies the second is complex, as (1 + √ m ) 2 ⩾ (1 − √ m ) 2 .

Original: [Equation] Case 2 follows automatically from the above two cases.
Revised: Case 2: This case follows automatically from the above two cases.

Original: A.4. Proof of Theorem
Revised: A.4. Proof of Theorem

Original: Proof We recall the conditions in (12) below:
Revised: Proof We recall the conditions in (13) below:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: First, by adding (18) and (19), we get:
Revised: First, by adding (19) and (20), we get:

Original: [Equation]
Revised: [Equation]

Original: Plugging (21) back into (18), we have:
Revised: Plugging (22) back into (19), we have:

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Plugging(21) and (22)  into (20), we have:
Revised: Plugging (22) and (23) into (21), we have:

Original: [Equation]
Revised: [Equation]

Original: Finally
Revised: Finally

Original: [Equation]
Revised: [Equation]

Original: to simplify (22) further
Revised: to simplify (23) further

Original: [Equation]
Revised: [Equation]

Original: from (23)
Revised: from (24)

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Hence, from (22),
Revised: Hence, from (23),

Original: [Equation]
Revised: [Equation]

Original: A.5. Proof of Corollary
Revised: A.5. Proof of Corollary

Original: Proof This result can be obtained either by plugging in c =
Revised: Proof This result can be obtained either by plugging in c =

Original: L − µto (23) and (24), or by using the same technique as the proof of Theorem 5.
Revised: L − µto (24) and (25), or by using the same technique as the proof of Theorem 5.