Original: E SCAPING S ADDLE P OINTS IN N ONCONVEX M INIMAX O PTIMIZATION VIA C UBIC -R EGULARIZED G RADIENT D ESCENT -A SCENT
Revised: E SCAPING S ADDLE P OINTS IN N ONCONVEX M INIMAX O PTIMIZATION VIA C UBIC -R EGULARIZED G RADIENT D ESCENT -A SCENT

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The gradient descent-ascent (GDA) algorithm has been widely applied to solve nonconvex minimax optimization problems.
Revised: The gradient descent-ascent (GDA) algorithm has been widely applied to solve nonconvex minimax optimization problems.

Original: However, the existing GDA-type algorithms can only ﬁnd ﬁrst-order stationary points of the envelope function of nonconvex minimax optimization problems, which does not rule out the possibility to get stuck at suboptimal saddle points.
Revised: However, the existing GDA-type algorithms can only ﬁnd ﬁrst-order stationary points of the envelope function of nonconvex minimax optimization problems, which does not rule out the possibility to get stuck at suboptimal saddle points.

Original: In this paper, we develop Cubic-GDA – the ﬁrst GDA-type algorithm for escaping strict saddle points in nonconvex-stronglyconcave minimax optimization.
Revised: In this paper, we develop Cubic-GDA – the ﬁrst Newton-type GDA algorithm for escaping strict saddle points in nonconvexstrongly-concave minimax optimization.

Original: Speciﬁcally, the algorithm uses gradient ascent to estimate the second-order information of the minimax objective function, and it leverages the cubic regularization technique to efﬁciently escape the strict saddle points.
Revised: Speciﬁcally, the algorithm uses gradient ascent to estimate the second-order information of the minimax objective function, and it leverages the cubic regularization technique to efﬁciently escape the strict saddle points.

Original: Under standard smoothness assumptions on the objective function, we show that Cubic-GDA admits an intrinsic potential function whose value monotonically decreases in the minimax optimization process.
Revised: Under standard smoothness assumptions on the objective function, we show that Cubic-GDA admits an intrinsic potential function whose value monotonically decreases in the minimax optimization process.

Original: Such a property leads to a desired global convergence of Cubic-GDA to a second-order stationary point at a sublinear rate.
Revised: Such a property leads to a desired global convergence of Cubic-GDA to a second-order stationary point at a sublinear rate.

Original: Moreover, we analyze the convergence rate of Cubic-GDA in the full spectrum of a gradient dominant-type nonconvex geometry.
Revised: Moreover, we analyze the convergence rate of Cubic-GDA in the full spectrum of a gradient dominant-type nonconvex geometry.

Original: Our result shows that CubicGDA achieves an orderwise faster convergence rate than the standard GDA for a wide spectrum of gradient dominant geometry.
Revised: Our result shows that Cubic-GDA achieves an orderwise faster convergence rate than the standard GDA for a wide spectrum of gradient dominant geometry.

Original: Our study bridges minimax optimization with second-order optimization and may inspire new developments along this direction.
Revised: Our study bridges minimax optimization with second-order optimization and may inspire new developments along this direction.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Nonconvex minimax optimization is a popular optimization framework that has broad applications in modern machine learning, including game theory (Ferreira et al., 2012), generative adversarial networks (Goodfellow et al., 2014), adversarial training (Sinha et al., 2017), reinforcement learning (Qiu et al., 2020;
Revised: Nonconvex minimax optimization is a popular optimization framework that has broad applications in modern machine learning, including game theory (Ferreira et al., 2012), generative adversarial networks (Goodfellow et al., 2014), adversarial training (Sinha et al., 2017), reinforcement learning (Qiu et al., 2020;

Original: Ho and Ermon, 2016; Song et al., 2018), etc.
Revised: Ho and Ermon, 2016; Song et al., 2018), etc.

Original: A standard nonconvex minimax optimization problem is shown below, where f is a smooth nonconvex function in x .
Revised: A standard nonconvex minimax optimization problem is shown below, where f is a smooth nonconvex function in x .

Original: [Equation]
Revised: [Equation]

Original: In the existing literature, many optimization algorithms have been developed to solve different types of minimax problems.
Revised: In the existing literature, many optimization algorithms have been developed to solve different types of minimax problems.

Original: Among them, a simple and popular algorithm is the gradient descent-ascent (GDA), which alternates between a gradient descent update on x and a gradient ascent update on y in each iteration.
Revised: Among them, a simple and popular algorithm is the gradient descent-ascent (GDA), which alternates between a gradient descent update on x and a gradient ascent update on y in each iteration.

Original: Speciﬁcally, the global convergence of GDA has been established for minimax problems under various types of global geometries, such as convex-concave-type geometry ( f is convex in x and concave in y )
Revised: Speciﬁcally, the global convergence of GDA has been established for minimax problems under various types of global geometries, such as convex-concave-type geometry ( f is convex in x and concave in y )

Original: (Nedi´c and Ozdaglar, 2009; Du and Hu, 2019; Mokhtari et al., 2020; Zhang and Wang, 2021), bi-linear geometry (Neumann, 1928; Robinson, 1951) and PolyakŁojasiewicz geometry (Nouiehed et al., 2019; Yang et al., 2020), yet these geometries are not satisﬁed by general nonconvex minimax problems.
Revised: (Nedi´c and Ozdaglar, 2009; Du and Hu, 2019; Mokhtari et al., 2020; Zhang and Wang, 2021), bi-linear geometry (Neumann, 1928; Robinson, 1951) and PolyakŁojasiewicz geometry (Nouiehed et al., 2019; Yang et al., 2020), yet these geometries are not satisﬁed by general nonconvex minimax problems.

Original: Recently, many studies proved the convergence of GDA in nonconvex minimax optimization for both nonconvex-concave problems (Lin et al., 2020; Nouiehed et al., 2019; Xu et al., 2020d) and nonconvex-strongly-concave problems (Lin et al., 2020; Xu et al., 2020d; Chen et al., 2021).
Revised: Recently, many studies proved the convergence of GDA in nonconvex minimax optimization for both nonconvex-concave problems (Lin et al., 2020; Nouiehed et al., 2019; Xu et al., 2020d) and nonconvex-strongly-concave problems (Lin et al., 2020; Xu et al., 2020d; Chen et al., 2021).

Original: In these studies, it has been shown that GDA converges sublinearly to a stationary point where the gradient of a certain envelop function of the minimax problem vanishes.
Revised: In these studies, it has been shown that GDA converges sublinearly to a stationary point where the gradient of a certain envelope function of the minimax problem vanishes.

Original: Although GDA can ﬁnd ﬁrst-order stationary points of nonconvex minimax problems, such a type of convergence guarantee does not rule out the possibility that GDA may get stuck at suboptimal saddle points of the envolope function, which are well known to be the major challenge for training high-dimensional machine learning models (Dauphin et al., 2014; Jin et al., 2017a; Zhou and Liang, 2018)
Revised: Although GDA can ﬁnd ﬁrst-order stationary points of nonconvex minimax problems, such a type of convergence guarantee does not rule out the possibility that GDA may get stuck at suboptimal saddle points of the envelope function, which are well known to be the major challenge for training high-dimensional machine learning models (Dauphin et al., 2014; Jin et al., 2017; Zhou and Liang, 2018).

Original: On the other hand, while numerous algorithms have been developed for escaping saddle points in conventional nonconvex optimization, e.g., ﬁrst-order algorithms (Ge et al., 2015; Jin et al., 2017b;
Revised: On the other hand, while numerous algorithms have been developed for escaping saddle points in conventional nonconvex optimization, e.g., ﬁrst-order algorithms (Ge et al., 2015; Jin et al., 2017;

Original: Carmon and Duchi, 2016; Liu and Yang, 2017) and second-order algorithms (Nesterov and Polyak, 2006; Agarwal et al., 2017; Yue et al., 2019;
Revised: Carmon and Duchi, 2016; Liu and Yang, 2017) and second-order algorithms (Nesterov and Polyak, 2006; Agarwal et al., 2017; Yue et al., 2019;

Original: Zhou et al., 2018), such a type of algorithm has not been developed for escaping saddle points in nonconvex minimax optimization.
Revised: Zhou et al., 2018), such a type of algorithm has not been developed for escaping saddle points in nonconvex minimax optimization.

Original: Therefore, we want to ask the following fundamental questions.
Revised: Therefore, we want to ask the following fundamental questions.

Original: • Q: How to develop a provably convergent GDA-type algorithm that can effectively escape saddle points in nonconvex minimax optimization?
Revised: • Q: How to develop a provably convergent Newton-type GDA algorithm that can effectively escape saddle points in nonconvex minimax optimization?

Original: How fast it converges?
Revised: How fast it converges?

Original: Developing and analyzing such an algorithm is nontrivial due to the following reasons: 1) we need to have a good understanding and characterization of both the ﬁrst-order and second-order information of nonconvex minimax problems; 2) we need to develop a computationally feasible and efﬁcient GDA algorithm that can leverage the local curvature of the function to escape saddle points; 3) we aim to develop a uniﬁed analysis framework that can characterize the convergence rate of this algorithm under different types of nonconvex geometry of the minimax problem.
Revised: Developing and analyzing such an algorithm is nontrivial due to the following reasons: 1) we need to have a good understanding and characterization of both the ﬁrst-order and second-order information of nonconvex minimax problems; 2) we need to develop a computationally feasible and efﬁcient GDA algorithm that can leverage the local curvature of the function to escape saddle points; 3) we aim to develop a uniﬁed analysis framework that can characterize the convergence rate of this algorithm under different types of nonconvex geometry of the minimax problem.

Original: In this paper, we provide comprehensive answers to these questions.
Revised: In this paper, we provide comprehensive answers to these questions.

Original: We develop the ﬁrst second-order
Revised: We develop the ﬁrst Newton-type

Original: GDA-type algorithm that escapes strict saddle points and converges to second-order stationary points in nonconvex-strongly-concave minimax optimization.
Revised: GDA algorithm that escapes strict saddle points and converges to second-order stationary points in nonconvex-strongly-concave minimax optimization.

Original: We also characterize the global and local convergence rates of this algorithm under various types of nonconvex geometry.
Revised: We also characterize the global and local convergence rates of this algorithm under various types of nonconvex geometry.

Original: We summarize our contributions as follows.
Revised: We summarize our contributions as follows.

Original:  O UR CONTRIBUTIONS
Revised: 1. O UR CONTRIBUTIONS

Original: We consider thefollowing nonconvex-strongly-concave minimax optimization problem. [Equation] where f is a twice-differentiable and nonconvex-strongly-concave function.
Revised: We consider the minimax optimization problem (P), where f is a twice-differentiable and nonconvexstrongly-concave function, and its gradient and Jacobian matrices are Lipschitz continuous.

Original: Deﬁne an envelope function Φ( x ) := max y ∈ R n f ( x, y ) .
Revised: Deﬁne an envelope function Φ( x ) := max y ∈ R n f ( x, y ) .

Original: The existing GDA algorithms can only ﬁnd ﬁrst-order stationary points that satisfy ∇ Φ( x ∗ )
Revised: The existing GDA algorithms can only ﬁnd ﬁrst-order stationary points that satisfy ∇ Φ( x ∗ )

Original: = 0 .
Revised: = 0 .

Original: In this paper, we develop a second-order GDA-type algorithm that converges to second-order stationary points of the nonconvex minimax problem (P), i.e.,
Revised: In this paper, we develop a Newton-type GDA algorithm that converges to second-order stationary points of the nonconvex minimax problem (P).

Original: [Equation]

Original: Speciﬁcally, we propose Cubic-GDA – a GDA-type algorithm that leverages the classical cubic regularization technique to escape saddle points.
Revised: Speciﬁcally, we propose Cubic-GDA – a Newton-type GDA algorithm that leverages the classical cubic regularization technique to escape saddle points.

Original: Different from the standard cubic regularization algorithm that uses the Hessian information of the function, the Hessian of Φ( x ) is not directly available in nonconvex minimax optimization, and hence we develop a rigorous and computationally feasible scheme in Cubic-GDA to estimate the Hessian.
Revised: Different from the standard cubic regularization algorithm that uses the Hessian information of the function, the Hessian of Φ( x ) is not directly available in nonconvex minimax optimization, and hence we develop a rigorous and computationally feasible scheme in Cubic-GDA to estimate the Hessian.

Original: We study the global convergence property of Cubic-GDA in general nonconvex-strongly-concave optimization.
Revised: We study the global convergence property of Cubic-GDA in general nonconvex-strongly-concave optimization.

Original: Speciﬁcally, we show that Cubic-GDA admits an intrinsic potential function H ( x, x (cid:48) , y )
Revised: Speciﬁcally, we show that Cubic-GDA admits an intrinsic potential function H ( x, x (cid:48) , y )

Original: (see Proposition 2), which monotonically decreases along the trajectory of Cubic-GDA.
Revised: (see Proposition 2), which monotonically decreases along the trajectory of Cubic-GDA.

Original: Based on the monotonicity of this potential function, we show that every limit point of the parameter sequence { x t } t generated by Cubic-GDA is a second-order stationary point of the minimax problem.
Revised: Based on the monotonicity of this potential function, we show that every limit point of the parameter sequence { x t } t generated by Cubic-GDA is a second-order stationary point of the minimax problem.

Original: We further analyze the aymptotic convergence rates of Cubic-GDA under a broad spectrum of the local nonconvex Łojasiewicz gradient geometry.
Revised: We further analyze the aymptotic convergence rates of Cubic-GDA under a broad spectrum of the local nonconvex Łojasiewicz gradient geometry.

Original: In this case, we show that Cubic-GDA converges to a unique limit point, which is a second-order stationary point.
Revised: In this case, we show that Cubic-GDA converges to a unique limit point, which is a second-order stationary point.

Original: Moreover, as the geometry parameter increases (i.e., sharper local geometry), the convergence rate of Cubic-GDA accelerates from sublinear convergence up to super-linear convergence, as we summarize in Table 1 below.
Revised: Moreover, as the geometry parameter increases (i.e., sharper local geometry), the convergence rate of Cubic-GDA accelerates from sublinear convergence up to super-linear convergence, as we summarize in Table 1 below.

Original: In particular, these convergence rates are orderwise faster than those of the standard GDA under the same type of nonconvex geometry (Chen et al., 2021) 1 .
Revised: In particular, these convergence rates are orderwise faster than those of the standard GDA under the same type of nonconvex geometry (Chen et al., 2021) 1 .

Original: [Table]
Revised: [Table]

Original: 1. R ELATED WORK
Revised: 1. R ELATED WORK

Original: Deterministic GDA algorithms: Yang et al.
Revised: Deterministic GDA algorithms: Many studies characterized the convergence of GDA in nonconvex minimax optimization. Speciﬁcally, Lin et al. (2020); Nouiehed et al.

Revised: ; Xu et al.

Original: (2020) studied an alternating gradient descent-ascent (AGDA) algorithm inwhich the gradient ascent step uses the current variable x t +1 instead of x t .
Revised: (2020d) studied the convergence of GDA in the nonconvex-concave setting whereas Lin et al. Xu et al. (2020d) focused on the nonconvex-strongly-concave setting. In these general nonconvex settings, it is shown that GDA converges to a certain stationary point at a sublinear rate.

Revised: Recently, Chen et al.

Revised: (2021) proved the parameter convergence of proximal-GDA in regularized nonconvex-stronglyconcave optimization under the Kurdyka-Łojasiewicz geometry.

Revised: The convergence rates obtained there are orderwise slower than that of Cubic-GDA.

Revised: Yang et al.

Revised: (2020) studied an alternating gradient descent-ascent (AGDA) algorithm in which the gradient ascent step uses the current variable x t + instead of x t .

Original: Xu et al.
Revised: Xu et al.

Original: (2020d) studied an alternating gradient projection algorithm which applies (cid:96) 2 regularizer to the local objective function of GDA followed by projection onto the constraint sets.
Revised: (2020d) studied an alternating gradient projection algorithm which applies (cid:96) 2 regularizer to the local objective function of GDA followed by projection onto the constraint sets.

Original: Daskalakis and Panageas (2018); Mokhtari et al.
Revised: Daskalakis and Panageas (2018); Mokhtari et al.

Original: ; Zhang and Wang (2021) analyzed optimistic gradient descent-ascent (OGDA) which applies negative momentum to accelerate GDA.
Revised: ; Zhang and Wang (2021) analyzed optimistic gradient descent-ascent (OGDA) which applies negative momentum to accelerate GDA.

Original: Mokhtari et al.
Revised: Mokhtari et al.

Original: (2020) also studied an extra-gradient algorithm which applies two-step GDA in each iteration.
Revised: (2020) also studied an extra-gradient algorithm which applies two-step GDA in each iteration.

Original: Nouiehed et al.
Revised: Nouiehed et al.

Original: (2019) studied multi-step GDA where multiple gradient ascent steps are performed, and they also studied the momentum-accelerated version.
Revised: (2019) studied multi-step GDA where multiple gradient ascent steps are performed, and they also studied the momentum-accelerated version.

Original: Cherukuri et al.
Revised: Cherukuri et al.

Original: ; Daskalakis and Panageas (2018); Jin et al.
Revised: Daskalakis and Panageas (2018); Jin et al.

Original: (2020) studied GDA in continuous time dynamics using differential equations.
Revised: (2020) studied GDA in continuous time dynamics using differential equations.

Original: Adolphs et al.
Revised: Adolphs et al.

Original: (2019) analyzed a second-order variant of the GDA algorithm.
Revised: (2019) analyzed a second-order variant of the GDA algorithm.

Revised: In a concurrent work (Luo and Chen, 2021), the authors proposed and studied the same Cubic-GDA algorithm.

Revised: They characterize the computation complexity under a special type of inexactness that approximates the inverse Jacobian using matrix Chebyshev polynomials.

Revised: As a comparison, this study focuses on analyzing the global and local convergence properties of Cubic-GDA.

Original: Stochastic GDA algorithms: Lin et al.
Revised: Stochastic GDA algorithms: Lin et al.

Original: ; Yang et al.
Revised: ; Yang et al.

Original: (2020) analyzed stochastic GDA and stochastic AGDA, which are direct extension of GDA and AGDA to the stochastic setting.
Revised: (2020) analyzed stochastic GDA and stochastic AGDA, which are direct extension of GDA and AGDA to the stochastic setting.

Original: Variance reduction techniques have been applied to stochastic minimax optimization, including SVRG-based (Du and Hu, 2019; Yang et al., 2020), SPIDER-based (Xu et al., 2020c), SREDA (Xu et al., 2020b),
Revised: Variance reduction techniques have been applied to stochastic minimax optimization, including SVRG-based (Du and Hu, 2019; Yang et al., 2020), SPIDER-based (Xu et al., 2020c), SREDA (Xu et al., 2020b),

Original: STORM (Qiu et al., 2020) and its gradient free version (Huang et al., 2020).
Revised: STORM (Qiu et al., 2020) and its gradient free version (Huang et al., 2020).

Original: Xie et al.
Revised: Xie et al.

Original: (2020) studied the complexity lower bound of ﬁrst-order stochastic algorithms for ﬁnite-sum minimax problem.
Revised: (2020) studied the complexity lower bound of ﬁrst-order stochastic algorithms for ﬁnite-sum minimax problem.

Original: Cubic regularization (CR): CR algorithm was ﬁrst proposed in (Nesterov and Polyak, 2006), in which the authors analyzed the convergence  of CR to second-order stationary points for nonconvex optimization.
Revised: Cubic regularization (CR): CR algorithm dates back to (Griewank, 1981), where global convergence of the algorithm is established. In Nesterov and Polyak (2006), the authors analyzed the convergence rate of CR to second-order stationary points for nonconvex optimization.

Original: In (Nesterov, 2008), the authors established the sub-linear convergence of CR for solving convex smooth problems, and they further proposed an accelerated version of CR with improved sub-linear convergence.
Revised: In (Nesterov, 2008), the authors established the sub-linear convergence of CR for solving convex smooth problems, and they further proposed an accelerated version of CR with improved sub-linear convergence.

Original: Recently, Yue et al.
Revised: Recently, Yue et al.

Original: (2019) studied the asymptotic convergence properties of CR under the error bound condition, and established the quadratic convergence of the iterates.
Revised: (2019) studied the asymptotic convergence properties of CR under the error bound condition, and established the quadratic convergence of the iterates.

Revised: Recently, Hallak and Teboulle (2020) proposed a framework of two directional method for ﬁnding second-order stationary points in general smooth nonconvex optimization.

Revised: This main idea of the algorithm is to search for a feasible direction toward the solution and is not based on cubic regularization.

Original: Several other works proposed different methods to solve the cubic subproblem of CR, e.g., (Agarwal et al., 2017; Carmon and Duchi, 2016; Cartis et al., 2011b).
Revised: Several other works proposed different methods to solve the cubic subproblem of CR, e.g., (Agarwal et al., 2017; Carmon and Duchi, 2016; Cartis et al., 2011b).

Original: Another line of work aimed at improving the computation efﬁciency of CR by solving the cubic subproblem with inexact gradient and Hessian information.
Revised: Another line of work aimed at improving the computation efﬁciency of CR by solving the cubic subproblem with inexact gradient and Hessian information.

Original: In particular, Ghadimi et al.
Revised: In particular, Ghadimi et al.

Original: ; Jin et al.

Original: (2017b) proposed an inexact CR for solving convex problem and nonconvex problems, respectively.
Revised: (2017) proposed an inexact CR for solving convex problem.

Original: Also, Cartis et al.
Revised: Also, Cartis et al.

Original: (2011a) proposed an adaptive inexact CR for nonconvex optimization, whereas Jiang et al.
Revised: (2011a) proposed an adaptive inexact CR for nonconvex optimization, whereas Jiang et al.

Original: (2017) further studied the accelerated version for convex optimization.
Revised: (2017) further studied the accelerated version for convex optimization.

Original: Several studies explored subsampling schemes to implement inexact CR algorithms, e.g., (Kohler and Lucchi, 2017; Xu et al., 2020a; Zhou and Liang, 2018; Wang et al., 2018).
Revised: Several studies explored subsampling schemes to implement inexact CR algorithms, e.g., (Kohler and Lucchi, 2017; Xu et al., 2020a; Zhou and Liang, 2018; Wang et al., 2018).

Original:  P ROBLEM F ORMULATION AND P RELIMINARIES
Revised:  P ROBLEM F ORMULATION AND P RELIMINARIES

Original: In this section, we introduce the problem formulation and present some preliminary results that will be used in the analysis.
Revised: In this section, we introduce the problem formulation and present some preliminary results that will be used in the analysis.

Original: We consider the following minimax optimization problem.

Original: [Equation]

Original: [Equation]

Original: P [Equation] where f : R m × R n → R is a twice-differentiable nonconvex-strongly-concave function with two variables. Deﬁne an envelope function Φ( x ) := max y ∈ R n f ( x, y ) .
Revised: Notation: For notation simplicity, we denote ∇ 1 f, ∇ 2 f as the gradients with respect to the ﬁrst and the second input arguments of f , respectively.

Original: Then the minimax problem (P) is equivalent to the following minimization problem.

Original: [Equation]

Original: As we show in item 2 of Proposition 1 later, this envelope function Φ( x ) is smooth and nonconvex. The existing GDA algorithms can only ﬁnd ﬁrst-order stationary points of the minimax problem that satisfy ∇ Φ( x ∗ ) = 0 . In this paper, we aim to develop a provably convergent algorithm that can ﬁnd second-order stationary points x ∗ of the function Φ( x ) that satisfy the following set of conditions.
Revised: We also denote ∇ 11 f, ∇ 22 f as the Jacobian matrices where the second-order derivatives are taken over the ﬁrst and second arguments of f , respectively.

Original: [Equation]

Original: In the existing literature, many optimization algorithms have been developed for ﬁnding secondorder stationary points in conventional nonconvex minimization problems.

Original: This includes ﬁrst-order algorithms (Ge et al., 2015; Jin et al., 2017b; Carmon and Duchi, 2016;

Original: Liu and Yang, 2017) and second-order algorithms (Nesterov and Polyak, 2006; Agarwal et al., 2017;

Original: Yue et al., 2019;

Original: Zhou et al., 2018).

Original: However, these algorithms are not directly applicable to solve the problem (P’), as the function Φ( x ) involves a special maximization structure and hence its speciﬁc function form Φ as well as the gradient ∇ Φ and Hessian ∇ 2 Φ are not available in practice.

Original: Instead, our algorithm design can only leverage information of the bi-variate function f . Notation: For notation simplicity, we denote ∇ 1 f, ∇ 2 f as the gradients with respect to the ﬁrst and the second input arguments of f , respectively. We also denote ∇ 11 f, ∇ 22 f as the Jacobian matrices where the second-order derivatives are taken over the ﬁrst and second arguments of f , respectively. Moreover, we denote ∇ 12 f as the Jacobian matrix where the second-order derivative is taken over the ﬁrst argument of f and followed by the second argument, and ∇ 21 f is deﬁned in a similar way.
Revised: Moreover, we denote ∇ 12 f as the Jacobian matrix where the second-order derivative is taken over the ﬁrst argument of f and followed by the second argument, and ∇ 21 f is deﬁned in a similar way.

Original: Throughout the paper, we adopt the following standard assumptions on the problem (P).
Revised: We consider the minimax optimization problem (P) that satisﬁes the following standard assumptions.

Original: Assumption 1.
Revised: Assumption 1.

Original: The minimax optimization problem (P) satisﬁes:
Revised: The minimax optimization problem (P) satisﬁes:

Original: Function f ( · , · ) is L 1 -smooth and function f ( x, · ) is µ -strongly concave for all ﬁxed x ; 2.
Revised: Function f ( · , · ) is L 1 -smooth and function f ( x, · ) is µ -strongly concave for all ﬁxed x ; 2.

Original: The Jacobian matrices ∇ 11 f , ∇ 12 f , ∇ 21 f , ∇ 22 f are L 2 -Lipschitz; 3.
Revised: The Jacobian matrices ∇ 11 f , ∇ 12 f , ∇ 21 f , ∇ 22 f are L 2 -Lipschitz; 3.

Original: Function Φ is bounded below and has compact sub-level sets.
Revised: Function Φ is bounded below and has compact sub-level sets.

Original: To elaborate, item 1 considers the class of nonconvex-strongly-concave functions f that has been widely studied in the minimax optimization literature (Lin et al., 2020; Jin et al., 2020; Xu et al., 2020d;
Revised: To elaborate, item 1 considers the class of nonconvex-strongly-concave functions f that has been widely studied in the minimax optimization literature (Lin et al., 2020; Jin et al., 2020; Xu et al., 2020d;

Original: Lu et al., 2020).
Revised: Lu et al., 2020).

Original: Items 2 assumes that the block Jacobian matrices of f are Lipschitz, which is a standard assumption for analyzing many second-order optimization algorithms (Nesterov and
Revised: Items 2 assumes that the block Jacobian matrices of f are Lipschitz, which is a standard assumption for analyzing many second-order optimization algorithms (Nesterov and

Original: Polyak, 2006; Agarwal et al., 2017; Yue et al., 2019).
Revised: Polyak, 2006; Agarwal et al., 2017; Yue et al., 2019).

Original: Moreover, item 3 guarantees that the minimax problem has at least one solution.
Revised: Moreover, item 3 guarantees that the minimax problem has at least one solution.

Original: By strong concavity of f ( x, · ) , it is clear that the minimizer y ∗ ( x ) :=
Revised: By strong concavity of f ( x, · ) , it is clear that the maximizer y ∗ ( x ) :=

Original: arg max y ∈ R n f ( x, y ) is unique for every x ∈ R m .
Revised: arg max y ∈ R n f ( x, y ) is unique for every x ∈ R m .

Original: In particular, if x ∗ is a second-order stationary point of Φ( x ) , then ( x ∗ , y ∗ ( x ∗ )) is the desired solution of the minimax problem (P). 
Revised: In particular, if x ∗ is a second-order stationary point of Φ( x ) , then ( x ∗ , y ∗ ( x ∗ )) is the desired solution of the minimax problem (P). Deﬁne an envelope function Φ( x ) := max y ∈ R n f ( x, y ) . Then the minimax problem (P) is equivalent to the minimization problem. min x ∈ R m Φ( x ) , where Φ( x ) = max y ∈ R n f ( x, y ) .

Original: Next, we present some important properties regarding the gradient and Jacobian matrices of the functions f ( x, y )  and  Φ( x  )  . 
Revised: As we show in item 2 of Proposition 1 later, this envelope function Φ( x ) is smooth and nonconvex. The existing GDA algorithms can only ﬁnd ﬁrst-order stationary points of the minimax problem that satisfy ∇ Φ( x ∗ ) = 0 . In this paper, we aim to develop a provably convergent algorithm that can ﬁnd second-order stationary points x ∗ of the function Φ( x ) that satisfy the following set of conditions.

Revised: [Equation]

Revised: In the existing literature, many optimization algorithms have been developed for ﬁnding secondorder stationary points in conventional nonconvex minimization problems.

Revised: This includes ﬁrst-order algorithms (Ge et al., 2015; Jin et al., 2017; Carmon and Duchi, 2016;

Revised: Liu and Yang, 2017) and second-order algorithms (Nesterov and Polyak, 2006; Agarwal et al., 2017;

Revised: Yue et al., 2019;

Revised: Zhou et al., 2018).

Revised: However, these algorithms are not directly applicable to solve the problem (P’), as the function Φ( x ) involves a special maximization structure and hence its speciﬁc function form Φ as well as the gradient ∇ Φ and Hessian ∇ 2 Φ are not available in practice.

Revised: Instead, our algorithm design can only leverage information of the bi-variate function f .

Revised: Next, we present some important properties regarding the gradient and Jacobian matrices of the functions f ( x, y ) and Φ( x ) .

Original: Throughout, we denote κ =
Revised: Throughout, we denote κ =

Original: L 1 /µ as the condition number.
Revised: L 1 /µ as the condition number.

Original: Proposition 1.
Revised: Proposition 1.

Original: Let Assumption 1 hold.
Revised: Let Assumption 1 hold.

Original: Then, the following statements hold.
Revised: Then, the following statements hold.

Original: Mapping y ∗ ( x ) is κ -Lipschitz continuous; 2.
Revised: Mapping y ∗ ( x ) is κ -Lipschitz continuous; 2.

Original: Function Φ( x ) is L 1 (1 + κ ) -smooth and ∇ Φ( x )
Revised: Function Φ( x ) is L 1 (1 + κ ) -smooth and ∇ Φ( x )

Original: = ∇ 1 f ( x, y ∗ ( x )) ; 3.
Revised: = ∇ 1 f ( x, y ∗ ( x )) ; 3.

Original: Deﬁne G ( x, y ) =
Revised: Deﬁne G ( x, y ) =

Original: ∇ 11 f ( x, y ) − ∇ 12 f ( x, y )
Revised: ∇ 11 f ( x, y ) − ∇ 12 f ( x, y )

Original: [ ∇ 22 f ( x, y )]
Revised: [ ∇ 22 f ( x, y )]

Original: − 1 ∇ 21 f ( x, y ) .
Revised: − 1 ∇ 21 f ( x, y ) .

Original: Then, G is a Lipschitz mapping with constant L G =
Revised: Then, G is a Lipschitz mapping with constant L G =

Original: L 2 (1 + κ ) 2 , i.e., (cid:107) G ( x (cid:48) , y (cid:48) )
Revised: L 2 (1 + κ ) 2 , i e., (cid:107) G ( x (cid:48) , y (cid:48) )

Original: − G ( x, y ) (cid:107) ≤
Revised: − G ( x, y ) (cid:107) ≤

Original: L G (cid:107) ( x (cid:48) , y (cid:48) ) − ( x, y ) (cid:107) ;4.
Revised: L G (cid:107) ( x (cid:48) , y (cid:48) ) − ( x, y ) (cid:107) ;

Original: The Hessian of Φ satisﬁes ∇ 2 Φ( x ) =
Revised: The Hessian of Φ satisﬁes ∇ 2 Φ( x ) =

Original: G ( x, y ∗ ( x )), which is Lipschitz continuous with constant L Φ =
Revised: G ( x, y ∗ ( x )) which is Lipschitz continuous with constant L Φ =

Original: L G (1 + κ ) =
Revised: L G (1 + κ ) =

Original: L 2 (1 + κ ) 3 .
Revised: L 2 ( + κ ) 3 .

Original: The ﬁrst two items characterize the gradient of the envelope function Φ in terms of the partial gradient of the bi-variate objective function f .
Revised: The ﬁrst two items characterize the gradient of the envelope function Φ in terms of the partial gradient of the bi-variate objective function f .

Original: They are proved in the previous work (Lin et al., 2020) and we include them for completeness.
Revised: They are proved in the previous work (Lin et al., 2020) and we include them for completeness.

Original: On the other hand, the last two items further characterize the Hessian of Φ in terms of the block Jacobian matrices of f .
Revised: On the other hand, the last two items further characterize the Hessian of Φ in terms of the block Jacobian matrices of f .

Original: As we present in the next section, the Lipschitz continuous Hessian ∇ 2 Φ( x ) allows us to develop a cubic-regularization-based algorithm for ﬁnding second-order stationary points.
Revised: As we present in the next section, the Lipschitz continuous Hessian ∇ 2 Φ( x ) allows us to develop a cubic-regularization-based algorithm for ﬁnding second-order stationary points.

Original: We also note that the proof of items 3 & 4 are not trivial.
Revised: We also note that the proof of items 3 & 4 are not trivial.

Original: Speciﬁcally, we need to ﬁrst develop bounds for the spectrum norm of the block Jacobian matrices in Lemma(see the ﬁrst page of the appendix), which helps prove the Lipschitz continuity of the G mapping in item 3.
Revised: Speciﬁcally, we need to ﬁrst develop bounds for the spectrum norm of the block Jacobian matrices in Lemma(see the ﬁrst page of the appendix), which helps prove the Lipschitz continuity of the G mapping in item 3.

Original: Moreover, we leverage the optimality condition of f ( x, · ) to derive an expression for the minimizer mapping y ∗  x ) (see (15) in the appendix), which is used to further prove item 4.
Revised: Moreover, we leverage the optimality condition of f ( x, · ) to derive an expression for the maximizer mapping y ∗ ( x ) (see (15) in the appendix), which is used to further prove item 4.

Original:  C UBIC -GDA: C UBIC -R EGULARIZED G RADIENT D ESCENT -A SCENT
Revised:  C UBIC -GDA: C UBIC -R EGULARIZED G RADIENT D ESCENT -A SCENT

Original: In this section, we propose a new Gradient Descent-Ascent (GDA) algorithm that leverages the cubic regularization technique (Nesterov and Polyak, 2006) to escape strict saddle points and ﬁnd second-order stationary points of the nonconvex minimax problem (P).
Revised: In this section, we propose a new Gradient Descent-Ascent (GDA) algorithm that leverages the cubic regularization technique (Nesterov and Polyak, 2006) to escape strict saddle points and ﬁnd second-order stationary points of the nonconvex minimax problem (P).

Original: Our algorithm design is inspired by the conventional cubic regularization algorithm (Nesterov and
Revised: Our algorithm design is inspired by the conventional cubic regularization algorithm (Nesterov and

Original: Polyak, 2006).
Revised: Polyak, 2006).

Original: Speciﬁcally, to ﬁnd a second-order stationary point of the envelope function Φ( x ) , the conventional cubic regularization algorithm would perform the following iterative update.
Revised: Speciﬁcally, to ﬁnd a second-order stationary point of the envelope function Φ( x ) , the conventional cubic regularization algorithm would perform the following iterative update.

Original: [Equation]
Revised: [Equation]

Original: where η x > 0 is a proper learning rate.
Revised: where η x > 0 is a proper learning rate.

Original: However, due to the special maximization structure of Φ , its gradient and Hessian have complex formulas (see Proposition 1) that involve the mapping y ∗ ( x ) , which cannot be computed exactly in practice.
Revised: However, due to the special maximization structure of Φ , its gradient and Hessian have complex formulas (see Proposition 1) that involve the mapping y ∗ ( x ) , which cannot be computed exactly in practice.

Original: Hence, we aim to develop a new algorithm to efﬁciently compute approximations of the gradient and Hessian of Φ and use them to perform the cubic regularization update.
Revised: Hence, we aim to develop a new algorithm to efﬁciently compute approximations of the gradient and Hessian of Φ and use them to perform the cubic regularization update.

Original: To perform the cubic regularization update in eq.
Revised: To perform the cubic regularization update in eq.

Original: (1), we need to compute ∇ Φ( x t ) = ∇ 1 f ( x t , y ∗ ( x t )) and ∇ 2 Φ( x t )
Revised: (1), we need to compute ∇ Φ( x t ) = ∇ 1 f ( x t , y ∗ ( x t )) and ∇ 2 Φ( x t )

Original: =
Revised: =

Original: G ( x t , y ∗ ( x t ))
Revised: G ( x t , y ∗ ( x t ))

Original: (by Proposition 1), both of which depend on the minimizer y ∗ ( x t ) of the function f ( x t , · ) .
Revised: (by Proposition 1), both of which depend on the maximizer y ∗ ( x t ) of the function f ( x t , · ) .

Original: Since f ( x t , · ) is strongly-concave, we can run N t iterations of gradient ascent to obtain an approximated minimizer (cid:101) y N t
Revised: Since f ( x t , · ) is strongly-concave, we can run N t iterations of gradient ascent to obtain an approximated maximizer (cid:101) y N t

Original: ≈ y ∗ ( x t ) , and then approximate ∇ Φ( x t ) , ∇ 2 Φ( x t ) using ∇ 1 f ( x t , (cid:101) y N t ) and G ( x t , (cid:101) y N t ) , respectively.
Revised: ≈ y ∗ ( x t ) , and then approximate ∇ Φ( x t ) , ∇ 2 Φ( x t ) using ∇ 1 f ( x t , (cid:101) y N t ) and G ( x t , (cid:101) y N t ) , respectively.

Original: Intuitively, these are good approximations due to two reasons: (i) (cid:101) y N t converges to y ∗ ( x t ) at a fast linear convergence rate; and (ii) both ∇ 1 f and
Revised: Intuitively, these are good approximations due to two reasons: (i) (cid:101) y N t converges to y ∗ ( x t ) at a fast linear convergence rate; and (ii) both ∇ 1 f and

Original: G are shown in Proposition 1 to be Lipschitz continuous in their second argument.
Revised: G are shown in Proposition 1 to be Lipschitz continuous in their second argument.

Original: We refer to this algorithm as Cubic-Regularized Gradient Descent-Ascent (Cubic-GDA) , and summarize its update rule in Algorithm 1 below.
Revised: We refer to this algorithm as Cubic-Regularized Gradient Descent-Ascent (Cubic-GDA) , and summarize its update rule in Algorithm 1 below.

Original: Algorithm
Revised: Algorithm

Original: Input: Initialize x 0 , y 0 and learning rates η x , η y .
Revised: Input: Initialize x 0 , y 0 and learning rates η x , η y .

Original: for t = 0 , 1 , 2 , . . .
Revised: for t = 0 , 1 , 2 , . . .

Original: , T − 1 do
Revised: , T − 1 do

Original: Initialize (cid:101) y 

Revised: Initialize (cid:101) y

Original: = y t .
Revised: = y t .

Original: for k = 0 , 1 , 2 , . . .
Revised: for k = 0 , 1 , 2 , . . .

Original: , N t − 1 do (cid:101)
Revised: , N t − 1 do (cid:101)

Original: y k +1 =
Revised: y k +1 =

Original: (cid:101)
Revised: (cid:101)

Original: y k
Revised: y k

Original: + η y ∇ 2 f ( x t , (cid:101) y k ) .
Revised: + η y ∇ 2 f ( x t , (cid:101) y k ) .

Original: end
Revised: end

Original: Set y t +1 = (cid:101)
Revised: Set y t +1 = (cid:101)

Original: y N t and compute G ( x t , y t +1 ) as follows: 
Revised: y N t and compute G ( x t , y t +1 ) as follows: G

Original: [Equation]
Revised: [Equation]

Original: , y t +1 )] − 1 ∇ 21 f ( x t , y t +1 ) .
Revised: t [Equation]

Original: x t
Revised: t

Original: [Table]
Revised: [Equation]

Original: ∈ arg
Revised: t [Equation] [Equation] +

Original: [Table]
Revised: [Equation]

Original: [Table]
Revised: x t +1 ∈ arg min x ∇ 1 f ( x t , y t +1 ) (cid:62) ( x − x t ) + 12 ( x − x t ) (cid:62) G ( x t , y t +1 ) ( x − x t ) + 1 6 η x (cid:107) x − x t (cid:107) 3 .

Original: endOutput: x T , y T .
Revised: endOutput: x T , y T .

Original: We further comment on the implementation of Cubic-GDA.
Revised: We further comment on the implementation of Cubic-GDA.

Original: We note that the Cubic-GDA updates in
Revised: We note that the Cubic-GDA updates in

Original: Algorithm 1 can be implemented in a computation efﬁcient way.
Revised: Algorithm 1 can be implemented in a computation efﬁcient way.

Original: First, note that Cubic-GDA involves an inner loop that performs gradient ascent updates.
Revised: First, note that Cubic-GDA involves an inner loop that performs gradient ascent updates.

Original: To guarantee the convergence of the algorithm, we prove in the next section that the number of inner iterations N t only needs to be kept at logarithm scale.
Revised: To guarantee the convergence of the algorithm, we prove in the next section that the number of inner iterations N t only needs to be kept at logarithm scale.

Original: Therefore, a few number of inner iterations sufﬁce to guarantee convergence in practice.
Revised: Therefore, a few number of inner iterations sufﬁce to guarantee convergence in practice.

Original: Second, the cubic regularization sub-problem can be efﬁciently solved by the gradient descent algorithm (Carmon and Duchi, 2016), and it involves computation of only Jacobian-vector product that can be efﬁciently computed by the existing machine learning platforms such as TensorFlow (Abadi et al., 2015) and
Revised: Second, the cubic regularization sub-problem can be efﬁciently solved by the gradient descent algorithm (Carmon and Duchi, 2016), and it involves computation of only Jacobian-vector product that can be efﬁciently computed by the existing machine learning platforms such as TensorFlow (Abadi et al., 2015) and

Original: PyTorch (Paszke et al., 2019).
Revised: PyTorch (Paszke et al., 2019).

Original: In particular, to compute the Hessian-vector product G ( x, y ) v for any vector v , one needs to compute the Jacobian-vector product ∇ 11 f ( x, y ) v and the matrix-vector product ∇ 12 f ( x, y )
Revised: In particular, to compute the Hessian-vector product G ( x, y ) v for any vector v , one needs to compute the Jacobian-vector product ∇ 11 f ( x, y ) v and the matrix-vector product ∇ 12 f ( x, y )

Original: [ ∇ 22 f ( x, y )]
Revised: [ ∇ 22 f ( x, y )]

Original: − 1 ∇ 21 f ( x, y ) v .
Revised: − 1 ∇ 21 f ( x, y ) v .

Original: We note that this matrix-vector product term can be computed as follows: ﬁrst compute the Jacobian-vector product b = ∇ 21 f ( x, y ) v ; Then, solve the invertible linear system ∇ 22 f ( x, y ) u
Revised: We note that this matrix-vector product term can be computed as follows: ﬁrst compute the Jacobian-vector product b = ∇ 21 f ( x, y ) v ; Then, solve the invertible linear system ∇ 22 f ( x, y ) u

Original: = b using any standard solver (e.g., conjugate gradient method), which involves iteratively computing Jacobian-vector products ∇ 22 f ( x, y ) w for some vector w ; Finally, compute the Jacobian-vector product ∇ 12 f ( x, y ) u .
Revised: = b using any standard solver (e.g., conjugate gradient method), which involves iteratively computing Jacobian-vector products ∇ 22 f ( x, y ) w for some vector w ; Finally, compute the Jacobian-vector product ∇ 12 f ( x, y ) u .

Original: Hence, one can call multiple
Revised: Hence, one can call multiple

Original: Jacobian-vector product oracles to solve the cubic regularization sub-problem.
Revised: Jacobian-vector product oracles to solve the cubic regularization sub-problem.

Original:  G LOBAL C ONVERGENCE P ROPERTIES OF C UBIC -GDA
Revised:  G LOBAL C ONVERGENCE P ROPERTIES OF C UBIC -GDA

Original: In this section, we study the global convergence properties of Cubic-GDA.
Revised: In this section, we study the global convergence properties of Cubic-GDA.

Original: Importantly, our analysis is based on characterizing an intrinsic potential function of the Cubic-GDA algorithm in nonconvex minimax optimization.
Revised: Importantly, our analysis is based on characterizing an intrinsic potential function of the Cubic-GDA algorithm in nonconvex minimax optimization.

Original: Recall that our goal is to ﬁnd a second-order stationary point of the function Φ( x ) .
Revised: Recall that our goal is to ﬁnd a second-order stationary point of the function Φ( x ) .

Original: Our next result shows that Cubic-GDA admits an intrinsic potential function that monotonically decreases in the optimization process.
Revised: Our next result shows that Cubic-GDA admits an intrinsic potential function that monotonically decreases in the optimization process.

Original: The proof of is included in Appendix B.
Revised: The proof of is included in Appendix B.

Original: Proposition 2.
Revised: Proposition 2.

Original: Let Assumption 1 hold.
Revised: Let Assumption 1 hold.

Original: Deﬁne the following potential function
Revised: Deﬁne the following potential function

Original: [Equation]
Revised: [Equation]

Original: and 
Revised: and denote H t : = H ( x t , x t − 1 , y t +1 ) . Choose N t ≥ max

Original: [Equation]
Revised: [Equation]

Revised: ln2 ,

Original: [Equation]
Revised: [Equation]

Original: ) (cid:107) /µ ] − 2ln (cid:107) x t − x t − 1 (cid:107) ln
Revised: [Equation]

Original: [ κ/ ( κ − 1)] and learning rates η x ≤ (8 L 1 + L Φ + 3 κ 7 L G ) − 1 , η y ≤ 2 L + µ .
Revised: [Equation] and learning rates η x ≤ [Equation] 2 L + µ .

Original: Then, the sequences { x t , y t } t generated by Cubic-GDA satisfy, for all t = 0 , 1 , 2 , ...
Revised: Then, the sequences { x t , y t } t generated by Cubic-GDA satisfy, for all t = 0 , 1 , 2 , ...

Original: [Equation]
Revised: [Equation]

Revised: Consequently, it holds that

Original: [Equation]
Revised: [Equation]

Original: Consequently, it holds that 
Revised: Remark 1. We note that the above key result can also be established for an inexact version of CubicGDA, which formulates the cubic subproblem with a general inexact gradient p t ≈ ∇ 1 f ( x t , y t +1 ) and inexact Jacobian P t ≈ G ( x t , y t +1 ) that satisfy the conditions

Original: [Equation]
Revised: [Equation]

Revised: These inexactness conditions are widely studied in the existing literature Cartis et al.

Revised: (2011b;a).

Revised: Under these inexact conditions, our proof of the above proposition remains unchanged, except that the coefﬁcients of the term (cid:107) x t +1 − x t (cid:107) 3 would be slightly different.

Original: Proposition 2 reveals that Cubic-GDA admits an intrinsic potential function H , which is the objective function Φ( x ) regularized by two cubic terms (cid:107) x (cid:48)
Revised: Proposition 2 reveals that Cubic-GDA admits an intrinsic potential function H , which is the objective function Φ( x ) regularized by two cubic terms (cid:107) x (cid:48)

Original: − x (cid:107) 3 , (cid:107) y − y ∗ ( x ) (cid:107) 3 .
Revised: − x (cid:107) 3 , (cid:107) y − y ∗ ( x ) (cid:107) 3 .

Original: Such a potential function is closely connected to the optimization goal.
Revised: Such a potential function is closely connected to the optimization goal.

Original: Speciﬁcally, consider a desired case where x t converges to a certain second-order stationary point x ∗ and y t converges to y ∗ ( x ∗ ) , it is clear that the potential function H t would converge to the corresponding function value Φ( x ∗ ) .
Revised: Speciﬁcally, consider a desired case where x t converges to a certain second-order stationary point x ∗ and y t converges to y ∗ ( x ∗ ) , it is clear that the potential function H t would converge to the corresponding function value Φ( x ∗ ) .

Original: Hence, minimizing the function Φ is equivalent to minimizing the potential function H .
Revised: Hence, minimizing the function Φ is equivalent to minimizing the potential function H .

Original: More importantly, Propositionshows that this potential function is monotonically decreasing along the optimization path of CubicGDA, implying that the algorithm continuously makes optimization progress.
Revised: More importantly, Propositionshows that this potential function is monotonically decreasing along the optimization path of CubicGDA, implying that the algorithm continuously makes optimization progress.

Original: By leveraging this property of the potential function, we are able to show that the parameter sequences generated by
Revised: By leveraging this property of the potential function, we are able to show that the parameter sequences generated by

Original: Cubic-GDA are asymptotically stable, i.e., x t +1 − x t → 0 , y t → y ∗ ( x t ) .
Revised: Cubic-GDA are asymptotically stable, i.e., x t +1 − x t → 0 , y t → y ∗ ( x t ) .

Original: Remark 1.
Revised: Remark 2.

Original: In each outer iteration t , we set the total number of inner gradient ascent iterations
Revised: In each outer iteration t , we set the total number of inner gradient ascent iterations

Original: N t based on (cid:107)∇ 2 f ( x t , y t ) (cid:107) and (cid:107) x
Revised: N t based on (cid:107)∇ 2 f ( x t , y t ) (cid:107) and (cid:107) x

Original: t − x t − 1 (cid:107) .
Revised: t − x t − 1 (cid:107) .

Original: Note that both of these two quantities can be easily computed right after iteration t − 1 .
Revised: Note that both of these two quantities can be easily computed right after iteration t − 1 .

Original: Based on Proposition 2, we are able to prove the convergence of { x t } t to a certain second-order stationary point, which we formally present in the next theorem.
Revised: Based on Proposition 2, we are able to prove the convergence of { x t } t to a certain second-order stationary point, which we formally present in the next theorem.

Original: The proof is included in Appendix C.
Revised: The proof is included in Appendix C.

Original: Theorem 1 (Global convergence) .
Revised: Theorem 1 (Global convergence) .

Original: Under the same conditions as those of Proposition 2, the CubicGDA has the following global convergence properties.
Revised: Under the same conditions as those of Proposition 2, the CubicGDA has the following global convergence properties.

Original: The function value sequence { Φ( x t ) } t converges to a ﬁnite limit H ∗ > −∞ ; 2.
Revised: The function value sequence { Φ( x t ) } t converges to a ﬁnite limit H ∗ > −∞ ; 2.

Original: The generated sequences { x t } t , { y t } t are bounded and have a compact sets of limit points.
Revised: The generated sequences { x t } t , { y t } t are bounded and have a compact sets of limit points.

Original: Every limit point x ∗ of { x t } t is a second-order stationary point of Φ , i
Revised: Every limit point x ∗ of { x t } t is a second-order stationary point of Φ , i

Original: e., ∇ Φ( x ∗ ) = 0 , ∇ 2 Φ( x ∗ ) (cid:23) 0 , and satisﬁes Φ( x ∗ ) =
Revised: e., ∇ Φ( x ∗ ) = 0 , ∇ 2 Φ( x ∗ ) (cid:23) 0 , and satisﬁes Φ( x ∗ ) =

Original: H ∗ .
Revised: H ∗ .

Original: The above theorem establishes the global convergence properties of Cubic-GDA.
Revised: The above theorem establishes the global convergence properties of Cubic-GDA.

Original: Speciﬁcally, item 1 shows that the function value sequence { Φ( x t ) } t converges to a ﬁnite limit H ∗ , which is also the limit of the potential function sequence { H t } t .
Revised: Speciﬁcally, item 1 shows that the function value sequence { Φ( x t ) } t converges to a ﬁnite limit H ∗ , which is also the limit of the potential function sequence { H t } t .

Original: Moreover, items 2 & 3 further show that all the limit points of { x t } t are second-order stationary points of the minimax problem, at which the function Φ achieves the constant value H ∗ .
Revised: Moreover, items 2 & 3 further show that all the limit points of { x t } t are second-order stationary points of the minimax problem, at which the function Φ achieves the constant value H ∗ .

Original: These results show that Cubic-GDA is guaranteed to ﬁnd second-order stationary points in nonconvex minimax optimization.
Revised: These results show that Cubic-GDA is guaranteed to ﬁnd second-order stationary points in nonconvex minimax optimization.

Original: By further leveraging the potential function characterized in Proposition 2, we obtain the following global convergence rate of Cubic-GDA to a second-order stationary point.
Revised: By further leveraging the potential function characterized in Proposition 2, we obtain the following global convergence rate of Cubic-GDA to a second-order stationary point.

Original: The proof is included in Appendix D. Throughout, we adopt the following standard measure of second-order stationary introduced in (Nesterov and Polyak, 2006).
Revised: The proof is included in Appendix D. Throughout, we adopt the following standard measure of second-order stationary introduced in (Nesterov and Polyak, 2006).

Original: [Equation]
Revised: [Equation]

Original: Intuitively, a smaller µ ( x ) means that the point x is closer to being second-order stationary.
Revised: Intuitively, a smaller µ ( x ) means that the point x is closer to being second-order stationary.

Original: Theorem 2 (Global convergence rate) .
Revised: Theorem 2 (Global convergence rate) .

Original: Under the same conditions as those of Proposition 2, the
Revised: Under the same conditions as those of Proposition 2, the

Original: Cubic-GDA converges at the following rate for all T ≥ H 0 − inf x ∈ R m. Φ( x ) L 1 /.
Revised: Cubic-GDA converges at the following rate for all T ≥ H 0 − inf x ∈ R m Φ( x ) L 2 κ 3 /

Original: [Equation]
Revised: [Equation]

Original: The above theorem shows that the ﬁrst-order stationary measure (cid:107)∇ Φ( x t ) (cid:107) converges at a sublinear rate O
Revised: The above theorem shows that the ﬁrst-order stationary measure (cid:107)∇ Φ( x t ) (cid:107) converges at a sublinear rate O

Original: ( T − 23 ) , and the second-order stationary measure − λ min (cid:0) ∇ 2 Φ( x ) (cid:1) converges at a sublinear rate O
Revised: ( T − 23 ) , and the second-order stationary measure − λ min (cid:0) ∇ 2 Φ( x ) (cid:1) converges at a sublinear rate O

Original: ( T − 13 ) .
Revised: ( T − 13 ) .

Original: Both results match the convergence rates of the cubic regularization algorithm for nonconvex minimization (Nesterov and Polyak, 2006).
Revised: Both results match the convergence rates of the cubic regularization algorithm for nonconvex minimization (Nesterov and Polyak, 2006).

Original: Therefore, by leveraging the curvature of the approximated Hessian matrix G ( x t , y t +1 ) , Cubic-GDA is able to escape strict saddle points of Φ at a fast rate.
Revised: Therefore, by leveraging the curvature of the approximated Hessian matrix G ( x t , y t +1 ) , Cubic-GDA is able to escape strict saddle points of Φ at a fast rate.

Original: We note that the proof of the global convergence results in Theorems 1 and 2 are critically based on the intrinsic potential function H that we characterized in Proposition 2.
Revised: We note that the proof of the global convergence results in Theorems 1 and 2 are critically based on the intrinsic potential function H that we characterized in Proposition 2.

Original: We elaborate our technical contribution as follows.
Revised: We elaborate our technical contribution as follows.

Original: • First, to identify the potential function, we need to characterize the per-iteration progress induced by the cubic regularization step.
Revised: • First, to identify the potential function, we need to characterize the per-iteration progress induced by the cubic regularization step.

Original: However, the cubic subproblem in Cubic-GDA is constructed by an inexact gradient ∇ 1 f ( x t , y t +1 ) and Hessian matrix G ( x t , y t +1 ) .
Revised: However, the cubic subproblem in Cubic-GDA is constructed by an inexact gradient ∇ 1 f ( x t , y t +1 ) and Hessian matrix G ( x t , y t +1 ) .

Original: Therefore, we need to properly choose the number of inner gradient ascent iterations N t to control the estimation error of both the gradient and Hessian approximations at a desired level.
Revised: Therefore, we need to properly choose the number of inner gradient ascent iterations N t to control the estimation error of both the gradient and Hessian approximations at a desired level.

Original: • Due to the inexactness of the gradient and Hessian matrix, the cubic regularization update of Cubic-GDA does not lead to a monotonically decreasing function value Φ( x t ) , as opposed to the original cubic regularization algorithm in nonconvex minimization (which uses exact gradient and Hessian).
Revised: • Due to the inexactness of the gradient and Hessian matrix, the cubic regularization update of Cubic-GDA does not lead to a monotonically decreasing function value Φ ( x t ) , as opposed to the original cubic regularization algorithm in nonconvex minimization (which uses exact gradient and Hessian).

Original: Hence, we construct and identify a decreasing potential function H instead.
Revised: Hence, we construct and identify a decreasing potential function H instead.

Original:  C ONVERGENCE A NALYSIS UNDER L OCAL N ONCONVEX G EOMETRY
Revised:  C ONVERGENCE A NALYSIS UNDER L OCAL N ONCONVEX G EOMETRY

Original: The (2) of Proposition 2 shows that Cubic-GDA has a special optimization dynamics and therefore its convergence rate is expected to be different from that of the vanilla GDA in nonconvex minimax optimization.
Revised: The (2) of Proposition 2 shows that Cubic-GDA has a special optimization dynamics and therefore its convergence rate is expected to be different from that of the vanilla GDA in nonconvex minimax optimization.

Original: In this section, we explore the convergence rates of Cubic-GDA under a broad spectrum of local nonconvex geometries characterized by the Łojasiewicz gradient inequality.
Revised: In this section, we explore the convergence rates of Cubic-GDA under a broad spectrum of local nonconvex geometries characterized by the Łojasiewicz gradient inequality.

Original: We ﬁrst introduce the Łojasiewicz gradient geometry of a function h .
Revised: We ﬁrst introduce the Łojasiewicz gradient geometry of a function h .

Original: Throughout, the point-to-set distance is denoted as dist Ω ( x ) :=
Revised: Throughout, the point-to-set distance is denoted as dist Ω ( x ) :=

Original: inf u ∈ Ω (cid:107) x − u (cid:107) .
Revised: inf u ∈ Ω (cid:107) x − u (cid:107) .

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A differentiable function h is said to satisfy the Łojasiewicz gradient geometry if for every compact set Ω of critical points on which h takes a constant value h Ω ∈ R , there exist ε, λ >such that for all x ∈ { z ∈ R m : dist Ω ( z ) <
Revised: A differentiable function h is said to satisfy the Łojasiewicz gradient geometry if for every compact set Ω of critical points on which h takes a constant value h Ω ∈ R , there exist ε, λ >such that for all x ∈ { z ∈ R m : dist Ω ( z ) <

Original: ε, h Ω < h ( z ) <
Revised: ε, h Ω < h [Equation]

Original:  h Ω + λ } , the following condition holds.
Revised: z ) < h Ω + λ } , the following condition holds.

Original: [Equation]
Revised: [Equation]

Original: where c > 0 is a universal constant and θ ∈ (1 , + ∞ ) is the geometry parameter.
Revised: where c > 0 is a universal constant and θ ∈ (1 , + ∞ ) is the geometry parameter.

Original: Intuitively, the Łojasiewicz gradient geometry is a gradient-dominant-type geometry that characterizes the local geometry of a nonconvex function around the set of critical points.
Revised: Intuitively, the Łojasiewicz gradient geometry is a gradient-dominant-type geometry that characterizes the local geometry of a nonconvex function around the set of critical points.

Original: In particular, it generalizes the Polyak-Łojasiewicz (PL) geometry that corresponds to the special case θ = 2 Łojasiewicz (1963);
Revised: In particular, it generalizes the Polyak-Łojasiewicz (PL) geometry that corresponds to the special case θ = 2 Łojasiewicz (1963);

Original: Karimi et al.
Revised: Karimi et al.

Original: In fact, a generalized version of the Łojasiewicz gradient geometry has been shown to hold for a large class of functions including sub-analytic functions, exponential functions and semi-algebraic functions, which cover most of the nonconvex functions encountered in machine learning applications (Zhou et al., 2016;
Revised: In fact, a generalized version of the Łojasiewicz gradient geometry has been shown to hold for a large class of functions including sub-analytic functions, exponential functions and semi-algebraic functions, which cover most of the nonconvex functions encountered in machine learning applications (Zhou et al., 2016;

Original: Yue et al., 2019; Zhou and Liang, 2017; Zhou et al., 2018).
Revised: Yue et al., 2019; Zhou and Liang, 2017; Zhou et al., 2018).

Revised: For example, consider the class of robust machine learning problems that involve the minimax problem

Revised: [Equation]

Revised: Here ( x i , y i ) is the i -th data sample that includes, e.g., an image x i and its label y i , ξ i denotes the adversarial image, h θ is a classiﬁcation model parameterized by θ , and (cid:96) denotes the loss function.

Revised: Such a problem is nonconvex-stongly-concave when λ is large.

Revised: In particular, as elaborated in the appendix of (Bolte et al., 2014), the envelop function Φ( x ) := max ξ i 1 n (cid:80) ni =1 (cid:96) ( h θ ( ξ i ) , y i ) 2 − λ 2 (cid:107) ξ i − a i (cid:107) 2 satisﬁes the local Lojasiewicz gradient geometry if it is semi-algebraic, which holds if every sample loss f ( θ, ξ i ) := (cid:96) ( h θ ( ξ i ) , y i ) 2 − λ 2 (cid:107) ξ i − a i (cid:107) 2 is semi-algebraic.

Original: By (2) of Proposition 2 and (29) (proved in Appendix E), we show that the potential function H of
Revised: By (2) of Proposition 2 and (30) (proved in Appendix E), we show that the potential function H of

Original: Cubic-GDA satisﬁes the following special optimization dynamics.
Revised: Cubic-GDA satisﬁes the following special optimization dynamics.

Original: [Equation]
Revised: [Equation]

Original: The above dynamics of Cubic-GDA involves higher-order terms than the dynamics of GDA, which takes the form H t +1 − H t ≤ −O ( (cid:107) x t +1 − x t (cid:107) 2 ) and (cid:107)∇ H t (cid:107) ≤ O ( (cid:107) x t +1 − x t (cid:107) )
Revised: The above dynamics of Cubic-GDA involves higher-order terms than the dynamics of GDA, which takes the form H t +1 − H t ≤ −O ( (cid:107) x t +1 − x t (cid:107) 2 ) and (cid:107)∇ H t (cid:107) ≤ O ( (cid:107) x t +1 − x t (cid:107) )

Original: (Chen et al., 2021).
Revised: (Chen et al., 2021).

Original: Thus, it is expected that Cubic-GDA achieves a faster convergence rate than GDA.
Revised: Thus, it is expected that Cubic-GDA achieves a faster convergence rate than GDA.

Original: On the other hand, compare with the dynamics of the cubic regularization algorithm
Revised: On the other hand, compare with the dynamics of the cubic regularization algorithm

Original: (Zhou et al., 2018), the gradient dynamic of Cubic-GDA in (5) involves an additional term (cid:107) x t − 1 − x t − 2 (cid:107) 2 that depends on the history, which is due to the inexact gradient and Hessian used in the cubic regularization step.
Revised: (Zhou et al., 2018), the gradient dynamic of Cubic-GDA in (5) involves an additional term (cid:107) x t − 1 − x t − 2 (cid:107) 2 that depends on the history, which is due to the inexact gradient and Hessian used in the cubic regularization step.

Original: With the optimization dynamics in (4) and (5) and by leveraging the Łojasiewicz gradient geometry, we are able to prove the following strengthened convergence result of Cubic-GDA.
Revised: With the optimization dynamics in (4) and (5) and by leveraging the Łojasiewicz gradient geometry, we are able to prove the following strengthened convergence result of Cubic-GDA.

Original: Theorem 3.
Revised: Theorem 3.

Original: Let Assumption 1 hold and assume that the potential function H satisﬁes the local Łojasiewicz gradient geometry.
Revised: Let Assumption 1 hold and assume that the potential function H satisﬁes the local Łojasiewicz gradient geometry.

Original: Choose the hyperparameters N t , η x , η y in the same way as Proposition 2.
Revised: Choose the hyperparameters N t , η x , η y in the same way as Proposition 2.

Original: Then, the sequences { ( x t , y t ) } t generated by Cubic-GDA have a unique limit point, which is a second-order stationary point of Φ .
Revised: Then, the sequences { ( x t , y t ) } t generated by Cubic-GDA have a unique limit point, which is a second-order stationary point of Φ .

Original: Recall that Theorem 1 only proves that every limit point of { x t } t is a second-order stationary point of the minimax problem.
Revised: Recall that Theorem 1 only proves that every limit point of { x t } t is a second-order stationary point of the minimax problem.

Original: Theorem 3 further strengthens Theorem 1 by showing that Cubic-GDA converges to a unique second-order stationary limit point under the Łojasiewicz gradient geometry.
Revised: Theorem 3 further strengthens Theorem 1 by showing that Cubic-GDA converges to a unique second-order stationary limit point under the Łojasiewicz gradient geometry.

Original: Next, we further study the asymptotic convergence rates of Cubic-GDA under different parameter ranges of the Łojasiewicz gradient geometry.
Revised: Next, we further study the asymptotic convergence rates of Cubic-GDA under different parameter ranges of the Łojasiewicz gradient geometry.

Original: We ﬁrst obtain the following function value convergence rate result, which strengthens the convergence rate result established in Theorem 2.
Revised: We ﬁrst obtain the following function value convergence rate result, which strengthens the convergence rate result established in Theorem 2.

Original: Throughout, tdenotes a sufﬁciently large positive integer and C 0 is a universal positive constant deﬁned as
Revised: Throughout, tdenotes a sufﬁciently large positive integer and C 0 is a universal positive constant deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The proof is included in Appendix F.
Revised: The proof is included in Appendix F.

Original: Theorem 4 (Funtion value convergence rate) .
Revised: Theorem 4 (Funtion value convergence rate) .

Original: Under the same conditions as those of Theorem 3, the sequence of potential function { H t } t converges to the limit H ∗ at the following rates.
Revised: Under the same conditions as those of Theorem 3, the sequence of potential function { H t } t converges to the limit H ∗ at the following rates.

Original: If the geometry parameter θ ∈ ( 32 , ∞ ) , then H t ↓ H ∗ super-linearly as
Revised: If the geometry parameter θ ∈ ( 32 , ∞ ) , then H t ↓ H ∗ super-linearly as

Original: [Equation]
Revised: [Equation]

Original: If the geometry parameter θ = 32 , then H t ↓ H ∗ linearly as
Revised: If the geometry parameter θ = 32 , then H t ↓ H ∗ linearly as

Original: [Equation]
Revised: [Equation]

Original: If the geometry parameter θ ∈ (1 , 32 ) , then H t ↓ H ∗ sub-linearly as
Revised: If the geometry parameter θ ∈ (1 , 32 ) , then H t ↓ H ∗ sub-linearly as

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 3.

Original: We note that if the Łojasiewicz gradient geometry holds globally, then the above asymptotic convergence rates become global convergence rates.
Revised: We note that if the Łojasiewicz gradient geometry holds globally, then the above asymptotic convergence rates become global convergence rates.

Original: The above theorem characterizes the convergence rates of the potential function of Cubic-GDA in the full spectrum of θ of the local Łojasiewicz gradient geometry.
Revised: The above theorem characterizes the convergence rates of the potential function of Cubic-GDA in the full spectrum of θ of the local Łojasiewicz gradient geometry.

Original: Speciﬁcally, it shows that a larger θ implies that the local geometry is sharper and hence leads to a faster convergence rate.
Revised: Speciﬁcally, it shows that a larger θ implies that the local geometry is sharper and hence leads to a faster convergence rate.

Original: In particular, as we summarize in Table 1 in the introduction section, the convergence rate of Cubic-GDA is orderwise faster than that of the vanilla GDA for a wide range of the parameter of the Łojasiewicz gradient geometry.
Revised: In particular, as we summarize in Table 1 in the introduction section, the convergence rate of Cubic-GDA is orderwise faster than that of the vanilla GDA for a wide range of the parameter of the Łojasiewicz gradient geometry.

Original: This demonstrates the advantage of leveraging higher-order information in nonconvex minimax-optimization.
Revised: This demonstrates the advantage of leveraging higher-order information in nonconvex minimax-optimization.

Original: As a byproduct, we also obtain the following asymptotic convergence rates of the parameter sequences generated by Cubic-GDA under the Łojasiewicz gradient geometry.
Revised: As a byproduct, we also obtain the following asymptotic convergence rates of the parameter sequences generated by Cubic-GDA under the Łojasiewicz gradient geometry.

Original: The proof is included in
Revised: The proof is included in

Original: Appendix G.
Revised: Appendix G.

Original: Theorem 5 (Parameter convergence rate) .
Revised: Theorem 5 (Parameter convergence rate) .

Original: Under the same conditions as those of Theorem 3, the sequences { x t , y t } t generated by Cubic-GDA converge to their limits x ∗ , y ∗ ( x ∗ ) respectively at the following rates.
Revised: Under the same conditions as those of Theorem 3, the sequences { x t , y t } t generated by Cubic-GDA converge to their limits x ∗ , y ∗ ( x ∗ ) respectively at the following rates.

Original: If the geometry parameter θ ∈ ( 32 , ∞ ) , then ( x t , y t ) → ( x ∗ , y ∗ ( x ∗ )) super-linearly as
Revised: If the geometry parameter θ ∈ ( 32 , ∞ ) , then ( x t , y t ) → ( x ∗ , y ∗ ( x ∗ )) super-linearly as

Original: [Equation]
Revised: [Equation]

Original: y [Equation] If the geometry parameter θ = 32 , then ( x t , y t ) → ( x ∗ , y ∗ ( x ∗ )) linearly asmax (cid:8) (cid:107) x t − x ∗ (cid:107) , (cid:107) y t − y ∗ ( x ∗ ) (cid:107) (cid:9) [Equation]
Revised: If the geometry parameter θ = 32 , then ( x t , y t ) → ( x ∗ , y ∗ ( x ∗ )) linearly as

Original: [Equation]
Revised: [Equation]

Original: If the geometry parameter θ ∈ (1 , 32 ) , then ( x t , y t ) → ( x ∗ , y ∗ ( x ∗ )) sub-linearly as(cid:107) x t − x ∗ (cid:107) ≤ O (cid:16) ( [Table] (cid:17) , (cid:107)
Revised: If the geometry parameter θ ∈ (1 , 32 ) , then ( x t , y t ) → ( x ∗ , y ∗ ( x ∗ )) sub-linearly as

Original: [Table]

Original: [Table]

Original: ) (cid:107)

Original: [Table]
Revised: [Equation]

Original: O (cid:16) ( t

Original: [Table]

Original: 0 ) −

Original: [Table]

Original: [Table]

Original: 2 θ ) (cid:17) , ∀ t ≥ t 0 .

Original: It can be seen that, similar to the convergence rate results of the function value sequence, the convergence rate of the parameter sequence is also affected by the parameterization of the local geometry.
Revised: It can be seen that, similar to the convergence rate results of the function value sequence, the convergence rate of the parameter sequence is also affected by the parameterization of the local geometry.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we take one step further toward improving the convergence guarantee of GDA-type algorithms in nonconvex minimax optimization.
Revised: In this paper [Equation] we take one step further toward improving the convergence guarantee of GDA-type algorithms in nonconvex minimax optimization.

Original: We develop a Cubic-GDA algorithm that leverages the second-order information and the cubic regularization technique to effectively escape strict saddle points in nonconvex minimax optimization.
Revised: We develop a Cubic-GDA algorithm that leverages the second-order information and the cubic regularization technique to effectively escape strict saddle points in nonconvex minimax optimization.

Original: Our key observation is that Cubic-GDA has an intrinsic potential function that monotonically decreases in the optimization process, and this leads to a guaranteed global convergence of the algorithm.
Revised: Our key observation is that Cubic-GDA has an intrinsic potential function that monotonically decreases in the optimization process, and this leads to a guaranteed global convergence of the algorithm.

Original: Moreover, our convergence analysis shows that
Revised: Moreover, our convergence analysis shows that

Original: Cubic-GDA achieves a faster convergence rate than the standard GDA for a wide spectrum of gradient dominant-type nonconvex geometries.
Revised: Cubic-GDA achieves a faster convergence rate than the standard GDA for a wide spectrum of gradient dominant-type nonconvex geometries.

Original: In the future study, we will develop stochastic variants of
Revised: In the future study, we will develop stochastic variants of

Original: Cubic-GDA to further improve its computation efﬁciency.
Revised: Cubic-GDA to further improve its computation efﬁciency.

Original: Another interesting direction is to apply momentum techniques to further accelerate the convergence of this algorithm.
Revised: Another interesting direction is to apply momentum techniques to further accelerate the convergence of this algorithm.

Original: Appendix Table of Contents
Revised: Appendix Table of Contents

Original: A Proof of Proposition
Revised: A Proof of Proposition

Original: B Proof of Proposition
Revised: B Proof of Proposition

Original: C Proof of Theorem
Revised: C Proof of Theorem

Original: D Proof of Theorem
Revised: D Proof of Theorem

Original: E Proof of Theorem
Revised: E Proof of Theorem

Original: F Proof of Theorem
Revised: F Proof of Theorem

Original: G Proof of Theorem
Revised: G Proof of Theorem

Original: In this supplementary material, we present the proof of all the results claimed in the paper.
Revised: In this supplementary material, we present the proof of all the results claimed in the paper.

Original: We ﬁrst prove the following auxiliary lemma that bounds the spectral norm of the Jacobian matrices.
Revised: We ﬁrst prove the following auxiliary lemma that bounds the spectral norm of the Jacobian matrices.

Original: Lemma 1.
Revised: Lemma 1.

Original: Let Assumption 1 hold.
Revised: Let Assumption 1 hold.

Original: Then, for any x ∈ R m and y ∈ R n , the Jacobian matrices of f ( x, y ) satisfy the following bounds.
Revised: Then, for any x ∈ R m and y ∈ R n , the Jacobian matrices of f ( x, y ) satisfy the following bounds.

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst prove eq.
Revised: We ﬁrst prove eq.

Original: Consider any x ∈ R m and y ∈ R n .
Revised: Consider any x ∈ R m and y ∈ R n .

Original: By Assumption 1 we know that f ( x, · ) is µ -strongy concave, which implies that −∇ 22 f ( x, y ) (cid:23) µI .
Revised: By Assumption 1 we know that f ( x, · ) is µ -strongy concave, which implies that −∇ 22 f ( x, y ) (cid:23) µI .

Original: Thus, we further conclude that
Revised: Thus, we further conclude that

Original: [Equation]
Revised: [Equation]

Original: Next, we prove eq.
Revised: Next, we prove eq.

Original: Consider any x, u ∈ R m and y ∈ R n , we have
Revised: Consider any x, u ∈ R m and y ∈ R n , we have

Original: [Equation]
Revised: [Equation]

Original: which implies that (cid:107)∇ 21 f ( x, y ) (cid:107) ≤ L 1 .
Revised: which implies that (cid:107)∇ 21 f ( x, y ) (cid:107) ≤ L 1 .

Original: Since f is twice differentiable and has continuous secondorder derivative, we have ∇ 12 f ( x, y ) (cid:62) = ∇ 21 f ( x, y ) , and hence eq.
Revised: Since f is twice differentiable and has continuous secondorder derivative, we have ∇ 12 f ( x, y ) (cid:62) = ∇ 21 f ( x, y ) , and hence eq.

Original: (13) follows.
Revised: (13) follows.

Original: A P ROOF OF P ROPOSITION
Revised: A P ROOF OF P ROPOSITION

Original: Proposition 1.
Revised: Proposition 1.

Original: Let Assumption 1 hold.
Revised: Let Assumption 1 hold.

Original: Then, the following statements hold.
Revised: Then, the following statements hold.

Original: Mapping y ∗ ( x ) is κ -Lipschitz continuous; 2.
Revised: Mapping y ∗ ( x ) is κ -Lipschitz continuous; 2.

Original: Function Φ( x ) is L 1 (1 + κ ) -smooth and ∇ Φ( x )
Revised: Function Φ( x ) is L 1 (1 + κ ) -smooth and ∇ Φ( x )

Original: = ∇ 1 f ( x, y ∗ ( x )) ; 3.
Revised: = ∇ 1 f ( x, y ∗ ( x )) ; 3.

Original: Deﬁne G ( x, y ) =
Revised: Deﬁne G ( x, y ) =

Original: ∇ 11 f ( x, y ) − ∇ 12 f ( x, y )
Revised: ∇ 11 f ( x, y ) − ∇ 12 f ( x, y )

Original: [ ∇ 22 f ( x, y )]
Revised: [ ∇ 22 f ( x, y )]

Original: − 1 ∇ 21 f ( x, y ) .
Revised: − 1 ∇ 21 f ( x, y ) .

Original: Then, G is a Lipschitz mapping with constant L G =
Revised: Then, G is a Lipschitz mapping with constant L G =

Original: L 2 (1 + κ ) 2 ,
Revised: L 2 (1 + κ ) 2 ,

Original: i
Revised: i

Original: e., (cid:107) G ( x (cid:48) , y (cid:48) )
Revised: e., (cid:107) G ( x (cid:48) , y (cid:48) )

Original: − G ( x, y ) (cid:107) ≤
Revised: − G ( x, y ) (cid:107) ≤

Original: L G (cid:107) ( x (cid:48) , y (cid:48) ) − ( x, y ) (cid:107) ;
Revised: L G (cid:107) ( x (cid:48) , y (cid:48) ) − ( x, y ) (cid:107) ;

Original: The Hessian of Φ satisﬁes ∇ 2 Φ( x ) =
Revised: The Hessian of Φ satisﬁes ∇ 2 Φ( x ) =

Original: G ( x, y ∗ ( x )) , which is Lipschitz continuous with constant L Φ =
Revised: G ( x, y ∗ ( x )) , which is Lipschitz continuous with constant L Φ =

Original: L G (1 + κ ) =
Revised: L G (1 + κ ) =

Original: L 2 (1 + κ ) 3 .
Revised: L 2 (1 + κ ) 3 .

Original: Proof.
Revised: Proof.

Original: The items 1 & 2 are proved in Chen et al.
Revised: The items 1 & 2 are proved in Chen et al.

Original: (2021); Lin et al.
Revised: (2021); Lin et al.

Original: We ﬁrst prove the item 3.
Revised: We ﬁrst prove the item 3.

Original: Consider any x, x
Revised: Consider any x, x

Original: (cid:48) ∈
Revised: (cid:48) ∈

Original: R m and y, y (cid:48) ∈ R n .
Revised: R m and y [Equation] y (cid:48) ∈ R n .

Original: For convenience we denote z =
Revised: For convenience we denote z =

Original: ( x, y ) and z (cid:48) =
Revised: ( x, y ) and z (cid:48) =

Original: ( x (cid:48) , y (cid:48) ) .
Revised: ( x (cid:48) , y (cid:48) ) .

Original: Then, by Assumption 1 and using the bounds of Lemma 1, we have that (cid:107) G
Revised: Then, by Assumption 1 and using the bounds of Lemma [Equation] we have that (cid:107) G

Original: ( x (cid:48) , y(cid:48) ) − G ( x, y ) (cid:107)
Revised: [Equation] x [Equation] , y

Original: [Equation]
Revised: [Equation]

Original: 11 f ( x (cid:48) , y (cid:48) ) − ∇ 11 f ( x, y ) (cid:107)
Revised: G [Equation] x, y ) [Equation]

Original: + (cid:107)∇ 12 f ( x (cid:48) , y(cid:48) ) 
Revised: y ) [Equation] [Equation] (cid:107)(cid:107)∇

Original: − ∇ 12 f ( x, y ) (cid:107)(cid:107) [ ∇ 22 f ( x (cid:48) , y (cid:48) )] − 1 (cid:107)(cid:107)∇ 21 f ( x (cid:48) , y (cid:48) )
Revised: [Equation] x [Equation] [Equation]

Original: (cid:107) +
Revised: [Equation]

Original: (cid:107)∇ 12 f ( x, y ) (cid:107)(cid:107) [ ∇ 22 f ( x (cid:48) , y (cid:48) )]
Revised: x [Equation] (cid:107)(cid:107)∇ [Equation]

Original: − 1 − [ ∇ 22 f ( x, y )] − 1 (cid:107)(cid:107)∇ 21 f ( x (cid:48) , y (cid:48) ) (cid:107) +(cid:107)∇ 12 f ( x, y ) (cid:107)(cid:107) [ ∇ 22 f ( x, y ) − 1 ] (cid:107)(cid:107)∇ 21 f ( x (cid:48) , y (cid:48)
Revised: +

Original: [Equation]
Revised: [Equation]

Original: Next
Revised: Next

Original: [Equation]
Revised: [Equation]

Original: we prove the item 4.
Revised: we prove the item 4.

Original: Consider any ﬁxed x ∈ R m
Revised: Consider any ﬁxed x ∈ R m

Original: [Equation]
Revised: [Equation]

Original: we know that f ( x, · ) achieves its maximum at y ∗ ( x ) , where the gradient vanishes,
Revised: we know that f ( x, · ) achieves its maximum at y ∗ ( x ) , where the gradient vanishes,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e., ∇ 2 f ( x, y ∗ ( x ))
Revised: e., ∇ 2 f ( x, y ∗ ( x ))

Original: = 0 .
Revised: = 0 .

Original: Thus [Equation] we further obtain that
Revised: Thus, we further obtain that

Original: [Equation]
Revised: [Equation]

Original: which implies that
Revised: which implies that

Original: [Equation]
Revised: [Equation]

Original: With the above equation
Revised: With the above equation

Original: [Equation]
Revised: [Equation]

Original: we take derivative of ∇ Φ( x )
Revised: we take derivative of ∇ Φ( x )

Original: = ∇ 1 f ( x, y ∗ ( x )) and obtain that
Revised: = ∇ 1 f ( x, y ∗ ( x )) and obtain that

Original: [Equation]
Revised: [Equation]

Original: Moreover, we have that
Revised: Moreover, we have that

Original: [Equation]
Revised: [Equation]

Original: where the last step uses the item 1.
Revised: where the last step uses the item 1.

Original: This proves the item 4.
Revised: This proves the item 4.

Original: B P ROOF OF P ROPOSITION
Revised: B P ROOF OF P ROPOSITION

Original: Proposition 2.
Revised: Proposition 2.

Original: Let Assumption 1 hold.
Revised: Let Assumption 1 hold.

Original: Deﬁne the following potential function
Revised: Deﬁne the following potential function

Revised: H

Revised: [Equation]

Revised: x

Original: [Equation]
Revised: [Equation]

Revised: x

Revised: [Equation]

Revised: y − y

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: and denote H t :
Revised: and denote H t :

Original: = H ( x t , x t − 1 , y t +1 ) .
Revised: = H ( x t , x t − 1 , y t +1 ) .

Original: Choose N t ≥ ln [ (cid:107)∇ 2 f ( x t ,y t ) (cid:107) /µ ] − 2ln
Revised: Choose N t ≥ max(ln2 , ln[ L 1 (cid:107)∇ 2 f ( x t ,y t ) (cid:107) /

Original:  (cid:107) x
Revised: ( L 2 µ )] − 2ln (cid:107) x

Original: t − x t − 1 (cid:107) ln
Revised: t − x t − 1 (cid:107) )

Original: [ κ/ ( κ − 1)] and learning rates η x ≤(8 L 1 + L Φ + 3 κ 7 L G ) − 1 , η y ≤ 2 L + µ .
Revised: ln[ κ/ ( κ − 1)] and learning rates η x ≤ 1 28 L 2 κ 3 , η y ≤ 2 L + µ .

Original: Then, the sequences { x t , y t } t generated by Cubic-GDA satisfy, for all t = 0 , 1 , 2 , ...
Revised: Then, the sequences { x t , y t } t generated by Cubic-GDA satisfy, for all t = 0 , 1 , 2 , ...

Original: [Equation]
Revised: [Equation]

Original: Consequently
Revised: Consequently

Original: [Equation]
Revised: [Equation]

Original: it holds that
Revised: it holds that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst bound the term (cid:107) y t +1 − y ∗ ( x t ) (cid:107) , which corresponds to the optimality gap of the maximization problem.
Revised: We ﬁrst bound the term (cid:107) y t +1 − y ∗ ( x t ) (cid:107) , which corresponds to the optimality gap of the maximization problem.

Original: Note that y ∗ ( x t ) ∈ Y is the unique maximizer of the strongly concave function f ( x t , y ) .
Revised: Note that y ∗ ( x t ) ∈ R n is the unique maximizer of the strongly concave function f ( x t , y ) .

Original: Note that y t +1 is obtained by applying N t gradient ascent steps starting from y t .
Revised: Note that y t +1 is obtained by applying N t gradient ascent steps starting from y t .

Original: Hence, By the convergence rate of gradient ascent algorithm under strong concavity, we conclude that with learning rate η y =
Revised: Hence, By the convergence rate of gradient ascent algorithm under strong concavity, we conclude that with learning rate η y =

Original: [Equation]
Revised: [Equation]

Original:  where the second inequality uses the hyperparameter choice
Revised: 19) where the second inequality uses the hyperparameter choice

Original: [Equation]
Revised: [Equation]

Original: as
Revised: as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: = [Equation] [Equation]
Revised: Moreover, eq.

Original: − y
Revised: (18) and the hyperparameter choice that N t ≥

Original: [Equation]
Revised: [Equation]

Original: ( x
Revised: ] imply that

Original: [Equation]
Revised: [Equation]

Original: ) (cid:107) . Moreover, eq. (18) and the fact that N t ≥ 1 imply that
Revised: where (i) denotes that a := y t +1 − y ∗ ( x t ) , b

Original: [Equation]
Revised: [Equation]

Original: where (i) denotes that a := y t +1 − y ∗ ( x t ) , b := y ∗ ( x t ) − y ∗ ( x t +1 ) ,θ := (1 + κ − 1 ) − 3 / 2 ∈ 0 , 1) , (ii) applies Jensen’s inequality to the convex function (cid:107) · (cid:107) 3 , (iii) uses the inequality that 1 − θ ≥ 12 κ and the fact that y ∗ is κ -Lipschitz.
Revised: = y ∗ ( x t ) − y ∗ ( x t +1 ) , (ii) applies Jensen’s inequality to the convex function (cid:107) · (cid:107) 3 .

Original: Based on Theorem 10 and Proposition 1 of Nesterov and Polyak (2006), we know that 
Revised: Based on Theorem 10 and Proposition 1 of Nesterov and Polyak (2006), we know that [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since x t +1 minimizes the following cubic sub-problem
Revised: Since x t +1 minimizes the following cubic sub-problem

Original: [Equation]
Revised: [Equation]

Original: we obtain the following optimality condition
Revised: we obtain the following optimality condition

Original: [Equation]
Revised: [Equation]

Original: [Equation] Next, by the Lipschitz Hessian of Φ 
Revised: Next, by the Lipschitz Hessian of Φ , we obtain that

Original: [Equation]
Revised: [Equation]

Original: we obtain that
Revised: [Table]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (i) uses Assumption 1, the item 4 of Proposition 1 and eq.
Revised: where (i) uses Assumption 1, the item 4 of Proposition 1 and eq.

Original: (22), (ii) uses eq.
Revised: (22), (ii) uses eq.

Original: (21) and the inequality that ab = ( a 3 / 2 a 3 / 2 b 3 ) 1 / 3 ≤ 23 a 3 / 2 +13 b 3  for any a, b ≥ 0  (based on AM-GM inequality), (iii) uses eq. 
Revised: (21) and the inequality that ab = (( Ca ) 3 / 2 ( Ca ) 3 / 2 ( b/C ) 3 ) 1 / 3 ≤ 23 ( Ca ) 3 / 2 + b 3 3 C 3 for any a, b ≥ 0 and C > 0 (based on AM-GM inequality), (iii) uses eq. (19) and L G =

Revised: L 2 (1 + κ ) 2 ≤ 4 L 2 κ 2 ,

Revised: L Φ =

Revised: L 2 (1 + κ ) 3 ≤ 8 L 2 κ 3 .

Original: Multiplying eq.
Revised: Multiplying eq.

Original: (20) with  LG κ 2/ 6 and adding it to eq.
Revised: (20) with 4 L 2 and adding it to eq.

Original: (23) yield that
Revised: (24) yield that

Original: [Equation]
Revised: [Equation]

Original: where (i) usesκ ≥ 1 and the condition that η x ≤(8 L 1 + L Φ + 3 κ 7 L G ) − 1 .
Revised: where (i) uses the condition that η x ≤ 1 28 L 2 κ 3 .

Original: The above inequality implies eq.
Revised: The above inequality implies eq.

Original: (2) by deﬁningH t := Φ
Revised: (2) by deﬁning

Original: [Equation]
Revised: [Equation]

Original: Next, summing eq.
Revised: Next, summing eq.

Original: (2) over t = 0 , 1 , ..., T − 1 , we obtain that for all T ≥ 1 ,L
Revised: (2) over t = 0 , 1 , ..., T − 1 , we obtain that for all T ≥ 1 ,

Original: [Equation]
Revised: [Equation]

Original: Letting T → ∞ yields that (cid:80) ∞ t =0 (cid:107) x t +1 − x t (cid:107) 3 < + ∞ , so lim t →∞ (cid:107) x t +1 − x t(cid:107) = 0 .
Revised: Letting T → ∞ yields that (cid:80) ∞ t =0 (cid:107) x t +1 − x t (cid:107) 3 < + ∞ and (cid:80) ∞ t =0 (cid:107) y t

Original: Then, eq.
Revised: +1 − y ∗ ( x t ) (cid:107) 3 < + ∞ , so lim t →∞ (cid:107) x t +1 − x t (cid:107) =

Original: (19) implies that lim t →∞ (cid:107) y t
Revised: lim t →∞ (cid:107) y t

Original: +1 − y ∗ ( x t ) (cid:107) =0 .
Revised: +1 − y ∗ ( x t ) (cid:107) = 0 .

Original: Hence, (cid:107) y t − y ∗ ( x t ) (cid:107) ≤ (cid:107) y t − y ∗ ( x t − 1 ) (cid:107) + (cid:107) y ∗ ( x t ) − y ∗ ( x t − 1 ) (cid:107) t → 0 using continuity of y ∗ , and (cid:107) y t
Revised: Hence, (cid:107) y t − y ∗ ( x t ) (cid:107) ≤ (cid:107) y t − y ∗ ( x t − 1 ) (cid:107) + (cid:107) y ∗ ( x t ) − y ∗ ( x t − 1 ) (cid:107) t → 0 using continuity of y ∗ , and (cid:107) y t

Original: +1 − y t (cid:107) ≤ (cid:107)
Revised: +1 − y t (cid:107) ≤ (cid:107)

Original: y t +1 − y ∗ ( x t ) (cid:107) + (cid:107) y t − y ∗ ( x t ) (cid:107) t → 0 .
Revised: y t +1 − y ∗ ( x t ) (cid:107) + (cid:107) y t − y ∗ ( x t ) (cid:107) t → 0 .

Original: C P ROOF OF T HEOREM
Revised: C P ROOF OF T HEOREM

Original: Theorem 1 (Global convergence) .
Revised: Theorem 1 (Global convergence) .

Original: Under the same conditions as those of Proposition 2, the CubicGDA has the following global convergence properties.
Revised: Under the same conditions as those of Proposition 2, the CubicGDA has the following global convergence properties.

Original: The function value sequence { Φ( x t ) } t converges to a ﬁnite limit H ∗ > −∞ ; 2.
Revised: The function value sequence { Φ( x t ) } t converges to a ﬁnite limit H ∗ > −∞ ; 2.

Original: The generated sequences { x t } t , { y t } t are bounded and have a compact sets of limit points.
Revised: The generated sequences { x t } t , { y t } t are bounded and have a compact sets of limit points.

Original: Every limit point x ∗ of { x t } t is a second-order stationary point of Φ , i.e., ∇ Φ( x ∗ ) = 0 , ∇ 2 Φ( x ∗ ) (cid:23) 0 , and satisﬁes Φ( x ∗ ) =
Revised: Every limit point x ∗ of { x t } t is a second-order stationary point of Φ , i e., ∇ Φ( x ∗ ) = 0 , ∇ 2 Φ( x ∗ ) (cid:23) 0 , and satisﬁes Φ( x ∗ ) =

Original: H ∗ .
Revised: H ∗ .

Original: Proof.
Revised: Proof.

Original: We ﬁrst prove item 1.
Revised: We ﬁrst prove item 1.

Original: We have shown in Proposition 2 that { H t } t is monotonically decreasing.
Revised: We have shown in Proposition 2 that { H t } t is monotonically decreasing.

Original: Also, Assumption 1 says that H t ≥ Φ( x t ) ≥ inf x ∈ R m Φ( x ) >
Revised: Also, Assumption 1 says that H t ≥ Φ( x t ) ≥ inf x ∈ R m Φ( x ) >

Original: −∞ .
Revised: −∞ .

Original: Therefore, we conclude that { H t } t converges to a ﬁnite limit H ∗ > −∞ .
Revised: Therefore, we conclude that { H t } t converges to a ﬁnite limit H ∗ > −∞ .

Original: Since Proposition 2 proves that (cid:107) x t − x t − 1 (cid:107) , (cid:107) y t
Revised: Since Proposition 2 proves that (cid:107) x t − x t − 1 (cid:107) , (cid:107) y t

Original: +

Revised: +

Original: −
Revised: −

Original: y ∗ ( x t ) (cid:107) t → 0 , we further conclude that Φ( x t ) t → H ∗ , which proves item 1.
Revised: y ∗ ( x t ) (cid:107) t → 0 , we further conclude that Φ( x t ) t → H ∗ , which proves item 1.

Original: Next, we prove item 2.
Revised: Next, we prove item 2.

Original: Note that Φ has compact sub-level sets.
Revised: Note that Φ has compact sub-level sets.

Original: Moreover, for all t , we have
Revised: Moreover, for all t , we have

Original: [Equation]
Revised: [Equation]

Original: Hence, the sequence { x t } t is bounded and thus has a compact set of limit points.
Revised: Hence, the sequence { x t } t is bounded and thus has a compact set of limit points.

Original: Since y ∗ is Lipschitz continuous by Proposition 1, {(cid:107) y ∗ ( x t ) (cid:107)} t is also bounded.
Revised: Since y ∗ is Lipschitz continuous by Proposition 1, {(cid:107) y ∗ ( x t ) (cid:107)} t is also bounded.

Original: Consequently, as (cid:107) y t − y ∗ ( x t ) (cid:107) → 0 , we conclude that { y t } t is also bounded and hence has a compact set of limit points.
Revised: Consequently, as (cid:107) y t − y ∗ ( x t ) (cid:107) → 0 , we conclude that { y t } t is also bounded and hence has a compact set of limit points.

Original: This proves item 2.
Revised: This proves item 2.

Original: Next, we will prove item 3.
Revised: Next, we will prove item 3.

Original: Suppose that x t k k → x ∗ along a certain sub-sequence { x t k } k .
Revised: Suppose that x t k k → x ∗ along a certain sub-sequence { x t k } k .

Original: We have that y t k +1 t → y ∗ ( x ∗ ) based on Proposition 2 and the continuity of y ∗ .
Revised: We have that y t k +1 t → y ∗ ( x ∗ ) based on Proposition 2 and the continuity of y ∗ .

Original: In addition, Propositionimplies that Φ , ∇ Φ , ∇ 2 Φ and G are continuous.
Revised: In addition, Propositionimplies that Φ , ∇ Φ , ∇ 2 Φ and G are continuous.

Original: Therefore, for every limit point x ∗ ,
Revised: Therefore, for every limit point x ∗ ,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: where (i) uses eq.

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: and (ii) uses eq.

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where (i) uses eq.

Original: (22) and (ii) uses eq.

Original: D P ROOF OF T HEOREM
Revised: D P ROOF OF T HEOREM

Original: Theorem 2 (Global convergence rate) .
Revised: Theorem 2 (Global convergence rate) .

Original: Under the same conditions as those of Proposition 2, the
Revised: Under the same conditions as those of Proposition 2, the

Original: Cubic-GDA converges at the following rate for all T ≥H 0 − inf x ∈ R m . Φ( x
Revised: Cubic-GDA converges at the following rate for all T ≥

Original: [Equation]
Revised: [Equation]

Original: L
Revised: Before proving Theorem

Original: [Equation]
Revised: [Equation]

Original: Before proving Theorem 2, we ﬁrst prove the following auxiliary lemma.
Revised: we ﬁrst prove the following auxiliary lemma.

Original: Lemma 2.
Revised: Lemma 2.

Original: For any symmetric matrices A, B ∈ R n × n , we have | λ min ( A ) − λ min ( B )
Revised: For any symmetric matrices A, B ∈ R n × n , we have | λ min ( A ) − λ min ( B )

Original: | ≤ (cid:107) A − B (cid:107) .
Revised: | ≤ (cid:107) A − B (cid:107) .

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: Note that (24) implies that
Revised: Note that (25) implies that

Original: [Equation]
Revised: [Equation]

Original: which further implies that there exists 1 ≤ t (cid:48) ≤ T − 

Revised: which further implies that there exists 1 ≤ t (cid:48) ≤ T −

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: On the other hand
Revised: On the other hand

Original: [Equation]
Revised: [Equation]

Original: equation (2.2) of Nesterov and Polyak (2006) implies that,
Revised: equation (2.2) of Nesterov and Polyak (2006) implies that,

Original: [Equation]
Revised: [Equation]

Original: Since ∇ Φ( x ) = ∇ 1 f ( x, y ∗ ( x )) , ∇ 2 Φ( x ) =
Revised: Since ∇ Φ( x ) = ∇ 1 f ( x, y ∗ ( x )) , ∇ 2 Φ( x ) =

Original: G ( x, y ∗ ( x ))
Revised: G ( x, y ∗ ( x ))

Original: [Equation]
Revised: [Equation]

Original: the above inequality implies that
Revised: the above inequality implies that

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original:  where (i) uses eq.
Revised: (28) where (i) uses eq.

Original: (22), item 1 of Assumption 1 and item 3 of Proposition 1, (ii) uses eq.
Revised: (22), item 1 of Assumption 1 and item 3 of Proposition 1, (ii) uses eq.

Original: (19), (iii) uses eq.

Original: (26) and L G =
Revised: (19) and L G =

Original: L Φ (1 + κ ) − 1 ≤  L Φ / 2  ,  and (iv) uses T ≥H 0 − inf x ∈ R m Φ
Revised: L 2 (1 + κ ) 2 ≤ 4 L 2 κ 2 , L Φ = L 2 (1 + κ ) 3 ≤ 8 L 2 κ 3 , (iii) uses eq. (27), and (iv) uses T ≥

Original: [Equation]
Revised: [Equation]

Original:  x 
Revised: inf x [Equation] R m Φ( x ) L [Equation]

Original: [Equation]
Revised: [Equation]

Original: Also, note that
Revised: Also, note that

Original: [Equation]
Revised: [Equation]

Original: where (i) uses Lemma 2, (ii) uses eq.
Revised: where (i) uses Lemma 2, (ii) uses eq.

Original: (21) and item 3 of Proposition 1, (iii) uses eq.
Revised: (21) and item 3 of Proposition 1, (iii) uses eq.

Original: (19), (iv) uses eq.
Revised: (19), (iv) uses eq.

Original: (26) and L G = L Φ (1 + κ ) − 1 ≤ L Φ / 2 , and (v) uses T ≥ H 0 − inf x ∈ R m Φ( x ) L 1 /.
Revised: (27), and (v) uses T ≥ H 0 − inf x ∈ R m Φ( x ) L 2 κ 3 /and L G = L 2 (1 + κ ) 2 ≤ 4 L 2 κ 2 .

Original: Equations(27) & (28)  imply Theorem 2.
Revised: Equations (28) & (29) imply Theorem 2.

Original: E P ROOF OF T HEOREM
Revised: E P ROOF OF T HEOREM

Original: Theorem 3.
Revised: Theorem 3.

Original: Let Assumption 1 hold and assume that the potential function H satisﬁes the local Łojasiewicz gradient geometry.
Revised: Let Assumption 1 hold and assume that the potential function H satisﬁes the local Łojasiewicz gradient geometry.

Original: Choose the hyperparameters N t , η x , η y in the same way as Proposition 2.
Revised: Choose the hyperparameters N t , η x , η y in the same way as Proposition 2.

Original: Then, the sequences { ( x t , y t ) } t generated by Cubic-GDA have a unique limit point, which is a second-order stationary point of Φ .
Revised: Then, the sequences { ( x t , y t ) } t generated by Cubic-GDA have a unique limit point, which is a second-order stationary point of Φ .

Original: Proof.
Revised: Proof.

Original: We ﬁrst derive a bound on (cid:107)∇
Revised: We ﬁrst derive a bound on (cid:107)∇

Original: [Equation]
Revised: [Equation]

Original: 2 as follows.
Revised: as follows.

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where (i) uses eq.
Revised: where (i) uses eq.

Original: (2.2) of Nesterov and Polyak (2006), the fact that ∇ 2 Φ is L Φ -Lipschitz, and eq.
Revised: (2.2) of Nesterov and Polyak (2006), the fact that ∇ 2 Φ is L Φ -Lipschitz, and eq.

Original: (15), (ii) uses Lemma 1 and ∇ 2 Φ( x )
Revised: (15), (ii) uses Lemma 1 and ∇ 2 Φ( x )

Original: =
Revised: =

Original: G ( x, y ∗ ( x )) from item 4 of Proposition 1, (iii) uses eq.
Revised: G ( x, y ∗ ( x )) from item 4 of Proposition 1, (iii) uses eq.

Original: (22), ∇ Φ( x )
Revised: (22), ∇ Φ( x )

Original: = ∇ 1 f ( x, y ∗ ( x )) from item 2 of Proposition 1 and the inequality that 1 + κ ≤ 2 κ , (iv) and (v) use eq. 
Revised: = ∇ 1 f ( x, y ∗ ( x )) from item 2 of Proposition 1 and the inequality that 1 + κ ≤ 2 κ , (iv) and (v) use eq. (19), and (v) uses L G = L 2 (1 + κ ) 2 ≤ 4 L 2 κ 2 , L Φ = L 2 (1 + κ ) 3 ≤ 8 L 2 κ 3 .

Original: Next, we prove the convergence of the sequence { x t } t under the assumption that H ( x, x (cid:48) , y ) satisﬁes the Łojasiewicz gradient geometry.
Revised: Next, we prove the convergence of the sequence { x t } t under the assumption that H ( x, x (cid:48) , y ) satisﬁes the Łojasiewicz gradient geometry.

Original: Recall that we have shown in the proof of Theorem 1 that: 1) { H t } t decreases monotonically to the ﬁnite limit H ∗ ; 2) for any limit point x ∗ , y ∗ of { x t } t , { y t } t , Φ( x ∗ ) =
Revised: Recall that we have shown in the proof of Theorem 1 that: 1) { H t } t decreases monotonically to the ﬁnite limit H ∗ ; 2) for any limit point x ∗ , y ∗ of { x t } t , { y t } t , Φ( x ∗ ) =

Original: H ∗ .
Revised: H ∗ .

Original: Hence, the Łojasiewicz gradient inequality (see Deﬁnition 1) holds after sufﬁciently large number of iterations, i.e., there exists t 1 ∈ N + such that for all t ≥ t 1 , H ( x t , x t − 1 , y t +1 ) −
Revised: Hence, the Łojasiewicz gradient inequality (see Deﬁnition 1) holds after sufﬁciently large number of iterations, i.e., there exists t 1 ∈ N + such that for all t ≥ t 1 , H ( x t , x t − 1 , y t +1 ) −

Original: H ∗ ≤ c (cid:107)∇ H ( x t , x t − 1 , y t +1 ) (cid:107) θ .
Revised: H ∗ ≤ c (cid:107)∇ H ( x t , x t − 1 , y t +1 ) (cid:107) θ .

Original: Equivalently,
Revised: Equivalently,

Original: [Equation]
Revised: [Equation]

Original: where we deﬁne the concave function that ϕ( s ) :
Revised: where we deﬁne the concave function that ϕ

Original: = c 1 /θ 1 − 1 /θ s 1 − 1 /θ (θ > 1 , s> 0) . In addition, since (cid:107) x t − x t − 1 (cid:107) t → 0 (Proposition 2), there exists t 2 ∈ N + such that for all t ≥ t 2 , (cid:107) x t − x t − [Equation] (cid:107) ,
Revised: ( s

Original: [Equation]
Revised: [Equation]

Original: x t

Original: [Equation] − x t [Equation] 2  (cid:107) ≤ 1 .
Revised: In addition, since (cid:107) x t − x t − 1 (cid:107) t → 0 (Proposition 2), there exists t 2 ∈ N + such that for all t ≥ t 2 , (cid:107) x t − x t − 1 (cid:107) ≤ 1 .

Original: Hence, rearranging the above inequality and utilizing eq.
Revised: Hence, rearranging the above inequality and utilizing eq.

Original: (29), we obtain that for all t ≥ t 0 :
Revised: (30), we obtain that for all t ≥ t 0 :

Original: =
Revised: =

Original: max( t 1 , t 2 ) ,
Revised: max( t 1 , t 2 ) ,

Original: [Equation]
Revised: [Equation]

Original: By concavity of the function ϕ ( s ) :
Revised: By concavity of the function ϕ ( s ) :

Original: = [Equation] [Equation] [Equation]
Revised: = c 1 /θ 1 − 1 /θ s 1 − 1 /θ ( θ > 1 , s > 0)

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: > [Equation] [Equation] > 0) , we know that
Revised: we know that

Original: [Equation]
Revised: [Equation]

Original: where (i) uses Proposition 2 and eq.
Revised: where (i) uses Proposition 2 and eq.

Original: [Equation]
Revised: [Equation]

Original: (ii) uses the inequality that a 2 + b 2 ≤ ( a + b ) 2 for any a, b ≥ 0 .
Revised: (ii) uses the inequality that a 2 + b 2 ≤ ( a + b ) 2 for any a, b ≥ 0 .

Original: Rearranging the above inequality yields that
Revised: Rearranging the above inequality yields that

Original: [Equation]
Revised: [Equation]

Revised: x t +

Revised: [Equation]

Revised: ≤ [ ϕ ( H t

Original: C
Revised: [Equation] H ∗ ) [Equation] t

Original: [Equation]
Revised: [Equation]

Revised: H

Original:  where the ﬁnal step uses the AM-GM inequality that ab 2 = (cid:2)3 (cid:112)
Revised: [Equation] [Equation] [Equation] where the ﬁnal step uses the AM-GM inequality that ab 2 = (cid:2)

Original: ( C 2 a )( b/C )( b/C ) (cid:3) 3 ≤ 127 ( C

Original: [Equation]
Revised: [Equation]

Original:  + 2 bC ) 3 for any a, b ≥ 0 and C > 0 (the value of C will be assigned later).
Revised: ( C 2 a )( b/C )( b/C ) (cid:3) 3 ≤ 127 ( C 2 a + 2 bC ) 3 for any a, b ≥ 0 and C > 0 (the value of C will be assigned later).

Original: Taking cubic root of both sides of the above inequality and telescoping it over t = t 0 , . .
Revised: Taking cubic root of both sides of the above inequality and telescoping it over t = t 0 , . .

Original: , T − 1 , we obtain that
Revised: , T − 1 , we obtain that

Original: [Equation]
Revised: [Equation]

Original: where the ﬁnal step uses the facts that H t − H ∗ ≥ 0 and that ϕ ( s ) is monotonically increasing.
Revised: where the ﬁnal step uses the facts that H t − H ∗ ≥ 0 and that ϕ ( s ) is monotonically increasing.

Original: Since the value of C > 0 is arbitrary, we can select large enough C such that
Revised: Since the value of C > 0 is arbitrary [Equation] we can select large enough C such that

Original: [Equation]
Revised: [Equation]

Original: L G κ 3 < 3 √ L 1 3 .
Revised: Hence [Equation]

Original: Hence, the inequality above further implies that
Revised: the inequality above further implies that

Original: [Equation]
Revised: [Equation]

Original: Letting T → ∞ concludes that
Revised: Letting T → ∞ concludes that

Original: [Equation]
Revised: [Equation]

Revised: Moreover

Original: Moreover, this implies that { x t } t is a Cauchy sequence and therefore converges to a certain limit, i.e., x t t → x ∗ .
Revised: [Equation] this implies that { x t } t is a Cauchy sequence and therefore converges to a certain limit, i [Equation] e., x t t → x ∗ .

Original: We have shown in Theorem 1 that any such limit point must be a second-order critical point of Φ .
Revised: We have shown in Theorem 1 that any such limit point must be a second-order critical point of Φ .

Original: Hence, we conclude that { x t } t converges to a certain second-order critical point x ∗ of Φ( x ) .
Revised: Hence, we conclude that { x t } t converges to a certain second-order critical point x ∗ of Φ( x ) .

Original: Also, note that (cid:107) y ∗ ( x t ) − y t (cid:107) t → 0 , x t t → x ∗ and y ∗ is a Lipschitz mapping, so we conclude that { y t } t converges to y ∗ ( x ∗ ) .
Revised: Also, note that (cid:107) y ∗ ( x t ) − y t (cid:107) t → 0 , x t t → x ∗ and y ∗ is a Lipschitz mapping, so we conclude that { y t } t converges to y ∗ ( x ∗ ) .

Original: Finally, the item 3 of Theorem 1 implies that x ∗ is a second-order critical point of Φ( x ) .
Revised: Finally, the item 3 of Theorem 1 implies that x ∗ is a second-order critical point of Φ( x ) .

Original: F P ROOF OF T HEOREM
Revised: F P ROOF OF T HEOREM

Original: Theorem 4 (Funtion value convergence rate) .
Revised: Theorem 4 (Funtion value convergence rate) .

Original: Under the same conditions as those of Theorem 3, the sequence of potential function { H t } t converges to the limit H ∗ at the following rates.
Revised: Under the same conditions as those of Theorem 3, the sequence of potential function { H t } t converges to the limit H ∗ at the following rates.

Original: If the geometry parameter θ ∈ ( 32 , ∞ ) , then H t ↓ H ∗ super-linearly as
Revised: If the geometry parameter θ ∈ ( 32 , ∞ ) , then H t ↓ H ∗ super-linearly as

Original: [Equation]
Revised: [Equation]

Original: If the geometry parameter θ = 32 , then H t ↓ H ∗ linearly as
Revised: If the geometry parameter θ = 32 , then H t ↓ H ∗ linearly as

Original: [Equation]
Revised: [Equation]

Original: If the geometry parameter θ ∈ (1 , 32 ) , then H t ↓ H ∗ sub-linearly as
Revised: If the geometry parameter θ ∈ (1 , 32 ) , then H t ↓ H ∗ sub-linearly as

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Equation (30) implies that there exists t 0 ∈ N + such that for any t ≥ t 0 ,
Revised: Equation (31) implies that there exists t 0 ∈ N + such that for any t ≥ t 0 ,

Original: [Equation]
Revised: [Equation]

Original: where (i) uses proposition
Revised: where (i) uses proposition

Original: [Equation]
Revised: [Equation]

Original: Deﬁningd t := H
Revised: Deﬁning

Original: [Equation]
Revised: [Equation]

Original:  and C[-
-]
Revised: , and C[-
-]

Original: = c 1 /θ L −

Original: [Equation]
Revised: [Equation]

Original: , the above inequality further becomes
Revised: the above inequality further becomes

Original: [Equation]
Revised: [Equation]

Original: where (i) applies Jensen’s inequality to the concave function ξ ( s )
Revised: where (i) applies Jensen’s inequality to the concave function ξ ( s )

Original: =
Revised: =

Original: s 2 / 3 , and (ii) uses the inequality that 2 1 / 3 max( C 1 , C 2 ) ≤ C 0 .
Revised: s 2 / 3 , and (ii) uses the inequality that 2 1 / 3 max( C 1 , C 2 ) ≤ C 0 .

Original: Next, we prove the convergence rates case by case.
Revised: Next, we prove the convergence rates case by case.

Original: (Case 1)
Revised: (Case 1)

Original: If θ ∈ ( 32 , + ∞ ) , since d t ≥ 0 , eq.
Revised: If θ ∈ ( 32 , + ∞ ) , since d t ≥ 0 , eq.

Original: (35) implies that for t ≥ t 0 ,
Revised: (36) implies that for t ≥ t 0 ,

Original: [Equation]
Revised: [Equation]

Original: 2 , which is equivalent to that
Revised: which is equivalent to that

Original: [Equation]
Revised: [Equation]

Original: Since d t ↓ 0 , C 3 θ 2 θ − 3d t 0 ≤ e − 1 for sufﬁciently large t 0 ∈ N + .
Revised: Since d t ↓ 0 , C [Equation] d t 0 ≤ e − 1 for sufﬁciently large t 0 ∈ N + .

Original: Hence, eq. (37) implies that for any k ∈ N +
Revised: Hence, eq. (38) implies that for any k ∈ N +

Original: [Equation]
Revised: [Equation]

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Note that θ ∈ ( 32 , + ∞ ) implies that 2 θ 3 > 1 , and thus the inequality above implies that H t ↓ H ∗ at the super-linear rate given by eq.
Revised: Note that θ ∈ ( 32 , + ∞ ) implies that 2 θ 3 > 1 , and thus the inequality above implies that H t ↓ H ∗ at the super-linear rate given by eq.

Original: (Case 2)
Revised: (Case 2)

Original: If θ = 32 , eq.
Revised: If θ = 32 , eq

Original: (36) implies that for t ≥ 
Revised: (37) implies that for t ≥ t

Original: t
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that d t 0 ≤ 1 for sufﬁciently large t 0 .
Revised: Note that d t 0 ≤ 1 for sufﬁciently large t 0 .

Original: Therefore, d t ↓ 0 (i.e., H ( z t ) ↓ H ∗ ) at the linear rate given by eq.
Revised: Therefore, d t ↓ 0 (i.e., H ( z t ) ↓ H ∗ ) at the linear rate given by eq.

Original: (Case 3)
Revised: (Case 3)

Original: If θ ∈ (1 , 32 ) , consider the following two subcases.
Revised: If θ ∈ (1 , 32 ) , consider the following two subcases.

Original: If d t − 2 ≤ 2 d t , denote ψ ( s ) = 2 θ 3 − 2 θ s 1 − 32 θ , then for any t ≥ t 0 ,
Revised: If d t − 2 ≤ 2 d t , denote ψ ( s ) = 2 θ 3 − 2 θ s 1 − 32 θ , then for any t ≥ t 0 ,

Original: [Equation]
Revised: [Equation]

Original: [Equation] where (i) uses d t ≤ d t − 2 , and − 32 (1 − θ ) < 0 , (ii) uses the following inequality implied by eq.
Revised: where (i) uses d t ≤ d t − 2 , and − 32 (1 − θ ) < 0 , (ii) uses the following inequality implied by eq.

Original: (36), and (iii) uses d t d t − 2 ≥ 12 and 32 θ ∈ (cid:0) 1 , 32 (cid:1) .
Revised: (37), and (iii) uses d t d t − 2 ≥ 12 and 32 θ ∈ (cid:0) 1 , 32 (cid:1) .

Original: If d t − 2 > 2 d t , then for any t ≥ t 0 ,
Revised: If d t − 2 > 2 d t , then for any t ≥ t 0 ,

Original: [Equation]
Revised: [Equation]

Original:  where we use 1 − 32 θ ∈ (cid:0) − 12 , 0 (cid:1), 2 θ 3 − 2 θ > 0 and d t ≤ d t 0 ≤ (2 C 0 ) 3 θ 3 − 2 θ
Revised: (41) where we use 1 − 32 θ ∈ (cid:0) [Equation] 0 (cid:1)

Original: [Equation]
Revised: [Equation]

Original: 2 θ
Revised: θ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: d t

Original: [Equation]
Revised: [Equation]

Original: 3 −
Revised: d

Original: [Equation]
Revised: [Equation]

Original:  2 θ 3 − 2 θ for sufﬁciently large t 0 ∈ N + .
Revised: [Equation] 2 θ (cid:104) 2 θ (cid:0) 1 − 2 1 − 32 θ (cid:1) 3 − 2 θ (cid:105) 2 θ 3 − 2 θ for sufﬁciently large t 0 ∈ N + .

Revised: Since at least one of eqs.

Original: Since at least one of eqs. (39) & (40)  holds, we have ψ ( d t ) − ψ ( d t − 2 ) ≥ (2 C 0 ) − 32 . 
Revised: (40) & (41) holds, we have ψ ( d t ) − ψ ( d t − 2 ) ≥ (2 C 0 ) − 32 . Hence, ψ [Equation] k +

Revised: [Equation]

Revised: +

Original: Hence, ψ ( d 2 k + t 0 +1 ) ≥ ψ ( d 2 k + t 0 ) ≥ ψ ( d t 0 ) + k (2 C 0 ) − 32 ≥ k (2 C 0 ) − 32 ; k ∈ N , which implies that ψ ( d t ) ≥ t − t 0 2 (2 C 0 ) − 32 By substituing the deﬁnition of ψ , the inequality above implies that H ( z t ) ↓ H ∗ in a sub-linear rate given by eq.
Revised: [Equation] ≥ ψ [Equation] N , which implies that ψ ( d t ) ≥ t − t 0 2 (2 C 0 ) − 32 By substituing the deﬁnition of ψ , the inequality above implies that H ( z t ) ↓ H ∗ in a sub-linear rate given by eq.

Original: G P ROOF OF T HEOREM
Revised: G P ROOF OF T HEOREM

Original: Theorem 5 (Parameter convergence rate) .
Revised: Theorem 5 (Parameter convergence rate) .

Original: Under the same conditions as those of Theorem 3, the sequences { x t , y t } t generated by Cubic-GDA converge to their limits x ∗ , y ∗ ( x ∗ ) respectively at the following rates.
Revised: Under the same conditions as those of Theorem 3, the sequences { x t , y t } t generated by Cubic-GDA converge to their limits x ∗ , y ∗ ( x ∗ ) respectively at the following rates.

Original: If the geometry parameter θ ∈ ( 32 , ∞ ) , then ( x t , y t ) → ( x ∗ , y ∗ ( x ∗ )) super-linearly as 
Revised: If the geometry parameter θ ∈ ( 32 , ∞ ) , then ( x t , y t ) → ( x ∗ , y ∗ ( x ∗ )) super-linearly as max (cid:8) (cid:107) x t − x ∗ (cid:107) , (cid:107) y t − y ∗ ( x ∗ ) (cid:107) (cid:9)

Original: [Equation]
Revised: ≤ O (cid:16) exp (cid:16) − 1 3 (cid:16) 2 θ 3 (cid:17) t − t 0 2 − 1 (cid:17)(cid:17) , ∀ t ≥ t 0 ; (9) 2.

Original: If the geometry parameter θ = 32 , then ( x t , y t ) → ( x ∗ , y ∗ ( x ∗ )) linearly as 
Revised: If the geometry parameter θ = 32 , then ( x t , y t ) → ( x ∗ , y ∗ ( x ∗ )) linearly as max (cid:8) (cid:107) x t − x ∗ (cid:107) , (cid:107) y t − y ∗ ( x ∗ ) (cid:107) (cid:9)

Original: [Equation]
Revised: [Equation]

Original: If the geometry parameter θ ∈ (1 , 32 ) , then ( x t , y t ) → ( x ∗ , y ∗ ( x ∗ )) sub-linearly as
Revised: If the geometry parameter θ ∈ (1 , 32 ) , then ( x t , y t ) → ( x ∗ , y ∗ ( x ∗ )) sub-linearly as

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Notice that eq.
Revised: Notice that eq.

Original: (33) still holds after increasing t 0 ,i.e., for T ≥ t ≥ t 0 and large enough C such that 23 C √ L 1 + L G , 23 C (cid:113) L Φ [Equation]
Revised: (34) still holds after increasing t 0 ,

Original: L
Revised: i

Original: [Equation]
Revised: [Equation]

Original: x +
Revised: e., for T ≥ t ≥ t 0 and large enough C such that

Original: [Equation]
Revised: [Equation]

Original: < 3 √ L 1 3 .

Original:  we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: Proposition 2 implies that
Revised: Proposition 2 implies that

Original: [Equation]
Revised: [Equation]

Original: where (i) uses H t  ≥ H t+1 ≥ H ∗ .
Revised: where (i) uses H t − 1 ≥ H t ≥ H ∗ .

Original: Therefore, for t ≥ t 0 , eqs.
Revised: Therefore, for t ≥ t 0 , eqs.

Original: (41) & (42)  imply that
Revised: (42) & (43) imply that

Original: [Equation]
Revised: [Equation]

Original: Next, we discuss case by case
Revised: Next, we discuss case by case.

Original: [Equation]

Original: (Case 1)
Revised: (Case 1)

Original: If [Equation] ) , then
Revised: If θ ∈ (cid:0) 32 , + ∞ ) , then

Original: [Equation]
Revised: [Equation]

Original: where the two ≤ use eqs.
Revised: where the two ≤ use eqs.

Original: (43) & (6) respectively.
Revised: (44) & (6) respectively.

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: where (i) uses eq.
Revised: where (i) uses eq.

Original: (19) and item 1 of Proposition 1, (ii) uses eq.
Revised: (19) and item 1 of Proposition 1, (ii) uses eq.

Original: (42) and (iii) uses eq.
Revised: (43) and (iii) uses eq.

Original: (Case 2)
Revised: (Case 2)

Original: If θ = 32 , then the proof is similar to that of Case 2, and eqs.
Revised: If θ = 32 , then the proof is similar to that of Case 2, and eqs.

Original: (44) & (46) still hold.
Revised: (45) & (47) still hold.

Original: The only difference is to use the convergence rate of H t − H ∗ given by eq.
Revised: The only difference is to use the convergence rate of H t − H ∗ given by eq.

Original: (7) instead of eq.
Revised: (7) instead of eq.

Original: (Case 3)
Revised: (Case 3)

Original: If θ ∈ (1 , 32 ) , then similar to the proof of Case 2, we obtain  that
Revised: If θ ∈ (1 , 32 ) , then similar to the proof of Case 2, we obtain from eq. (44) that

Original: [Equation]
Revised: [Equation]

Original: where the two ≤ use eqs.
Revised: where the two ≤ use eqs.

Original: (42) & (6) respectively.
Revised: (43) & (6) respectively.

Original: Then, as eq. (46) still holds, we have
Revised: Then, as eq. (47) still holds, we have

Original: [Equation]
Revised: [Equation]