Original: Quasi-Newton Methods for Saddle Point Problems
Revised: Quasi-Newton Methods for Saddle Point Problems

Original: Abstract
Revised: Abstract

Original: This paper studies quasi-Newton methods for strongly-convex-strongly-concavesaddle point problems.
Revised: This paper studies quasi-Newton methods for strongly-convex-strongly-concave saddle point problems.

Original: We propose random Broyden family updates, which haveexplicit local superlinear convergence rate of O (cid:0)(cid:0) 1 − 1 / ( nκ 2 ) (cid:1) k
Revised: We propose random Broyden family updates, which have explicit local superlinear convergence rate of O ((1 − 1 / ( d κ 2 ))

Original:  ( k − 1) / 2 (cid:1) , where nis the dimension of the problem, κ is the condition number and k is the number ofiterations.
Revised: k ( k − 1) / 2 ) , where n is the dimension of the problem, κ is the condition number and k is the number of iterations.

Original: The design and analysis of proposed algorithm are based on estimatingthe square of indefinite Hessian matrix, which is different from classical quasiNewton methods in convex optimization.
Revised: The design and analysis of proposed algorithm are based on estimating the square of indefinite Hessian matrix, which is different from classical quasiNewton methods in convex optimization.

Original: We also present two specific Broydenfamily algorithms with BFGS-type and SR1-type updates, which enjoy the fasterlocal convergence rate of O (cid:0)(cid:0) 1 − 1 /n (cid:1) k ( k − 1) / 2 (cid:1) .
Revised: We also present two specific Broyden family algorithms with BFGS-type and SR1-type updates, which enjoy the faster local convergence rate of O ((1 − 1 /d ) k ( k − 1) / 2 ) .

Original: Our numerical experiments showproposed algorithms outperform classical first-order methods.
Revised: Our numerical experiments show proposed algorithms outperform classical first-order methods

Original:  Introduction
Revised:  Introduction

Original: In this paper we focus on the following smooth saddle point problem
Revised: In this paper, we focus on the following smooth saddle point problem

Original: [Equation]
Revised: [Equation]

Original: where f is strongly-convex in x and strongly-concave in y .
Revised: where f is strongly-convex in x and strongly-concave in y .

Original: We target to find the saddle point ( x ∗ , y ∗ )which holds that
Revised: We target to find the saddle point ( x ∗ , y ∗ ) which holds that

Original: [Equation]
Revised: [Equation]

Original: ) for all x ∈ R n x and y ∈ R n y .
Revised: for all x ∈ R d x and y ∈ R d y .

Original: This formulation is widely used in game theory [1, 38], AUCmaximization [15, 41], fairness-aware machine learning [42], robust optimization [2, 11, 13, 35],empirical risk minimization [44] and reinforcement learning [10].
Revised: This formulation is widely used in game theory [1, 39], AUC maximization [15, 43], fairness-aware machine learning [44], robust optimization [2, 11, 13, 36], empirical risk minimization [46] and reinforcement learning [10].

Original: There are a great number of first-order optimization algorithms for solving problem (1), includingextragradient method [18, 37], optimistic gradient descent ascent [7, 27], proximal point method[30] and dual extrapolation [23].
Revised: There are a great number of first-order optimization algorithms for solving problem (1), including extragradient method [18, 38], optimistic gradient descent ascent [7, 28], proximal point method [31] and dual extrapolation [24].

Original: These algorithms iterate with first-order oracle and achieve linearconvergence.
Revised: These algorithms iterate with first-order oracle and achieve linear convergence.

Original: Lin et al.
Revised: Lin et al.

Original: [20], Wang and Li [39] used Catalyst acceleration to reduce the complexityfor unbalanced saddle point problem, nearly matching the lower bound of first-order algorithms [25,43] under some specific assumptions.
Revised: [21], Wang and Li [40] used Catalyst acceleration to reduce the complexity for unbalanced saddle point problem, nearly matching the lower bound of first-order algorithms [26, 45] under some specific assumptions.

Original: Compared with first-order methods, second-order methodsusually enjoy superior convergence in numerical optimization.
Revised: Compared with first-order methods, second-order methods usually enjoy superior convergence in numerical optimization.

Original: Huang et al.
Revised: Huang et al.

Original: [16] extended cubicregularized Newton (CRN) method [22, 23] to solve saddle point problem (1), which has quadraticlocal convergence.
Revised: [16] extended cubic regularized Newton (CRN) method [23, 24] to solve saddle point problem (1), which has quadratic local convergence.

Original: However, each iteration of CRN requires accessing the exact Hessian matrix andsolving the corresponding linear systems.
Revised: However, each iteration of CRN requires accessing the exact Hessian matrix and solving the corresponding linear systems.

Original: These steps arise O (cid:0) n 3 x + n 3 y (cid:1) time complexity, which istoo expensive for high dimensional problems.
Revised: These steps arise O (cid:0) d 3 x + d 3 y (cid:1) time complexity, which is too expensive for high dimensional problems.

Original: Quasi-Newton methods [3–5, 8, 36] are popular ways to avoid accessing exact second-order information applied in standard Newton methods.
Revised: Quasi-Newton methods [3–5, 8, 37] are popular ways to avoid accessing exact second-order information applied in standard Newton methods.

Original: They approximate the Hessian matrix based on the Broydenfamily updating formulas [3], which significantly reduces the computational cost.
Revised: They approximate the Hessian matrix based on the Broyden family updating formulas [3], which significantly reduces the computational cost.

Original: These algorithmsare well studied for convex optimization.
Revised: These algorithms

Original: The famous quasi-Newton methods including DavidonFletcher-Powell (DFP) method [8, 12], Broyden-Fletcher-Goldfarb-Shanno (BFGS) method [4, 5, 36]

Original: [Table]
Revised: [Table]

Original: BFGS/SR1 (Alg. 6 and 7) [Table] and symmetric rank 1 (SR1) method [3, 8] enjoy local superlinear convergence [6, 9, 28] when the objective function is strongly-convex.
Revised: are well studied for convex optimization. The famous quasi-Newton methods including DavidonFletcher-Powell (DFP) method [8, 12], Broyden-Fletcher-Goldfarb-Shanno (BFGS) method [4, 5, 37] and symmetric rank 1 (SR1) method [3, 8] enjoy local superlinear convergence [6, 9, 29] when the objective function is strongly-convex.

Original: Recently, Rodomanov and Nesterov [31, 32, 33] proposed greedyand random variants of quasi-Newton methods, which first achieves non-asymptotic superlinear convergence.
Revised: Recently, Rodomanov and Nesterov [32, 33, 34] proposed greedy and random variants of quasi-Newton methods, which first achieves non-asymptotic superlinear convergence.

Original: Later, Lin et al.
Revised: Later, Lin et al.

Original: [19] established a better convergence rate which is condition-number-free.
Revised: [20] established a better convergence rate which is condition-number-free.

Original: Jin and Mokhtari [17], Ye et al.
Revised: Jin and Mokhtari [17], Ye et al.

Original: [40] showed the non-asymptotic superlinear convergence rate alsoholds for classical DFP, BFGS and SR1 methods.
Revised: [42] showed the non-asymptotic superlinear convergence rate also holds for classical DFP, BFGS and SR1 methods.

Original: In this paper, we study quasi-Newton methods for saddle point problem (1).
Revised: In this paper, we study quasi-Newton methods for saddle point problem (1).

Original: Note that when the
Revised: Note that when the

Original: Hessian matrix of the objective function is indefinite, the existing Broyden family update formulasand their convergence analysis cannot be applied directly.
Revised: Hessian matrix of the objective function is indefinite, the existing Broyden family update formulas and their convergence analysis cannot be applied directly.

Original: To overcome this issue, we propose avariant framework of random quasi-Newton methods for saddle point problems, which approximatesthe square of the Hessian matrix during the iteration.
Revised: To overcome this issue, we propose a variant framework of random quasi-Newton methods for saddle point problems, which approximates the square of the Hessian matrix during the iteration.

Original: Our theoretical analysis characterizes theconvergence rate by the gradient norm, rather than the weighted norm of gradient used in convexoptimization [17, 19, 31–33, 40].
Revised: Our theoretical analysis characterizes the convergence rate by the gradient norm, rather than the weighted norm of gradient used in convex optimization [17, 20, 32–34, 42].

Original: We summarize the theoretical results for proposed algorithms in
Revised: We summarize the theoretical results for proposed algorithms in

Original: Table 1.
Revised: Table 1.

Original: The local convergence behaviors for all of the algorithms have two periods.
Revised: The local convergence behaviors for all of the algorithms have two periods.

Original: The first periodhas k 0 iterations with a linear convergence rate O ((1 − 1 /κ 2 ) k 0 ) .
Revised: The first period has k 0 iterations with a linear convergence rate O ((1 − 1 / κ 2 ) k 0 ) .

Original: The second one enjoys superlinearconvergence, that is, for general Broyden family methods, we have O ((1 − 1 / ( nκ 2 + 1))
Revised: The second one enjoys superlinear convergence, that is, • For general Broyden family methods, we have O (cid:0)(cid:0) 1 − 1 / ( d κ 2 + 1) (cid:1) k

Original: k ( k − 1) / 2 )and for BFGS method and SR1 method, we have O ((1 − 1 / ( n + 1))
Revised: ( k − 1) / 2 (cid:1) . • For BFGS method and SR1 method, we have O (cid:0)(cid:0) 1 − 1 / ( d + 1) (cid:1) k

Original: k ( k − 1) / 2 ) , which is conditionnumber-free.
Revised: ( k − 1) / 2 (cid:1) , which is condition-number-free.

Original: Paper Organization
Revised: Paper Organization

Original: In Section 2, we state the notation and preliminaries of this paper.
Revised: In Section 2, we state the notation and preliminaries of this paper.

Original: In Section3, we first propose random quasi-Newton methods for quadratic saddle point problem which enjoyslocal superlinear convergence.
Revised: In Section 3, we first propose random quasi-Newton methods for quadratic saddle point problem which enjoys local superlinear convergence.

Original: Then we extend it to solve general strongly-convex-strongly-concavesaddle point problems.
Revised: Then we extend it to solve general strongly-convex-strongly-concave saddle point problems.

Original: In Section 4, we provide numerical experiments to validate our algorithmson popular machine learning models.
Revised: In Section 5, we provide numerical experiments to validate our algorithms on popular machine learning models.

Original: All proofs, experiment details and extensions are deferred toappendix.
Revised: All proofs, experiment details and extensions are deferred to appendix.

Original:  Notation and Preliminaries
Revised:  Notation and Preliminaries

Original: We use ∥·∥ to present spectral norm and Euclidean norm of matrix and vector respectively.
Revised: We use ∥·∥ to present spectral norm and Euclidean norm of matrix and vector respectively.

Original: We denotethe standard basis for R n by { e 1 , . . .
Revised: We denote the standard basis for R d by { e 1 , . . .

Original: , e n } and let I be the identity matrix.
Revised: , e d } and let I be the identity matrix.

Original: The trace of a squarematrix is denoted by tr ( · ) .
Revised: The trace of a square matrix is denoted by tr ( · ) .

Original: Given two positive definite matrices G and H , we define their innerproduct as ⟨ G , H ⟩ def = tr ( GH ) .
Revised: Given two positive definite matrices G and H , we define their inner product as ⟨ G , H ⟩ def = tr ( GH ) .

Original: We introduce the quantity σ H ( G ) def = ⟨ H − 1 , G − H ⟩ = ⟨ H − 1 , G ⟩ − n to measure how well doesmatrix G approximate matrix H .
Revised: We introduce the quantity σ H ( G ) def = ⟨ H − 1 , G − H ⟩ = ⟨ H − 1 , G ⟩ − d , which is used to measure how well does matrix G approximate matrix H .

Original: If we further suppose G ⪰ H , then according toRodomanov and
Revised: If we further suppose G ⪰ H , then according to

Original:  Nesterov [31] it holds that G − H ⪯ ⟨ H − 1 , G − H ⟩ H =
Revised: Rodomanov and Nesterov [32] it holds that G − H ⪯ ⟨ H − 1 , G − H ⟩ H =

Original: σ H ( G ) H .
Revised: σ H ( G ) H .

Original: Using the notation of problem (1), we let z =
Revised: Using the notation of problem (1), we let z =

Original: [ x ; y ] ∈ R n where n def = n x + n y and denote thegradient and Hessian matrix of f at ( x , y ) as g ( z ) ∈ R n and ˆ H ( z ) ∈ R n × n .
Revised: [ x ; y ] ∈ R d where d def = d x + d y and denote the gradient and Hessian matrix of f at ( x , y ) as g ( z ) ∈ R d and ˆ H ( z ) ∈ R d × d .

Original: We suppose the saddle point problem (1) satisfies the following assumptions.
Revised: We suppose the saddle point problem (1) satisfies the following assumptions.

Original: Assumption 2.1.
Revised: Assumption 2.1.

Original: The objective function f ( x , y ) is twice differentiable and has L -Lipschitz continuous gradient and L 2 -Lipschitz continuous Hessian , i.e., there exists constants L > 0 and L 2 > 0 such that for any z
Revised: The objective function f ( x , y ) is twice differentiable and has L -Lipschitz continuous gradient and L 2 -Lipschitz continuous Hessian , i.e., there exists constants L > 0 and L 2 >such that for any z

Original: = [ x ; y ] , z ′ =
Revised: = [ x ; y ] , z ′ =

Original: [ x ′ ; y ′ ]
Revised: [ x ′ ; y ′ ]

Original: ∈ R n , we have ∥ g ( z ) − g
Revised: ∈ R d , we have ∥ g ( z ) − g

Original: ( z ′ )
Revised: ( z ′ )

Original: ∥
Revised: ∥

Original: ≤ L ∥
Revised: ≤ L ∥

Original: z − z ′ ∥ and∥ ˆ H ( z ) − ˆ H ( z ′ )
Revised: z − z ′ ∥ and ∥ ˆ H ( z ) − ˆ H ( z ′ )

Original: ∥
Revised: ∥

Original: ≤ L 2 ∥ z − z ′ ∥ .
Revised: ≤ L 2 ∥ z − z ′ ∥ .

Original: Assumption 2.2.
Revised: Assumption 2.2.

Original: The objective function f ( x , y ) is twice differentiable, µ -strongly-convex in xand µ -strongly-concave in y , i.e., there exists constant µ > 0 such that ∇ 2 xx f ( x , y )
Revised: The objective function f ( x , y ) is twice differentiable, µ -strongly-convex in x and µ -strongly-concave in y , i.e., there exists constant µ > 0 such that ∇ 2 xx f ( x , y )

Original: ⪰ µ I [-and∇ 
-] yy f ( x , y ) ⪯ − µ I for any 
Revised: ⪰ µ I [-and∇ 
-]and ∇ 2 yy f ( x , y ) ⪯ − µ I for any (

Original: ( x , y ) ∈ R n x × R n y .
Revised: x , y ) ∈ R d x × R d y .

Original: The L -lipschitz continuous of gradient means the spectral norm of Hessian matrix ˆ H ( z ) can be upperbounded
Revised: The L -lipschitz continuous of gradient means the spectral norm of Hessian matrix ˆ H ( z ) can be upper bounded, that is ∥

Original: , that is ∥ ˆ H ( z ) ∥ ≤ L .
Revised: ˆ H ( z ) ∥ ≤ L .

Original: Additionally, the condition number of the objective function is definedas κ def = L/µ .
Revised: Additionally, the condition number of the objective function is defined as κ def = L/µ .

Original:  Quasi-Newton Methods for Saddle Point Problems
Revised:  Quasi-Newton Methods for Saddle Point Problems

Original: In this section, we focus on designing quasi-Newton methods for saddle point problem and showingtheir superlinear local convergence rate.
Revised: In this section, we focus on designing quasi-Newton methods for saddle point problem and showing their superlinear local convergence rate.

Original: The update rule of standard Newton’s method can be writtenas z + = z − (cid:0)
Revised: The update rule of standard Newton’s method can be written as z + = z − (cid:0)

Original: ˆ H ( z ) (cid:1) − 1 g ( z ) for solving problem (1).
Revised: ˆ H ( z ) (cid:1) − 1 g ( z ) for solving problem (1).

Original: It has quadratic local convergence, but takes
Revised: It has quadratic local convergence, but takes

Original: O (cid:0) n 3 (cid:1) time complexity per iteration.
Revised: O (cid:0) d 3 (cid:1) time complexity per iteration.

Original: For convex minimization, quasi-Newton methods including
Revised: For convex minimization, quasi-Newton methods including

Original: BFGS/SR1 [3–5, 8, 36] and their variants [19, 31, 32, 40] focus on approximating the Hessian whichcan reduce the computational cost to O (cid:0) n 2 (cid:1) for each round.
Revised: BFGS/SR1 [3–5, 8, 37] and their variants [20, 32, 33, 42] focus on approximating the Hessian which can reduce the computational cost to O (cid:0) d 2 (cid:1) for each round.

Original: However, all these algorithms andrelated convergence analysis are based on the assumption that the Hessian matrix is positive definite,which is not suitable for our saddle point problems since ˆ H ( z ) is indefinite.
Revised: However, all these algorithms and related convergence analysis are based on the assumption that the Hessian matrix is positive definite, which is not suitable for our saddle point problems since ˆ H ( z ) is indefinite.

Original: We introduce the auxiliary matrix H ( z ) be the square of Hessian H ( z ) def = (cid:0) ˆ H ( z ) (cid:1) 2 .
Revised: We introduce the auxiliary matrix H ( z ) be the square of Hessian H ( z ) def = (cid:0) ˆ H ( z ) (cid:1) 2 .

Original: The followinglemma shows that H ( z ) is positive definite.
Revised: The following lemma shows that H ( z ) is positive definite.

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: Under Assumption 2.1 and 2.2, we have µ 2 I ⪯ H ( z ) ⪯
Revised: Under Assumption 2.1 and 2.2, we have µ 2 I ⪯ H ( z ) ⪯

Original: L 2 I for all z ∈ R n .
Revised: L 2 I for all z ∈ R d .

Original: Hence, we can reformulate the update of Newton’s method by
Revised: Hence, we can reformulate the update of Newton’s method by

Original: [Equation]
Revised: [Equation]

Original: Then it is natural to characterize the second-order information by estimating the auxiliary matrix
Revised: Then it is natural to characterize the second-order information by estimating the auxiliary matrix

Original: H ( z ) , rather than the indefinite Hessian ˆ H ( z ) .
Revised: H ( z ) , rather than the indefinite Hessian ˆ H ( z ) .

Original: If one can obtain a symmetric positive definite matrix
Revised: If one can obtain a symmetric positive definite matrix

Original: G ∈ R n × n as an estimator for H ( z ) , the update rule of (2) can be approximated by
Revised: G ∈ R d × d as an estimator for H ( z ) , the update rule of (2) can be approximated by

Original: [Equation]
Revised: [Equation]

Original: The remainder of this section introduce strategies to construct G , resulting the quasi-Newton methodsfor saddle point problem with local superlinear convergence.
Revised: The remainder of this section introduce strategies to construct G , resulting the quasi-Newton methods for saddle point problem with local superlinear convergence.

Original: We should point out the implementationof iteration (3) is unnecessary to construct Hessian matrix
Revised: We should point out the implementation of iteration (3) is unnecessary to construct Hessian matrix

Original: ˆ H ( z ) explicitly, since we are only interestedin the Hessian-vector product
Revised: ˆ H ( z ) explicitly, since we are only interested in the Hessian-vector product

Original: ˆ H ( z ) g
Revised: ˆ H ( z ) g

Original: ( z ) which can be computed efficiently [26, 34].
Revised: ( z ) which can be computed efficiently [27, 35].

Original: 3.1 The Broyden Family Updates
Revised: 3.1 The Broyden Family Updates

Original: We first review some basic results for quasi-Newton methods in convex optimization.
Revised: We first review some basic results for quasi-Newton methods in convex optimization.

Original: We introducethe Broyden family [24, Section 6.3] of quasi-Newton updates for approximating an positive definitematrix H ∈ R n × n by using the information of current estimator G ∈R n × n .
Revised: We introduce the Broyden family [25, Section 6.3] of quasi-Newton updates for approximating an positive definite matrix H ∈ R d × d by using the information of current estimator G ∈

Original: Definition 3.2.
Revised: Definition 3.2.

Original: Suppose two positive definite matrices H , G ∈ R n × n satisfy H ⪯ G .
Revised: Suppose two positive definite matrices H , G ∈ R d × d satisfy H ⪯ G .

Original: For anyu ∈ R n , if Gu =
Revised: For any u ∈ R d , if Gu =

Original: Hu , we define Broyd τ ( G , H , u ) def = H .
Revised: Hu , we define Broyd τ ( G , H , u ) def = H .

Original: Otherwise, we define
Revised: Otherwise, we define

Original: [Equation]
Revised: [Equation]

Original: The different choices of parameter τ for above formula contain several popular quasi-Newton updates:
Revised: The different choices of parameter τ for above formula contain several popular quasi-Newton updates:

Original: • For τ = u ⊤ Hu / (cid:0) u ⊤ Gu (cid:1) ∈ [0 , 1] , it corresponds to the BFGS update
Revised: • For τ = u ⊤ Hu / (cid:0) u ⊤ Gu (cid:1) ∈ [0 , 1] , it corresponds to the BFGS update

Original: [Equation]
Revised: [Equation]

Revised: • For τ = 0 , it corresponds to the SR1 update

Original: [Equation]
Revised: [Equation]

Original: Now we introduce the update rule [19, 31] by choosing u asu ∼ N ( 0 , I ) or u ∼ Unif (cid:0) S n − 1 (cid:1) .
Revised: Now we introduce the update rule [20, 32] by choosing u as

Original: [Equation]
Revised: [Equation]

Original: The following lemma shows applying the update rule (4) with (7) leads to a new estimator withtighter error bound in the measure of σ H ( · ) .
Revised: The following lemma shows applying the update rule (4) with (7) leads to a new estimator with tighter error bound in the measure of σ H ( · ) .

Original: Lemma 3.3 (Modified from Lin et al.
Revised: Lemma 3.3 (Modified from Lin et al.

Original: [19, Theorem 3.1]) .
Revised: [20, Theorem 3.1]) .

Original: Suppose two positive definite matricesµ 2 I ⪯ H ⪯ L 2 I and G ∈ R n × n satisfy that H ⪯ G .
Revised: Suppose two positive definite matrices µ 2 I ⪯ H ⪯ L 2 I and G ∈ R d × d satisfy that H ⪯ G .

Original: Let G + =
Revised: Let G + =

Original: Broyd τ ( G , H , u )) , where u is chosen random method as (7) , then for any τ ∈ [0 , 1] , we have E [ σ H ( G + )] ≤ (cid:0) 1 − 1 / ( nκ 2 ) (cid:1)
Revised: Broyd τ ( G , H , u )) , where u is chosen random method as (7) , then for any τ ∈ [0 , 1] , we have E [ σ H ( G + )] ≤ (cid:0) 1 − 1 / ( d κ 2 ) (cid:1)

Original: σ H ( G )
Revised: σ H ( G )

Original: For specific Broyden family updates BFGS and SR1 shown in (5) and (6), the update can achievea better convergence result.
Revised: For specific Broyden family updates BFGS and SR1 shown in (5) and (6), the update can achieve a better convergence result.

Original: Concretely, for BFGS method, we first find L such that G − 1 =
Revised: Concretely, for BFGS method, we first find L such that G − 1 =

Original: L ⊤ L ,where L is an upper triangular matrix.
Revised: L ⊤ L , where L is an upper triangular matrix.

Original: This step can be implemented with O ( n 2 ) complexity [19,
Revised: This step can be implemented with O ( d 2 ) complexity [20,

Original: Proposition 1].
Revised: Proposition 1].

Original: We present the subroutine for factorizing G − 1 in Algorithm 1 and give its detailedimplementation in the appendix.
Revised: We present the subroutine for factorizing G − 1 in Algorithm 1 and give its detailed implementation in the appendix.

Original: And we use the direction L ⊤ u instead of u for the BFGS update.
Revised: And we use the direction L ⊤ u instead of u for the BFGS update.

Original: Applying the BFGS update rule (5) with formula (7), we obtain a condition-number-free result asfollows.
Revised: Applying the BFGS update rule (5) with formula (7), we obtain a condition-number-free result as follows.

Original: Lemma 3.4 (Modified from Lin et al.
Revised: Lemma 3.4 (Modified from Lin et al.

Original: [19, Theorem 4.2]) .
Revised: [20, Theorem 4.2]) .

Original: Suppose two positive definite matrices
Revised: Suppose two positive definite matrices

Original: H , G ∈ R n × n satisfy H ⪯ G .
Revised: H , G ∈ R d × d satisfy H ⪯ G .

Original: Let G + =
Revised: Let G + =

Original: BFGS (cid:0) G , H , L ⊤ u (cid:1) , where u is chosen by the random method in (7) and L is an upper triangular matrix such that G − 1 =
Revised: BFGS (cid:0) G , H , L ⊤ u (cid:1) , where u is chosen by the random method in (7) and L is an upper triangular matrix such that G − 1 =

Original: L ⊤ L .
Revised: L ⊤ L .

Original: Then we have123 E [ σ H ( G + )] ≤ (1 − 1 /n )
Revised: Then we have E [ σ H ( G + )] ≤ (1 − 1 /d )

Original: σ H ( G ) .
Revised: σ H ( G ) .

Original: Remark 3.5 .
Revised: Remark 3.5 .

Original: Note that the step of conducting G − 1 =
Revised: Note that the step of conducting G − 1 =

Original: L ⊤ L requires QR decomposition of rank-1change matrix which requires O (26 n 2 ) flops [14, Section 12.5.1].
Revised: L ⊤ L requires QR decomposition of rankchange matrix which requires O (26 d 2 ) flops [14, Section 12.5.1].

Original: We do not recommend using this
Revised: We do not recommend using this

Original: BFGS update strategy in practice when n is large.
Revised: BFGS update strategy in practice when n is large.

Original: The convergence of the SR1 update can be characterized by the measure τ H ( G ) def = tr( G − H ) .
Revised: The convergence of the SR1 update can be characterized by the measure τ H ( G ) def = tr( G − H ) .

Original: Applying the SR1 update rule (6) with formula (7), the convergence also holds a condition-numberfree result.
Revised: Applying the SR1 update rule (6) with formula (7), the convergence also holds a condition-numberfree result.

Original: Lemma 3.6 (Modified from Lin et al.
Revised: Lemma 3.6 (Modified from Lin et al.

Original: [19, Theorem 4.1]) .
Revised: [20, Theorem 4.1]) .

Original: Suppose two positive definite matrices
Revised: Suppose two positive definite matrices

Original: H , G ∈ R n × n satisfy H ⪯ G .
Revised: H , G ∈ R d × d satisfy H ⪯ G .

Original: Let G + =
Revised: Let G + =

Original: SR1 ( G , H , u ) , where u is chosen by the random methodin (7) .
Revised: SR1 ( G , H , u ) , where u is chosen by the random method in (7) .

Original: Then we have E [ τ H ( G + )] ≤ (1 − 1 /n )
Revised: Then we have E [ τ H ( G + )] ≤ (1 − 1 /d )

Original: τ H ( G ) .
Revised: τ H ( G ) .

Original: 3.2 Algorithms for Quadratic Saddle Point Problems
Revised: 3.2 Algorithms for Quadratic Saddle Point Problems

Original: We now solve simple quadratic saddle point problem that f ( x , y )
Revised: We now solve simple quadratic saddle point problem that f ( x , y )

Original: =
Revised: =

Original: 12 z ⊤ Az − b ⊤ z in (1)wherez
Revised: 12 z ⊤ Az − b ⊤ z in (1)

Original:  =
Revised: where z =

Original: [ x ; y ] , b ∈ R n , A ∈ R n × n is symmetric and n = n x + n y .
Revised: [ x ; y ] , b ∈ R d , A ∈ R d × d is symmetric and d = d x + d y .

Original: We suppose A could be partitioned 
Revised: We suppose A could be partitioned as A =

Original: 1: Input: z 0 ∈ R n , G 0 = L 2 I , L 0 = L − 1 I 2: for k = 0 , 1 . . . 3: z k +1 =
Revised: [Equation]

Original: z k − G − 1 k ˆ Hg ( z k ) 4: Randomly choose ˜ u k from (7) 5: u k = L ⊤ k ˜ u k 6:
Revised: A

Original: G k +1 = BFGS ( G k , H , u k ) 7: L k +1 = Fast-Chol ( H , L k , u k ) 8: end for
Revised: [Equation]

Original: 1: Input: z 0 ∈ R n , [-G 0 = L 
 I K ≤ n +2: for k = 0-] , 1 . .
Revised: (cid:21) where the sub-matrices A xx ∈ R d x × d x , [-G 0 = L 
 I K ≤ n +2: for k = 0-]A xy ∈ R d x × d y , A yx ∈ R d y × d x

Original: , K 3: z k +1 =
Revised: 3: z k +1 =

Original: z k −G − 1 k ˆ Hg ( z k ) 4:
Revised: −

Original: Randomly choose u k from (7) 5: G k +1 ={++
+}
Revised: Randomly choose u k from (7) and update G k {++
+}

Original: SR1 (cid:0) G k , H , u k (cid:1) 6: end for as A =(cid:20)
Revised: =

Original: A xx
Revised: Broyd τ k

Original: A xy A yx
Revised: G k H u

Original: A yy (cid:21) where the sub-matrices A xx ∈ R n x × n x , A xy ∈ R n x × n y , A yx ∈ R n y × n xand A yy ∈ R n y × n y satisfy A xx ⪰ µ I , A yy ⪯ − µ I and ∥ A ∥ ≤ L .
Revised: 5: end for and A yy ∈ R d y × d y satisfy A xx ⪰ µ I , A yy ⪯ − µ I and ∥ A ∥ ≤ L .

Original: Using notations introduced in
Revised: Using notations introduced in

Original: Section 2, we have z =
Revised: Section 2, we have z =

Original: [ x ; y ] , g ( z ) =
Revised: [ x ; y ] , g ( z ) =

Original: Az − b , ˆ H def = ˆ H ( z )
Revised: Az − b , ˆ H def = ˆ H ( z )

Original: =
Revised: =

Original: A and H def = H ( z ) =
Revised: A and H def = H ( z ) =

Original: A 2 .
Revised: A 2 .

Original: We present the detailed procedure of random quasi-Newton methods for quadratic saddle pointproblem by using the Broyden family update, BFGS update and SR1 update in Algorithm 2, 3 and 4respectively.
Revised: We present the detailed procedure of random quasi-Newton methods for quadratic saddle point problem by using the Broyden family update, BFGS update and SR1 update in Algorithm 2, 3 andrespectively.

Original: We define λ k as the gradient norm at z k for our convergence analysis, that is λ k def =
Revised: We define λ k as the gradient norm at z k for our convergence analysis, that is λ k def =

Original: ∥ g ( z k ) ∥ .
Revised: ∥ g ( z k ) ∥ .

Original: Thedefinition of λ k in this paper is different from the measure used in convex optimization [19, 31] 1 , butit also holds the similar property as follows.
Revised: The definition of λ k in this paper is different from the measure used in convex optimization [20, 32] 2 , but it also holds the similar property as follows.

Original: Lemma 3.7.
Revised: Lemma 3.7.

Original: Assume we have η k ≥ 1 and G k ∈ R n × n such that H ⪯ G k ⪯ η k H for Algorithm 2,3 and 4, then we have λ k +1 ≤ (1 − 1 /η k ) λ k .
Revised: Assume we have η k ≥ 1 and G k ∈ R d × d such that H ⪯ G k ⪯ η k H for Algorithm 2, 3 and 4, then we have λ k +1 ≤ (1 − 1 /η k ) λ k .

Original: The next theorem states the assumptions of Lemma 3.7 always holds with η k = κ 2 ≥ 1 , which meansλ k converges to 0 linearly.
Revised: The next theorem states the assumptions of Lemma 3.7 always holds with η k = κ 2 ≥ 1 , which means λ k converges to 0 linearly.

Original: Theorem 3.8.
Revised: Theorem 3.8.

Original: For all k ≥ 0 , Algorithm 2, 3, 4 hold that H ⪯ G k ⪯ κ 2 H and λ k ≤ (cid:0) 1 − 1 /κ 2 (cid:1) k λ 0 .
Revised: For all k ≥ 0 , Algorithm 2, 3, 4 hold that H ⪯ G k ⪯ κ 2 H and λ k ≤ (cid:0) 1 − 1 / κ 2 (cid:1) k λ 0 .

Original: Lemma 3.7 also implies superlinear convergence can be obtained if there exists η k which convergesto 1.
Revised: Lemma 3.7 also implies superlinear convergence can be obtained if there exists η k which converges to 1.

Original: Applying Lemma 3.3, 3.4 and 3.6, we can show it holds for proposed algorithms.
Revised: Applying Lemma 3.3, 3.4 and 3.6, we can show it holds for proposed algorithms.

Original: Theorem 3.9.
Revised: Theorem 3.9.

Original: Solving quadratic saddle point problem by the proposed quasi-Newton Algorithms,for all k ≥ 0 , we have:
Revised: Solving quadratic saddle point problem by the proposed quasi-Newton Algorithms, for all k ≥ 0 , we have:

Original: For the Broyden family method (Algorithm 2), we have E [ λ k +1 /λ k ] ≤ (1 − 1 / ( nκ 2 ))
Revised: For the Broyden family method (Algorithm 2), we have E [ λ k +1 /λ k ] ≤ (cid:0) 1 − 1 / ( d κ 2 ) (cid:1) k

Original: k nκ 2 .2.
Revised: d κ 2 . 2.

Original: For the BFGS method (Algorithm 3), we have E [ λ k +1 /λ k ] ≤ (1 − 1 /n )
Revised: For the BFGS method (Algorithm 3), we have E [ λ k +1 /λ k ] ≤ (1 − 1 /d )

Original: k nκ 2 . 
Revised: k d κ 2 . 3.

Original: For the SR1 method (Algorithm 4), we have E [ λ k +1 /λ k ] ≤ (1 − k/n ) nκ 4 .
Revised: For the SR1 method (Algorithm 4), we have E [ λ k +1 /λ k ] ≤ (1 − k/d ) d κ 4 .

Original: Combining the results of Theorem 3.8 and 3.9, we achieve the two-stages convergence behavior, thatis, the algorithm has the global linear convergence and local superlinear convergence.
Revised: Combining the results of Theorem 3.8 and 3.9, we achieve the two-stages convergence behavior, that is, the algorithm has the global linear convergence and local superlinear convergence.

Original: We leave theformal description in appendix.
Revised: We leave the formal description in appendix.

Original: 3.3 Algorithms for General Saddle Point Problems
Revised: 3.3 Algorithms for General Saddle Point Problems

Original: In this section, we consider the general saddle point problem where f ( x , y ) in (1) satisfies Assumption 2.1 and 2.2.
Revised: In this section, we consider the general saddle point problem where f ( x , y ) in (1) satisfies Assumption 2.1 and 2.2.

Original: We propose quasi-Newton methods for solving the problem with local superlinearconvergence and O (cid:0) n 2 (cid:1) time complexity for each iteration.
Revised: We propose quasi-Newton methods for solving the problem with local superlinear convergence and O (cid:0) d 2 (cid:1) time complexity for each iteration.

Original: 3.3.1 Algorithms

Original: The key idea of designing quasi-Newton methods for saddle point problems is approximating theauxiliary matrix H ( z ) def = (cid:0) ˆ H ( z ) (cid:1) 2 to characterize the second-order information.

Original: Since the Hessian

Original: 1: Input: z 0 ∈ R n , G 0 ⪰ H , τ k ∈ [0 , 1] and M ≥ 0 . 2: for k = 0 , 1 . . . 3: z k +1 = z k − G − 1 k{+
+}
Revised: 1: Input: z 0 ∈ R d , G {+
+}

Original: ˆ H k g k 4:

Original: Compute r k = ∥

Original: z k +1 − z k ∥ and ˜ G k

Original: =

Original: (1 + Mr k ) G k 5:

Original: [-Randomly choose u k from (7) and update G k +
-]
Revised: [-Randomly choose u k from (7) and update G k +
-]

Original: =
Revised: =

Original: Broyd τ k ( ˜ G k , H k +1 , u k ) . 6: end for 1: Input: z 0 ∈ R n , G 0 ⪰ H , M ≥ 0 .
Revised: L 2 I , L 0 =

Original: 2: L0 = G − 1 / 23: for k = 0 , 1 . . .
Revised: L − 1 I 2: for k = 0 , 1 . . .

Original:  z k +1 =
Revised: 3: z k +1 =

Original: z k − G − 1 k ˆ H k g k 5: r k = ∥ z k +1 − z k ∥ 6: ˜ G k
Revised: z k − G − 1 k ˆ Hg ( z k ) 4:

Original: = (1 + Mr k ) G k 7:
Revised: Randomly choose ˜ u k from (7) 5: u k =

Original: ˜ L  k [-= L k / √ 
-]
Revised: L ⊤ k [-= L k / √ 
-]˜ u

Original: + Mr k 8:
Revised: k 6:

Original: Randomly choose u k from (7) 9:

Original: G k +1 =
Revised: G k +1 =

Original: BFGS( ˜ G k , Hk +1 ,˜ L k u k ) 10:
Revised: BFGS ( G k , H , u k ) 7:

Original: L k +1 =
Revised: L k +1 =

Original: Fast-Chol ( Hk +1 ,˜ L k ,˜ L k u k ) 11: end for
Revised: Fast-Chol ( H , L k , u k ) 8: end for

Original: 1: Input: z 0 ∈ R n , G 0 ⪰ H and M ≥2: for k{+
+} = 0 , 1 . . .{+L 
+}
Revised: 1: Input: z 0 ∈ R d , G {+
+} = {+L 
+}

Original: 3: z k+1 = z k − G − 1 k
Revised: I K ≤ d +2: for k = 0 , 1 . .

Original: ˆ H k g k 4: r k = ∥
Revised: , K 3:

Original: z k +1 − z k ∥ 5: ˜ G  k = (1 + Mr k ) G k 6:
Revised: z k +1 = z k − G − 1 k ˆ Hg ( z k ) 4:

Original: Randomly choose u k from (7) 7:
Revised: Randomly choose u k from (7) 5:

Original: G k +1 =
Revised: G k +1 =

Revised: SR1 (cid:0) G k , H , u k (cid:1) 6: end for

Revised: 3.3.1 Algorithms

Revised: The key idea of designing quasi-Newton methods for saddle point problems is approximating the auxiliary matrix H ( z ) def = (cid:0) ˆ H ( z ) (cid:1) 2 to characterize the second-order information.

Original: SR1( ˜ G k , H k +1 , u k ) 8: end for of f is Lipschitz continuous and bounded by Assumption 2.1 and 2.2, which means the auxiliarymatrix operator H ( z ) is also Lipschitz continuous.
Revised: Since the Hessian of f is Lipschitz continuous and bounded by Assumption 2.1 and 2.2, which means the auxiliary matrix operator H ( z ) is also Lipschitz continuous.

Original: Lemma 3.10.
Revised: Lemma 3.10.

Original: Under Assumption 2.1 and 2.2, we have H ( z ) is 2 LL 2 -Lipschitz continuous.
Revised: Under Assumption 2.1 and 2.2, we have H ( z ) is 2 LL 2 -Lipschitz continuous.

Original: Combining Lemma 3.1 and 3.10, we achieve the following properties of H ( z ) , which analogize thestrongly self-concordance in convex optimization [31].
Revised: Combining Lemma 3.1 and 3.10, we achieve the following properties of H ( z ) , which analogize the strongly self-concordance in convex optimization [32].

Original: Lemma 3.11.
Revised: Lemma 3.11.

Original: Under Assumption 2.1 and 2.2, for all z , z ′ , w ∈ R n , the auxiliary matrix operator
Revised: Under Assumption 2.1 and 2.2, for all z , z ′ , w ∈ R d , the auxiliary matrix operator

Original: H ( · ) satisfies H ( z ) − H ( z ′ )
Revised: H ( · ) satisfies H ( z ) − H ( z ′ )

Original: ⪯ M ∥
Revised: ⪯ M ∥

Original: z − z ′ ∥ H ( w )  where M = 2 κ 2 L 2 /L .
Revised: z − z ′ ∥ H ( w ) , where M = 2 κ 2 L 2 /L .

Original: Corollary 3.12.
Revised: Corollary 3.12.

Original: Let z , z ′ ∈ R n and r = ∥
Revised: Let z , z ′ ∈ R d and r = ∥

Original: z ′ − z ∥ .
Revised: z ′ − z ∥ .

Original: Suppose the objective function f satisfies
Revised: Suppose the objective function f satisfies

Original: Assumption 2.1 and 2.2, then for all z , z ′ ∈ R n and M = 2 κ 2 L 2 /L , the auxiliary matrix operator
Revised: Assumption 2.1 and 2.2, then for all z , z ′ ∈ R d and M = 2 κ 2 L 2 /L , the auxiliary matrix operator

Original: H ( · ) holds thatH ( z ) /
Revised: H ( · ) holds that

Original: (1 + Mr ) ⪯ H ( z ′ ) ⪯ (1 + Mr ) H ( z ) .
Revised: [Equation]

Original: Different from the quadratic case, the auxiliary matrix H ( z ) is not fixed for general saddle pointproblem.
Revised: Different from the quadratic case, the auxiliary matrix H ( z ) is not fixed for general saddle point problem.

Original: Based on the smoothness of H ( z ) , we apply Corollary 3.12 to generalize Lemma 3.13 asfollows.
Revised: Based on the smoothness of H ( z ) , we apply Corollary 3.12 to generalize Lemma 3.13 as follows.

Original: Lemma 3.13.
Revised: Lemma 3.13.

Original: Let z ∈ R n and G ∈ R n × n be a positive definite matrix
Revised: Let z ∈ R d and G ∈ R d × d be a positive definite matrix

Original: such that H ( z ) ⪯ G ⪯ η H ( z )for some η ≥ 1 .
Revised: such that H ( z ) ⪯ G ⪯ η H ( z ) for some η ≥ 1 .

Original: In addition, define z + ∈ R n and r = ∥
Revised: In addition, define z + ∈ R d and r = ∥

Original: z + − z ∥ , then for all u ∈ R n , τ ∈ [0 , 1]and M = 2 κ 2 L 2 /L , we have H ( z + ) ⪯ Broyd τ ( ˜ G , H ( z + ) , u ) ⪯ (1 + Mr ) 2 η H ( z + ) where˜ G def =
Revised: z + − z ∥ , then for all u ∈ R d , τ ∈ [0 , 1] and M = 2 κ 2 L 2 /L , we have [Equation]

Original:  (1 + Mr ) G ⪰ H ( z + ) .
Revised: where ˜ G def = (1 + Mr ) G ⪰ H ( z + ) .

Original: Lemma 3.13 implies it is reasonable to have the algorithms using G k
Revised: Lemma 3.13 implies it is reasonable to have the algorithms using G k

Original: +

Revised: +

Original: =
Revised: =

Original: Broyd τ k
Revised: Broyd τ k

Original: ( ˜ G k , H k +1 , u k )with ˜ G k
Revised: ( ˜ G k , H k +1 , u k ) with ˜ G k

Original: = (1 + Mr k ) G k and r k = ∥
Revised: = (1 + Mr k ) G k and r k = ∥

Original: z k +1 − z k ∥ .
Revised: z k +1 − z k ∥ .

Original: Similarly, we can also achieve G k +1 by such˜ G k for specific BFGS and SR1 update.
Revised: Similarly, we can also achieve G k +1 by such ˜ G k for specific BFGS and SR1 update.

Original: Combining this with iteration (3), we propose severalquasi-Newton methods for general strongly-convex-strongly-concave saddle point problems.
Revised: Combining this with iteration (3), we propose several quasi-Newton methods for general strongly-convex-strongly-concave saddle point problems.

Original: Thedetails are shown in Algorithm 5, 6 and 7 for Broyden family, BFGS and SR1 updates respectively.
Revised: The details are shown in Algorithm 5, 6 and 7 for Broyden family, BFGS and SR1 updates respectively.

Original: 3.3.2 Convergence Analysis
Revised: 3.3.2 Convergence Analysis

Original: Let us consider the convergence guarantee for algorithms proposed in Section 3.3.1.
Revised: Let us consider the convergence guarantee for algorithms proposed in Section 3.3.1.

Original: We introducethe following notations to simplify the presentation.
Revised: We introduce the following notations to simplify the presentation.

Original: We let { z k } be the sequence generated from
Revised: We let { z k } be the sequence generated from

Original: Algorithm 5, 6 or 7 and denoteg k def = g
Revised: Algorithm 5, 6 or 7 and denote

Original:  ( z k ) ,r k def = ∥
Revised: [Equation] k def = g ( z k ) ,

Original: z k +1 − z k ∥ , ˆ H k def = ˆ H ( z k ) and H k def = (cid:0) ˆ H ( z k ) (cid:1) 2 .
Revised: [Equation]

Original: We still use gradient norm λ k def = ∥∇ f
Revised: We still use gradient norm λ k def = ∥∇ f

Original: ( z k ) ∥ for analysis and establish the relationship between λ kand λ k +1 , which is shown in Lemma 3.14. 
Revised: ( z k ) ∥ for analysis and establish the relationship between λ k and λ k +1 , which is shown in Lemma 3.14. 1: Input: z R d , G , τ

Revised: 1] and M

Revised: 2: for k = 0 ,

Original: Lemma 3.14.
Revised: = G H g Compute r k = ∥

Revised: z k +1 − z k ∥ and ˜ G

Revised: = (1 + Mr k ) G k 5:

Original: {+Randomly choose u k from (7) and update G k +
+}
Revised: {+Randomly choose u k from (7) and update G k +
+}

Revised: =

Revised: Broyd τ k

Original: Using Algorithm 5, 6 and 7, suppose we have H k ⪯ G k ⪯ η k H k  ,  for some η k ≥ 1and let β
Revised: ( ˜ G k , H k 1 , u k ) . 6: end for Randomly choose u k from (7) 9:

Original: = L 2 / (2 µ 2 ) , then we have λ k +1 ≤ (1 − 1 /η k )
Revised: Fast 11: end for Randomly choose 8: end for

Original: λ k + βλ 2 k and r k ≤ (1 /µ ) 
Revised: Lemma 3.14. Using Algorithm 5, 6 and 7, suppose we have H k ⪯ G k ⪯ η k H k , for some η k ≥and let β = L 2 / (2 µ 2 ) , then we have

Original: λ k .
Revised: [Equation] and

Original:  Rodomanov and Nesterov [31, Lemma 4.3] derive a result similar to Lemma 3.14 for minimizingthe strongly-convex function ˆ f ( · ) on the different measure λ ˆ f ( · ) def =
Revised: [Equation] Rodomanov and Nesterov [32, Lemma 4.3] derive a result similar to Lemma 3.14 for minimizing the strongly-convex function ˆ f ( · ) on the different measure λ ˆ f ( · ) def =

Original: ⟨∇ ˆ f ( · ) , ∇ 2 ˆ f ( · ) − 1 ∇ ˆ f ( · ) ⟩ .[-
-]
Revised: ⟨∇ ˆ f ( · ) , ∇ 2 ˆ f ( · ) − 1 ∇ ˆ f ( · ) ⟩ .[-
-]

Original: Note that our algorithms are based on the iteration rule z k +

Revised: Note that our algorithms are based on the iteration rule z k +

Original: =
Revised: =

Original: z k − G − 1 k ˆ H k g k .
Revised: z k − G − 1 k ˆ H k g k .

Original: Compared withquasi-Newton methods for convex optimization, there exists an additional term ˆ H k between G − 1 kand g k , which leads to the fact that the convergence analysis based on λ ˆ f ( z k ) is difficult.
Revised: Compared with quasi-Newton methods for convex optimization, there exists an additional term ˆ H k between G − 1 k and g k , which leads to the fact that the convergence analysis based on λ ˆ f ( z k ) is difficult.

Original: Fortunately,we find using gradient norm λ k directly makes the analysis achievable.
Revised: Fortunately, we find using gradient norm λ k directly makes the analysis achievable.

Original: For further analysis, we also denote σ k def = σ H k
Revised: For further analysis, we also denote σ k def = σ H k

Original: ( G k ) =
Revised: ( G k ) =

Original: ⟨ H − 1 k , G k ⟩ − n.
Revised: ⟨ H − 1 k , G k ⟩ − d.

Original: Then we establish the linear convergence for the first period of iterations, which can be viewed as theextension of Theorem 3.8.
Revised: Then we establish the linear convergence for the first period of iterations, which can be viewed as the extension of Theorem 3.8.

Original: Note that the following result holds for Algorithm 5, 6 and 7; and it doesnot depend on the choice of u k .
Revised: Note that the following result holds for Algorithm 5, 6 and 7; and it does not depend on the choice of u k .

Original: Theorem 3.15.
Revised: Theorem 3.15.

Original: Using Algorithm 5, 6 and 7 by G 

Revised: Using Algorithm 5, 6 and 7 by G

Original: =
Revised: =

Original: L 2 I and M = 2 κ 2 L 2 /L , suppose the initialpoint z 0 is sufficiently close to z ∗
Revised: L 2 I and M = 2 κ 2 L 2 /L , suppose the initial point z 0 is sufficiently close to z ∗

Original: such that Mλ 0 /µ ≤ ln b/ (4 bκ 2 ) , where 1 < b < 5 , then wehave λ k ≤ (cid:0) 1 − 1 /(2 bκ 2 ) (cid:1)
Revised: such that Mλ 0 /µ ≤ ln b/ (4 b κ 2 ) with 1 < b < 5 , then we have λ k ≤ (cid:0) 1 − 1 / 2 b κ 2 (cid:1)

Original: k λ 0 and H k ⪯ G k ⪯ exp (cid:16) 2 (cid:80)
Revised: k λ 0 and H k ⪯ G k ⪯ exp (cid:16) 2 (cid:80)

Original: k − 1 i =

Revised: k − 1 i =

Original: ρ i (cid:17) κ 2 H k ⪯ bκ 2 H k for all k ≥ 0 ,where ρ k def =
Revised: ρ i (cid:17) κ 2 H k ⪯ b κ 2 H k for all k ≥ 0 , where ρ k def =

Original: Mλ k /µ .
Revised: Mλ k /µ .

Original: We now analyze how σ k changes after one iteration to show the local superlinear convergence forthe Broyden family method (Algorithm 5) and the BFGS method (Algorithm 6).
Revised: We now analyze how σ k changes after one iteration to show the local superlinear convergence for the Broyden family method (Algorithm 5) and the BFGS method (Algorithm 6).

Original: Recall that σ k isdefined to measure how well does matrix G k approximate H k .
Revised: Recall that σ k is defined to measure how well does matrix G k approximate H k .

Original: Lemma 3.16.
Revised: Lemma 3.16.

Original: Solving the general strongly-convex-strongly-concave saddle point problem (1) under
Revised: Solving the general strongly-convex-strongly-concave saddle point problem (1) under

Original: Assumption 2.1 and 2.2 by our quasi-Newton algorithms and supposing the sequences G 0 , · · · , G kgenerated by the Algorithm 5 and 6 are given, then we have the following results for σ k for all k:
Revised: Assumption 2.1 and 2.2 by our quasi-Newton algorithms and supposing the sequences G 0 , · · · , G k generated by the Algorithm 5 and 6 are given, then we have the following results for σ k for all k

Original: The random Broyden family method (Algorithm 5) holds thatE [ σ k +1 ] ≤ (cid:0) 1 − 1 / ( nκ 2 ) (cid:1) (1 + Mr k ) 2 ( σ k + 2 nMr k / (1 + Mr k )) .
Revised: The random Broyden family method (Algorithm 5) holds that

Original:  The random BFGS method (Algorithm 6) holds thatE [ σ k +1 ] ≤ (1 − 1 /n )
Revised: [Equation] The random BFGS method (Algorithm 6) holds that

Original: (1 + Mr k ) 2 ( σ k + 2 nMr k / (1 + Mr k )) .
Revised: [Equation]

Original: The analysis for the SR1 method is based on constructing η k such that η k = tr ( G k − H k 
Revised: The analysis for the SR1 method is based on constructing η k such that η k = tr( G k − H k ) /

Original: ) / tr( H k  and the technical details are showed in appendix.
Revised: tr( H k ) and the technical details are showed in appendix.

Original: Based on Lemma 3.16, one can show that ouralgorithms enjoy the local superlinear convergence for the general saddle point problems.
Revised: Based on Lemma 3.16, one can show that our algorithms enjoy the local superlinear convergence for the general saddle point problems.

Original: Theorem 3.17.
Revised: Theorem 3.17.

Original: Solving general saddle point problem (1) under Assumption 2.1 and 2.2 by proposedrandom quasi-Newton methods (Algorithm 5, 6 and 7) by G 

Revised: Solving general saddle point problem (1) under Assumption 2.1 and 2.2 by proposed random quasi-Newton methods (Algorithm 5, 6 and 7) by G

Original: =
Revised: =

Original: L 2 I and M = 2 κ 2 L 2 /L , we havethe following results for all k > 0 :
Revised: L 2 I and M = 2 κ 2 L 2 /L , we have the following results for all k > 0 :

Original: For the random Broyden family method (Algorithm 5), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+2 n ) κ 2 ,we have E [ λ k +1 /λ k ] ≤ (cid:0) 1 − 1 / ( nκ 2 ) (cid:1) k2 nκ 2 .2.
Revised: For the random Broyden family method (Algorithm 5), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+2 d ) κ 2 , we have E [ λ k +1 /λ k ] ≤ (cid:0) 1 − 1 / ( d κ 2 ) (cid:1) k

Revised: 2 d κ 2 . 2.

Original: For the random BFGS method (Algorithm 6), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+2 n ) κ 2 , we have E [ λ k +1 /λ k ] ≤ (1 − 1 /n )
Revised: For the random BFGS method (Algorithm 6), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+ d ) κ 2 , we have E [ λ k +1 /λ k ] ≤ (1 − 1 /d )

Original: k 2 nκ 2 .3.
Revised: k 2 d κ 2 . 3.

Original: For the random SR1 method (Algorithm 7), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+2 nκ 2 ) κ 2 , we have E [ λ k +1 /λ k ] ≤ (1 − 1 /n )
Revised: For the random SR1 method (Algorithm 7), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+2 d κ 2 ) κ 2 , we have E [ λ k +1 /λ k ] ≤ (1 − 1 /d )

Original: k 2 nκ 4 .
Revised: k 2 d κ 4 .

Original: Finally, combining the results of Theorem 3.15 and 3.17, we can prove the algorithms achieve thetwo-stages convergence behaviors as follows.
Revised: Finally, combining the results of Theorem 3.15 and 3.17, we can prove the algorithms achieve the two-stages convergence behaviors as follows.

Original: Corollary 3.18.
Revised: Corollary 3.18.

Original: Solving the general saddle point problem (1) under Assumption 2.1 and 2.2 byour random quasi-Newton methods (Algorithm 5, 6 and 7) with G 

Revised: Solving the general saddle point problem (1) under Assumption 2.1 and 2.2 by our random quasi-Newton methods (Algorithm 5, 6 and 7) with G

Original: =
Revised: =

Original: L 2 I and M = 2 κ 2 L 2 /L , ifthe initial point is sufficiently close to the saddle point such that Mλ 0 /µ ≤ ln 2 / (8 κ 2 ) , then withprobability 1 − δ for any δ ∈ (0 , 1) , we have the following results:
Revised: L 2 I and M = 2 κ 2 L 2 /L , if the initial point is sufficiently close to the saddle point such that Mλ 0 /µ ≤ ln 2 / (8 κ 2 ) , then with probability 1 − δ for any δ ∈ (0 , 1) , we have the following results:

Original: The random Broyden family method (Algorithm 5) holds [-for k 
-]
Revised: The random Broyden family method (Algorithm 5) holds [-for k 
-]that

Original: {+[Equation] for all k 
+} = O (cid:0)nκ 2 ln( nκ/δ ) (cid:1) andall k ≥ 0 that λ k 0 + k ≤ (cid:16) 1 − 1 nκ 2 +1 (cid:17) k ( k − 1)(cid:0) 12 (cid:1)
Revised: {+[Equation] for all k 
+} = O (cid:0)

Original: k (cid:0) 1 − 14 κ 2  (cid:1)  k  0 λ 0 .2.
Revised: d κ 2 ln ( d κ /δ ) (cid:1) and k ≥ 0 .

Original: The random BFGS/SR1 method (Algorithm 6/7) holds [-for k 
-]
Revised: The random BFGS/SR1 method (Algorithm 6/7) holds [-for k 
-]that

Original: {+[Equation] for all k 
+} = O (cid:0) max (cid:8) n, κ 2 (cid:9) ln( nκ/δ ) (cid:1)and all k ≥ 0 that λ k 0 + k ≤ (cid:16) 1 − 1 n +1 (cid:17) k
Revised: {+[Equation] for all k 
+} = O (cid:0) max (cid:8) d, κ 2 (cid:9) ln ( d κ /δ ) (cid:1) and k ≥ 0 .

Original: ( k − 1)(cid:0) 12 (cid:1)
Revised: The relationship between our quasi-Newton methods (Algorithm 5, 6 and 7) and existing first-order method for minimax problem is similar to the one for minimization problem. For our BFGS and SR methods, the dependency on κ 2 only appears on the first period of linear convergence, which matches the convergence rate of gradient descent ascent.

Original: k (cid:0) 1 − 14 κ 2 (cid:1) k 0 λ 0 .
Revised: As an analogy, minimizing strongly-convex function by quasi-Newton methods [20, 32] has dependency on κ in the first period of linear convergence, which matches the convergence rate of gradient descent. Our quasi-Newton methods also can be extended for solving non-linear equations, and the two-stage convergence behavior still hold.

Original:   Numerical Experiments
Revised: We provide the detailed discussion and the comparison with related work [19, 41] in Appendix F. [Figure]  Numerical Experiments

Original: In this section, we conduct the experiments on machine learning applications of AUC Maximizationand adversarial debiasing to verify our theory.
Revised: In this section, we conduct the experiments on machine learning applications of AUC Maximization and adversarial debiasing to verify our theory.

Original: We refer to Algorithm 2 and 5 with parameterτ k =
Revised: We refer to Algorithm 2 and 5 with parameter τ k =

Original: u ⊤ H
Revised: u ⊤ H

Original: k +1 / ( u ⊤ G k u ) as RaBFGSv1-Q and RaBFGSv1-G; refer to Algorithm 3, 4, 6 and 7 as
Revised: k +1 / ( u ⊤ G k u ) as RaBFGSv1-Q and RaBFGSv1-G; refer to Algorithm 3, 4, 6 and 7 as

Original: RaBFGSv2-Q, RaSR1-Q, RaBFGSv2-G and RaSR1-G respectively.
Revised: RaBFGSv2-Q, RaSR1-Q, RaBFGSv2-G and RaSR1-G respectively.

Original: We compare these proposedalgorithms with classical first-order method extragradient (EG) [18, 37].
Revised: We compare these proposed algorithms with classical first-order method extragradient (EG) [18, 38].

Original: 4.1 AUC Maximization
Revised: 5.1 AUC Maximization

Original: AUC maximization [15, 41] aims to find the classifier w ∈ R d on the training set { a i , b i } mi =1 wherea{+ni =
+}
Revised: AUC maximization [15, 43] aims to find the classifier w ∈ R m on the training set { a i , b i } {+ni =
+}

Original:  i ∈ R d and b i ∈ { +1 , − 1 } .
Revised: where a i ∈ R d and b i ∈ { +1 , − 1 } .

Original: We denote m + be the number of positive instances and p = m + /m .
Revised: We denote n + be the number of positive instances and p = n + /n .

Original: The minimax formulation for AUC maximization can be written as
Revised: The minimax formulation for AUC maximization can be written as

Original: [Equation]
Revised: [Equation]

Original: where x = [ w ; u ; v ] ∈ R d +2 , λ > 0 is the regularization parameter and f i is defined as
Revised: where x = [ w ; u ; v ] ∈ R m +2 , λ > 0 is the regularization parameter and f i is defined as

Original: [Equation]
Revised: [Equation]

Original: The objective function of AUC maximization is quadratic, hence we conduct the algorithms in Section3.2 (Algorithm 2, 3 and 4) for this model.
Revised: The objective function of AUC maximization is quadratic, hence we conduct the algorithms in Section 3.2 (Algorithm 2, 3 and 4) for this model.

Original: We set λ = 100 /m and evaluate all algorithms on threereal-world datasets “a9a”, “w8a” and “sido0”.
Revised: We set λ = 100 /n and evaluate all algorithms on three real-world datasets “a9a”, “w8a” and “sido0”.

Original: The dimension of the problem isn = d  +3 .
Revised: The dimension of the problem is d = m +

Original: Theresults of iteration numbers against ∥ g ( z ) ∥ 2 and CPU time against ∥ g ( z ) ∥ 2 are presented in Figure 1.
Revised: The results of iteration numbers against ∥ g ( z ) ∥ 2 and CPU time against ∥ g ( z ) ∥ 2 are presented in Figure 1.

Original: These results show that our algorithms perform better than the EG method.
Revised: These results show that our algorithms perform better than the EG method.

Original: 4.2 Adversarial Debiasing
Revised: 5.2 Adversarial Debiasing

Original: Adversarial learning [21, 42] can be used on fairness-aware machine learning issues.
Revised: Adversarial learning [22, 44] can be used on fairness-aware machine learning issues.

Original: Give the trainingset { a i , b i , c i } mi =1 where a i ∈ R d contains all input variables, b i ∈ R is the output and c i ∈ R is theinput variable which we want to protect and make it unbiased.
Revised: Give the training set { a i , b i , c i } ni =1 where a i ∈ R d contains all input variables, b i ∈ R is the output and c i ∈ R is the input variable which we want to protect and make it unbiased.

Original: Our experiments are based on thefairness-aware binary classification dataset “adult”, “bank market” and “law school”[29], leading tob i , c i ∈ { +1 , − 1 } .
Revised: Our experiments are based on the [Figure] fairness-aware binary classification dataset “adult”, “bank market” and “law school”[30], leading to b i , c i ∈ { +1 , − 1 } .

Original: The model try to minimax the function
Revised: The model is formulated by the minimax problem

Original: [Equation]
Revised: [Equation]

Original: where l 1 , l 2 are the logit functions: logit( a , b, c )
Revised: where l 1 , l 2 are the logit functions: logit( a , b, c )

Original: = log(1 + exp( − b a ⊤ c )) .
Revised: = log(1 + exp( − b a ⊤ c )) .

Original: We set the parametersβ, λ and γ as 0 . 5 , 10 − 4 and 10 − 4 respectively.
Revised: We set the parameters β, λ and γ as 0 . 5 , 10 − 4 and 10 − 4 respectively.

Original: The dimension of the problem isn = d + 1 .
Revised: The dimension of the problem is d = m +1 .

Original: Since theobjective function is non-quadratic, we conduct the proposed algorithms in Section 3.3 (Algorithm 5,6 and 7) here.
Revised: Since the objective function is non-quadratic, we conduct the proposed algorithms in Section 3.3 (Algorithm 5, 6 and 7) here.

Original: We use extragradient as warm up to achieve the local condition for proposed algorithms.
Revised: We use extragradient as warm up to achieve the local condition for proposed algorithms.

Original: The results of iteration numbers against ∥ g ( z ) ∥ 2 and CPU time against ∥ g ( z ) ∥ 2 are presented in
Revised: The results of iteration numbers against ∥ g ( z ) ∥ 2 and CPU time against ∥ g ( z ) ∥ 2 are presented in

Original: Figure 2, which indicate that our algorithms significantly outperform the baseline algorithm.[Figure] [Figure]
Revised: Figure 2, which indicate that our algorithms significantly outperform the baseline algorithm.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we propose quasi-Newton methods for solving strongly-convex-strongly-concave saddlepoint problems.
Revised: In this work, we propose quasi-Newton methods for solving strongly-convex-strongly-concave saddle point problems.

Original: We characterize the second-order information by approximating the square of
Revised: We characterize the second-order information by approximating the square of

Original: Hessian matrix, which avoids the issue of dealing with the indefinite Hessian directly.
Revised: Hessian matrix, which avoids the issue of dealing with the indefinite Hessian directly.

Original: We presentthe explicit local superlinear convergence rates for Broyden’s family update and faster convergencerates for two specific methods: SR1 and BFGS updates.
Revised: We present the explicit local superlinear convergence rates for Broyden’s family update and faster convergence rates for two specific methods: SR1 and BFGS updates.

Original: However, our algorithms still require tocompute the exact gradient and store the approximate Hessian.
Revised: However, our algorithms still require to compute the exact gradient and store the approximate Hessian by O ( d 2 ) space complexity.

Original: In future work, it is interesting todesign the stochastic algorithms to further reduce the computational cost of the iteration.
Revised: In future work, it is interesting to design the stochastic algorithms to further reduce the computational cost of the iteration.

Original: It is alsopossible to design the limited-memory quasi-Newton methods to make our ideas work more scalablesaddle point problems. 
Revised: It is also possible to design the limited-memory quasi-Newton methods for more scalable saddle point problems. Acknowledgements

Revised: We would like to thank the anonymous reviewer for pointing out the mistakes in our previous proof.

Revised: We would also like to thank Tong Zhang and John C.S. Lui for giving useful suggestions.

Revised: This work is supported by National Natural Science Foundation of China (No. 62206058) and Shanghai Sailing

Revised: Program (22YF1402900).

Original: Checklist
Revised: Checklist

Original: For

Original:  all authors.. (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section 2.(b)
Revised: [Yes] See Section 2.

Original:  Did you include complete proofs 
Revised: (b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A-D. 3.

Original:  (a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We present them in the supplemental material.
Revised: [Yes] We present them in the supplemental material.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] We present them in Section 4 and in Appendix E. (c) Did you report error bars (e
Revised: [Yes] We present them in Section 5 and in Appendix E. (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi- ments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix E. 
Revised: [Yes] See Appendix E. 4.

Original: If you are using existing assets (e g , code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 4 andAppendix E. (b) Did you mention the license of the assets?
Revised: [Yes] See Section 5 and Appendix E. (b) Did you mention the license of the assets?

Original: [Yes] See Section 4 and Appendix E, the data is public available.
Revised: [Yes] See Section 5 and Appendix E, the data is public available.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]See Appendix E. (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] See Appendix E. (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]