Original: Quasi-Newton Methods for Saddle Point Problems
Revised: Quasi-Newton Methods for Saddle Point Problems

Original: Abstract
Revised: Abstract

Original: This paper studies quasi-Newton methods for strongly-convex-strongly-concavesaddle point problems.
Revised: This paper studies quasi-Newton methods for strongly-convex-strongly-concavesaddle point problems.

Original: We propose random Broyden family updates, which haveexplicit local superlinear convergence rate of O (cid:0)(cid:0) 1 − 1 / ( nκ 2 ) (cid:1) k
Revised: We propose random Broyden family updates, which haveexplicit local superlinear convergence rate of O (cid:0)(cid:0) 1 − 1 / ( nκ 2 ) (cid:1) k

Original: ( k − 1) / 2 (cid:1) , where nis the dimension of the problem, κ is the condition number and k is the number ofiterations.
Revised: ( k − 1) / 2 (cid:1) , where nis the dimension of the problem, κ is the condition number and k is the number ofiterations.

Original: The design and analysis of proposed algorithm are based on estimatingthe square of indefinite Hessian matrix, which is different from classical quasiNewton methods in convex optimization.
Revised: The design and analysis of proposed algorithm are based on estimatingthe square of indefinite Hessian matrix, which is different from classical quasiNewton methods in convex optimization.

Original: We also present two specific Broydenfamily algorithms with BFGS-type and SR1-type updates, which enjoy the fasterlocal convergence rate of O (cid:0)(cid:0) 1 − 1 /n (cid:1) k ( k − 1) / 2 (cid:1) .
Revised: We also present two specific Broydenfamily algorithms with BFGS-type and SR1-type updates, which enjoy the fasterlocal convergence rate of O (cid:0)(cid:0) 1 − 1 /n (cid:1) k ( k − 1) / 2 (cid:1) .

Original: Our numerical experiments showproposed algorithms outperform classical first-order methods.
Revised: Our numerical experiments showproposed algorithms outperform classical first-order methods.

Original:  Introduction
Revised:  Introduction

Original: In this paper we focus on the following smooth saddle point problem
Revised: In this paper we focus on the following smooth saddle point problem

Original: [Equation]
Revised: [Equation]

Original: where f is strongly-convex in x and strongly-concave in y .
Revised: where f is strongly-convex in x and strongly-concave in y .

Original: We target to find the saddle point ( x ∗ , y ∗ )which holds that
Revised: We target to find the saddle point ( x ∗ , y ∗ )which holds that

Original: [Equation]
Revised: [Equation]

Original: ) for all x ∈ R n x and y ∈ R n y .
Revised: ) for all x ∈ R n x and y ∈ R n y .

Original: This formulation is widely used in game theory [1, 38], AUCmaximization [15, 41], fairness-aware machine learning [42], robust optimization [2, 11, 13, 35],empirical risk minimization [44] and reinforcement learning [10].
Revised: This formulation is widely used in game theory [1, 38], AUCmaximization [15, 41], fairness-aware machine learning [42], robust optimization [2, 11, 13, 35],empirical risk minimization [44] and reinforcement learning [10].

Original: There are a great number of first-order optimization algorithms for solving problem (1), includingextragradient method [18, 37], optimistic gradient descent ascent [7, 27], proximal point method[30] and dual extrapolation [23].
Revised: There are a great number of first-order optimization algorithms for solving problem (1), includingextragradient method [18, 37], optimistic gradient descent ascent [7, 27], proximal point method[30] and dual extrapolation [23].

Original: These algorithms iterate with first-order oracle and achieve linearconvergence.
Revised: These algorithms iterate with first-order oracle and achieve linearconvergence.

Original: Lin et al.
Revised: Lin et al.

Original: [20], Wang and Li [39] used Catalyst acceleration to reduce the complexityfor unbalanced saddle point problem, nearly matching the lower bound of first-order algorithms [25,43] under some specific assumptions.
Revised: [20], Wang and Li [39] used Catalyst acceleration to reduce the complexityfor unbalanced saddle point problem, nearly matching the lower bound of first-order algorithms [25,43] under some specific assumptions.

Original: Compared with first-order methods, second-order methodsusually enjoy superior convergence in numerical optimization.
Revised: Compared with first-order methods, second-order methodsusually enjoy superior convergence in numerical optimization.

Original: Huang et al.
Revised: Huang et al.

Original: [16] extended cubicregularized Newton (CRN) method [22, 23] to solve saddle point problem (1), which has quadraticlocal convergence.
Revised: [16] extended cubicregularized Newton (CRN) method [22, 23] to solve saddle point problem (1), which has quadraticlocal convergence.

Original: However, each iteration of CRN requires accessing the exact Hessian matrix andsolving the corresponding linear systems.
Revised: However, each iteration of CRN requires accessing the exact Hessian matrix andsolving the corresponding linear systems.

Original: These steps arise O (cid:0) n 3 x + n 3 y (cid:1) time complexity, which istoo expensive for high dimensional problems.
Revised: These steps arise O (cid:0) n 3 x + n 3 y (cid:1) time complexity, which istoo expensive for high dimensional problems.

Original: Quasi-Newton methods [3–5, 8, 36] are popular ways to avoid accessing exact second-order information applied in standard Newton methods.
Revised: Quasi-Newton methods [3–5, 8, 36] are popular ways to avoid accessing exact second-order information applied in standard Newton methods.

Original: They approximate the Hessian matrix based on the Broydenfamily updating formulas [3], which significantly reduces the computational cost.
Revised: They approximate the Hessian matrix based on the Broydenfamily updating formulas [3], which significantly reduces the computational cost.

Original: These algorithmsare well studied for convex optimization.
Revised: These algorithmsare well studied for convex optimization.

Original: The famous quasi-Newton methods including DavidonFletcher-Powell (DFP) method [8, 12], Broyden-Fletcher-Goldfarb-Shanno (BFGS) method [4, 5, 36]
Revised: The famous quasi-Newton methods including DavidonFletcher-Powell (DFP) method [8, 12], Broyden-Fletcher-Goldfarb-Shanno (BFGS) method [4, 5, 36]

Original: [Table]
Revised: [Table]

Original: BFGS/SR1 (Alg. 6 and 7)
Revised: BFGS/SR1 (Alg. 6 and 7)

Original: [Table]
Revised: [Table]

Original: and symmetric rank 1 (SR1) method [3, 8] enjoy local superlinear convergence [6, 9, 28] when the objective function is strongly-convex.
Revised: and symmetric rank 1 (SR1) method [3, 8] enjoy local superlinear convergence [6, 9, 28] when the objective function is strongly-convex.

Original: Recently, Rodomanov and Nesterov [31, 32, 33] proposed greedyand random variants of quasi-Newton methods, which first achieves non-asymptotic superlinear convergence.
Revised: Recently, Rodomanov and Nesterov [31, 32, 33] proposed greedyand random variants of quasi-Newton methods, which first achieves non-asymptotic superlinear convergence.

Original: Later, Lin et al.
Revised: Later, Lin et al.

Original: [19] established a better convergence rate which is condition-number-free.
Revised: [19] established a better convergence rate which is condition-number-free.

Original: Jin and Mokhtari [17], Ye et al.
Revised: Jin and Mokhtari [17], Ye et al.

Original: [40] showed the non-asymptotic superlinear convergence rate alsoholds for classical DFP, BFGS and SR1 methods.
Revised: [40] showed the non-asymptotic superlinear convergence rate alsoholds for classical DFP, BFGS and SR1 methods.

Original: In this paper, we study quasi-Newton methods for saddle point problem (1).
Revised: In this paper, we study quasi-Newton methods for saddle point problem (1).

Original: Note that when the
Revised: Note that when the

Original: Hessian matrix of the objective function is indefinite, the existing Broyden family update formulasand their convergence analysis cannot be applied directly.
Revised: Hessian matrix of the objective function is indefinite, the existing Broyden family update formulasand their convergence analysis cannot be applied directly.

Original: To overcome this issue, we propose avariant framework of random quasi-Newton methods for saddle point problems, which approximatesthe square of the Hessian matrix during the iteration.
Revised: To overcome this issue, we propose avariant framework of random quasi-Newton methods for saddle point problems, which approximatesthe square of the Hessian matrix during the iteration.

Original: Our theoretical analysis characterizes theconvergence rate by the gradient norm, rather than the weighted norm of gradient used in convexoptimization [17, 19, 31–33, 40].
Revised: Our theoretical analysis characterizes theconvergence rate by the gradient norm, rather than the weighted norm of gradient used in convexoptimization [17, 19, 31–33, 40].

Original: We summarize the theoretical results for proposed algorithms in
Revised: We summarize the theoretical results for proposed algorithms in

Original: Table 1.
Revised: Table 1.

Original: The local convergence behaviors for all of the algorithms have two periods.
Revised: The local convergence behaviors for all of the algorithms have two periods.

Original: The first periodhas k 0 iterations with a linear convergence rate O ((1 − 1 /κ 2 ) k 0 ) .
Revised: The first periodhas k 0 iterations with a linear convergence rate O ((1 − 1 /κ 2 ) k 0 ) .

Original: The second one enjoys superlinearconvergence, that is, for general Broyden family methods, we have O ((1 − 1 / ( nκ 2 + 1))
Revised: The second one enjoys superlinearconvergence, that is, for general Broyden family methods, we have O ((1 − 1 / ( nκ 2 + 1))

Original: k ( k − 1) / 2 )and for BFGS method and SR1 method, we have O ((1 − 1 / ( n + 1))
Revised: k ( k − 1) / 2 )and for BFGS method and SR1 method, we have O ((1 − 1 / ( n + 1))

Original: k ( k − 1) / 2 ) , which is conditionnumber-free.
Revised: k ( k − 1) / 2 ) , which is conditionnumber-free.

Original: Paper Organization
Revised: Paper Organization

Original: In Section 2, we state the notation and preliminaries of this paper.
Revised: In Section 2, we state the notation and preliminaries of this paper.

Original: In Section3, we first propose random quasi-Newton methods for quadratic saddle point problem which enjoyslocal superlinear convergence.
Revised: In Section3, we first propose random quasi-Newton methods for quadratic saddle point problem which enjoyslocal superlinear convergence.

Original: Then we extend it to solve general strongly-convex-strongly-concavesaddle point problems.
Revised: Then we extend it to solve general strongly-convex-strongly-concavesaddle point problems.

Original: In Section 4, we provide numerical experiments to validate our algorithmson popular machine learning models.
Revised: In Section 4, we provide numerical experiments to validate our algorithmson popular machine learning models.

Original: All proofs, experiment details and extensions are deferred toappendix.
Revised: All proofs, experiment details and extensions are deferred toappendix.

Original:  Notation and Preliminaries
Revised:  Notation and Preliminaries

Original: We use ∥·∥ to present spectral norm and Euclidean norm of matrix and vector respectively.
Revised: We use ∥·∥ to present spectral norm and Euclidean norm of matrix and vector respectively.

Original: We denotethe standard basis for R n by { e 1 , . . .
Revised: We denotethe standard basis for R n by { e 1 , . . .

Original: , e n } and let I be the identity matrix.
Revised: , e n } and let I be the identity matrix.

Original: The trace of a squarematrix is denoted by tr ( · ) .
Revised: The trace of a squarematrix is denoted by tr ( · ) .

Original: Given two positive definite matrices G and H , we define their innerproduct as ⟨ G , H ⟩ def = tr ( GH ) .
Revised: Given two positive definite matrices G and H , we define their innerproduct as ⟨ G , H ⟩ def = tr ( GH ) .

Original: We introduce the quantity σ H ( G ) def = ⟨ H − 1 , G − H ⟩ = ⟨ H − 1 , G ⟩ − n to measure how well doesmatrix G approximate matrix H .
Revised: We introduce the quantity σ H ( G ) def = ⟨ H − 1 , G − H ⟩ = ⟨ H − 1 , G ⟩ − n to measure how well doesmatrix G approximate matrix H .

Original: If we further suppose G ⪰ H , then according to Rodomanov and
Revised: If we further suppose G ⪰ H , then according to Rodomanov and

Original: Nesterov [31] it holds that G − H ⪯ ⟨ H − 1 , G − H ⟩ H =
Revised: Nesterov [31] it holds that G − H ⪯ ⟨ H − 1 , G − H ⟩ H =

Original: σ H ( G ) H .
Revised: σ H ( G ) H .

Original: Using the notation of problem (1), we let z =
Revised: Using the notation of problem (1), we let z =

Original: [ x ; y ] ∈ R n where n def = n x + n y and denote thegradient and Hessian matrix of f at ( x , y ) as g ( z ) ∈ R n and ˆ H ( z ) ∈ R n × n .
Revised: [ x ; y ] ∈ R n where n def = n x + n y and denote thegradient and Hessian matrix of f at ( x , y ) as g ( z ) ∈ R n and ˆ H ( z ) ∈ R n × n .

Original: We suppose the saddle point problem (1) satisfies the following assumptions.
Revised: We suppose the saddle point problem (1) satisfies the following assumptions.

Original: Assumption 2.1.
Revised: Assumption 2.1.

Original: The objective function f ( x , y ) is twice differentiable and has L -Lipschitz continuous gradient and L 2 -Lipschitz continuous Hessian , i.e., there exists constants L > 0 and L 2 > 0 such that for any z
Revised: The objective function f ( x , y ) is twice differentiable and has L -Lipschitz continuous gradient and L 2 -Lipschitz continuous Hessian , i.e., there exists constants L > 0 and L 2 > 0 such that for any z

Original: = [ x ; y ] , z ′ =
Revised: = [ x ; y ] , z ′ =

Original: [ x ′ ; y ′ ]
Revised: [ x ′ ; y ′ ]

Original: ∈ R n , we have ∥ g ( z ) − g
Revised: ∈ R n , we have ∥ g ( z ) − g

Original: ( z ′ )
Revised: ( z ′ )

Original: ∥
Revised: ∥

Original: ≤ L ∥
Revised: ≤ L ∥

Original: z − z ′ ∥
Revised: z − z ′ ∥

Original: and∥ ˆ H ( z ) − ˆ H ( z ′ )
Revised: and∥ ˆ H ( z ) − ˆ H ( z ′ )

Original: ∥
Revised: ∥

Original: ≤ L 2 ∥ z − z ′ ∥ .
Revised: ≤ L 2 ∥ z − z ′ ∥ .

Original: Assumption 2.2.
Revised: Assumption 2.2.

Original: The objective function f ( x , y ) is twice differentiable, µ -strongly-convex in xand µ -strongly-concave in y , i.e., there exists constant µ > 0 such that ∇ 2 xx f ( x , y )
Revised: The objective function f ( x , y ) is twice differentiable, µ -strongly-convex in xand µ -strongly-concave in y , i.e., there exists constant µ > 0 such that ∇ 2 xx f ( x , y )

Original: ⪰ µ I and∇ 

Revised: ⪰ µ I and∇

Original: yy f
Revised: yy f

Original: ( x , y ) ⪯ − µ I for any
Revised: ( x , y ) ⪯ − µ I for any

Original: ( x , y ) ∈ R n x × R n y .
Revised: ( x , y ) ∈ R n x × R n y .

Original: The L -lipschitz continuous of gradient means the spectral norm of Hessian matrix ˆ H ( z ) can be upperbounded
Revised: The L -lipschitz continuous of gradient means the spectral norm of Hessian matrix ˆ H ( z ) can be upperbounded

Original: , that is ∥ ˆ H ( z ) ∥ ≤ L .
Revised: , that is ∥ ˆ H ( z ) ∥ ≤ L .

Original: Additionally, the condition number of the objective function is definedas κ def = L/µ .
Revised: Additionally, the condition number of the objective function is definedas κ def = L/µ .

Original:  Quasi-Newton Methods for Saddle Point Problems
Revised:  Quasi-Newton Methods for Saddle Point Problems

Original: In this section, we focus on designing quasi-Newton methods for saddle point problem and showingtheir superlinear local convergence rate.

Original: The update rule of standard Newton’s method can be writtenas z + = z − (cid:0)
Revised: The update rule of standard Newton’s method can be written as z + = z − (cid:0)

Original: ˆ H ( z ) (cid:1) − 1 g ( z ) for solving problem (1).
Revised: ˆ H ( z ) (cid:1) − 1 g ( z ) forsolving problem (1).

Original: It has quadratic local convergence, but takes O (cid:0) n 3 (cid:1) time complexity per iteration.
Revised: It has quadratic local convergence, but takes O (cid:0) n 3 (cid:1) time complexity periteration.

Original: For convex minimization, quasi-Newton methods including BFGS/SR1 [3–5, 8, 36] and their variants [19, 31, 32, 40] focus on approximating the Hessian whichcan reduce the computational cost to O (cid:0) n 2 (cid:1) for each round.
Revised: For convex minimization, quasi-Newton methods including BFGS/SR1 [3–5, 8, 36] andtheir variants [19, 31, 32, 40] focus on approximating the Hessian which can reduce the computationalcost to O (cid:0) n 2 (cid:1) for each round.

Original: However, all these algorithms andrelated convergence analysis are based on the assumption that the Hessian matrix is positive definite,which is not suitable for our saddle point problems since ˆ H ( z ) is indefinite.
Revised: However, all these algorithms and related convergence analysis arebased on the assumption that the Hessian matrix is positive definite, which is not suitable for oursaddle point problems since ˆ H ( z ) is indefinite.

Original: We introduce the auxiliary matrix H ( z ) be the square of Hessian H ( z ) def = (cid:0) ˆ H ( z ) (cid:1) 2 .
Revised: We introduce the auxiliary matrix H ( z ) be the square of Hessian [Equation]

Original: The followinglemma shows that H ( z ) is positive definite.
Revised: The following lemma shows that H ( z ) is positive definite.

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: Under Assumption 2.1 and 2.2, we have µ 2 I ⪯ H ( z ) ⪯
Revised: Under Assumption 2.1 and 2.2, we have µ 2 I ⪯ H ( z ) ⪯

Original: L 2 I for all z ∈ R n .
Revised: L 2 I for all z ∈ R n .

Original: Hence, we can reformulate the update of Newton’s method by
Revised: Hence, we can reformulate the update of Newton’s method by

Original: [Equation]
Revised: [Equation]

Original: Then it is natural to characterize the second-order information by estimating the auxiliary matrix
Revised: Then it is natural to characterize the second-order information by estimating the auxiliary matrix

Original: H ( z ) , rather than the indefinite Hessian ˆ H ( z ) .
Revised: H ( z ) , rather than the indefinite Hessian ˆ H ( z ) .

Original: If one can obtain a symmetric positive definite matrix
Revised: If one can obtain a symmetric positive definite matrix

Original: G ∈ R n × n as an estimator for H ( z ) , the update rule of (2) can be approximated by
Revised: G ∈ R n × n as an estimator for H ( z ) , the update rule of (2) can be approximated by

Original: [Equation]
Revised: [Equation]

Original: The remainder of this section introduce strategies to construct G , resulting the quasi-Newton methodsfor saddle point problem with local superlinear convergence.
Revised: The remainder of this section introduce strategies to construct G , resulting the quasi-Newton methodsfor saddle point problem with local superlinear convergence.

Original: We should point out the implementationof iteration (3) is unnecessary to construct Hessian matrix
Revised: We should point out the implementationof iteration (3) is unnecessary to construct Hessian matrix

Original: ˆ H ( z ) explicitly, since we are only interestedin the Hessian-vector product
Revised: ˆ H ( z ) explicitly, since we are only interestedin the Hessian-vector product

Original: ˆ H ( z ) g
Revised: ˆ H ( z ) g

Original: ( z ) which can be computed efficiently [26, 34].
Revised: ( z ) which can be computed efficiently [26, 34].

Original: 3.1 The Broyden Family Updates
Revised: 3.1 The Broyden Family Updates

Original: We first review some basic results for quasi-Newton methods in convex optimization. We introducethe Broyden family [24, Section 6.3] of quasi-Newton updates for approximating an positive definitematrix H ∈ R n × n by using the information of current estimator G ∈ R n × n .
Revised: We first introduce the Broyden family [24, Section 6.3] of quasi-Newton updates for approximatingan positive definite matrix H ∈ R n × n by using the information of current estimator G ∈ R n × n .

Original: Definition 3.2.
Revised: Definition 3.2.

Original: Suppose two positive definite matrices H , G ∈ R n × n satisfy H ⪯ G .
Revised: Suppose two positive definite matrices H , G ∈ R n × n satisfy H ⪯ G .

Original: For anyu ∈ R n , if Gu
Revised: For anyu ∈ R n , if Gu

Original: =
Revised: =

Original: Hu , we define Broyd τ ( G , H , u ) def = H .
Revised: Hu , we define Broyd τ ( G , H , u ) def = H .

Original: Otherwise, we define
Revised: Otherwise, we define

Original: [Equation]
Revised: [Equation]

Original: The different choices of parameter τ for above formula contain several popular quasi-Newton updates:
Revised: The different choices of parameter τ for above formula contain several popular quasi-Newton updates:

Original: • For τ = u ⊤ Hu / (cid:0) u ⊤ Gu (cid:1) ∈ [0 , 1] , it corresponds to the BFGS update
Revised: • For τ = u ⊤ Hu / (cid:0) u ⊤ Gu (cid:1) ∈ [0 , 1] , it corresponds to the BFGS update

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Now we introduce the update rule [19, 31] by choosing u asu ∼ N ( 0 , I ) or u ∼ Unif (cid:0) S n − 1 (cid:1) .
Revised: Now we introduce the update rule [19, 31] by choosing u asu ∼ N ( 0 , I ) or u ∼ Unif (cid:0) S n − 1 (cid:1) .

Original: [Equation]
Revised: [Equation]

Original: The following lemma shows applying the update rule (4) with (7) leads to a new estimator withtighter error bound in the measure of σ H ( · ) .
Revised: The following lemma shows applying the update rule (4) with (7) leads to a new estimator withtighter error bound in the measure of σ H ( · ) .

Original: Lemma 3.3 (Modified from Lin et al.
Revised: Lemma 3.3 (Modified from Lin et al.

Original: [19, Theorem 3.1]) .
Revised: [19, Theorem 3.1]) .

Original: Suppose two positive definite matricesµ 2 I ⪯ H ⪯ L 2 I and G ∈ R n × n satisfy that H ⪯ G .
Revised: Suppose two positive definite matricesµ 2 I ⪯ H ⪯ L 2 I and G ∈ R n × n satisfy that H ⪯ G .

Original: Let G + =
Revised: Let G + =

Original: Broyd τ ( G , H , u )) , where u is chosen random method as (7) , then for any τ ∈ [0 , 1] , we have E [ σ H ( G + )] ≤ (cid:0) 1 − 1 / ( nκ 2 ) (cid:1)
Revised: Broyd τ ( G , H , u )) , where u is chosen random method as (7) , then for any τ ∈ [0 , 1] , we have E [ σ H ( G + )] ≤ (cid:0) 1 − 1 / ( nκ 2 ) (cid:1)

Original: σ H ( G )
Revised: σ H ( G )

Original: For specific Broyden family updates BFGS and SR1 shown in (5) and (6), the update can achievea better convergence result.
Revised: For specific Broyden family updates BFGS and SR1 shown in (5) and (6), the update can achievea better convergence result.

Original: Concretely, for BFGS method, we first find L such that G − 1 =
Revised: Concretely, for BFGS method, we first find L such that G − 1 =

Original: L ⊤ L ,where L is an upper triangular matrix.
Revised: L ⊤ L ,where L is an upper triangular matrix.

Original: This step can be implemented with O ( n 2 ) complexity [19,
Revised: This step can be implemented with O ( n 2 ) complexity [19,

Original: Proposition 1].
Revised: Proposition 1].

Original: We present the subroutine for factorizing G − 1 in Algorithm 1 and give its detailedimplementation in the appendix.
Revised: We present the subroutine for factorizing G − 1 in Algorithm 1 and give its detailedimplementation in the appendix.

Original: And we use the direction L ⊤ u instead of u for the BFGS update.
Revised: And we use the direction L ⊤ u instead of u for the BFGS update.

Original: Applying the BFGS update rule (5) with formula (7), we obtain a condition-number-free result asfollows.
Revised: Applying the BFGS update rule (5) with formula (7), we obtain a condition-number-free result asfollows.

Original: Lemma 3.4 (Modified from Lin et al.
Revised: Lemma 3.4 (Modified from Lin et al.

Original: [19, Theorem 4.2]) .
Revised: [19, Theorem 4.2]) .

Original: Suppose two positive definite matrices
Revised: Suppose two positive definite matrices

Original: H , G ∈ R n × n satisfy H ⪯ G .
Revised: H , G ∈ R n × n satisfy H ⪯ G .

Original: Let G + =
Revised: Let G + =

Original: BFGS (cid:0) G , H , L ⊤ u (cid:1) , where u is chosen by the random method in (7) and L is an upper triangular matrix such that G − 1 =
Revised: BFGS (cid:0) G , H , L ⊤ u (cid:1) , where u is chosen by the random method in (7) and L is an upper triangular matrix such that G − 1 =

Original: L ⊤ L .
Revised: L ⊤ L .

Original: Then we have 123 E [ σ H ( G + )] ≤ (1 − 1 /n )
Revised: Then we have 121 E [ σ H ( G + )] ≤ (1 − 1 /n )

Original: σ H ( G ) .
Revised: σ H ( G ) .

Original: Remark 3.5 .
Revised: Remark 3.5 .

Original: Note that the step of conducting G − 1 =
Revised: Note that the step of conducting G − 1 =

Original: L ⊤ L requires QR decomposition of rank-1change matrix which requires O (26 n 2 ) flops [14, Section 12.5.1].
Revised: L ⊤ L requires QR decomposition of rank-1change matrix which requires O (26 n 2 ) flops [14, Section 12.5.1].

Original: We do not recommend using this
Revised: We do not recommend using this

Original: BFGS update strategy in practice when n is large.
Revised: BFGS update strategy in practice when n is large.

Original: The convergence of the SR1 update can be characterized by the measure τ H ( G ) def = tr( G − H ) .
Revised: The convergence of the SR1 update can be characterized by the measure τ H ( G ) def = tr( G − H ) .

Original: Applying the SR1 update rule (6) with formula (7), the convergence also holds a condition-numberfree result.
Revised: Applying the SR1 update rule (6) with formula (7), the convergence also holds a condition-numberfree result.

Original: Lemma 3.6 (Modified from Lin et al.
Revised: Lemma 3.6 (Modified from Lin et al.

Original: [19, Theorem 4.1]) .
Revised: [19, Theorem 4.1]) .

Original: Suppose two positive definite matrices
Revised: Suppose two positive definite matrices

Original: H , G ∈ R n × n satisfy H ⪯ G .
Revised: H , G ∈ R n × n satisfy H ⪯ G .

Original: Let G + =
Revised: Let G + =

Original: SR1 ( G , H , u ) , where u is chosen by the random methodin (7) .
Revised: SR1 ( G , H , u ) , where u is chosen by the random methodin (7) .

Original: Then we have E [ τ H ( G + )] ≤ (1 − 1 /n )
Revised: Then we have E [ τ H ( G + )] ≤ (1 − 1 /n )

Original: τ
Revised: τ

Original: H ( G ) .
Revised: H ( G ) .

Original: 3.2 Algorithms for Quadratic Saddle Point Problems
Revised: 3.2 Algorithms for Quadratic Saddle Point Problems

Original: We now solve simple quadratic saddle point problem that f ( x , y )
Revised: We now solve simple quadratic saddle point problem that f ( x , y )

Original: =
Revised: =

Original: 12 z ⊤ Az − b ⊤ z in (1) wherez
Revised: 12 z ⊤ Az − b ⊤ z in (1) wherez

Original: =
Revised: =

Original: [ x ; y ] , b ∈ R n , A ∈ R n × n is symmetric and n = n x + n y .
Revised: [ x ; y ] , b ∈ R n , A ∈ R n × n is symmetric and n = n x + n y .

Original: We suppose A could be partitioned
Revised: We suppose A could be partitioned

Original: 1: Input: z 0 ∈ R n , G 0 =
Revised: 1: Input: z 0 ∈ R n , G 0 =

Original: L 2 I , L 0 =
Revised: L 2 I , L 0 =

Original: L − 1 I 2: for k = 0 , 1 . . .
Revised: L − 1 I 2: for k = 0 , 1 . . .

Original: 3: z k +1 =
Revised: 3: z k +1 =

Original: z k − G − 1 k ˆ Hg ( z k ) 4:
Revised: z k − G − 1 k ˆ Hg ( z k ) 4:

Original: Randomly choose ˜ u k from (7) 5: u k =
Revised: Randomly choose ˜ u k from (7) 5: u k =

Original: L ⊤ k ˜ u
Revised: L ⊤ k ˜ u

Original: k 6:
Revised: k 6:

Original: G k +1 =
Revised: G k +1 =

Original: BFGS ( G k , H , u k ) 7:
Revised: BFGS ( G k , H , u k ) 7:

Original: L k +1 =
Revised: L k +1 =

Original: Fast-Chol ( H , L k , u k ) 8: end for
Revised: Fast-Chol ( H , L k , u k ) 8: end for

Original: 1: Input: z 0 ∈ R n , G 0 =
Revised: 1: Input: z 0 ∈ R n , G 0 =

Original: L 

Revised: L

Original: I
Revised: I

Original: K ≤ n +2: for k = 0 , 1 . .
Revised: K ≤ n +2: for k = 0 , 1 . .

Original: , K 3:
Revised: , K 3:

Original: z k +1 =
Revised: z k +1 =

Original: z k − G − 1 k ˆ Hg ( z k ) 4:
Revised: z k − G − 1 k ˆ Hg ( z k ) 4:

Original: Randomly choose u k from (7) 5:
Revised: Randomly choose u k from (7) 5:

Original: G k +1 =
Revised: G k +1 =

Original: SR1 (cid:0) G k , H , u k (cid:1) 6: end for as A = (cid:20)
Revised: SR1 (cid:0) G k , H , u k (cid:1) 6: end for as A = (cid:20)

Original: A xx
Revised: A xx

Original: A xy A yx
Revised: A xy A yx

Original: A yy (cid:21) where the sub-matrices A xx ∈ R n x × n x , A xy ∈ R n x × n y , A yx ∈ R n y × n xand A yy ∈ R n y × n y satisfy A xx ⪰ µ I , A yy ⪯ − µ I and ∥ A ∥ ≤ L .
Revised: A yy (cid:21) where the sub-matrices A xx ∈ R n x × n x , A xy ∈ R n x × n y , A yx ∈ R n y × n xand A yy ∈ R n y × n y satisfy A xx ⪰ µ I , A yy ⪯ − µ I and ∥ A ∥ ≤ L .

Original: Using notations introduced in
Revised: Using notations introduced in

Original: Section 2, we have z =
Revised: Section 2, we have z =

Original: [ x ; y ] , g ( z ) =
Revised: [ x ; y ] , g ( z ) =

Original: Az − b , ˆ H def = ˆ H ( z )
Revised: Az − b , ˆ H def = ˆ H ( z )

Original: =
Revised: =

Original: A and H def = H ( z ) =
Revised: A and H def = H ( z ) =

Original: A 2 .
Revised: A 2 .

Original: We present the detailed procedure of random quasi-Newton methods for quadratic saddle pointproblem by using the Broyden family update, BFGS update and SR1 update in Algorithm 2, 3 and 4respectively.
Revised: We present the detailed procedure of random quasi-Newton methods for quadratic saddle pointproblem by using the Broyden family update, BFGS update and SR1 update in Algorithm 2, 3 and 4respectively.

Original: We define λ k as the gradient norm at z k for our convergence analysis, that is λ k def =
Revised: We define λ k as the gradient norm at z k for our convergence analysis, that is λ k def =

Original: ∥ g ( z k ) ∥ .
Revised: ∥ g ( z k ) ∥ .

Original: Thedefinition of λ k in this paper is different from the measure used in convex optimization [19, 31] 1 , butit also holds the similar property as follows.
Revised: Thedefinition of λ k in this paper is different from the measure used in convex optimization [19, 31] 1 , butit also holds the similar property as follows.

Original: Lemma 3.7.
Revised: Lemma 3.7.

Original: Assume we have η k ≥ 1 and G k ∈ R n × n such that H ⪯ G k ⪯ η k H for Algorithm 2,3 and 4, then we have λ k +1 ≤ (1 − 1 /η k ) λ k .
Revised: Assume we have η k ≥ 1 and G k ∈ R n × n such that H ⪯ G k ⪯ η k H for Algorithm 2,3 and 4, then we have λ k +1 ≤ (1 − 1 /η k ) λ k .

Original: The next theorem states the assumptions of Lemma 3.7 always holds with η k = κ 2 ≥ 1 , which meansλ k converges to 0 linearly.
Revised: The next theorem states the assumptions of Lemma 3.7 always holds with η k = κ 2 ≥ 1 , which meansλ k converges to 0 linearly.

Original: Theorem 3.8.
Revised: Theorem 3.8.

Original: For all k ≥ 0 , Algorithm 2, 3, 4 hold that H ⪯ G k ⪯ κ 2 H and λ k ≤ (cid:0) 1 − 1 /κ 2 (cid:1) k λ 0 .
Revised: For all k ≥ 0 , Algorithm 2, 3, 4 hold that H ⪯ G k ⪯ κ 2 H and λ k ≤ (cid:0) 1 − 1 /κ 2 (cid:1) k λ 0 .

Original: Lemma 3.7 also implies superlinear convergence can be obtained if there exists η k which convergesto 1.
Revised: Lemma 3.7 also implies superlinear convergence can be obtained if there exists η k which convergesto 1.

Original: Applying Lemma 3.3, 3.4 and 3.6, we can show it holds for proposed algorithms.
Revised: Applying Lemma 3.3, 3.4 and 3.6, we can show it holds for proposed algorithms.

Original: Theorem 3.9.
Revised: Theorem 3.9.

Original: Solving quadratic saddle point problem by the proposed quasi-Newton Algorithms,for all k ≥ 0 , we have:
Revised: Solving quadratic saddle point problem by the proposed quasi-Newton Algorithms,for all k ≥ 0 , we have:

Original: For the Broyden family method (Algorithm 2), we have E [ λ k +1 /λ k ] ≤ (1 − 1 / ( nκ 2 ))
Revised: For the Broyden family method (Algorithm 2), we have E [ λ k +1 /λ k ] ≤ (1 − 1 / ( nκ 2 ))

Original: k nκ 2 .2.
Revised: k nκ 2 .2.

Original: For the BFGS method (Algorithm 3), we have E [ λ k +1 /λ k ] ≤ (1 − 1 /n )
Revised: For the BFGS method (Algorithm 3), we have E [ λ k +1 /λ k ] ≤ (1 − 1 /n )

Original: k nκ 2 .
Revised: k nκ 2 .

Original: For the SR1 method (Algorithm 4), we have E [ λ k +1 /λ k ] ≤ (1 − k/n ) nκ 4 .
Revised: For the SR1 method (Algorithm 4), we have E [ λ k +1 /λ k ] ≤ (1 − k/n ) nκ 4 .

Original: Combining the results of Theorem 3.8 and 3.9, we achieve the two-stages convergence behavior, thatis, the algorithm has the global linear convergence and local superlinear convergence.
Revised: Combining the results of Theorem 3.8 and 3.9, we achieve the two-stages convergence behavior, thatis, the algorithm has the global linear convergence and local superlinear convergence.

Original: We leave theformal description in appendix.
Revised: We leave theformal description in appendix.

Original: 3.3 Algorithms for General Saddle Point Problems
Revised: 3.3 Algorithms for General Saddle Point Problems

Original: In this section, we consider the general saddle point problem where f ( x , y ) in (1) satisfies Assumption 2.1 and 2.2.
Revised: In this section, we consider the general saddle point problem where f ( x , y ) in (1) satisfies Assumption 2.1 and 2.2.

Original: We propose quasi-Newton methods for solving the problem with local superlinearconvergence and O (cid:0) n 2 (cid:1) time complexity for each iteration.
Revised: We propose quasi-Newton methods for solving the problem with local superlinearconvergence and O (cid:0) n 2 (cid:1) time complexity for each iteration.

Original: 3.3.1 Algorithms
Revised: 3.3.1 Algorithms

Original: The key idea of designing quasi-Newton methods for saddle point problems is approximating theauxiliary matrix H ( z ) def = (cid:0) ˆ H ( z ) (cid:1) 2 to characterize the second-order information.
Revised: The key idea of designing quasi-Newton methods for saddle point problems is approximating theauxiliary matrix H ( z ) def = (cid:0) ˆ H ( z ) (cid:1) 2 to characterize the second-order information.

Original: Since the Hessian
Revised: Since the Hessian

Original: 1: Input: z 0 ∈ R n , G 0 ⪰ H , τ k ∈ [0 , 1] and M ≥ 0 .
Revised: 1: Input: z 0 ∈ R n , G 0 ⪰ H , τ k ∈ [0 , 1] and M ≥ 0 .

Original: 2: for k = 0 , 1 . . .
Revised: 2: for k = 0 , 1 . . .

Original: 3: z k +1 =
Revised: 3: z k +1 =

Original: z k − G − 1 k
Revised: z k − G − 1 k

Original: ˆ H k g k 4:
Revised: ˆ H k g k 4:

Original: Compute r k = ∥
Revised: Compute r k = ∥

Original: z k +1 − z k ∥ and ˜ G k
Revised: z k +1 − z k ∥ and ˜ G k

Original: =
Revised: =

Original: (1 + Mr k ) G k 5:
Revised: (1 + Mr k ) G k 5:

Original: Randomly choose u k from (7) and update G k +

Revised: Randomly choose u k from (7) and update G k +

Original: =
Revised: =

Original: Broyd τ k
Revised: Broyd τ k

Original: ( ˜ G k , H k +1 , u k ) .
Revised: ( ˜ G k , H k +1 , u k ) .

Original: 6: end for
Revised: 6: end for

Original: 1: Input: z 0 ∈ R n , G 0 ⪰ H , M ≥ 0 .
Revised: 1: Input: z 0 ∈ R n , G 0 ⪰ H , M ≥ 0 .

Original: 2: L 0 =
Revised: 2: L 0 =

Original: G − 1 / 23: for k = 0 , 1 . . .
Revised: G − 1 / 23: for k = 0 , 1 . . .

Original: z k +1 =
Revised: z k +1 =

Original: z k − G − 1 k
Revised: z k − G − 1 k

Original: ˆ H k g k 5: r k = ∥
Revised: ˆ H k g k 5: r k = ∥

Original: z k +1 − z k ∥ 6: ˜ G k
Revised: z k +1 − z k ∥ 6: ˜ G k

Original: =
Revised: =

Original: (1 + Mr k ) G k 7:
Revised: (1 + Mr k ) G k 7:

Original: ˜ L k =
Revised: ˜ L k =

Original: L k / √ 

Revised: L k / √

Original: + Mr k 8:
Revised: + Mr k 8:

Original: Randomly choose u k from (7) 9:
Revised: Randomly choose u k from (7) 9:

Original: G k +1 =
Revised: G k +1 =

Original: BFGS( ˜ G k , H k +1 , ˜
Revised: BFGS( ˜ G k , H k +1 , ˜

Original: L k u k ) 10:
Revised: L k u k ) 10:

Original: L k +1 =
Revised: L k +1 =

Original: Fast-Chol ( H k +1 ,
Revised: Fast-Chol ( H k +1 ,

Original: ˜ L k , ˜
Revised: ˜ L k , ˜

Original: L k u k ) 11: end for
Revised: L k u k ) 11: end for

Original: 1: Input: z 0 ∈ R n , G 0 ⪰ H and M ≥2: for k = 0 , 1 . . .
Revised: 1: Input: z 0 ∈ R n , G 0 ⪰ H and M ≥2: for k = 0 , 1 . . .

Original: 3: z k +1 =
Revised: 3: z k +1 =

Original: z k − G − 1 k
Revised: z k − G − 1 k

Original: ˆ H k g k 4: r k = ∥
Revised: ˆ H k g k 4: r k = ∥

Original: z k +1 − z k ∥ 5: ˜
Revised: z k +1 − z k ∥ 5: ˜

Original: G k = (1 + Mr k ) G k 6:
Revised: G k = (1 + Mr k ) G k 6:

Original: Randomly choose u k from (7) 7:
Revised: Randomly choose u k from (7) 7:

Original: G k +1 =
Revised: G k +1 =

Original: SR1( ˜ G k , H k +1 , u k ) 8: end for of f is Lipschitz continuous and bounded by Assumption 2.1 and 2.2, which means the auxiliarymatrix operator H ( z ) is also Lipschitz continuous.
Revised: SR1( ˜ G k , H k +1 , u k ) 8: end for of f is Lipschitz continuous and bounded by Assumption 2.1 and 2.2, which means the auxiliarymatrix operator H ( z ) is also Lipschitz continuous.

Original: Lemma 3.10.
Revised: Lemma 3.10.

Original: Under Assumption 2.1 and 2.2, we have H ( z ) is 2 LL 2 -Lipschitz continuous.
Revised: Under Assumption 2.1 and 2.2, we have H ( z ) is 2 LL 2 -Lipschitz continuous.

Original: Combining Lemma 3.1 and 3.10, we achieve the following properties of H ( z ) , which analogize thestrongly self-concordance in convex optimization [31].
Revised: Combining Lemma 3.1 and 3.10, we achieve the following properties of H ( z ) , which analogize thestrongly self-concordance in convex optimization [31].

Original: Lemma 3.11.
Revised: Lemma 3.11.

Original: Under Assumption 2.1 and 2.2, for all z , z ′ , w ∈ R n , the auxiliary matrix operator
Revised: Under Assumption 2.1 and 2.2, for all z , z ′ , w ∈ R n , the auxiliary matrix operator

Original: H ( · ) satisfies H ( z ) − H ( z ′ )
Revised: H ( · ) satisfies H ( z ) − H ( z ′ )

Original: ⪯ M ∥
Revised: ⪯ M ∥

Original: z − z ′ ∥ H ( w ) where M = 2 κ 2 L 2 /L .
Revised: z − z ′ ∥ H ( w ) where M = 2 κ 2 L 2 /L .

Original: Corollary 3.12.
Revised: Corollary 3.12.

Original: Let z , z ′ ∈ R n and r = ∥
Revised: Let z , z ′ ∈ R n and r = ∥

Original: z ′ − z ∥ .
Revised: z ′ − z ∥ .

Original: Suppose the objective function f satisfies
Revised: Suppose the objective function f satisfies

Original: Assumption 2.1 and 2.2, then for all z , z ′ ∈ R n and M = 2 κ 2 L 2 /L , the auxiliary matrix operator
Revised: Assumption 2.1 and 2.2, then for all z , z ′ ∈ R n and M = 2 κ 2 L 2 /L , the auxiliary matrix operator

Original: H ( · ) holds that H ( z ) /
Revised: H ( · ) holds that H ( z ) /

Original: (1 + Mr ) ⪯ H ( z ′ ) ⪯ (1 + Mr ) H ( z ) .
Revised: (1 + Mr ) ⪯ H ( z ′ ) ⪯ (1 + Mr ) H ( z ) .

Original: Different from the quadratic case, the auxiliary matrix H ( z ) is not fixed for general saddle pointproblem.
Revised: Different from the quadratic case, the auxiliary matrix H ( z ) is not fixed for general saddle pointproblem.

Original: Based on the smoothness of H ( z ) , we apply Corollary 3.12 to generalize Lemma 3.13 asfollows.
Revised: Based on the smoothness of H ( z ) , we apply Corollary 3.12 to generalize Lemma 3.13 asfollows.

Original: Lemma 3.13.
Revised: Lemma 3.13.

Original: Let z ∈ R n and G ∈ R n × n be a positive definite matrix
Revised: Let z ∈ R n and G ∈ R n × n be a positive definite matrix

Original: such that H ( z ) ⪯ G ⪯ η H ( z )for some η ≥ 1 .
Revised: such that H ( z ) ⪯ G ⪯ η H ( z )for some η ≥ 1 .

Original: In addition, define z + ∈ R n and r = ∥
Revised: In addition, define z + ∈ R n and r = ∥

Original: z + − z ∥ , then for all u ∈ R n , τ ∈ [0 , 1]and M = 2 κ 2 L 2 /L , we have H ( z + ) ⪯ Broyd τ ( ˜ G , H ( z + ) , u ) ⪯ (1 + Mr ) 2 η H ( z + ) where˜ G def =
Revised: z + − z ∥ , then for all u ∈ R n , τ ∈ [0 , 1]and M = 2 κ 2 L 2 /L , we have H ( z + ) ⪯ Broyd τ ( ˜ G , H ( z + ) , u ) ⪯ (1 + Mr ) 2 η H ( z + ) where˜ G def =

Original: (1 + Mr ) G ⪰ H
Revised: (1 + Mr ) G ⪰ H

Original: ( z + ) .
Revised: ( z + ) .

Original: Lemma 3.13 implies it is reasonable to have the algorithms using G k
Revised: Lemma 3.13 implies it is reasonable to have the algorithms using G k

Original: +

Revised: +

Original: =
Revised: =

Original: Broyd τ k
Revised: Broyd τ k

Original: ( ˜ G k , H k +1 , u k )with ˜ G k
Revised: ( ˜ G k , H k +1 , u k )with ˜ G k

Original: = (1 + Mr k ) G k and r k = ∥
Revised: = (1 + Mr k ) G k and r k = ∥

Original: z k +1 − z k ∥ .
Revised: z k +1 − z k ∥ .

Original: Similarly, we can also achieve G k +1 by such˜ G k for specific BFGS and SR1 update.
Revised: Similarly, we can also achieve G k +1 by such˜ G k for specific BFGS and SR1 update.

Original: Combining this with iteration (3), we propose severalquasi-Newton methods for general strongly-convex-strongly-concave saddle point problems.
Revised: Combining this with iteration (3), we propose severalquasi-Newton methods for general strongly-convex-strongly-concave saddle point problems.

Original: Thedetails are shown in Algorithm 5, 6 and 7 for Broyden family, BFGS and SR1 updates respectively.
Revised: Thedetails are shown in Algorithm 5, 6 and 7 for Broyden family, BFGS and SR1 updates respectively.

Original: 3.3.2 Convergence Analysis
Revised: 3.3.2 Convergence Analysis

Original: Let us consider the convergence guarantee for algorithms proposed in Section 3.3.1.
Revised: Let us consider the convergence guarantee for algorithms proposed in Section 3.3.1.

Original: We introducethe following notations to simplify the presentation.
Revised: We introducethe following notations to simplify the presentation.

Original: We let { z k } be the sequence generated from
Revised: We let { z k } be the sequence generated from

Original: Algorithm 5, 6 or 7 and denote g k def = g
Revised: Algorithm 5, 6 or 7 and denote g k def = g

Original: ( z k ) , r k def = ∥
Revised: ( z k ) , r k def = ∥

Original: z k +1 − z k ∥ , ˆ H k def = ˆ H ( z k ) and H k def = (cid:0) ˆ H ( z k ) (cid:1) 2 .
Revised: z k +1 − z k ∥ , ˆ H k def = ˆ H ( z k ) and H k def = (cid:0) ˆ H ( z k ) (cid:1) 2 .

Original: We still use gradient norm λ k def = ∥∇ f
Revised: We still use gradient norm λ k def = ∥∇ f

Original: ( z k ) ∥ for analysis and establish the relationship between λ kand λ k +1 , which is shown in Lemma 3.14.
Revised: ( z k ) ∥ for analysis and establish the relationship between λ kand λ k +1 , which is shown in Lemma 3.14.

Original: Lemma 3.14.
Revised: Lemma 3.14.

Original: Using Algorithm 5, 6 and 7, suppose we have H k ⪯ G k ⪯ η k H k , for some η k ≥ 1and let β
Revised: Using Algorithm 5, 6 and 7, suppose we have H k ⪯ G k ⪯ η k H k , for some η k ≥ 1and let β

Original: = L 2 / (2 µ 2 ) , then we have λ k +1 ≤ (1 − 1 /η k )
Revised: = L 2 / (2 µ 2 ) , then we have λ k +1 ≤ (1 − 1 /η k )

Original: λ k + βλ 2 k and r k ≤ (1 /µ )
Revised: λ k + βλ 2 k and r k ≤ (1 /µ )

Original: λ k .
Revised: λ k .

Original: Rodomanov and Nesterov [31, Lemma 4.3] derive a result similar to Lemma 3.14 for minimizingthe strongly-convex function ˆ f ( · ) on the different measure λ ˆ f ( · ) def =
Revised: Rodomanov and Nesterov [31, Lemma 4.3] derive a result similar to Lemma 3.14 for minimizingthe strongly-convex function ˆ f ( · ) on the different measure λ ˆ f ( · ) def =

Original: ⟨∇ ˆ f ( · ) , ∇ 2 ˆ f ( · ) − 1 ∇ ˆ f ( · ) ⟩ . 

Revised: ⟨∇ ˆ f ( · ) , ∇ 2 ˆ f ( · ) − 1 ∇ ˆ f ( · ) ⟩ .

Original: Note that our algorithms are based on the iteration rule z k +

Revised: Note that our algorithms are based on the iteration rule z k +

Original: =
Revised: =

Original: z k − G − 1 k ˆ H k g k .
Revised: z k − G − 1 k ˆ H k g k .

Original: Compared withquasi-Newton methods for convex optimization, there exists an additional term ˆ H k between G − 1 kand g k , which leads to the fact that the convergence analysis based on λ ˆ f ( z k ) is difficult.
Revised: Compared withquasi-Newton methods for convex optimization, there exists an additional term ˆ H k between G − 1 kand g k , which leads to the fact that the convergence analysis based on λ ˆ f ( z k ) is difficult.

Original: Fortunately,we find using gradient norm λ k directly makes the analysis achievable.
Revised: Fortunately,we find using gradient norm λ k directly makes the analysis achievable.

Original: For further analysis, we also denote σ k def = σ H k
Revised: For further analysis, we also denote σ k def = σ H k

Original: ( G k ) =
Revised: ( G k ) =

Original: ⟨ H − 1 k , G k ⟩ − n.
Revised: ⟨ H − 1 k , G k ⟩ − n.

Original: Then we establish the linear convergence for the first period of iterations, which can be viewed as theextension of Theorem 3.8.
Revised: Then we establish the linear convergence for the first period of iterations, which can be viewed as theextension of Theorem 3.8.

Original: Note that the following result holds for Algorithm 5, 6 and 7; and it doesnot depend on the choice of u k .
Revised: Note that the following result holds for Algorithm 5, 6 and 7; and it doesnot depend on the choice of u k .

Original: Theorem 3.15.
Revised: Theorem 3.15.

Original: Using Algorithm 5, 6 and 7 by G 

Revised: Using Algorithm 5, 6 and 7 by G

Original: =
Revised: =

Original: L 2 I and M = 2 κ 2 L 2 /L , suppose the initialpoint z 0 is sufficiently close to z ∗
Revised: L 2 I and M = 2 κ 2 L 2 /L , suppose the initialpoint z 0 is sufficiently close to z ∗

Original: such that Mλ 0 /µ ≤ ln b/ (4 bκ 2 ) , where 1 < b < 5 , then wehave λ k ≤ (cid:0) 1 − 1 / (2 bκ 2 ) (cid:1)
Revised: such that Mλ 0 /µ ≤ ln b/ (4 bκ 2 ) , where 1 < b < 5 , then wehave λ k ≤ (cid:0) 1 − 1 / (2 bκ 2 ) (cid:1)

Original: k λ 0 and H k ⪯ G k ⪯ exp (cid:16) 2 (cid:80)
Revised: k λ 0 and H k ⪯ G k ⪯ exp (cid:16) 2 (cid:80)

Original: k − 1 i =

Revised: k − 1 i =

Original: ρ i (cid:17) κ 2 H k ⪯ bκ 2 H k for all k ≥ 0 ,where ρ k def =
Revised: ρ i (cid:17) κ 2 H k ⪯ bκ 2 H k for all k ≥ 0 ,where ρ k def =

Original: Mλ k /µ .
Revised: Mλ k /µ .

Original: We now analyze how σ k changes after one iteration to show the local superlinear convergence forthe Broyden family method (Algorithm 5) and the BFGS method (Algorithm 6).
Revised: We now analyze how σ k changes after one iteration to show the local superlinear convergence forthe Broyden family method (Algorithm 5) and the BFGS method (Algorithm 6).

Original: Recall that σ k isdefined to measure how well does matrix G k approximate H k .
Revised: Recall that σ k isdefined to measure how well does matrix G k approximate H k .

Original: Lemma 3.16.
Revised: Lemma 3.16.

Original: Solving the general strongly-convex-strongly-concave saddle point problem (1) under
Revised: Solving the general strongly-convex-strongly-concave saddle point problem (1) under

Original: Assumption 2.1 and 2.2 by our quasi-Newton algorithms and supposing the sequences G 0 , · · · , G kgenerated by the Algorithm 5 and 6 are given, then we have the following results for σ k for all k :
Revised: Assumption 2.1 and 2.2 by our quasi-Newton algorithms and supposing the sequences G 0 , · · · , G kgenerated by the Algorithm 5 and 6 are given, then we have the following results for σ k for all k :

Original: The random Broyden family method (Algorithm 5) holds thatE [ σ k +1 ] ≤ (cid:0) 1 − 1 / ( nκ 2 ) (cid:1) (1 + Mr k ) 2 ( σ k + 2 nMr k / (1 + Mr k )) .
Revised: The random Broyden family method (Algorithm 5) holds thatE [ σ k +1 ] ≤ (cid:0) 1 − 1 / ( nκ 2 ) (cid:1) (1 + Mr k ) 2 ( σ k + 2 nMr k / (1 + Mr k )) .

Original: The random BFGS method (Algorithm 6) holds thatE [ σ k +1 ] ≤ (1 − 1 /n )
Revised: The random BFGS method (Algorithm 6) holds thatE [ σ k +1 ] ≤ (1 − 1 /n )

Original: (1 + Mr k ) 2 ( σ k + 2 nMr k / (1 + Mr k )) .
Revised: (1 + Mr k ) 2 ( σ k + 2 nMr k / (1 + Mr k )) .

Original: The analysis for the SR1 method is based on constructing η k such that η k = tr
Revised: The analysis for the SR1 method is based on constructing η k such that η k = tr

Original: ( G
Revised: ( G

Original: k − H k
Revised: k − H k

Original: ) / tr( H
Revised: ) / tr( H

Original: k
Revised: k

Original: and the technical details are showed in appendix.
Revised: and the technical details are showed in appendix.

Original: Based on Lemma 3.16, one can show that ouralgorithms enjoy the local superlinear convergence for the general saddle point problems.
Revised: Based on Lemma 3.16, one can show that ouralgorithms enjoy the local superlinear convergence for the general saddle point problems.

Original: Theorem 3.17.
Revised: Theorem 3.17.

Original: Solving general saddle point problem (1) under Assumption 2.1 and 2.2 by proposedrandom quasi-Newton methods (Algorithm 5, 6 and 7) by G 

Revised: Solving general saddle point problem (1) under Assumption 2.1 and 2.2 by proposedrandom quasi-Newton methods (Algorithm 5, 6 and 7) by G

Original: =
Revised: =

Original: L 2 I and M = 2 κ 2 L 2 /L , we havethe following results for all k > 0 :
Revised: L 2 I and M = 2 κ 2 L 2 /L , we havethe following results for all k > 0 :

Original: For the random Broyden family method (Algorithm 5), if λ 0 satisfies that Mλ 0 µ ≤
Revised: For the random Broyden family method (Algorithm 5), if λ 0 satisfies that Mλ 0 µ ≤

Original: ln2 8(1+2 n ) κ 2 ,we have E [ λ k +1 /λ k ] ≤ (cid:0) 1 − 1 / ( nκ 2 ) (cid:1) k 2 nκ 2 .2.
Revised: ln2 8(1+2 n ) κ 2 ,we have E [ λ k +1 /λ k ] ≤ (cid:0) 1 − 1 / ( nκ 2 ) (cid:1) k 2 nκ 2 .2.

Original: For the random BFGS method (Algorithm 6), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+2 n ) κ 2 , we have
Revised: For the random BFGS method (Algorithm 6), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+2 n ) κ 2 , we have

Original: E [ λ k +1 /λ k ] ≤ (1 − 1 /n )
Revised: E [ λ k +1 /λ k ] ≤ (1 − 1 /n )

Original: k 2 nκ 2 .3.
Revised: k 2 nκ 2 .3.

Original: For the random SR1 method (Algorithm 7), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+2 nκ 2 ) κ 2 , we have
Revised: For the random SR1 method (Algorithm 7), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+2 nκ 2 ) κ 2 , we have

Original: E [ λ k +1 /λ k ] ≤ (1 − 1 /n )
Revised: E [ λ k +1 /λ k ] ≤ (1 − 1 /n )

Original: k 2 nκ 4 .
Revised: k 2 nκ 4 .

Original: Finally, combining the results of Theorem 3.15 and 3.17, we can prove the algorithms achieve thetwo-stages convergence behaviors as follows.
Revised: Finally, combining the results of Theorem 3.15 and 3.17, we can prove the algorithms achieve thetwo-stages convergence behaviors as follows.

Original: Corollary 3.18.
Revised: Corollary 3.18.

Original: Solving the general saddle point problem (1) under Assumption 2.1 and 2.2 byour random quasi-Newton methods (Algorithm 5, 6 and 7) with G 

Revised: Solving the general saddle point problem (1) under Assumption 2.1 and 2.2 byour random quasi-Newton methods (Algorithm 5, 6 and 7) with G

Original: =
Revised: =

Original: L 2 I and M = 2 κ 2 L 2 /L , ifthe initial point is sufficiently close to the saddle point such that Mλ 0 /µ ≤ ln 2 / (8 κ 2 ) , then withprobability 1 − δ for any δ ∈ (0 , 1) , we have the following results:
Revised: L 2 I and M = 2 κ 2 L 2 /L , ifthe initial point is sufficiently close to the saddle point such that Mλ 0 /µ ≤ ln 2 / (8 κ 2 ) , then withprobability 1 − δ for any δ ∈ (0 , 1) , we have the following results:

Original: The random Broyden family method (Algorithm 5) holds for k 

Revised: The Random Broyden family method (Algorithm 5) holds for k

Original: = O (cid:0) nκ 2 ln( nκ/δ ) (cid:1) andall k ≥ 0 that λ k 0 + k ≤ (cid:16) 1 − 1 nκ 2 +1 (cid:17) k
Revised: = O (cid:0) nκ 2 ln( nκ/δ ) (cid:1) andall k ≥ 0 that λ k 0 + k ≤ (cid:16) 1 − 1 nκ 2 +1 (cid:17) k

Original: ( k − 1)(cid:0) 12 (cid:1)
Revised: ( k − 1)(cid:0) 12 (cid:1)

Original: k (cid:0) 1 − 14 κ 2 (cid:1) k 0 λ 0 .2.
Revised: k (cid:0) 1 − 14 κ 2 (cid:1) k 0 λ 0 .2.

Original: The random BFGS/SR1 method (Algorithm 6/7) holds for k 

Revised: The random BFGS/SR1 method (Algorithm 6/7) holds for k

Original: = O (cid:0) max (cid:8) n, κ 2 (cid:9) ln( nκ/δ ) (cid:1)and all k ≥ 0 that λ k 0 + k ≤ (cid:16) 1 − 1 n +1 (cid:17) k
Revised: = O (cid:0) max (cid:8) n, κ 2 (cid:9) ln( nκ/δ ) (cid:1)and all k ≥ 0 that λ k 0 + k ≤ (cid:16) 1 − 1 n +1 (cid:17) k

Original: ( k − 1)(cid:0) 12 (cid:1)
Revised: ( k − 1)(cid:0) 12 (cid:1)

Original: k (cid:0) 1 − 14 κ 2 (cid:1) k 0 λ 0 .
Revised: k (cid:0) 1 − 14 κ 2 (cid:1) k 0 λ 0 .

Original:  Numerical Experiments
Revised:  Numerical Experiments

Original: In this section, we conduct the experiments on machine learning applications of AUC Maximizationand adversarial debiasing to verify our theory.
Revised: In this section, we conduct the experiments on machine learning applications of AUC Maximizationand adversarial debiasing to verify our theory.

Original: We refer to Algorithm 2 and 5 with parameterτ k =
Revised: We refer to Algorithm 2 and 5 with parameterτ k =

Original: u ⊤ H
Revised: u ⊤ H

Original: k +1 / ( u ⊤ G k u ) as RaBFGSv1-Q and RaBFGSv1-G; refer to Algorithm 3, 4, 6 and 7 as
Revised: k +1 / ( u ⊤ G k u ) as RaBFGSv1-Q and RaBFGSv1-G; refer to Algorithm 3, 4, 6 and 7 as

Original: RaBFGSv2-Q, RaSR1-Q, RaBFGSv2-G and RaSR1-G respectively.
Revised: RaBFGSv2-Q, RaSR1-Q, RaBFGSv2-G and RaSR1-G respectively.

Original: We compare these proposedalgorithms with classical first-order method extragradient (EG) [18, 37].
Revised: We compare these proposedalgorithms with classical first-order method extragradient (EG) [18, 37].

Original: 4.1 AUC Maximization
Revised: 4.1 AUC Maximization

Original: AUC maximization [15, 41] aims to find the classifier w ∈ R d on the training set { a i , b i } mi =1 wherea
Revised: AUC maximization [15, 41] aims to find the classifier w ∈ R d on the training set { a i , b i } mi =1 wherea

Original: i ∈
Revised: i ∈

Original: R d and b i ∈ { +1 , − 1 } .
Revised: R d and b i ∈ { +1 , − 1 } .

Original: We denote m + be the number of positive instances and p = m
Revised: We denote m + be the number of positive instances and p = m

Original: + /m .
Revised: + /m .

Original: The minimax formulation for AUC maximization can be written as
Revised: The minimax formulation for AUC maximization can be written as

Original: [Equation]
Revised: [Equation]

Original: where x = [ w ; u ; v ] ∈ R d +2 , λ > 0 is the regularization parameter and f i is defined as
Revised: where x = [ w ; u ; v ] ∈ R d +2 , λ > 0 is the regularization parameter and f i is defined as

Original: [Equation]
Revised: [Equation]

Original: The objective function of AUC maximization is quadratic, hence we conduct the algorithms in Section3.2 (Algorithm 2, 3 and 4) for this model.
Revised: The objective function of AUC maximization is quadratic, hence we conduct the algorithms in Section3.2 (Algorithm 2, 3 and 4) for this model.

Original: We set λ = 100 /m and evaluate all algorithms on threereal-world datasets “a9a”, “w8a” and “sido0”.
Revised: We set λ = 100 /m and evaluate all algorithms on threereal-world datasets “a9a”, “w8a” and “sido0”.

Original: The dimension of the problem is n = d + 3 .
Revised: The dimension of the problem is n = d + 3 .

Original: Theresults of iteration numbers against ∥ g ( z ) ∥ 2 and CPU time against ∥ g ( z ) ∥ 2 are presented in Figure 1.
Revised: Theresults of iteration numbers against ∥ g ( z ) ∥ 2 and CPU time against ∥ g ( z ) ∥ 2 are presented in Figure 1.

Original: These results show that our algorithms perform better than the EG method.
Revised: These results show that our algorithms perform better than the EG method.

Original: 4.2 Adversarial Debiasing
Revised: 4.2 Adversarial Debiasing

Original: Adversarial learning [21, 42] can be used on fairness-aware machine learning issues.
Revised: Adversarial learning [21, 42] can be used on fairness-aware machine learning issues.

Original: Give the trainingset { a i , b i , c i } mi =1 where a i ∈ R d contains all input variables, b i ∈ R is the output and c i ∈ R is theinput variable which we want to protect and make it unbiased.
Revised: Give the trainingset { a i , b i , c i } mi =1 where a i ∈ R d contains all input variables, b i ∈ R is the output and c i ∈ R is theinput variable which we want to protect and make it unbiased.

Original: Our experiments are based on thefairness-aware binary classification dataset “adult”, “bank market” and “law school”[29], leading tob i , c i ∈ { +1 , − 1 } .
Revised: Our experiments are based on thefairness-aware binary classification dataset “adult”, “bank market” and “law school”[29], leading tob i , c i ∈ { +1 , − 1 } .

Original: The model try to minimax the function
Revised: The model try to minimax the function

Original: [Equation]
Revised: [Equation]

Original: where l 1 , l 2 are the logit functions: logit( a , b, c )
Revised: where l 1 , l 2 are the logit functions: logit( a , b, c )

Original: = log(1 + exp( − b a ⊤ c )) .
Revised: = log(1 + exp( − b a ⊤ c )) .

Original: We set the parametersβ, λ and γ as 0 . 5 , 10 − 4 and 10 − 4 respectively.
Revised: We set the parametersβ, λ and γ as 0 . 5 , 10 − 4 and 10 − 4 respectively.

Original: The dimension of the problem is n = d + 1 .
Revised: The dimension of the problem is n = d + 1 .

Original: Since theobjective function is non-quadratic, we conduct the proposed algorithms in Section 3.3 (Algorithm 5,6 and 7) here.
Revised: Since theobjective function is non-quadratic, we conduct the proposed algorithms in Section 3.3 (Algorithm 5,6 and 7) here.

Original: We use extragradient as warm up to achieve the local condition for proposed algorithms.
Revised: We use extragradient as warm up to achieve the local condition for proposed algorithms.

Original: The results of iteration numbers against ∥ g ( z ) ∥ 2 and CPU time against ∥ g ( z ) ∥ 2 are presented in
Revised: The results of iteration numbers against ∥ g ( z ) ∥ 2 and CPU time against ∥ g ( z ) ∥ 2 are presented in

Original: Figure 2, which indicate that our algorithms significantly outperform the baseline algorithm.
Revised: Figure 2, which indicate that our algorithms significantly outperform the baseline algorithm.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we propose quasi-Newton methods for solving strongly-convex-strongly-concave saddlepoint problems.
Revised: In this work, we propose quasi-Newton methods for solving strongly-convex-strongly-concave saddlepoint problems.

Original: We characterize the second-order information by approximating the square of
Revised: We characterize the second-order information by approximating the square of

Original: Hessian matrix, which avoids the issue of dealing with the indefinite Hessian directly.
Revised: Hessian matrix, which avoids the issue of dealing with the indefinite Hessian directly.

Original: We presentthe explicit local superlinear convergence rates for Broyden’s family update and faster convergencerates for two specific methods: SR1 and BFGS updates.
Revised: We presentthe explicit local superlinear convergence rates for Broyden’s family update and faster convergencerates for two specific methods: SR1 and BFGS updates.

Original: However, our algorithms still require tocompute the exact gradient and store the approximate Hessian.
Revised: However, our algorithms still require tocompute the exact gradient and store the approximate Hessian.

Original: In future work, it is interesting todesign the stochastic algorithms to further reduce the computational cost of the iteration.
Revised: In future work, it is interesting todesign the stochastic algorithms to further reduce the computational cost of the iteration.

Original: It is alsopossible to design the limited-memory quasi-Newton methods to make our ideas work more scalablesaddle point problems.
Revised: It is alsopossible to design the limited-memory quasi-Newton methods to make our ideas work more scalablesaddle point problems.

Original: Checklist
Revised: Checklist

Original: For all authors..
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section 2.(b)
Revised: [Yes] See Section 2.(b)

Original: Did you include complete proofs 
Revised: Did you include complete proofs of

Original: (a) Did you include the code, data, and instructions needed to reproduce the main exper-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main exper-

Original: imental results (either in the supplemental material or as a URL)?
Revised: imental results (either in the supplemental material or as a URL)?

Original: [Yes] We present
Revised: [Yes] We present

Original: them in the supplemental material.
Revised: them in the supplemental material.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] We present them in Section 4 and in Appendix E.
Revised: [Yes] We present them in Section 4 and in Appendix E.(c)

Original: (c) Did you report error bars (e
Revised: Did you report error bars (e

Original: g., with respect to the random seed after running experi-
Revised: g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [No]
Revised: [No]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix E.
Revised: [Yes] See Appendix E.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g , code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 4 andAppendix E.
Revised: [Yes] See Section 4 and Appendix E.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] See Section 4 and Appendix E, the data is public available.
Revised: [Yes] See Section 4 and Appendix E, thedata is public available.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]See Appendix E.
Revised: [Yes]See Appendix E.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A