Original: Quasi-Newton Methods for Saddle Point Problems
Revised: Quasi-Newton Methods for Saddle Point Problems

Original: Abstract
Revised: Abstract

Original: This paper studies quasi-Newton methods for strongly-convex-strongly-concave saddle point problems.
Revised: This paper studies quasi-Newton methods for strongly-convex-strongly-concave saddle point problems.

Original: We propose random Broyden family updates, which have explicit local superlinear convergence rate of O ((1 − 1 / ( d κ 2 ))
Revised: We propose random Broyden family updates, which have explicit local superlinear convergence rate of O ((1 − 1 / ( d κ 2 ))

Original: k ( k − 1) / 2 ) , where n is the dimension of the problem, κ is the condition number and k is the number of iterations.
Revised: k ( k − 1) / 2 ) , where n is the dimension of the problem, κ is the condition number and k is the number of iterations.

Original: The design and analysis of proposed algorithm are based on estimating the square of indefinite Hessian matrix, which is different from classical quasiNewton methods in convex optimization.
Revised: The design and analysis of proposed algorithm are based on estimating the square of indefinite Hessian matrix, which is different from classical quasiNewton methods in convex optimization.

Original: We also present two specific Broyden family algorithms with BFGS-type and SR1-type updates, which enjoy the faster local convergence rate of O ((1 − 1 /d ) k
Revised: We also present two specific Broyden family algorithms with BFGS-type and SR1-type updates, which enjoy the faster local convergence rate of O ((1 − 1 /d ) k

Original: ( k − 1) / 2 ) .
Revised: ( k − 1) / 2 ) .

Original: Our numerical experiments show proposed algorithms outperform classical first-order methods
Revised: Our numerical experiments show proposed algorithms outperform classical first-order methods

Original:  Introduction
Revised:  Introduction

Original: In this paper, we focus on the following smooth saddle point problem
Revised: In this paper, we focus on the following smooth saddle point problem

Original: [Equation]
Revised: [Equation]

Original: where f is strongly-convex in x and strongly-concave in y .
Revised: where f is strongly-convex in x and strongly-concave in y .

Original: We target to find the saddle point ( x ∗ , y ∗ ) which holds that
Revised: We target to find the saddle point ( x ∗ , y ∗ ) which holds that

Original: [Equation]
Revised: [Equation]

Original: for all x ∈ R
Revised: for all x ∈ R

Original: d x and y ∈ R d y .
Revised: d x and y ∈ R d y .

Original: This formulation is widely used in game theory [1, 39], AUC maximization [15, 43], fairness-aware machine learning [44], robust optimization [2, 11, 13, 36], empirical risk minimization [46] and reinforcement learning [10].
Revised: This formulation is widely used in game theory [1, 39], AUC maximization [15, 43], fairness-aware machine learning [44], robust optimization [2, 11, 13, 36], empirical risk minimization [46] and reinforcement learning [10].

Original: There are a great number of first-order optimization algorithms for solving problem (1), including extragradient method [18, 38], optimistic gradient descent ascent [7, 28], proximal point method [31] and dual extrapolation [24].
Revised: There are a great number of first-order optimization algorithms for solving problem (1), including extragradient method [18, 38], optimistic gradient descent ascent [7, 28], proximal point method [31] and dual extrapolation [24].

Original: These algorithms iterate with first-order oracle and achieve linear convergence.
Revised: These algorithms iterate with first-order oracle and achieve linear convergence.

Original: Lin et al.
Revised: Lin et al.

Original: [21], Wang and Li [40] used Catalyst acceleration to reduce the complexity for unbalanced saddle point problem, nearly matching the lower bound of first-order algorithms [26, 45] under some specific assumptions.
Revised: [21], Wang and Li [40] used Catalyst acceleration to reduce the complexity for unbalanced saddle point problem, nearly matching the lower bound of first-order algorithms [26, 45] under some specific assumptions.

Original: Compared with first-order methods, second-order methods usually enjoy superior convergence in numerical optimization.
Revised: Compared with first-order methods, second-order methods usually enjoy superior convergence in numerical optimization.

Original: Huang et al.
Revised: Huang et al.

Original: [16] extended cubic regularized Newton (CRN) method [23, 24] to solve saddle point problem (1), which has quadratic local convergence.
Revised: [16] extended cubic regularized Newton (CRN) method [23, 24] to solve saddle point problem (1), which has quadratic local convergence.

Original: However, each iteration of CRN requires accessing the exact Hessian matrix and solving the corresponding linear systems.
Revised: However, each iteration of CRN requires accessing the exact Hessian matrix and solving the corresponding linear systems.

Original: These steps arise O (cid:0) d 3 x + d 3 y (cid:1) time complexity, which is too expensive for high dimensional problems.
Revised: These steps arise O (cid:0) d 3 x + d 3 y (cid:1) time complexity, which is too expensive for high dimensional problems.

Original: Quasi-Newton methods [3–5, 8, 37] are popular ways to avoid accessing exact second-order information applied in standard Newton methods.
Revised: Quasi-Newton methods [3–5, 8, 37] are popular ways to avoid accessing exact second-order information applied in standard Newton methods.

Original: They approximate the Hessian matrix based on the Broyden family updating formulas [3], which significantly reduces the computational cost.
Revised: They approximate the Hessian matrix based on the Broyden family updating formulas [3], which significantly reduces the computational cost.

Original: These algorithms
Revised: These algorithms

Original: [Table]
Revised: [Table]

Original: are well studied for convex optimization.
Revised: are well studied for convex optimization.

Original: The famous quasi-Newton methods including DavidonFletcher-Powell (DFP) method [8, 12], Broyden-Fletcher-Goldfarb-Shanno (BFGS) method [4, 5, 37] and symmetric rank 1 (SR1) method [3, 8] enjoy local superlinear convergence [6, 9, 29] when the objective function is strongly-convex.
Revised: The famous quasi-Newton methods including DavidonFletcher-Powell (DFP) method [8, 12], Broyden-Fletcher-Goldfarb-Shanno (BFGS) method [4, 5, 37] and symmetric rank 1 (SR1) method [3, 8] enjoy local superlinear convergence [6, 9, 29] when the objective function is strongly-convex.

Original: Recently, Rodomanov and Nesterov [32, 33, 34] proposed greedy and random variants of quasi-Newton methods, which first achieves non-asymptotic superlinear convergence.
Revised: Recently, Rodomanov and Nesterov [32, 33, 34] proposed greedy and random variants of quasi-Newton methods, which first achieves non-asymptotic superlinear convergence.

Original: Later, Lin et al.
Revised: Later, Lin et al.

Original: [20] established a better convergence rate which is condition-number-free.
Revised: [20] established a better convergence rate which is condition-number-free.

Original: Jin and Mokhtari [17], Ye et al.
Revised: Jin and Mokhtari [17], Ye et al.

Original: [42] showed the non-asymptotic superlinear convergence rate also holds for classical DFP, BFGS and SR1 methods.
Revised: [42] showed the non-asymptotic superlinear convergence rate also holds for classical DFP, BFGS and SR1 methods.

Original: In this paper, we study quasi-Newton methods for saddle point problem (1).
Revised: In this paper, we study quasi-Newton methods for saddle point problem (1).

Original: Note that when the
Revised: Note that when the

Original: Hessian matrix of the objective function is indefinite, the existing Broyden family update formulas and their convergence analysis cannot be applied directly.
Revised: Hessian matrix of the objective function is indefinite, the existing Broyden family update formulas and their convergence analysis cannot be applied directly.

Original: To overcome this issue, we propose a variant framework of random quasi-Newton methods for saddle point problems, which approximates the square of the Hessian matrix during the iteration.
Revised: To overcome this issue, we propose a variant framework of random quasi-Newton methods for saddle point problems, which approximates the square of the Hessian matrix during the iteration.

Original: Our theoretical analysis characterizes the convergence rate by the gradient norm, rather than the weighted norm of gradient used in convex optimization [17, 20, 32–34, 42].
Revised: Our theoretical analysis characterizes the convergence rate by the gradient norm, rather than the weighted norm of gradient used in convex optimization [17, 20, 32–34, 42].

Original: We summarize the theoretical results for proposed algorithms in
Revised: We summarize the theoretical results for proposed algorithms in

Original: Table 1.
Revised: Table 1.

Original: The local convergence behaviors for all of the algorithms have two periods.
Revised: The local convergence behaviors for all of the algorithms have two periods.

Original: The first period has k 0 iterations with a linear convergence rate O ((1 − 1 / κ 2 ) k 0 ) .
Revised: The first period has k 0 iterations with a linear convergence rate O ((1 − 1 / κ 2 ) k 0 ) .

Original: The second one enjoys superlinear convergence, that is,
Revised: The second one enjoys superlinear convergence, that is,

Original: • For general Broyden family methods, we have O (cid:0)(cid:0) 1 − 1 / ( d κ 2 + 1) (cid:1) k
Revised: • For general Broyden family methods, we have O (cid:0)(cid:0) 1 − 1 / ( d κ 2 + 1) (cid:1) k

Original: ( k − 1) / 2 (cid:1) .
Revised: ( k − 1) / 2 (cid:1) .

Original: • For BFGS method and SR1 method, we have O (cid:0)(cid:0) 1 − 1 / ( d + 1) (cid:1) k
Revised: • For BFGS method and SR1 method, we have O (cid:0)(cid:0) 1 − 1 / ( d + 1) (cid:1) k

Original: ( k − 1) / 2 (cid:1) , which is condition-number-free.
Revised: ( k − 1) / 2 (cid:1) , which is condition-number-free.

Original: Paper Organization
Revised: Paper Organization

Original: In Section 2, we state the notation and preliminaries of this paper.
Revised: In Section 2, we state the notation and preliminaries of this paper.

Original: In Section 3, we first propose random quasi-Newton methods for quadratic saddle point problem which enjoys local superlinear convergence.
Revised: In Section 3, we first propose random quasi-Newton methods for quadratic saddle point problem which enjoys local superlinear convergence.

Original: Then we extend it to solve general strongly-convex-strongly-concave saddle point problems.
Revised: Then we extend it to solve general strongly-convex-strongly-concave saddle point problems.

Original: In Section 5, we provide numerical experiments to validate our algorithms on popular machine learning models.
Revised: In Section 5, we provide numerical experiments to validate our algorithms on popular machine learning models.

Original: All proofs, experiment details and extensions are deferred to appendix.
Revised: All proofs, experiment details and extensions are deferred to appendix.

Original:  Notation and Preliminaries
Revised:  Notation and Preliminaries

Original: We use ∥·∥ to present spectral norm and Euclidean norm of matrix and vector respectively.
Revised: We use ∥·∥ to present spectral norm and Euclidean norm of matrix and vector respectively.

Original: We denote the standard basis for R d by { e 1 , . . .
Revised: We denote the standard basis for R d by { e 1 , . . .

Original: , e d } and let I be the identity matrix.
Revised: , e d } and let I be the identity matrix.

Original: The trace of a square matrix is denoted by tr ( · ) .
Revised: The trace of a square matrix is denoted by tr ( · ) .

Original: Given two positive definite matrices G and H , we define their inner product as ⟨ G , H ⟩ def = tr ( GH ) .
Revised: Given two positive definite matrices G and H , we define their inner product as ⟨ G , H ⟩ def = tr ( GH ) .

Original: We introduce the quantity σ H ( G ) def = ⟨ H − 1 , G − H ⟩ = ⟨ H − 1 , G ⟩ − d , which is used to measure how well does matrix G approximate matrix H .
Revised: We introduce the quantity σ H ( G ) def = ⟨ H − 1 , G − H ⟩ = ⟨ H − 1 , G ⟩ − d , which is used to measure how well does matrix G approximate matrix H .

Original: If we further suppose G ⪰ H , then according to
Revised: If we further suppose G ⪰ H , then according to

Original: Rodomanov and Nesterov [32] it holds that G − H ⪯ ⟨ H − 1 , G − H ⟩ H
Revised: Rodomanov and Nesterov [32] it holds that G − H ⪯ ⟨ H − 1 , G − H ⟩ H

Original: =
Revised: =

Original: σ H ( G ) H .
Revised: σ H ( G ) H .

Original: Using the notation of problem (1), we let z =
Revised: Using the notation of problem (1), we let z =

Original: [ x ; y ] ∈ R d where d def = d x + d y and denote the gradient and Hessian matrix of f at ( x , y ) as g ( z ) ∈ R d and ˆ H ( z ) ∈ R d × d .
Revised: [ x ; y ] ∈ R d where d def = d x + d y and denote the gradient and Hessian matrix of f at ( x , y ) as g ( z ) ∈ R d and ˆ H ( z ) ∈ R d × d .

Original: We suppose the saddle point problem (1) satisfies the following assumptions.
Revised: We suppose the saddle point problem (1) satisfies the following assumptions.

Original: Assumption 2.1.
Revised: Assumption 2.1.

Original: The objective function f ( x , y ) is twice differentiable and has L -Lipschitz continuous gradient and L 2 -Lipschitz continuous Hessian , i.e., there exists constants L > 0 and L 2 >such that for any z = [ x ; y ] , z ′ =
Revised: The objective function f ( x , y ) is twice differentiable and has L -Lipschitz continuous gradient and L 2 -Lipschitz continuous Hessian , i.e., there exists constants L > 0 and L 2 > 0 such that for any z = [ x ; y ] , z ′ =

Original: [ x ′ ; y ′ ]
Revised: [ x ′ ; y ′ ]

Original: ∈ R d , we have ∥ g ( z ) − g
Revised: ∈ R d , we have ∥ g ( z ) − g

Original: ( z ′ )
Revised: ( z ′ )

Original: ∥
Revised: ∥

Original: ≤ L ∥
Revised: ≤ L ∥

Original: z − z ′ ∥ and ∥ ˆ H ( z ) − ˆ H ( z ′ )
Revised: z − z ′ ∥ and ∥ ˆ H ( z ) − ˆ H ( z ′ )

Original: ∥
Revised: ∥

Original: ≤ L 2 ∥ z − z ′ ∥ .
Revised: ≤ L 2 ∥ z − z ′ ∥ .

Original: Assumption 2.2.
Revised: Assumption 2.2.

Original: The objective function f ( x , y ) is twice differentiable, µ -strongly-convex in x and µ -strongly-concave in y , i.e., there exists constant µ > 0 such that ∇ 2 xx f ( x , y )
Revised: The objective function f ( x , y ) is twice differentiable, µ -strongly-convex in x and µ -strongly-concave in y , i.e., there exists constant µ > 0 such that ∇ 2 xx f ( x , y )

Original: ⪰ µ I and ∇ 2 yy f ( x , y ) ⪯ − µ I for any (
Revised: ⪰ µ I and ∇ 2 yy f ( x , y ) ⪯ − µ I for any (

Original: x , y ) ∈ R d x × R d y .
Revised: x , y ) ∈ R d x × R d y .

Original: The L -lipschitz continuous of gradient means the spectral norm of Hessian matrix
Revised: The L -Lipschitz continuous of gradient means the spectral norm of Hessian matrix

Original: ˆ H ( z ) can be upper bounded, that is ∥
Revised: ˆ H ( z ) can be upper bounded, that is ∥

Original: ˆ H ( z ) ∥ ≤ L .
Revised: ˆ H ( z ) ∥ ≤ L .

Original: Additionally, the condition number of the objective function is defined as κ def = L/µ .
Revised: Additionally, the condition number of the objective function is defined as κ def = L/µ .

Original:  Quasi-Newton Methods for Saddle Point Problems
Revised:  Quasi-Newton Methods for Saddle Point Problems

Original: In this section, we focus on designing quasi-Newton methods for saddle point problem and showing their superlinear local convergence rate.
Revised: In this section, we focus on designing quasi-Newton methods for saddle point problem and showing their superlinear local convergence rate.

Original: The update rule of standard Newton’s method can be written as z + = z − (cid:0)
Revised: The update rule of standard Newton’s method can be written as z + = z − (cid:0)

Original: ˆ H ( z ) (cid:1) − 1 g ( z ) for solving problem (1).
Revised: ˆ H ( z ) (cid:1) − 1 g ( z ) for solving problem (1).

Original: It has quadratic local convergence, but takes
Revised: It has quadratic local convergence, but takes

Original: O (cid:0) d 3 (cid:1) time complexity per iteration.
Revised: O ( d 3 ) time complexity per iteration.

Original: For convex minimization, quasi-Newton methods including
Revised: For convex minimization, quasi-Newton methods including

Original: BFGS/SR1 [3–5, 8, 37] and their variants [20, 32, 33, 42] focus on approximating the Hessian which can reduce the computational cost to O (cid:0) d 2 (cid:1) for each round.
Revised: BFGS/SR1 [3–5, 8, 37] and their variants [20, 32, 33, 42] focus on approximating the Hessian which can reduce the computational cost to O (cid:0) d 2 (cid:1) for each round.

Original: However, all these algorithms and related convergence analysis are based on the assumption that the Hessian matrix is positive definite, which is not suitable for our saddle point problems since ˆ H ( z ) is indefinite.
Revised: However, all these algorithms and related convergence analysis are based on the assumption that the Hessian matrix is positive definite, which is not suitable for our saddle point problems since ˆ H ( z ) is indefinite.

Original: We introduce the auxiliary matrix H ( z ) be the square of Hessian H ( z ) def = (cid:0) ˆ H ( z ) (cid:1) 2 .
Revised: We introduce the auxiliary matrix H ( z ) be the square of Hessian H ( z ) def = (cid:0) ˆ H ( z ) (cid:1) 2 .

Original: The following lemma shows that H ( z ) is positive definite.
Revised: The following lemma shows that H ( z ) is positive definite.

Original: Lemma 3.1.
Revised: Lemma 3.1.

Original: Under Assumption 2.1 and 2.2, we have µ 2 I ⪯ H ( z ) ⪯
Revised: Under Assumption 2.1 and 2.2, we have µ 2 I ⪯ H ( z ) ⪯

Original: L 2 I for all z ∈ R d .
Revised: L 2 I for all z ∈ R d .

Original: Hence, we can reformulate the update of Newton’s method by
Revised: Hence, we can reformulate the update of Newton’s method by

Original: [Equation]
Revised: [Equation]

Original: Then it is natural to characterize the second-order information by estimating the auxiliary matrix
Revised: Then it is natural to characterize the second-order information by estimating the auxiliary matrix

Original: H ( z ) , rather than the indefinite Hessian ˆ H ( z ) .
Revised: H ( z ) , rather than the indefinite Hessian ˆ H ( z ) .

Original: If one can obtain a symmetric positive definite matrix
Revised: If one can obtain a symmetric positive definite matrix

Original: G ∈ R d × d as an estimator for H ( z ) , the update rule of (2) can be approximated by
Revised: G ∈ R d × d as an estimator for H ( z ) , the update rule of (2) can be approximated by

Original: [Equation]
Revised: [Equation]

Original: The remainder of this section introduce strategies to construct G , resulting the quasi-Newton methods for saddle point problem with local superlinear convergence.
Revised: The remainder of this section introduce strategies to construct G , resulting the quasi-Newton methods for saddle point problem with local superlinear convergence.

Original: We should point out the implementation of iteration (3) is unnecessary to construct Hessian matrix
Revised: We should point out the implementation of iteration (3) is unnecessary to construct Hessian matrix

Original: ˆ H ( z )
Revised: ˆ H ( z )

Original: explicitly, since we are only interested in the Hessian-vector product
Revised: explicitly, since we are only interested in the Hessian-vector product

Original: ˆ H ( z ) g
Revised: ˆ H ( z ) g

Original: ( z ) which can be computed efficiently [27, 35].
Revised: ( z ) which can be computed efficiently [27, 35].

Original: 3.1 The Broyden Family Updates
Revised: 3.1 The Broyden Family Updates

Original: We first review some basic results for quasi-Newton methods in convex optimization.
Revised: We first review some basic results for quasi-Newton methods in convex optimization.

Original: We introduce the Broyden family [25, Section 6.3] of quasi-Newton updates for approximating an positive definite matrix
Revised: We introduce the Broyden family [25, Section 6.3] of quasi-Newton updates for approximating an positive definite matrix

Original: H ∈ R d × d by using the information of current estimator G ∈ 
Revised: H ∈ R d × d by using the information of current estimator G ∈ R d × d .

Original: Definition 3.2.
Revised: Definition 3.2.

Original: Suppose two positive definite matrices H , G ∈ R d
Revised: Suppose two positive definite matrices H ∈ R d × d and G ∈ R d

Original: × d satisfy H ⪯ G .
Revised: × d satisfy H ⪯ G .

Original: For any u ∈ R d , if Gu =
Revised: For any u ∈ R d , if Gu =

Original: Hu , we define Broyd τ ( G , H , u ) def = H .
Revised: Hu , we define Broyd τ ( G , H , u ) def = H .

Original: Otherwise, we define
Revised: Otherwise, we define

Original: [Equation]
Revised: [Equation]

Original: The different choices of parameter τ for above formula contain several popular quasi-Newton updates:
Revised: The different choices of parameter τ for above formula contain several popular quasi-Newton updates:

Original: • For τ = u ⊤ Hu / (cid:0) u ⊤ Gu (cid:1) ∈ [0 , 1] , it corresponds to the BFGS update
Revised: • For τ = u ⊤ Hu / (cid:0) u ⊤ Gu (cid:1) ∈ [0 , 1] , it corresponds to the BFGS update

Original: [Equation]
Revised: [Equation]

Original: • For τ = 0 , it corresponds to the SR1 update
Revised: • For τ = 0 , it corresponds to the SR1 update

Original: [Equation]
Revised: [Equation]

Original: Now we introduce the update rule [20, 32] by choosing u as
Revised: Now we introduce the update rule [20, 32] by choosing u as

Original: [Equation]
Revised: [Equation]

Original: The following lemma shows applying the update rule (4) with (7) leads to a new estimator with tighter error bound in the measure of σ H ( · ) .
Revised: The following lemma shows applying the update rule (4) with (7) leads to a new estimator with tighter error bound in the measure of σ H ( · ) .

Original: Lemma 3.3 (Modified from Lin et al.
Revised: Lemma 3.3 (Modified from Lin et al.

Original: [20, Theorem 3.1]) .
Revised: [20, Theorem 3.1]) .

Original: Suppose two positive definite matrices µ 2 I ⪯ H ⪯ L 2 I and G ∈ R d × d satisfy that H ⪯ G .
Revised: Suppose two positive definite matrices µ 2 I ⪯ H ⪯ L 2 I and G ∈ R d × d satisfy that H ⪯ G .

Original: Let G + =
Revised: Let G + =

Original: Broyd τ ( G , H , u )) , where u is chosen random method as (7) , then for any τ ∈ [0 , 1] , we have E [ σ H ( G + )] ≤ (cid:0) 1 − 1 / ( d κ 2 ) (cid:1)
Revised: Broyd τ ( G , H , u )) , where u is chosen random method as (7) , then for any τ ∈ [0 , 1] , we have E [ σ H ( G + )] ≤ (cid:0) 1 − 1 / ( d κ 2 ) (cid:1)

Original: σ H ( G )
Revised: σ H ( G )

Original: For specific Broyden family updates BFGS and SR1 shown in (5) and (6), the update can achieve a better convergence result.
Revised: For specific Broyden family updates BFGS and SR1 shown in (5) and (6), the update can achieve a better convergence result.

Original: Concretely, for BFGS method, we first find L such that G − 1 =
Revised: Concretely, for BFGS method, we first find L such that G − 1 =

Original: L ⊤ L , where L is an upper triangular matrix.
Revised: L ⊤ L , where L is an upper triangular matrix.

Original: This step can be implemented with O ( d 2 ) complexity [20,
Revised: This step can be implemented with O ( d 2 ) complexity [20,

Original: Proposition 1].
Revised: Proposition 1].

Original: We present the subroutine for factorizing G − 1 in Algorithm 1 and give its detailed implementation in the appendix.
Revised: We present the subroutine for factorizing G − 1 in Algorithm 1 and give its detailed implementation in the appendix.

Original: And we use the direction L ⊤ u instead of u for the BFGS update.
Revised: And we use the direction L ⊤ u instead of u for the BFGS update.

Original: Applying the BFGS update rule (5) with formula (7), we obtain a condition-number-free result as follows.
Revised: Applying the BFGS update rule (5) with formula (7), we obtain a condition-number-free result as follows.

Original: Lemma 3.4 (Modified from Lin et al.
Revised: Lemma 3.4 (Modified from Lin et al.

Original: [20, Theorem 4.2]) .
Revised: [20, Theorem 4.2]) .

Original: Suppose two positive definite matrices
Revised: Suppose two positive definite matrices

Original: H , G ∈ R d × d satisfy H ⪯ G .
Revised: H ∈ R d × d and G ∈ R d × d satisfy H ⪯ G .

Original: Let G + =
Revised: Let G + =

Original: BFGS (cid:0) G , H , L ⊤ u (cid:1) , where u is chosen by the random method in (7) and L is an upper triangular matrix such that G − 1 =
Revised: BFGS (cid:0) G , H , L ⊤ u (cid:1) , where u is chosen by the random method in (7) and L is an upper triangular matrix such that G − 1 =

Original: L ⊤ L .
Revised: L ⊤ L .

Original: Then we have E [ σ H ( G + )] ≤ (1 − 1 /d )
Revised: Then we have E [ σ H ( G + )] ≤ (1 − 1 /d )

Original: σ H ( G ) .
Revised: σ H ( G ) .

Original: Remark 3.5 .
Revised: Remark 3.5 .

Original: Note that the step of conducting G − 1 =
Revised: Note that the step of conducting G − 1 =

Original: L ⊤ L requires QR decomposition of rankchange matrix which requires O (26 d 2 ) flops [14, Section 12.5.1].
Revised: L ⊤ L requires QR decomposition of rankchange matrix which requires O (26 d 2 ) flops [14, Section 12.5.1].

Original: We do not recommend using this
Revised: We do not recommend using this

Original: BFGS update strategy in practice when n is large.
Revised: BFGS update strategy in practice when n is large.

Original: The convergence of the SR1 update can be characterized by the measure τ H ( G ) def = tr( G − H ) .
Revised: The convergence of the SR1 update can be characterized by the measure τ H ( G ) def = tr( G − H ) .

Original: Applying the SR1 update rule (6) with formula (7), the convergence also holds a condition-numberfree result.
Revised: Applying the SR1 update rule (6) with formula (7), the convergence also holds a condition-numberfree result.

Original: Lemma 3.6 (Modified from Lin et al.
Revised: Lemma 3.6 (Modified from Lin et al.

Original: [20, Theorem 4.1]) .
Revised: [20, Theorem 4.1]) .

Original: Suppose two positive definite matrices
Revised: Suppose two positive definite matrices

Original: H , G ∈ R d × d satisfy H ⪯ G .
Revised: H ∈ R d × d and G ∈ R d × d satisfy H ⪯ G .

Original: Let G + =
Revised: Let G + =

Original: SR1 ( G , H , u ) , where u is chosen by the random method in (7) .
Revised: SR1 ( G , H , u ) , where u is chosen by the random method in (7) .

Original: Then we have E [ τ H ( G + )] ≤ (1 − 1 /d )
Revised: Then we have E [ τ H ( G + )] ≤ (1 − 1 /d )

Original: τ H ( G ) .
Revised: τ H ( G ) .

Original: 3.2 Algorithms for Quadratic Saddle Point Problems
Revised: 3.2 Algorithms for Quadratic Saddle Point Problems

Original: We now solve simple quadratic saddle point problem that f ( x , y )
Revised: We now solve simple quadratic saddle point problem that f ( x , y )

Original: =
Revised: =

Original: 12 z ⊤ Az − b ⊤ z in (1)
Revised: 12 z ⊤ Az − b ⊤ z in (1)

Original: where z =
Revised: where z =

Original: [ x ; y ] , b ∈ R d ,
Revised: [ x ; y ] , b ∈ R d ,

Original: A ∈ R d × d is symmetric and d = d x + d y .
Revised: A ∈ R d × d is symmetric and d = d x + d y .

Original: We suppose A could be partitioned as A = 
Revised: We suppose A could be partitioned as A = (cid:20) A xx

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original:  (cid:21) where the sub-matrices A xx ∈ R d x × d x , A xy ∈ R d x × d y , A yx ∈ R d y × d x
Revised: A yy (cid:21) where the sub-matrices A xx ∈ R d x × d x , A xy ∈ R d x × d y , A yx ∈ R d y × d x

Original: Randomlychoose u[-k from (7) and update G k +
 =-]
Revised: Randomly u[-k from (7) and update G k +
 =-]

Original: Broydτ k G H u 5: end for and A yy ∈ R d y × d y satisfy A xx ⪰ µ I , A yy ⪯ − µ I and ∥ A ∥ ≤ L .
Revised: Broyd for and A yy ∈ R d y × d y satisfy A xx ⪰ µ I , A yy ⪯ − µ I and ∥ A ∥ ≤ L .

Original: Using notations introduced in
Revised: Using notations introduced in

Original: Section 2, we have z =
Revised: Section 2, we have z =

Original: [ x ; y ] , g ( z ) =
Revised: [ x ; y ] , g ( z ) =

Original: Az − b , ˆ H def = ˆ H ( z )
Revised: Az − b , ˆ H def = ˆ H ( z )

Original: =
Revised: =

Original: A and H def = H ( z ) =
Revised: A and H def = H ( z ) =

Original: A 2 .
Revised: A 2 .

Original: We present the detailed procedure of random quasi-Newton methods for quadratic saddle point problem by using the Broyden family update, BFGS update and SR1 update in Algorithm 2, 3 andrespectively.
Revised: We present the detailed procedure of random quasi-Newton methods for quadratic saddle point problem by using the Broyden family update, BFGS update and SR1 update in Algorithm 2, 3 andrespectively.

Original: We define λ k as the gradient norm at z k for our convergence analysis, that is λ k def =
Revised: We define λ k as the gradient norm at z k for our convergence analysis, that is λ k def =

Original: ∥ g ( z k ) ∥ .
Revised: ∥ g ( z k ) ∥ .

Original: The definition of λ k in this paper is different from the measure used in convex optimization [20, 32] 2 , but it also holds the similar property as follows.
Revised: The definition of λ k in this paper is different from the measure used in convex optimization [20, 32] 2 , but it also holds the similar property as follows.

Original: Lemma 3.7.
Revised: Lemma 3.7.

Original: Assume we have η k ≥ 1 and G k ∈ R d × d such that H ⪯ G k ⪯ η k H for Algorithm 2, 3 and 4, then we have λ k +1 ≤ (1 − 1 /η k ) λ k .
Revised: Assume we have η k ≥ 1 and G k ∈ R d × d such that H ⪯ G k ⪯ η k H for Algorithm 2-4, then we have λ k +1 ≤ (1 − 1 /η k ) λ k .

Original: The next theorem states the assumptions of Lemma 3.7 always holds with η k = κ 2 ≥ 1 , which means λ k converges to 0 linearly.
Revised: The next theorem states the assumptions of Lemma 3.7 always holds with η k = κ 2 ≥ 1 , which means λ k converges to 0 linearly.

Original: Theorem 3.8.
Revised: Theorem 3.8.

Original: For all k ≥ 0 , Algorithm 2, 3, 4 hold that λ k ≤ (cid:0) 1 − 1 κ 2 (cid:1) k λ 0 and H ⪯ G k ≤ κ .
Revised: For all k ≥ 0 , Algorithm 2, 3, 4 hold that λ k ≤ (cid:0) 1 − 1 κ 2 (cid:1) k λ 0 and H ⪯ G k ≤ κ .

Original: Lemma 3.7 also implies superlinear convergence can be obtained if there exists η k which converges to 1.
Revised: Lemma 3.7 also implies superlinear convergence can be obtained if there exists η k which converges to 1.

Original: Applying Lemma 3.3, 3.4 and 3.6, we can show it holds for proposed algorithms.
Revised: Applying Lemma 3.3, 3.4 and 3.6, we can show it holds for proposed algorithms.

Original: Theorem 3.9.
Revised: Theorem 3.9.

Original: Solving quadratic saddle point problem by the proposed quasi-Newton Algorithms, for all k ≥ 0 , we have:
Revised: Solving quadratic saddle point problem by the proposed quasi-Newton Algorithms, for all k ≥ 0 , we have:

Original: For the Broyden family method (Algorithm 2), we have E [ λ k +1 /λ k ] ≤ (cid:0) 1 − 1 / ( d κ 2 ) (cid:1) k
Revised: For the Broyden family method (Algorithm 2), we have E [ λ k +1 /λ k ] ≤ (cid:0) 1 − 1 / ( d κ 2 ) (cid:1) k

Original: d κ 2 . 2.
Revised: d κ 2 . 2.

Original: For the BFGS method (Algorithm 3), we have E [ λ k +1 /λ k ] ≤ (1 − 1 /d )
Revised: For the BFGS method (Algorithm 3), we have E [ λ k +1 /λ k ] ≤ (1 − 1 /d )

Original: k d κ 2 . 3.
Revised: k d κ 2 . 3.

Original: For the SR1 method (Algorithm 4), we have E [ λ k +1 /λ k ] ≤ (1 − k/d ) d κ 4 .
Revised: For the SR1 method (Algorithm 4), we have E [ λ k +1 /λ k ] ≤ (1 − k/d ) d κ 4 .

Original: Combining the results of Theorem 3.8 and 3.9, we achieve the two-stages convergence behavior, that is, the algorithm has the global linear convergence and local superlinear convergence.
Revised: Combining the results of Theorem 3.8 and 3.9, we achieve the two-stages convergence behavior, that is, the algorithm has the global linear convergence and local superlinear convergence.

Original: We leave the formal description in appendix.
Revised: We leave the formal description in appendix.

Original: 3.3 Algorithms for General Saddle Point Problems
Revised: 3.3 Algorithms for General Saddle Point Problems

Original: In this section, we consider the general saddle point problem where f ( x , y ) in (1) satisfies Assumption 2.1 and 2.2.
Revised: In this section, we consider the general saddle point problem where f ( x , y ) in (1) satisfies Assumption 2.1 and 2.2.

Original: We propose quasi-Newton methods for solving the problem with local superlinear convergence and O (cid:0) d 2 (cid:1) time complexity for each iteration.
Revised: We propose quasi-Newton methods for solving the problem with local superlinear convergence and O ( d 2 ) time complexity for each iteration.

Original: 1: Input: z 0 ∈ R d , G 

Revised: 1: Input: z 0 ∈ R d , G

Original: =
Revised: =

Original: L 2 I , L 0 =
Revised: L 2 I , L 0 =

Original: L − 1 I 2: for k = 0 , 1 . . .
Revised: L − 1 I 2: for k = 0 , 1 . . .

Original: 3: z k +1 =
Revised: 3: z k +1 =

Original: z k − G − 1 k ˆ Hg ( z k ) 4:
Revised: z k − G − 1 k ˆ Hg ( z k ) 4:

Original: Randomly choose ˜ u k from (7) 5: u k =
Revised: Randomly choose ˜ u k from (7) 5: u k =

Original: L ⊤ k ˜ u
Revised: L ⊤ k ˜ u

Original: k 6:
Revised: k 6:

Original: G k +1 =
Revised: G k +1 =

Original: BFGS ( G k , H , u k ) 7:
Revised: BFGS ( G k , H , u k ) 7:

Original: L k +1 =
Revised: L k +1 =

Original: Fast-Chol ( H , L k , u k ) 8: end for
Revised: Fast-Chol ( H , L k , u k ) 8: end for

Original: 1: Input: z 0 ∈ R d , G 

Revised: 1: Input: z 0 ∈ R d , G

Original: =
Revised: =

Original: L 

Revised: L

Original: I
Revised: I

Original: K ≤ d +2: for k = 0 , 1 . .
Revised: K ≤ d +2: for k = 0 , 1 . .

Original: , K 3:
Revised: , K 3:

Original: z k +1 =
Revised: z k +1 =

Original: z k − G − 1 k ˆ Hg ( z k ) 4:
Revised: z k − G − 1 k ˆ Hg ( z k ) 4:

Original: Randomly choose u k from (7) 5:
Revised: Randomly choose u k from (7) 5:

Original: G k +1 =
Revised: G k +1 =

Original: SR1 (cid:0) G k , H , u k (cid:1) 6: end for
Revised: SR1 (cid:0) G k , H , u k (cid:1) 6: end for

Original: 3.3.1 Algorithms
Revised: 3.3.1 Algorithms

Original: The key idea of designing quasi-Newton methods for saddle point problems is approximating the auxiliary matrix H ( z ) def = (cid:0) ˆ H ( z ) (cid:1) 2 to characterize the second-order information.
Revised: The key idea of designing quasi-Newton methods for saddle point problems is approximating the auxiliary matrix H ( z ) def = (cid:0) ˆ H ( z ) (cid:1) 2 to characterize the second-order information.

Original: Since the Hessian of f is Lipschitz continuous and bounded by Assumption 2.1 and 2.2, which means the auxiliary matrix operator H ( z ) is also Lipschitz continuous.
Revised: Since the Hessian of f is Lipschitz continuous and bounded by Assumption 2.1 and 2.2, which means the auxiliary matrix operator H ( z ) is also Lipschitz continuous.

Original: Lemma 3.10.
Revised: Lemma 3.10.

Original: Under Assumption 2.1 and 2.2, we have H ( z ) is 2 LL 2 -Lipschitz continuous.
Revised: Under Assumption 2.1 and 2.2, we have H ( z ) is 2 LL 2 -Lipschitz continuous.

Original: Combining Lemma 3.1 and 3.10, we achieve the following properties of H ( z ) , which analogize the strongly self-concordance in convex optimization [32].
Revised: Combining Lemma 3.1 and 3.10, we achieve the following properties of H ( z ) , which analogize the strongly self-concordance in convex optimization [32].

Original: Lemma 3.11.
Revised: Lemma 3.11.

Original: Under Assumption 2.1 and 2.2, for all z , z ′ , w ∈ R d , the auxiliary matrix operator
Revised: Under Assumption 2.1 and 2.2, for all z , z ′ , w ∈ R d , the auxiliary matrix operator

Original: H ( · ) satisfies H ( z ) − H ( z ′ )
Revised: H ( · ) satisfies H ( z ) − H ( z ′ )

Original: ⪯ M ∥
Revised: ⪯ M ∥

Original: z − z ′ ∥ H ( w ) , where M = 2 κ 2 L 2 /L .
Revised: z − z ′ ∥ H ( w ) , where M = 2 κ 2 L 2 /L .

Original: Corollary 3.12.
Revised: Corollary 3.12.

Original: Let z , z ′ ∈ R d and r = ∥
Revised: Let z , z ′ ∈ R d and r = ∥

Original: z ′ − z ∥ .
Revised: z ′ − z ∥ .

Original: Suppose the objective function f satisfies
Revised: Suppose the objective function f satisfies

Original: Assumption 2.1 and 2.2, then for all z , z ′ ∈ R d and M = 2 κ 2 L 2 /L , the auxiliary matrix operator
Revised: Assumption 2.1 and 2.2, then for all z , z ′ ∈ R d and M = 2 κ 2 L 2 /L , the auxiliary matrix operator

Original: H ( · ) holds that
Revised: H ( · ) holds that

Original: [Equation]
Revised: [Equation]

Original: Different from the quadratic case, the auxiliary matrix H ( z ) is not fixed for general saddle point problem.
Revised: Different from the quadratic case, the auxiliary matrix H ( z ) is not fixed for general saddle point problem.

Original: Based on the smoothness of H ( z ) , we apply Corollary 3.12 to generalize Lemma 3.13 as follows.
Revised: Based on the smoothness of H ( z ) , we apply Corollary 3.12 to generalize Lemma 3.13 as follows.

Original: Lemma 3.13.
Revised: Lemma 3.13.

Original: Let z ∈ R d and G ∈ R
Revised: Let z ∈ R d and G ∈ R

Original: d
Revised: d

Original: ×
Revised: ×

Original: d be a positive definite matrix
Revised: d be a positive definite matrix

Original: such that H ( z ) ⪯ G ⪯ η H ( z ) for some η ≥ 1 .
Revised: such that H ( z ) ⪯ G ⪯ η H ( z ) for some η ≥ 1 .

Original: In addition, define z + ∈ R d and r = ∥
Revised: In addition, define z + ∈ R d and r = ∥

Original: z + − z ∥ , then for all u ∈ R d , τ ∈ [0 , 1] and
Revised: z + − z ∥ , then for all u ∈ R d , τ ∈ [0 , 1] and

Original: M = 2 κ 2 L 2 /L , we have
Revised: M = 2 κ 2 L 2 /L , we have

Original: [Equation]
Revised: [Equation]

Original: where ˜ G def = (1 + Mr ) G ⪰ H ( z + ) .
Revised: where ˜ G def = (1 + Mr ) G ⪰ H ( z + ) .

Original: Lemma 3.13 implies it is reasonable to have the algorithms using G k
Revised: Lemma 3.13 implies it is reasonable to have the algorithms using G k

Original: +

Revised: +

Original: =
Revised: =

Original: Broyd τ k
Revised: Broyd τ k

Original: ( ˜ G k , H k +1 , u k ) with ˜ G k
Revised: ( ˜ G k , H k +1 , u k ) with ˜ G k

Original: = (1 + Mr k ) G k and r k = ∥
Revised: = (1 + Mr k ) G k and r k = ∥

Original: z k +1 − z k ∥ .
Revised: z k +1 − z k ∥ .

Original: Similarly, we can also achieve G k +1 by such ˜
Revised: Similarly, we can also achieve G k +1 by such ˜

Original: G k for specific BFGS and SR1 update.
Revised: G k for specific BFGS and SR1 update.

Original: Combining this with iteration (3), we propose several quasi-Newton methods for general strongly-convex-strongly-concave saddle point problems.
Revised: Combining this with iteration (3), we propose several quasi-Newton methods for general strongly-convex-strongly-concave saddle point problems.

Original: The details are shown in Algorithm 5, 6 and 7 for Broyden family, BFGS and SR1 updates respectively.
Revised: The details are shown in Algorithm 5, 6 and 7 for Broyden family, BFGS and SR1 updates respectively.

Original: 3.3.2 Convergence Analysis
Revised: 3.3.2 Convergence Analysis

Original: Let us consider the convergence guarantee for algorithms proposed in Section 3.3.1.
Revised: Let us consider the convergence guarantee for algorithms proposed in Section 3.3.1.

Original: We introduce the following notations to simplify the presentation.
Revised: We introduce the following notations to simplify the presentation.

Original: We let { z k } be the sequence generated from
Revised: We let { z k } be the sequence generated from

Original: Algorithm 5, 6 or 7 and denote
Revised: Algorithm 5, 6 or 7 and denote

Original: [Equation]
Revised: [Equation]

Original: k def = g ( z k ) ,
Revised: k def = g ( z k ) ,

Original: [Equation]
Revised: [Equation]

Original: We still use gradient norm λ k def = ∥∇ f
Revised: We still use gradient norm λ k def = ∥∇ f

Original: ( z k ) ∥ for analysis and establish the relationship between λ k and λ k +1 , which is shown in Lemma 3.14.
Revised: ( z k ) ∥ for analysis and establish the relationship between λ k and λ k +1 , which is shown in Lemma 3.14.

Original: 1: Input: z
Revised: 1: Input: z

Original: R d , G
Revised: R d , G

Original: , τ
Revised: , τ

Original: 1] and M
Revised: 1] and M

Original: 2: for k = 0 ,
Revised: 2: for k = 0 ,

Original: =
Revised: =

Original: G
Revised: G

Original: H g
Revised: H g

Original: Compute r k = ∥
Revised: Compute r k = ∥

Original: z k +1 − z k ∥ and ˜ G
Revised: z k +1 − z k ∥ and ˜ G

Original: = (1 + Mr k ) G k 5:
Revised: = (1 + Mr k ) G k 5:

Original: Randomly choose u k from (7) and update G k +

Revised: Randomly choose u k from (7) and update G k +

Original: =
Revised: =

Original: Broyd τ k
Revised: Broyd τ k

Original: ( ˜ G k , H k
Revised: ( ˜ G k , H k

Original: 1 , u k ) .
Revised: 1 , u k ) .

Original: 6: end for
Revised: 6: end for

Original: Randomly choose u k from (7) 9:
Revised: Randomly choose u k from (7) 9:

Original: Fast
Revised: Fast

Original: 11: end for
Revised: 11: end for

Original: Randomly choose
Revised: Randomly choose

Original: 8: end for
Revised: 8: end for

Original: Lemma 3.14.
Revised: Lemma 3.14.

Original: Using Algorithm 5, 6 and 7, suppose we have H k ⪯ G k ⪯ η k H k , for some η k ≥and let β
Revised: Using Algorithm 5, 6 and 7, suppose we have H k ⪯ G k ⪯ η k H k , for some η k ≥and let β

Original: =
Revised: =

Original: L 2 / (2 µ 2 ) , then we have
Revised: L 2 / (2 µ 2 ) , then we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Rodomanov and Nesterov [32, Lemma 4.3] derive a result similar to Lemma 3.14 for minimizing the strongly-convex function ˆ f ( · ) on the different measure λ ˆ f ( · ) def =
Revised: Rodomanov and Nesterov [32, Lemma 4.3] derive a result similar to Lemma 3.14 for minimizing the strongly-convex function ˆ f ( · ) on the different measure λ ˆ f ( · ) def =

Original: ⟨∇ ˆ f ( · ) , ∇ 2 ˆ f ( · ) − 1 ∇ ˆ f ( · ) ⟩ .
Revised: ⟨∇ ˆ f ( · ) , ∇ 2 ˆ f ( · ) − 1 ∇ ˆ f ( · ) ⟩ .

Original: Note that our algorithms are based on the iteration rule z k +

Revised: Note that our algorithms are based on the iteration rule z k +

Original: =
Revised: =

Original: z k − G − 1 k ˆ H k g k .
Revised: z k − G − 1 k ˆ H k g k .

Original: Compared with quasi-Newton methods for convex optimization, there exists an additional term ˆ H k between G − 1 k and g k , which leads to the fact that the convergence analysis based on λ ˆ f ( z k ) is difficult.
Revised: Compared with quasi-Newton methods for convex optimization, there exists an additional term ˆ H k between G − 1 k and g k , which leads to the fact that the convergence analysis based on λ ˆ f ( z k ) is difficult.

Original: Fortunately, we find using gradient norm λ k directly makes the analysis achievable.
Revised: Fortunately, we find using gradient norm λ k directly makes the analysis achievable.

Original: For further analysis, we also denote σ k def = σ H k
Revised: For further analysis, we also denote σ k def = σ H k

Original: ( G k ) =
Revised: ( G k ) =

Original: ⟨ H − 1 k , G k ⟩ − d.
Revised: ⟨ H − 1 k , G k ⟩ − d.

Original: Then we establish the linear convergence for the first period of iterations, which can be viewed as the extension of Theorem 3.8.
Revised: Then we establish the linear convergence for the first period of iterations, which can be viewed as the extension of Theorem 3.8.

Original: Note that the following result holds for Algorithm 5, 6 and 7; and it does not depend on the choice of u k .
Revised: Note that the following result holds for Algorithm 5, 6 and 7; and it does not depend on the choice of u k .

Original: Theorem 3.15.
Revised: Theorem 3.15.

Original: Using Algorithm 5, 6 and 7 by G 

Revised: Using Algorithm 5, 6 and 7 by G

Original: =
Revised: =

Original: L 2 I and M = 2 κ 2 L 2 /L , suppose the initial point z 0 is sufficiently close to z ∗
Revised: L 2 I and M = 2 κ 2 L 2 /L , suppose the initial point z 0 is sufficiently close to z ∗

Original: such that Mλ 0 /µ ≤ ln b/ (4 b κ 2 ) with 1 < b < 5 , then we have λ k ≤ (cid:0) 1 − 1 / 2 b κ 2 (cid:1)
Revised: such that Mλ 0 /µ ≤ ln b/ (4 b κ 2 ) with 1 < b < 5 , then we have λ k ≤ (cid:0) 1 − 1 / 2 b κ 2 (cid:1)

Original: k λ 0 and H k ⪯ G k ⪯ exp (cid:16) 2 (cid:80)
Revised: k λ 0 and H k ⪯ G k ⪯ exp (cid:16) 2 (cid:80)

Original: k − 1 i =

Revised: k − 1 i =

Original: ρ i (cid:17) κ 2 H k ⪯ b κ 2 H k for all k ≥ 0 , where ρ k def =
Revised: ρ i (cid:17) κ 2 H k ⪯ b κ 2 H k for all k ≥ 0 , where ρ k def =

Original: Mλ k /µ .
Revised: Mλ k /µ .

Original: We now analyze how σ k changes after one iteration to show the local superlinear convergence for the Broyden family method (Algorithm 5) and the BFGS method (Algorithm 6).
Revised: We now analyze how σ k changes after one iteration to show the local superlinear convergence for the Broyden family method (Algorithm 5) and the BFGS method (Algorithm 6).

Original: Recall that σ k is defined to measure how well does matrix G k approximate H k .
Revised: Recall that σ k is defined to measure how well does matrix G k approximate H k .

Original: Lemma 3.16.
Revised: Lemma 3.16.

Original: Solving the general strongly-convex-strongly-concave saddle point problem (1) under
Revised: Solving the general strongly-convex-strongly-concave saddle point problem (1) under

Original: Assumption 2.1 and 2.2 by our quasi-Newton algorithms and supposing the sequences G 0 , · · · , G k generated by the Algorithm 5 and 6 are given, then we have the following results for σ k for all k
Revised: Assumption 2.1 and 2.2 by our quasi-Newton algorithms and supposing the sequences G 0 , · · · , G k generated by the Algorithm 5 and 6 are given, then we have the following results for σ k for all k

Original: The random Broyden family method (Algorithm 5) holds that
Revised: The random Broyden family method (Algorithm 5) holds that

Original: [Equation]
Revised: [Equation]

Original: The random BFGS method (Algorithm 6) holds that
Revised: The random BFGS method (Algorithm 6) holds that

Original: [Equation]
Revised: [Equation]

Original: The analysis for the SR1 method is based on constructing η k such that η k = tr( G k − H k ) /
Revised: The analysis for the SR1 method is based on constructing η k such that η k = tr( G k − H k ) /

Original: tr( H k ) and the technical details are showed in appendix.
Revised: tr( H k ) and the technical details are showed in appendix.

Original: Based on Lemma 3.16, one can show that our algorithms enjoy the local superlinear convergence for the general saddle point problems.
Revised: Based on Lemma 3.16, one can show that our algorithms enjoy the local superlinear convergence for the general saddle point problems.

Original: Theorem 3.17.
Revised: Theorem 3.17.

Original: Solving general saddle point problem (1) under Assumption 2.1 and 2.2 by proposed random quasi-Newton methods (Algorithm 5, 6 and 7) by G 

Revised: Solving general saddle point problem (1) under Assumption 2.1 and 2.2 by proposed random quasi-Newton methods (Algorithm 5, 6 and 7) by G

Original: =
Revised: =

Original: L 2 I and M = 2 κ 2 L 2 /L , we have the following results for all k > 0 :
Revised: L 2 I and M = 2 κ 2 L 2 /L , we have the following results for all k > 0 :

Original: For the random Broyden family method (Algorithm 5), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+2 d ) κ 2 , we have E [ λ k +1 /λ k ] ≤ (cid:0) 1 − 1 / ( d κ 2 ) (cid:1) k
Revised: For the random Broyden family method (Algorithm 5), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+2 d ) κ 2 , we have E [ λ k +1 /λ k ] ≤ (cid:0) 1 − 1 / ( d κ 2 ) (cid:1) k

Original: 2 d κ 2 . 2.
Revised: 2 d κ 2 . 2.

Original: For the random BFGS method (Algorithm 6), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+ d ) κ 2 , we have E [ λ k +1 /λ k ] ≤ (1 − 1 /d )
Revised: For the random BFGS method (Algorithm 6), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+ d ) κ 2 , we have E [ λ k +1 /λ k ] ≤ (1 − 1 /d )

Original: k
Revised: k

Original: 2 d κ 2 . 3.
Revised: 2 d κ 2 . 3.

Original: For the random SR1 method (Algorithm 7), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+2 d κ 2 ) κ 2 , we have E [ λ k +1 /λ k ] ≤ (1 − 1 /d )
Revised: For the random SR1 method (Algorithm 7), if λ 0 satisfies that Mλ 0 µ ≤ ln2 8(1+2 d κ 2 ) κ 2 , we have E [ λ k +1 /λ k ] ≤ (1 − 1 /d )

Original: k 2 d κ 4 .
Revised: k 2 d κ 4 .

Original: Finally, combining the results of Theorem 3.15 and 3.17, we can prove the algorithms achieve the two-stages convergence behaviors as follows.
Revised: Finally, combining the results of Theorem 3.15 and 3.17, we can prove the algorithms achieve the two-stages convergence behaviors as follows.

Original: Corollary 3.18.
Revised: Corollary 3.18.

Original: Solving the general saddle point problem (1) under Assumption 2.1 and 2.2 by our random quasi-Newton methods (Algorithm 5, 6 and 7) with G 

Revised: Solving the general saddle point problem (1) under Assumption 2.1 and 2.2 by our random quasi-Newton methods (Algorithm 5, 6 and 7) with G

Original: =
Revised: =

Original: L 2 I and M = 2 κ 2 L 2 /L , if the initial point is sufficiently close to the saddle point such that Mλ 0 /µ ≤ ln 2 / (8 κ 2 ) , then with probability 1 − δ for any δ ∈ (0 , 1) , we have the following results:
Revised: L 2 I and M = 2 κ 2 L 2 /L , if the initial point is sufficiently close to the saddle point such that Mλ 0 /µ ≤ ln 2 / (8 κ 2 ) , then with probability 1 − δ for any δ ∈ (0 , 1) , we have the following results:

Original: The random Broyden family method (Algorithm 5) holds that
Revised: The random Broyden family method (Algorithm 5) holds that

Original: [Equation]
Revised: [Equation]

Original: for all k 

Revised: for all k

Original: = O (cid:0)
Revised: = O (cid:0)

Original: d κ 2 ln
Revised: d κ 2 ln

Original: ( d κ /δ ) (cid:1) and k ≥ 0 .
Revised: ( d κ /δ ) (cid:1) and k ≥ 0 .

Original: The random BFGS/SR1 method (Algorithm 6/7) holds that
Revised: The random BFGS/SR1 method (Algorithm 6/7) holds that

Original: [Equation]
Revised: [Equation]

Original: for all k 

Revised: for all k

Original: = O (cid:0) max (cid:8) d, κ 2 (cid:9) ln
Revised: = O (cid:0) max (cid:8) d, κ 2 (cid:9) ln

Original: ( d κ /δ ) (cid:1) and k ≥ 0 .
Revised: ( d κ /δ ) (cid:1) and k ≥ 0 .

Original:  Discussion
Revised:  Discussion

Original: The relationship between our quasi-Newton methods (Algorithm 5, 6 and 7) and existing first-order method for minimax problem is similar to the one for minimization problem.
Revised: The relationship between our quasi-Newton methods (Algorithm 5, 6 and 7) and existing first-order method for minimax problem is similar to the one for minimization problem.

Original: For our BFGS and SR methods, the dependency on κ 2 only appears on the first period of linear convergence, which matches the convergence rate of gradient descent ascent.
Revised: For our BFGS and SR methods, the dependency on κ 2 only appears on the first period of linear convergence, which matches the convergence rate of gradient descent ascent.

Original: As an analogy, minimizing strongly-convex function by quasi-Newton methods [20, 32] has dependency on κ in the first period of linear convergence, which matches the convergence rate of gradient descent.
Revised: As an analogy, minimizing strongly-convex function by quasi-Newton methods [20, 32] has dependency on κ in the first period of linear convergence, which matches the convergence rate of gradient descent.

Original: Our quasi-Newton methods also can be extended for solving non-linear equations, and the two-stage convergence behavior still hold.
Revised: Our quasi-Newton methods also can be extended for solving non-linear equations, and the two-stage convergence behavior still hold.

Original: We provide the detailed discussion and the comparison with related work [19, 41] in Appendix F
Revised: We provide the detailed discussion and the comparison with related work [19, 41] in Appendix F.

Original: [Figure]
Revised: [Figure]

Original:  Numerical Experiments
Revised:  Numerical Experiments

Original: In this section, we conduct the experiments on machine learning applications of AUC Maximization and adversarial debiasing to verify our theory.
Revised: In this section, we conduct the experiments on machine learning applications of AUC Maximization and adversarial debiasing to verify our theory.

Original: We refer to Algorithm 2 and 5 with parameter τ k =
Revised: We refer to Algorithm 2 and 5 with parameter τ k =

Original: u ⊤ H
Revised: u ⊤ H

Original: k +1 / ( u ⊤ G k u ) as RaBFGSv1-Q and RaBFGSv1-G; refer to Algorithm 3, 4, 6 and 7 as
Revised: k +1 / ( u ⊤ G k u ) as RaBFGSv1-Q and RaBFGSv1-G; refer to Algorithm 3, 4, 6 and 7 as

Original: RaBFGSv2-Q, RaSR1-Q, RaBFGSv2-G and RaSR1-G respectively.
Revised: RaBFGSv2-Q, RaSR1-Q, RaBFGSv2-G and RaSR1-G respectively.

Original: We compare these proposed algorithms with classical first-order method extragradient (EG) [18, 38].
Revised: We compare these proposed algorithms with classical first-order method extragradient (EG) [18, 38].

Original: 5.1 AUC Maximization
Revised: 5.1 AUC Maximization

Original: AUC maximization [15, 43] aims to find the classifier w ∈ R m on the training set { a i , b i } ni =

Revised: AUC maximization [15, 43] aims to find the classifier w ∈ R m on the training set { a i , b i } ni =

Original: where a i ∈ R d and b i ∈ { +1 , − 1 } .
Revised: where a i ∈ R d and b i ∈ { +1 , − 1 } .

Original: We denote n + be the number of positive instances and p = n + /n .
Revised: We denote n + be the number of positive instances and p = n + /n .

Original: The minimax formulation for AUC maximization can be written as
Revised: The minimax formulation for AUC maximization can be written as

Original: [Equation]
Revised: [Equation]

Original: where x = [ w ; u ; v ] ∈ R m +2 , λ > 0 is the regularization parameter and f i is defined as
Revised: where x = [ w ; u ; v ] ∈ R m +2 , λ > 0 is the regularization parameter and f i is defined as

Original: [Equation]
Revised: [Equation]

Original: The objective function of AUC maximization is quadratic, hence we conduct the algorithms in Section 3.2 (Algorithm 2, 3 and 4) for this model.
Revised: The objective function of AUC maximization is quadratic, hence we conduct the algorithms in Section 3.2 (Algorithm 2, 3 and 4) for this model.

Original: We set λ = 100 /n and evaluate all algorithms on three real-world datasets “a9a”, “w8a” and “sido0”.
Revised: We set λ = 100 /n and evaluate all algorithms on three real-world datasets “a9a”, “w8a” and “sido0”.

Original: The dimension of the problem is d = m + 
Revised: The dimension of the problem is d = m + 3 .

Original: The results of iteration numbers against ∥ g ( z ) ∥ 2 and CPU time against ∥ g ( z ) ∥ 2 are presented in Figure 1.
Revised: The results of iteration numbers against ∥ g ( z ) ∥ 2 and CPU time against ∥ g ( z ) ∥ 2 are presented in Figure 1.

Original: These results show that our algorithms perform better than the EG method.
Revised: These results show that our algorithms perform better than the EG method.

Original: 5.2 Adversarial Debiasing
Revised: 5.2 Adversarial Debiasing

Original: Adversarial learning [22, 44] can be used on fairness-aware machine learning issues.
Revised: Adversarial learning [22, 44] can be used on fairness-aware machine learning issues.

Original: Give the training set { a i , b i , c i } ni =1  where a i ∈ R d contains all input variables, b i ∈ R is the output and c i ∈ R is the input variable which we want to protect and make it unbiased.
Revised: Give the training set { a i , b i , c i } ni =1 , where a i ∈ R d contains all input variables, b i ∈ R is the output and c i ∈ R is the input variable which we want to protect and make it unbiased.

Original: Our experiments are based on the
Revised: Our experiments are based on the

Original: [Figure]
Revised: [Figure]

Original: fairness-aware binary classification dataset “adult”, “bank market” and “law school”[30], leading to b i , c i ∈ { +1 , − 1 } .
Revised: fairness-aware binary classification dataset “adult”, “bank market” and “law school”[30], leading to b i , c i ∈ { +1 , − 1 } .

Original: The model is formulated by the minimax problem
Revised: The model is formulated by the minimax problem

Original: [Equation]
Revised: [Equation]

Original: where l 1 , l 2 are the logit functions: logit( a , b, c )
Revised: where l 1 , l 2 are the logit functions: logit( a , b, c )

Original: = log(1 + exp( − b a ⊤ c )) .
Revised: = log(1 + exp( − b a ⊤ c )) .

Original: We set the parameters β, λ and γ as 0 . 5 , 10 − 4 and 10 − 4 respectively.
Revised: We set the parameters β, λ and γ as 0 . 5 , 10 − 4 and 10 − 4 respectively.

Original: The dimension of the problem is d = m +1 .
Revised: The dimension of the problem is d = m + 1 .

Original: Since the objective function is non-quadratic, we conduct the proposed algorithms in Section 3.3 (Algorithm 5, 6 and 7) here.
Revised: Since the objective function is non-quadratic, we conduct the proposed algorithms in Section 3. (Algorithm 5, 6 and 7) here.

Original: We use extragradient as warm up to achieve the local condition for proposed algorithms.
Revised: We use extragradient as warm up to achieve the local condition for proposed algorithms.

Original: The results of iteration numbers against ∥ g ( z ) ∥ 2 and CPU time against ∥ g ( z ) ∥ 2 are presented in Figure 2, which indicate that our algorithms significantly outperform the baseline algorithm.
Revised: The results of iteration numbers against ∥ g ( z ) ∥ 2 and CPU time against ∥ g ( z ) ∥are presented in Figure 2, which indicate that our algorithms significantly outperform the baseline algorithm.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we propose quasi-Newton methods for solving strongly-convex-strongly-concave saddle point problems.
Revised: In this work, we propose quasi-Newton methods for solving strongly-convex-strongly-concave saddle point problems.

Original: We characterize the second-order information by approximating the square of
Revised: We characterize the second-order information by approximating the square of

Original: Hessian matrix, which avoids the issue of dealing with the indefinite Hessian directly.
Revised: Hessian matrix, which avoids the issue of dealing with the indefinite Hessian directly.

Original: We present the explicit local superlinear convergence rates for Broyden’s family update and faster convergence rates for two specific methods: SR1 and BFGS updates.
Revised: We present the explicit local superlinear convergence rates for Broyden’s family update and faster convergence rates for two specific methods: SR1 and BFGS updates.

Original: However, our algorithms still require to compute the exact gradient and store the approximate Hessian by O ( d 2 ) space complexity.
Revised: However, our algorithms still require to compute the exact gradient and store the approximate Hessian by O ( d 2 ) space complexity.

Original: In future work, it is interesting to design the stochastic algorithms to further reduce the computational cost of the iteration.
Revised: In future work, it is interesting to design the stochastic algorithms to further reduce the computational cost of the iteration.

Original: It is also possible to design the limited-memory quasi-Newton methods for more scalable saddle point problems.
Revised: It is also possible to design the limited-memory quasi-Newton methods for more scalable saddle point problems.

Original: Acknowledgements
Revised: Acknowledgements

Original: We would like to thank the anonymous reviewer for pointing out the mistakes in our previous proof.
Revised: We would like to thank the anonymous reviewer for pointing out the mistakes in our previous proof.

Original: We would also like to thank Tong Zhang and John C.S. Lui for giving useful suggestions.
Revised: We would also like to thank Tong Zhang and John C.S. Lui for giving useful suggestions.

Original: This work is supported by National Natural Science Foundation of China (No. 62206058) and Shanghai Sailing
Revised: This work is supported by National Natural Science Foundation of China (No. 62206058) and Shanghai Sailing

Original: Program (22YF1402900).
Revised: Program (22YF1402900).

Original: Checklist
Revised: Checklist

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: [Yes] See Section 6.
Revised: [Yes] See Section 6.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 2.
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Section 2.
Revised: [Yes] See Section 2.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix A-D. 3.
Revised: [Yes] See Appendix A-D. 3.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We present them in the supplemental material.
Revised: [Yes] We present them in the supplemental material.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?
Revised: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] We present them in Section 5 and in Appendix E. (c) Did you report error bars (e
Revised: [Yes] We present them in Section 5 and in Appendix E. (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: g., with respect to the random seed after running experiments multiple times)?

Original: [No] (d) Did you include the total amount of compute and the type of resources used (e
Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e

Original: g., type of GPUs, internal cluster, or cloud provider)?
Revised: g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix E. 4.
Revised: [Yes] See Appendix E. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 5 and Appendix E. (b) Did you mention the license of the assets?
Revised: [Yes] See Section 5 and Appendix E. (b) Did you mention the license of the assets?

Original: [Yes] See Section 5 and Appendix E, the data is public available.
Revised: [Yes] See Section 5 and Appendix E, the data is public available.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] See Appendix E. (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [Yes] See Appendix E. (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] 5.
Revised: [N/A] 5.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]