Original: HOUDINI: Escaping from Moderately Constrained Saddles
Revised: HOUDINI: Escaping from Moderately Constrained Saddles

Original: author names withheld

Original: Under Review for OPT

Original: Abstract
Revised: Abstract

Original: We give polynomial time algorithms for escaping from high-dimensional saddle points under a moderate number of constraints.
Revised: We give polynomial time algorithms for escaping from high-dimensional saddle points under a moderate number of constraints.

Original: Given gradient access to a smooth function f : R d → R we show that (noisy) gradient descent methods can escape from saddle points under a logarithmic number of inequality constraints.
Revised: Given gradient access to a smooth function f : R d → R we show that (noisy) gradient descent methods can escape from saddle points under a logarithmic number of inequality constraints.

Original: This constitutes progress (without reliance on NP-oracles or altering the definitions to only account for certain constraints) on the main open question of the breakthrough work of Ge et al.
Revised: This constitutes progress (without reliance on NP-oracles or altering the definitions to only account for certain constraints) on the main open question of the breakthrough work of Ge et al.

Original: [16] who showed an analogous result for unconstrained and equality-constrained problems.
Revised: [16] who showed an analogous result for unconstrained and equality-constrained problems.

Original: Our results hold for both regular and stochastic gradient descent.
Revised: Our results hold for both regular and stochastic gradient descent.

Original: 1. Introduction
Revised: 1. Introduction

Original: Achieving convergence of gradient descent to an (approximate) local minimum is a central question in non-convex optimization for machine learning.
Revised: Achieving convergence of gradient descent to an (approximate) local minimum is a central question in non-convex optimization for machine learning.

Original: In recent years, breakthrough progress starting with the work of Ge et al.
Revised: In recent years, breakthrough progress starting with the work of Ge et al.

Original: [16] has led to a flurry of results in this area (see e.g. Carmon and Duchi [9, 10], Carmon et al.
Revised: [16] has led to a flurry of results in this area (see e.g. Carmon and Duchi [9, 10], Carmon et al.

Original: [11], Du et al.
Revised: [11], Du et al.

Original: [14], Jin et al.
Revised: [14], Jin et al.

Original: [19, 20, 21], Mokhtari et al.
Revised: [19, 20, 21], Mokhtari et al.

Original: [23], Staib et al.
Revised: [23], Staib et al.

Original: [31]), culminating in almost optimal bounds [34].
Revised: [31]), culminating in almost optimal bounds [34].

Original: However, despite this success a key open question of [16] still remains unanswered – can gradient methods efficiently escape from saddle points in constrained non-convex optimization?
Revised: However, despite this success a key open question of [16] still remains unanswered – can gradient methods efficiently escape from saddle points in constrained non-convex optimization?

Original: In fact, even basic linear inequality constraints still remain an obstacle: “Dealing with inequality constraints is left as future work” Ge et al.
Revised: In fact, even basic linear inequality constraints still remain an obstacle: “Dealing with inequality constraints is left as future work” Ge et al.

Original: This is due to the NP-hardness of the related copositivity problem [24], which corresponds to the case when the number of constraints is linear in the dimension.
Revised: This is due to the NP-hardness of the related copositivity problem [24], which corresponds to the case when the number of constraints is linear in the dimension.

Original: In this paper, we make progress on this open question in the case when the number of constraints depends moderately on the dimension.
Revised: In this paper, we make progress on this open question in the case when the number of constraints depends moderately on the dimension.

Original: Consider a feasible set defined by k linear inequality constraints: S = { x ∈ R d | Ax ≤ b } , where A ∈ R k × d and b ∈ R k .
Revised: Consider a feasible set defined by k linear inequality constraints: S = { x ∈ R d | Ax ≤ b } , where A ∈ R k × d and b ∈ R k .

Original: Let B d ( x , r ) be a d -dimensional closed ball of radius r centred at x .
Revised: Let B d ( x , r ) be a d -dimensional closed ball of radius r centred at x .

Original: We write B ( x , r ) when the dimension is clear from the context and drop the first parameter when x = 0 .
Revised: We write B ( x , r ) when the dimension is clear from the context and drop the first parameter when x = 0 .

Original: Our goal is to minimize the objective function f : R d → R over S , i.e. min x ∈ S f
Revised: Our goal is to minimize the objective function f : R d → R over S , i.e. min x ∈ S f

Original: ( x ) .
Revised: ( x ) .

Original: We first introduce standard assumptions of boundedness and smoothness.
Revised: We first introduce standard assumptions of boundedness and smoothness.

Original: A function f is bounded if it takes values in [ − 1 , 1] which can always be achieved by rescaling.
Revised: A function f is bounded if it takes values in [ − 1 , 1] which can always be achieved by rescaling.

Original: Assumption 1 (Smoothness)
Revised: Assumption 1 (Smoothness)

Original: The objective function f satisfies the following properties:
Revised: The objective function f satisfies the following properties:

Original: (First order) f has an L -Lipschitz gradient ( f is L -smooth): ∥∇ f ( x ) − ∇ f ( y )
Revised: (First order) f has an L -Lipschitz gradient ( f is L -smooth): ∥∇ f ( x ) − ∇ f ( y )

Original: ∥
Revised: ∥

Original: ≤ L ∥ x − y ∥ , ∀ x , y ∈ R d . 2.
Revised: ≤ L ∥ x − y ∥ , ∀ x , y ∈ R d . 2.

Original: (Second order) f has a ρ -Lipschitz Hessian: ∥∇ 2 f ( x ) − ∇
Revised: (Second order) f has a ρ -Lipschitz Hessian: ∥∇ 2 f ( x ) − ∇

Original: 2 f ( y ) ∥
Revised: 2 f ( y ) ∥

Original: ≤ ρ ∥ x − y ∥ , ∀ x , y ∈ R d .
Revised: ≤ ρ ∥ x − y ∥ , ∀ x , y ∈ R d .

Original: Definition 1 (Local minimum) For S ⊆ R d , let f : R d → R .
Revised: Definition 1 (Local minimum) For S ⊆ R d , let f : R d → R .

Original: A point x ⋆ is a local minimum of f in S if and only if there exists r > 0 such that f ( x ) ≥ f
Revised: A point x ⋆ is a local minimum of f in S if and only if there exists r > 0 such that f ( x ) ≥ f

Original: ( x ⋆ ) for all x ∈ S ∩ B ( x ⋆ , r ) .
Revised: ( x ⋆ ) for all x ∈ S ∩ B ( x ⋆ , r ) .

Original: Since finding a local minimum is NP-hard even in the unconstrained case (see e.g. Anandkumar and Ge [3] and the references within) the notion of a local minimum is typically relaxed as follows.
Revised: Since finding a local minimum is NP-hard even in the unconstrained case (see e.g. Anandkumar and Ge [3] and the references within), the notion of a local minimum is typically relaxed, a natural relaxation being the following:

Original: Definition 2 (Approximate local minimum) For S ⊆ R d and f : R d → R a point x ⋆ is a ( δ, r ) -approximate local minimum if f ( x ) ≥ f
Revised: Definition 2 (Approximate local minimum) For S ⊆ R d and f : R d → R a point x ⋆ is a ( δ, r ) -approximate local minimum if f ( x ) ≥ f

Original: ( x ⋆ ) − δ for all x ∈ S ∩ B ( x ⋆ , r ) .
Revised: ( x ⋆ ) − δ for all x ∈ S ∩ B ( x ⋆ , r ) .

Original: For smooth functions one can define stationary points in terms of the gradient and the eigenvalues instead:
Revised: For smooth functions, a common approach is to instead define stationary points in terms of the gradient and the eigenvalues, which, as shown below, is motivated by the Taylor expansion of f :

Original: Definition 3 ([21, 25])
Revised: Definition 3 ([21, 25])

Original: A point x is an ε -second-order stationary point ( ε -SOSP) if ∥∇ f ( x ) ∥ <
Revised: A point x is an ε -second-order stationary point ( ε -SOSP) if ∥∇ f ( x ) ∥ <

Original: ε and λ min ( ∇ 2 f ( x )) > −√ ρε , where λ min denotes the smallest eigenvalue.
Revised: ε and λ min ( ∇ 2 f ( x )) > − √ ρε , where λ min denotes the smallest eigenvalue of the Hessian.

Original: When applying this definition to the constrained case, eigenvectors and eigenvalues are not well-defined since there might be no eigenvectors inside the feasible set, while an escaping direction might exist.
Revised: When applying this definition to the constrained case, eigenvectors and eigenvalues are not well-defined, since, even when an escaping direction exists, there might be no eigenvectors inside the feasible set.

Original: Moreover, for f ( x ) = − 12 ∥ x
Revised: Moreover, for f ( x ) = − 12 ∥ x

Original: ∥ 2 and any compact feasible set, the Hessian is − I at any point with λ min ( − I )
Revised: ∥ 2 and any compact feasible set, the Hessian is − I at any point with λ min ( − I )

Original: = − 1 .
Revised: = − 1 .

Original: Hence an ε -SOSP doesn’t exist according to the Definition 3, even though a local minimum exists.
Revised: Hence an ε -SOSP doesn’t exist according to the Definition 3, even though a local minimum exists.

Original: In fact, Definition 3 arises from the Taylor expansion, which justifies the choice of √ ρϵ as the bound on the smallest eigenvalue.
Revised: In fact, Definition 3 arises from the Taylor expansion, which justifies the choice of √ ρϵ as the bound on the smallest eigenvalue.

Original: If the function has a ρ -Lipschitz Hessian:
Revised: If the function has a ρ -Lipschitz Hessian:

Original: [Equation]
Revised: [Equation]

Original: To guarantee that the discrepancy between the function and its quadratic approximation is small relative to δ (from Definition 2), a natural choice of r is 3 (cid:112) δ / ρ , which bounds the discrepancy with Θ( δ ) .
Revised: To guarantee that the discrepancy between the function and its quadratic approximation is small relative to δ (from Definition 2), a natural choice of r is 3 (cid:112) δ / ρ , which bounds the discrepancy with Θ( δ ) .

Original: Therefore, based on the quadratic approximation, one can distinguish a ( δ, r ) -approximate local minimum from not a ( cδ, r ) -approximate local minimum for c < 1 .
Revised: Therefore, based on the quadratic approximation, one can distinguish a ( δ, r ) -approximate local minimum from not a ( cδ, r ) -approximate local minimum for c < 1 .

Original: By setting this r and selecting ε = 3 (cid:112) δ 2 ρ , we have √ ρε = 3 (cid:112) δρ 2 and for any h ∈ B ( x , r ) (see Appendix A):
Revised: By setting this r and selecting ε = 3 (cid:112) δ 2 ρ , we have √ ρε = 3 (cid:112) δρ 2 and for any h ∈ B ( x , r ) (see Appendix A):

Original: [Equation]
Revised: [Equation]

Original: Using the ball radius discussed above we arrive at the following version of Definition 2:
Revised: Using the ball radius discussed above we arrive at the following version of Definition 2:

Original: Definition 4 (Approximate SOSP)
Revised: Definition 4 (Approximate SOSP)

Original: For S ⊆ R d , let f : R d
Revised: For S ⊆ R d , let f : R d

Original: → R be a twice-differentiable function with a ρ -Lipschitz Hessian.
Revised: → R be a twice-differentiable function with a ρ -Lipschitz Hessian.

Original: A point x ⋆ is a δ -second-order stationary point ( δ -SOSP) if for r = 3 (cid:112) δ / ρ :
Revised: A point x ⋆ is a δ -second-order stationary point ( δ -SOSP) if for r = 3 (cid:112) δ / ρ :

Original: [Equation]
Revised: [Equation]

Original: 1.1. Our results and related work
Revised: 1.1. Our Results

Original: Our results hold for stochastic gradient descent (SGD):
Revised: Our results hold for stochastic gradient descent (SGD) under the following standard assumption:

Original: Assumption 2 Access to a stochastic gradient oracle g ( x ) :
Revised: Assumption 2 Access to a stochastic gradient oracle g ( x ) :

Original: (Unbiased expectation) E
Revised: (Unbiased expectation) E

Original: [ g ( x )]
Revised: [ g ( x )]

Original: = ∇ f ( x ) .
Revised: = ∇ f ( x ) .

Original: (Variance) E [ ∥ g ( x ) − ∇ f ( x ) ∥ 2 ] ≤ σ 2 .
Revised: (Variance) E [ ∥ g ( x ) − ∇ f ( x ) ∥ 2 ] ≤ σ 2 .

Original: Our main result is the following theorem which quantifies the complexity of finding an approximate SOSP under a moderate number of linear inequality constraints, showing that this problem is solvable in polynomial time for k
Revised: Our main result is the following theorem which quantifies the complexity of finding an approximate SOSP under a moderate number of linear inequality constraints, showing that this problem is solvable in polynomial time for k

Original: = O (log d ) .
Revised: = O (log d ) .

Original: We refer to a function as ( L, ρ ) -smooth if it satisfies
Revised: We refer to a function as ( L, ρ ) -smooth if it satisfies

Original: Assumption 1 and simply second-order smooth if both smoothness parameters are constant.
Revised: Assumption 1 and simply second-order smooth if both smoothness parameters are constant.

Original: Theorem 5  Let S be a set defined by an intersection of k linear inequality constraints.
Revised: Theorem 5 (Theorem 8) Let S be a set defined by an intersection of k linear inequality constraints.

Original: Let f be a second-order smooth bounded function.
Revised: Let f be a second-order smooth bounded function.

Original: Given access to a stochastic gradient oracle satisfying Assumption 2, there exists an algorithm which for any δ > 0 finds a δ -SOSP in ˜ O
Revised: Given access to a stochastic gradient oracle satisfying Assumption 2, there exists an algorithm which for any δ > 0 finds a δ -SOSP in ˜ O

Original: ( 1 δ ( d 3 2 k + d 2 σ 2 δ 4 / 3 )) time using ˜ O ( dδ (1+ dσ 2 δ 4 / 3 )) stochastic gradient oracle calls.
Revised: ( 1 δ ( d 3 2 k + d 2 σδ 4 / 3 )) time using ˜ O ( 1 δ ( d + d 3 σ 2 δ 4 / 3 )) stochastic gradient oracle calls.

Original: In the deterministic gradient case ( σ = 0 ), the time complexity is ˜ O
Revised: In the deterministic gradient case ( σ = 0 ), the time complexity is ˜ O

Original: ( d 3 2 k δ ) and the number of gradient oracle calls is ˜ O
Revised: ( d 3 2 k δ ) and the number of gradient oracle calls is ˜ O

Original: ( dδ ) .
Revised: ( dδ ) .

Original: The exponential dependence of time complexity on k in our results (not required in the oracle calls) is most likely unavoidable due to the following hardness result, which implies that when k
Revised: The exponential dependence of time complexity on k in our results (not required in the oracle calls) is most likely unavoidable due to the following hardness result, which implies that when k

Original: = d then the complexity of this problem can’t be polynomial in d under the standard hardness assumptions.
Revised: = d then the complexity of this problem can’t be polynomial in d under the standard hardness assumptions.

Original: Remark 6 (Matrix copositivity [24]) For a quadratic function f ( x ) =
Revised: Remark 6 (Matrix copositivity [24]) For a quadratic function f ( x ) =

Original: x T Mx subject to constraints x i ≥ 0 for all i , it is NP-hard to decide whether there exists a solution with f ( x ) > 0 .
Revised: x T Mx subject to constraints x i ≥ 0 for all i , it is NP-hard to decide whether there exists a solution with f ( x ) > 0 .

Revised: 1.2. Related Work

Original: Related results in convex optimization are covered in Boyd and Vandenberghe [7], Bubeck et al.
Revised: Related results in convex optimization are covered in Boyd and Vandenberghe [7], Bubeck et al.

Original: Among related results in non-convex optimization here we only focus on the algorithms which only use gradient information.
Revised: Among related results in non-convex optimization here we only focus on the algorithms which only use gradient information.

Original: Unconstrained optimization
Revised: Unconstrained optimization

Original: Recall that an ϵ -first-order stationary point ( ϵ -FOSP) is defined so that ∥∇ f ( x ) ∥ ≤ ϵ .
Revised: Recall that an ϵ -first-order stationary point ( ϵ -FOSP) is defined so that ∥∇ f ( x ) ∥ ≤ ϵ .

Original: Analyses of convergence to an ε -FOSP are a cornerstone of non-convex optimization (see e.g. classic texts Bertsekas [5], Nocedal and Wright [26]).
Revised: Analyses of convergence to an ε -FOSP are a cornerstone of non-convex optimization (see e.g. classic texts Bertsekas [5], Nocedal and Wright [26]).

Original: Quantitative analysis of convergence to an ϵ
Revised: Quantitative analysis of convergence to an ϵ

Original: -SOSP
Revised: -SOSP

Original: (Definition 3) started with the breakthrough work by Ge et al.
Revised: (Definition 3) started with the breakthrough work by Ge et al.

Original: [16] further refined in Carmon and Duchi [9, 10], Jin et al.
Revised: [16] further refined in Carmon and Duchi [9, 10], Jin et al.

Original: [19, 20, 21] and most recently in Zhang and Li [34], who show an almost optimal bound.
Revised: [19, 20, 21] and most recently in Zhang and Li [34], who show an almost optimal bound.

Original: Due to the prevalence of SGD in deep learning, stochastic methods have attracted the most attention (see Allen-Zhu [1], Allen-Zhu and Li [2], Fang et al.
Revised: Due to the prevalence of SGD in deep learning, stochastic methods have attracted the most attention (see Allen-Zhu [1], Allen-Zhu and Li [2], Fang et al.

Original: [15], Tripuraneni et al.
Revised: [15], Tripuraneni et al.

Original: [32], Xu et al.
Revised: [32], Xu et al.

Original: [33], Zhou and Gu [35], Zhou et al.
Revised: [33], Zhou and Gu [35], Zhou et al.

Original: [36] for the case of Lipschitz gradients and Daneshmand et al.
Revised: [36] for the case of Lipschitz gradients and Daneshmand et al.

Original: [13], Ge et al.
Revised: [13], Ge et al.

Original: [16] for non-Lipschitz gradients).
Revised: [16] for non-Lipschitz gradients).

Original: For an in-depth summary of the previous work on unconstrained non-convex optimization we refer the reader to Jin et al.
Revised: For an in-depth summary of the previous work on unconstrained non-convex optimization we refer the reader to Jin et al.

Original: Constrained optimization
Revised: Constrained optimization

Original: The case of equality constraints is typically reducible to the unconstrained case by using Lagrangian multipliers (see e.g. Ge et al.
Revised: The case of equality constraints is typically reducible to the unconstrained case by using Lagrangian multipliers (see e.g. Ge et al.

Original: However, the general constrained case is substantially more challenging since even the definitions of stationarity require a substantial revision.
Revised: However, the general constrained case is substantially more challenging since even the definitions of stationarity require a substantial revision.

Original: For first-order convergence a rich literature exists, covering projected gradient,
Revised: For first-order convergence a rich literature exists, covering projected gradient,

Original: Frank-Wolfe, cubic regularization, etc (see e.g. Mokhtari et al.
Revised: Frank-Wolfe, cubic regularization, etc (see e.g. Mokhtari et al.

Original: [23] and the references within).
Revised: [23] and the references within).

Original: For second-order convergence, the landscape of existing work is substantially sparser due to
Revised: For second-order convergence, the landscape of existing work is substantially sparser due to

Original: NP-hardness (Remark 6, [24]).
Revised: NP-hardness (Remark 6, [24]).

Original: A large body of work focuses on achieving convergence using various forms of
Revised: A large body of work focuses on achieving convergence using various forms of

Original: NP-oracles (see e.g. Bian et al.
Revised: NP-oracles (see e.g. Bian et al.

Original: [6], Cartis et al.
Revised: [6], Cartis et al.

Original: [12], Haeser et al.
Revised: [12], Haeser et al.

Original: [17], Mokhtari et al.
Revised: [17], Mokhtari et al.

Original: [23], Nouiehed and Razaviyayn [27]), while another approach is to define stationarity in terms of tight constraints only [4, 22].
Revised: [23], Nouiehed and Razaviyayn [27]), while another approach is to define stationarity in terms of tight constraints only [4, 22].

Original: Relationship with other definitions of SOSP
Revised: Relationship with other definitions of SOSP

Original: As discussed in Remark 6, second-order constrained optimization is NP-hard due to the hardness of the matrix copositivity problem.
Revised: As discussed in Remark 6, second-order constrained optimization is NP-hard due to the hardness of the matrix copositivity problem.

Original: Definitions of constrained SOSP in the previous work fall into two categories: 1) definitions that only consider active constraints (“active constraints only” definitions), 2) definitions that preserve the NP-hardness of the problem and rely on NP-oracles to achieve polynomial-time convergence: 1.
Revised: Definitions of constrained SOSP in the previous work fall into two categories: 1) definitions that only consider active constraints (“active constraints only” definitions), 2) definitions that preserve the NP-hardness of the problem and rely on NP-oracles to achieve polynomial-time convergence: 1.

Original: (“Active constraints only”)
Revised: (“Active constraints only”)

Original: In Avdiukhin et al.
Revised: In Avdiukhin et al.

Original: [4], Lu et al.
Revised: [4], Lu et al.

Original: [22] definitions analogous to
Revised: [22] definitions analogous to

Original: Definition 3, and the second-order conditions are given with respect to the set of active (i.e. tight for the current iterate) constraints.
Revised: Definition 3, and the second-order conditions are given with respect to the set of active (i.e. tight for the current iterate) constraints.

Original: This allows bypassing the NP-hardness since the point at which the hardness of the copositivity problem applies now becomes a stationary point by definition.
Revised: This allows bypassing the NP-hardness since the point at which the hardness of the copositivity problem applies now becomes a stationary point by definition.

Original: (NP-hard)
Revised: (NP-hard)

Original: In the results relying on NP-oracles (e.g. [6, 17, 23, 27]) the complexity is shifted to solving black-box quadratic optimization problems of a certain type.
Revised: In the results relying on NP-oracles (e.g. [6, 17, 23, 27]) the complexity is shifted to solving black-box quadratic optimization problems of a certain type.

Original: A key advantage of these types of approaches is that they can handle an arbitrary number of constraints and hence promising in certain machine learning applications.
Revised: A key advantage of these types of approaches is that they can handle an arbitrary number of constraints and hence promising in certain machine learning applications.

Original: What is currently lacking in the state of the art is a quantitative analysis of the complexity of convergence to a second-order stationary point, which shows full dependence on both the dimension and accuracy while defining stationarity with respect to the full set of constraints, instead of just active constraints only 2 .
Revised: What is currently lacking in the state of the art is a quantitative analysis of the complexity of convergence to a second-order stationary point, which shows full dependence on both the dimension and accuracy while defining stationarity with respect to the full set of constraints, instead of just active constraints only 2 .

Original: Our goal in Theorem 5 is to address this gap and give such an analysis.
Revised: Our goal in Theorem 5 is to address this gap and give such an analysis.

Original: 2. Main Result
Revised: 2. Main Result

Original: We address the NP-hardness of the copositivity problem by focusing on the case of a moderate number of constraints and arguing that it can be addressed using gradient-based methods.
Revised: We address the NP-hardness of the copositivity problem by focusing on the case of a moderate number of constraints and arguing that it can be addressed using gradient-based methods.

Original: In order to streamline the presentation, we first focus in Section 2.1 on the key challenge of escaping from a saddle point in a corner defined by the constraints when the underlying function is simply quadratic (see details in Section B).
Revised: In order to streamline the presentation, we first focus in Section 2.1 on the key challenge of escaping from a saddle point in a corner defined by the constraints when the underlying function is simply quadratic (see details in Section B).

Original: This is already enough to capture some of the key contributions.
Revised: This is already enough to capture some of the key contributions.

Original: Our general result is presented in Section 2.2, while more technical details and the full algorithm are given in Appendix C.
Revised: Our general result is presented in Section 2.2, while more technical details and the full algorithm are given in Appendix C.

Original: 2.1. Quadratic corner saddle point
Revised: 2.1. Quadratic corner saddle point

Original: In this simplified scenario we consider the case when the function is quadratic without a linear term and the saddle point is located in the corner of the constrained space.
Revised: In this simplified scenario we consider the case when the function is quadratic without a linear term and the saddle point is located in the corner of the constrained space.

Original: The NP-hardness comes from the fact that the point we aim to find can lie in the intersection of an arbitrary subset of constraints.
Revised: The NP-hardness comes from the fact that the point we aim to find can lie in the intersection of an arbitrary subset of constraints.

Original: By doing an exhaustive search over this set of constraints (Algorithm 1) and enforcing them throughout the search process we are able to reduce to a setting similar to the equalityconstrained case (Algorithm 2).
Revised: By doing an exhaustive search over this set of constraints (Algorithm 1) and enforcing them throughout the search process we are able to reduce to a setting similar to the equalityconstrained case (Algorithm 2).

Original: We show different subsets of constraints enforced by the algorithm
Revised: We show different subsets of constraints enforced by the algorithm

Original: −− 0 . 0 .
Revised: −− 0 . 0 .

Original: − 1 −−   x y (
Revised: − 1 −−   x y (

Original: a ) Feasible set with two inequality constraints:
Revised: a ) Feasible set with two inequality constraints:

Original: x ≤ 0 , y ≤ 0 .
Revised: x ≤ 0 , y ≤ 0 .

Original: −− 0 . 0 .
Revised: −− 0 . 0 .

Original: − 1 −−   x y
Revised: − 1 −−   x y

Original: −− 0 . 0 .
Revised: −− 0 . 0 .

Original: − 1 −−   x y
Revised: − 1 −−   x y

Original: ( c ) Active constraint x = 0 (green) with an escape direction (0 , − 1) .
Revised: ( c ) Active constraint x = 0 (green) with an escape direction (0 , − 1) .

Original: As shown in Lemma 15, out of two escape directions (0 , − 1) and (0 , 1) in this constraint, at least one (0 , − 1) (blue) lies in the feasible set, and is found by the algorithm.
Revised: As shown in Lemma 15, out of two escape directions (0 , − 1) and (0 , 1) in this constraint, at least one (0 , − 1) (blue) lies in the feasible set, and is found by the algorithm.

Original: −− 0 . 0 .
Revised: −− 0 . 0 .

Original: −  −−
Revised: −  −−

Original:   x y ( d
Revised:   x y ( d

Original: ) When no constraints are active, the algorithm finds red directions (negative eigenvectors) outside the feasible set.
Revised: ) When no constraints are active, the algorithm finds red directions (negative eigenvectors) outside the feasible set.

Original: Note that escape directions with no active constraints exist (blue).
Revised: Note that escape directions with no active constraints exist (blue).

Original: Lemma 15 guarantees that we find them in some constrained space (Fig-
Revised: Lemma 15 guarantees that we find them in some constrained space (Fig-

Original: ure 1(c)
Revised: ure 1(c)

Original: √
Revised: √

Original: [Equation]
Revised: [Equation]

Original: + 12 y
Revised: + 12 y

Original: [Equation]
Revised: [Equation]

Original: 2 − ( − 12 x + √ y ) constraints x, y ≤ 0 .
Revised: 2 − ( − 12 x + √ y ) constraints x, y ≤ 0 .

Original: f is a composition of g ( x, y ) =
Revised: f is a composition of g ( x, y ) =

Original: x 2 − y 2 and rotation by in an example in Figure 1.
Revised: x 2 − y 2 and rotation by in an example in Figure 1.

Original: The key challenge is making this argument formal and arguing that this process converges to a constrained approximate SOSP as in Definition 4.
Revised: The key challenge is making this argument formal and arguing that this process converges to a constrained approximate SOSP as in Definition 4.

Original: This relies on performing a robust analysis of the properties of the points which lead to the hardness of copositivity.
Revised: This relies on performing a robust analysis of the properties of the points which lead to the hardness of copositivity.

Original: In particular, we show that after we guess the set of constraints correctly, the problem reduces to finding the smallest eigenvector (Lemma 14).
Revised: In particular, we show that after we guess the set of constraints correctly, the problem reduces to finding the smallest eigenvector (Lemma 14).

Original: An exact error analysis of the eigenvector process (Lemma 15,
Revised: An exact error analysis of the eigenvector process (Lemma 15,

Original: Lemma 16) is then required to complete the proof of the main theorem for this case.
Revised: Lemma 16) is then required to complete the proof of the main theorem for this case.

Original: Theorem 7 (Quadratic Corner Saddle Point Escape) Let δ, r > 0 .
Revised: Theorem 7 (Quadratic Corner Saddle Point Escape) Let δ, r > 0 .

Original: Let f ( x ) = 12 x ⊤ Mx with λ | max | ( M ) ≤ L and let S = { x | Ax ≤ 0 } be defined by k linear inequalies.
Revised: Let f ( x ) = 12 x ⊤ Mx with λ | max | ( M ) ≤ L and let S = { x | Ax ≤ 0 } be defined by k linear inequalies.

Original: If min x ∈B ( r ) ∩ S f ( x ) < − δ , then there exists an Algorithm which with probability at least 1 − ξ finds a point x ∈ S ∩ B ( r ) with f ( x ) < − Ω( δ ) using O (cid:16)
Revised: If min x ∈B ( r ) ∩ S f ( x ) < − δ , then there exists an Algorithm which with probability at least 1 − ξ finds a point x ∈ S ∩ B ( r ) with f ( x ) < − Ω( δ ) using O (cid:16)

Original: Lr 2 k 2 k δ log 1 ξ (cid:17) deterministic gradient oracle calls.
Revised: Lr 2 k 2 k δ log 1 ξ (cid:17) deterministic gradient oracle calls.

Original: The full proof of Theorem 7 is presented in Appendix B.
Revised: The full proof of Theorem 7 is presented in Appendix B.

Original: 2.2. General case
Revised: 2.2. General case

Original: In the algorithm for the general case, we address the assumptions made in the quadratic corner saddle point case, while also handling the stochasticity in the gradient.
Revised: In the algorithm for the general case, we address the assumptions made in the quadratic corner saddle point case, while also handling the stochasticity in the gradient.

Original: The latter part is standard and is handled via variance reduction in Appendix C.3.
Revised: The latter part is standard and is handled via variance reduction in Appendix C.3.

Original: The full algorithm iterates the escape subroutine (Algorithm 3) until an escaping point is found.
Revised: The full algorithm iterates the escape subroutine (Algorithm 3) until an escaping point is found.

Original: The escape subroutine first approximates the Hessian matrix using the gradient oracle and then performs an exhaustive search over the set of active constraints at the escaping point in a way similar to the quadratic corner case.
Revised: The escape subroutine first approximates the Hessian matrix using the gradient oracle and then performs an exhaustive search over the set of active constraints at the escaping point in a way similar to the quadratic corner case.

Original: After the correct subset of constraints is fixed, the current iterate needs to be projected on this set of constraints, which also necessitates a recomputation of various related parameters.
Revised: After the correct subset of constraints is fixed, the current iterate needs to be projected on this set of constraints, which also necessitates a recomputation of various related parameters.

Original: When this is done, Algorithm(analogous to Algorithm 2 from the quadratic corner case) finds an escape direction for the resulting quadratic function.
Revised: When this is done, Algorithm(analogous to Algorithm 2 from the quadratic corner case) finds an escape direction for the resulting quadratic function.

Original: The algorithm tries to find an escaping direction within a ball of radius r = 3 (cid:112) δ / ρ (from
Revised: The algorithm tries to find an escaping direction within a ball of radius r = 3 (cid:112) δ / ρ (from

Original: Definition 4), where the function is well approximated by a quadratic one (by the smoothness assumption, as discussed above).
Revised: Definition 4), where the function is well approximated by a quadratic one (by the smoothness assumption, as discussed above).

Original: However, compared with the previous section, the function can have a non-negligible linear term – either due to the gradient or as a result of projection on the affine plane – which significantly complicates the algorithm and analysis.
Revised: However, compared with the previous section, the function can have a non-negligible linear term – either due to the gradient or as a result of projection on the affine plane – which significantly complicates the algorithm and analysis.

Original: We consider the following cases:
Revised: We consider the following cases:

Original: Case 1.
Revised: Case 1.

Original: When the linear term is very large, then either the optimum
Revised: When the linear term is very large, then either the optimum

Original: lies on the boundary of the feasible set S (which is not possible in the case when we guessed the active constraints correctly) or the linear term dominates the quadratic term and is sufficient to escape from a saddle point.
Revised: lies on the boundary of the feasible set S (which is not possible in the case when we guessed the active constraints correctly) or the linear term dominates the quadratic term and is sufficient to escape from a saddle point.

Original: In particular, this case covers the situation when the objective is sensitive to the change in the argument (the quadratic term is not sensitive due to the smoothness condition, which bounds its largest eigenvalue).
Revised: In particular, this case covers the situation when the objective is sensitive to the change in the argument (the quadratic term is not sensitive due to the smoothness condition, which bounds its largest eigenvalue).

Original: Case 2.
Revised: Case 2.

Original: When the optimum lies in the interior of the ball of radius r , none of the constraints are active, and hence we can find the unique unconstrained critical point directly: we find a point where the gradient is zero, which only requires solving a linear system.
Revised: When the optimum lies in the interior of the ball of radius r , none of the constraints are active, and hence we can find the unique unconstrained critical point directly: we find a point where the gradient is zero, which only requires solving a linear system.

Original: Case 3.
Revised: Case 3.

Original: When the optimum lies on the boundary of the ball of radius r , the only active constraint is c ( y ) =
Revised: When the optimum lies on the boundary of the ball of radius r , the only active constraint is c ( y ) =

Original: ∥
Revised: ∥

Original: y − x ∥ 2 − r 2 = 0 .
Revised: y − x ∥ 2 − r 2 = 0 .

Original: By KKT conditions, for y to be the minimizer, there must exist µ such that ∇ f ( y ) =
Revised: By KKT conditions, for y to be the minimizer, there must exist µ such that ∇ f ( y ) =

Original: µ ∇ c ( y ) .
Revised: µ ∇ c ( y ) .

Original: We show that for each µ , there exists a unique y ( µ ) satisfying the condition above, and only for O ( d ) values of µ , y ( µ ) lies on the boundary of the ball, resulting in
Revised: We show that for each µ , there exists a unique y ( µ ) satisfying the condition above, and only for O ( d ) values of µ , y ( µ ) lies on the boundary of the ball, resulting in

Original: O ( d ) candidate solutions.
Revised: O ( d ) candidate solutions.

Original: To find these candidates, we first diagonalize the matrix using an orthogonal transformation.
Revised: To find these candidates, we first diagonalize the matrix using an orthogonal transformation.

Original: This gives a quadratic function whose critical points on the boundary of the ball can be found explicitly (up to the required precision) as roots of certain polynomials.
Revised: This gives a quadratic function whose critical points on the boundary of the ball can be found explicitly (up to the required precision) as roots of certain polynomials.

Original: Here we note that the diagonalization performed in this case is the most computationally expensive step in the algorithm, resulting in polynomial dependence on the dimension.
Revised: Here we note that the diagonalization performed in this case is the most computationally expensive step in the algorithm, resulting in polynomial dependence on the dimension.

Original: We formalize our main result in the following theorem:
Revised: We formalize our main result in the following theorem:

Original: Theorem 8 Let S =
Revised: Theorem 8 Let S =

Original: { x | Ax ≤ b } be a set defined by an intersection of k linear inequality constraints.
Revised: { x | Ax ≤ b } be a set defined by an intersection of k linear inequality constraints.

Original: Let f satisfy Assumptions 1 and 2 and let min x ∈ S
Revised: Let f satisfy Assumptions 1 and 2 and let min x ∈ S

Original: f ( x ) =
Revised: f ( x ) =

Original: f ⋆ .
Revised: f ⋆ .

Original: Then there exists an algorithm which for δ > 0 finds a δ -SOSP in ˜ O
Revised: Then there exists an algorithm which for δ > 0 finds a δ -SOSP in ˜ O

Original: ( f ( x 0 ) − f
Revised: ( f ( x 0 ) − f

Original: ⋆ δ d 3 (2 k + σ 2 δ 4 / 3 )) time using ˜ O
Revised: ⋆ δ d 3 (2 k + σ 2 δ 4 / 3 )) time using ˜ O

Original: ( f ( x 0 ) − f ⋆ δ ( d + d 3 σδ 4 / 3 )) stochastic gradient oracle calls.
Revised: ( f ( x 0 ) − f ⋆ δ ( d + d 3 σδ 4 / 3 )) stochastic gradient oracle calls.

Original: Next, we outline the proof of Theorem 8.
Revised: Next, we outline the proof of Theorem 8.

Original: If x is not a δ -SOSP, our algorithm finds a point y ∈ S ∩ B ( x , r ) which significantly decreases function value: f ( y ) <
Revised: If x is not a δ -SOSP, our algorithm finds a point y ∈ S ∩ B ( x , r ) which significantly decreases function value: f ( y ) <

Original: f ( x ) − Ω( δ ) .
Revised: f ( x ) − Ω( δ ) .

Original: Therefore, if x 0 is the initial point, our algorithm requires O ( f ( x 0 )
Revised: Therefore, if x 0 is the initial point, our algorithm requires O ( f ( x 0 )

Original: − f ⋆ δ ) iterations.
Revised: − f ⋆ δ ) iterations.

Original: As we show in Appendix C.1, we can consider a quadratic approximation of the objective.
Revised: As we show in Appendix C.1, we can consider a quadratic approximation of the objective.

Original: By guessing which constraints are active at the minimizer x ⋆ and enforcing these constraints, we restrict the function to some affine subspace A .
Revised: By guessing which constraints are active at the minimizer x ⋆ and enforcing these constraints, we restrict the function to some affine subspace A .

Original: By parameterizing A , we eliminate enforced constraints, and, since the rest of the constraints are not active at x ⋆ , we need to optimize a quadratic function in the intersection of a ball and linear inequality constraints.
Revised: By parameterizing A , we eliminate enforced constraints, and, since the rest of the constraints are not active at x ⋆ , we need to optimize a quadratic function in the intersection of a ball and linear inequality constraints.

Original: The full proof is presented in Appendix C.
Revised: The full proof is presented in Appendix C.

Original: In conclusion, we prove that, given a logarithmic number of constraints, it’s possible to escape from constrained saddle points in polynomial time using stochastic gradient oracles.
Revised: In conclusion, we prove that, given a logarithmic number of constraints, it’s possible to escape from constrained saddle points in polynomial time using stochastic gradient oracles.

Original: Appendix A. Notation and Standard Facts
Revised: Appendix A. Notation and Standard Facts

Original: Notation
Revised: Notation

Original: For a set S let Int S be its interior and ∂S be its boundary.
Revised: For a set S let Int S be its interior and ∂S be its boundary.

Original: For x ∈ R d and S ⊆ R d ,
Revised: For x ∈ R d and S ⊆ R d ,

Original: Proj S ( x ) = argmin y ∈ S ∥ x − y ∥ is the projection of x on S .
Revised: Proj S ( x ) = argmin y ∈ S ∥ x − y ∥ is the projection of x on S .

Original: For a square matrix M with eigenvalues λ 1 ≤ . . .
Revised: For a square matrix M with eigenvalues λ 1 ≤ . . .

Original: ≤ λ d , we denote λ min ( M ) =
Revised: ≤ λ d , we denote λ min ( M ) =

Original: λ 1 and λ | max | ( M ) =
Revised: λ 1 and λ | max | ( M ) =

Original: max( | λ 1 | , | λ d | ) .
Revised: max( | λ 1 | , | λ d | ) .

Original: For S = { x ∈ R d |
Revised: For S = { x ∈ R d |

Original: Ax ≤ b } and x ∈ S , we say that i -th constraint is active at x if A ⊤ i x
Revised: Ax ≤ b } and x ∈ S , we say that i -th constraint is active at x if A ⊤ i x

Original: = b i , where A i is the i -th row of A .
Revised: = b i , where A i is the i -th row of A .

Original: ˜ O notation hides polylogarithmic dependence on all parameters, including error probability.
Revised: ˜ O notation hides polylogarithmic dependence on all parameters, including error probability.

Original: Fact 9 (Efficient projection on a polyhedron)
Revised: Fact 9 (Efficient projection on a polyhedron)

Original: For an arbitrary point x ∈ R d and a set S
Revised: For an arbitrary point x ∈ R d and a set S

Original: = { x ∈
Revised: = { x ∈

Original: R d | Ax ≤ b } – an intersection of k linear inequality constraints, we can find a projection of x onto
Revised: R d | Ax ≤ b } – an intersection of k linear inequality constraints, we can find a projection of x onto

Original: S in time poly( d ) .
Revised: S in time poly( d ) .

Original: Fact 10 (Convergence of power iteration)
Revised: Fact 10 (Convergence of power iteration)

Original: Let
Revised: Let

Original: M ∈ R
Revised: M ∈ R

Original: d × d be a PSD matrix with largest eigenvalue λ | max | .
Revised: d × d be a PSD matrix with largest eigenvalue λ | max | .

Original: Let a sequence { x t } ∞ t =0 in R d be defined according to the power iteration: x 0 ∼
Revised: Let a sequence { x t } ∞ t =0 in R d be defined according to the power iteration: x 0 ∼

Original: N ( 0 , I ) and x t +1 =
Revised: N ( 0 , I ) and x t +1 =

Original: Mx t .
Revised: Mx t .

Original: Then for δ < λ | max | we have x ⊤ T Mx T ∥ x T ∥ 2 > δ with probability 1 − O ( ξ ) after T
Revised: Then for δ < λ | max | we have x ⊤ T Mx T ∥ x T ∥ 2 > δ with probability 1 − O ( ξ ) after T

Original: = O (cid:16) log( dδ ) − log( ξ ( λ | max | − δ )
Revised: = O (cid:16) log( dδ ) − log( ξ ( λ | max | − δ )

Original: ) log λ | max | / δ (cid:17) iterations.
Revised: ) log λ | max | / δ (cid:17) iterations.

Original: Proof If λ i < δ , the corresponding component of the vector decreases exponentially with the rate of at least δ / λ | max | and is eventually dominated by the component corresponding to λ | max | .
Revised: Proof If λ i < δ , the corresponding component of the vector decreases exponentially with the rate of at least δ / λ | max | and is eventually dominated by the component corresponding to λ | max | .

Original: This leaves us only with components corresponding to λ
Revised: This leaves us only with components corresponding to λ

Original: i ≥ δ , finishing the proof.
Revised: i ≥ δ , finishing the proof.

Original: Let λ 1 , . . .
Revised: Let λ 1 , . . .

Original: , λ d be the eigenvalues of M and let e 1 , . . .
Revised: , λ d be the eigenvalues of M and let e 1 , . . .

Original: , e d be the corresponding eigenvectors.
Revised: , e d be the corresponding eigenvectors.

Original: Then x 0 = (cid:80) di
Revised: Then x 0 = (cid:80) di

Original: =1 ⟨ x 0 , e i ⟩ e i and x T = (cid:80) di
Revised: =1 ⟨ x 0 , e i ⟩ e i and x T = (cid:80) di

Original: =1 ⟨ x 0 , e i ⟩ λ
Revised: =1 ⟨ x 0 , e i ⟩ λ

Original: Ti e i .
Revised: Ti e i .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: It suffices to show that the numerator in the last term is non-negative.
Revised: It suffices to show that the numerator in the last term is non-negative.

Original: Clearly, in the summation, the terms with λ i ≥ δ are positive and the terms with λ
Revised: Clearly, in the summation, the terms with λ i ≥ δ are positive and the terms with λ

Original: i < δ are negative.
Revised: i < δ are negative.

Original: Hence, we must lower-bound ⟨ x 0 , e i ⟩ 2 when λ i ≥ δ and upper-bound ⟨ x 0 , e i ⟩ 2 when λ i < δ .
Revised: Hence, we must lower-bound ⟨ x 0 , e i ⟩ 2 when λ i ≥ δ and upper-bound ⟨ x 0 , e i ⟩ 2 when λ i < δ .

Original: Since x 0 ∼ N ( 0 , I ) , the following holds for all i ∈ [ d ] :
Revised: Since x 0 ∼ N ( 0 , I ) , the following holds for all i ∈ [ d ] :

Original: • With probability 1 − O ( ξ / d ) , |⟨ x 0 , e i ⟩| ≥ ξ / d . • With probability 1 − O ( ξ / d ) , |⟨ x 0 , e i ⟩| ≤ log d / ξ .
Revised: • With probability 1 − O ( ξ / d ) , |⟨ x 0 , e i ⟩| ≥ ξ / d . • With probability 1 − O ( ξ / d ) , |⟨ x 0 , e i ⟩| ≤ log d / ξ .

Original: By the union bound, these inequalities hold for all i with probability 1 − O ( ξ ) .
Revised: By the union bound, these inequalities hold for all i with probability 1 − O ( ξ ) .

Original: For i such that
Revised: For i such that

Original: λ
Revised: λ

Original: i = λ | max | , we have
Revised: i = λ | max | , we have

Original: [Equation]
Revised: [Equation]

Original: For all i such that λ i < δ , we have:
Revised: For all i such that λ i < δ , we have:

Original: [Equation]
Revised: [Equation]

Original: i ⟩ 2 ( λ i / λ | max | ) 2 T ( λ i − δ ) ≥ − δ ( δ / λ | max
Revised: i ⟩ 2 ( λ i / λ | max | ) 2 T ( λ i − δ ) ≥ − δ ( δ / λ | max

Original: [Equation]
Revised: [Equation]

Original: ) 2 T log 2 d / ξ .
Revised: ) 2 T log 2 d / ξ .

Original: Since there are at most d such terms, to guarantee that (cid:80) di
Revised: Since there are at most d such terms, to guarantee that (cid:80) di

Original: [Equation]
Revised: [Equation]

Original: − δ
Revised: − δ

Original: [Equation]
Revised: [Equation]

Original: ≥
Revised: ≥

Original: [Equation]
Revised: [Equation]

Original: , it suffices to have:
Revised: , it suffices to have:

Original: [Equation]
Revised: [Equation]

Original: which holds when
Revised: which holds when

Original: [Equation]
Revised: [Equation]

Original: Fact 11 (Coordinate-wise median trick) Let ξ (1) , . . .
Revised: Fact 11 (Coordinate-wise median trick) Let ξ (1) , . . .

Original: , ξ ( n ) be independent random variables in
Revised: , ξ ( n ) be independent random variables in

Original: R d sampled from the same distribution with mean E [ ξ ( i ) ]
Revised: R d sampled from the same distribution with mean E [ ξ ( i ) ]

Original: = µ and variance E [ ∥ ξ (1) − µ ∥ 2 ] ≤ σ 2 .
Revised: = µ and variance E [ ∥ ξ (1) − µ ∥ 2 ] ≤ σ 2 .

Original: If ζ is a coordinate-wise median of ξ 1 , . . . , ξ n , then
Revised: If ζ is a coordinate-wise median of ξ 1 , . . . , ξ n , then

Original: [Equation]
Revised: [Equation]

Original: Proof The proof idea is to apply the median trick to each coordinate separately.
Revised: Proof The proof idea is to apply the median trick to each coordinate separately.

Original: For each ξ ( i ) we have:
Revised: For each ξ ( i ) we have:

Original: [Equation]
Revised: [Equation]

Original: For each coordinate j , let σ 2 j = E
Revised: For each coordinate j , let σ 2 j = E

Original: [( ξ ( i ) j − µ j ) 2 ] .
Revised: [( ξ ( i ) j − µ j ) 2 ] .

Original: Using Chebyshev’s inequality, we have:
Revised: Using Chebyshev’s inequality, we have:

Original: [Equation]
Revised: [Equation]

Original: Let X i,j be random variables such that X i,j = 1 if ( ξ ( i ) j − µ j ) 2 ≥ 4 σ 2 j and X i,j = 0 otherwise.
Revised: Let X i,j be random variables such that X i,j = 1 if ( ξ ( i ) j − µ j ) 2 ≥ 4 σ 2 j and X i,j = 0 otherwise.

Original: From the above inequality, we have E [ (cid:80) ni =
Revised: From the above inequality, we have E [ (cid:80) ni =

Original: 1 X i,j ] ≤ n 4 .
Revised: 1 X i,j ] ≤ n 4 .

Original: Since X 1 ,j , . . .
Revised: Since X 1 ,j , . . .

Original: , X n,j are bounded and independent, by the Chernoff bound we have:
Revised: , X n,j are bounded and independent, by the Chernoff bound we have:

Original: [Equation]
Revised: [Equation]

Original: Hence, for any j , with probability 1 − e − Ω( n ) at most half of X i,j are 1 .
Revised: Hence, for any j , with probability 1 − e − Ω( n ) at most half of X i,j are 1 .

Original: Therefore, at least half of ξ (1) j , . .
Revised: Therefore, at least half of ξ (1) j , . .

Original: . , ξ ( n ) j lie in [ µ j − σ j , µ j + σ j ] , and hence the same holds for their median ζ j .
Revised: . , ξ ( n ) j lie in [ µ j − σ j , µ j + σ j ] , and hence the same holds for their median ζ j .

Original: Taking the union bound over all coordinates, with probability 1 − d · e − Ω( n ) we have:
Revised: Taking the union bound over all coordinates, with probability 1 − d · e − Ω( n ) we have:

Original: [Equation]
Revised: [Equation]

Original: Appendix B. Quadratic Corner Saddle Point Case
Revised: Appendix B. Quadratic Corner Saddle Point Case

Original: We introduce the key ideas of the analysis in a simplified setting when: 1) the function f is quadratic, 2) the gradient is small, 3) the current iterate is located in a corner of the constraint space.
Revised: We introduce the key ideas of the analysis in a simplified setting when: 1) the function f is quadratic, 2) the gradient is small, 3) the current iterate is located in a corner of the constraint space.

Original: Intuitively, this represents the key challenge of the constrained saddle escape problem since its NP-hardness comes from the hardness of the matrix copositivity problem in Remark 6 (i.e. the function is exactly quadratic and has no gradient at the current iterate which lies in the intersection of all constraints).
Revised: Intuitively, this represents the key challenge of the constrained saddle escape problem since its NP-hardness comes from the hardness of the matrix copositivity problem in Remark 6 (i.e. the function is exactly quadratic and has no gradient at the current iterate which lies in the intersection of all constraints).

Original: We refer to this setting as the Quadratic Corner Saddle Point problem defined formally below.
Revised: We refer to this setting as the Quadratic Corner Saddle Point problem defined formally below.

Original: By shifting the coordinate system, w.l.o.g. we can assume that the saddle point is 0 and f ( 0 )
Revised: By shifting the coordinate system, w.l.o.g. we can assume that the saddle point is 0 and f ( 0 )

Original: = 0 3 .
Revised: = 0 3 .

Original: Ifis a ( δ, r )-QCSP
Revised: Ifis a ( δ, r )-QCSP

Original: (as defined below), the objective can be decreased by δ within a ball of radius r .
Revised: (as defined below), the objective can be decreased by δ within a ball of radius r .

Original: Definition 12 (Quadratic Corner Saddle Point) Let S
Revised: Definition 12 (Quadratic Corner Saddle Point) Let S

Original: =
Revised: =

Original: { x | Ax ≤ 0 } .
Revised: { x | Ax ≤ 0 } .

Original: For δ, r > 0 and function f ( x ) = 12 x ⊤ Mx , we say that a point 0 is a:
Revised: For δ, r > 0 and function f ( x ) = 12 x ⊤ Mx , we say that a point 0 is a:

Original: • ( δ, r )-Quadratic
Revised: • ( δ, r )-Quadratic

Original: Corner Saddle Point (( δ, r )-QCSP) if min x ∈B ( r ) ∩ S f ( x ) < − δ .
Revised: Corner Saddle Point (( δ, r )-QCSP) if min x ∈B ( r ) ∩ S f ( x ) < − δ .

Original: • ( δ, r )-boundary QCSP if min x ∈B ( r ) ∩ ∂S f ( x ) < − δ .
Revised: • ( δ, r )-boundary QCSP if min x ∈B ( r ) ∩ ∂S f ( x ) < − δ .

Original: input : Starting point x , feasible set S
Revised: input : Starting point x , feasible set S

Original: = { y |
Revised: = { y |

Original: A
Revised: A

Original: ( y − x ) ≤ 0 ∈ R k } parameters: δ and r from definition of ( δ, r ) -QCSP for I ∈ 2 [ k ] – every subset of constraints do // Optimize in A
Revised: ( y − x ) ≤ 0 ∈ R k } parameters: δ and r from definition of ( δ, r ) -QCSP for I ∈ 2 [ k ] – every subset of constraints do // Optimize in A

Original: A ← { y | A ⊤ i ( y − x ) = 0 for i ∈
Revised: A ← { y | A ⊤ i ( y − x ) = 0 for i ∈

Original: I} , where A i is the i -th row of A y ← F IND I NSIDE C ORNER ( x , A )
Revised: I} , where A i is the i -th row of A y ← F IND I NSIDE C ORNER ( x , A )

Original: if y ∈ S and f ( y ) < f ( x ) − δ 2 then return y return ⊥
Revised: if y ∈ S and f ( y ) < f ( x ) − δ 2 then return y return ⊥

Original: Algorithm 2: F IND I NSIDE C ORNER ( x , A ) input : Corner x , affine subspace A with x ∈ A parameters: δ and r from Definition 12, step size η = 1 L , number of iterations T =
Revised: Algorithm 2: F IND I NSIDE C ORNER ( x , A ) input : Corner x , affine subspace A with x ∈ A parameters: δ and r from Definition 12, step size η = 1 L , number of iterations T =

Original: ˜ O
Revised: ˜ O

Original: ( Lr 2 δ )
Revised: ( Lr 2 δ )

Original: Sample ξ ∼ N ( 0 , I ) , x 0 ← Proj A ( x + ξ ) for t = 0 , . .
Revised: Sample ξ ∼ N ( 0 , I ) , x 0 ← Proj A ( x + ξ ) for t = 0 , . .

Original: , T − 1 do //
Revised: , T − 1 do //

Original: Power method step x t +1 ← Proj A ( x t − η ( ∇ f ( x t ) − ∇ f ( x )))
Revised: Power method step x t +1 ← Proj A ( x t − η ( ∇ f ( x t ) − ∇ f ( x )))

Original: e ← r x T − x ∥ x T − x ∥ return x + e
Revised: e ← r x T − x ∥ x T − x ∥ return x + e

Original: In this section, we show how to escape from a ( δ, r ) -QCSP, namely we prove the following
Revised: In this section, we show how to escape from a ( δ, r ) -QCSP, namely we prove the following

Original: Theorem:
Revised: Theorem:

Original: Theorem 13 (Theorem 7 restated) Let δ, r > 0 .
Revised: Theorem 13 (Theorem 7 restated) Let δ, r > 0 .

Original: Let f ( x ) = 12 x ⊤ Mx with λ | max | ( M ) ≤ L and let S = { x | Ax ≤ 0 } be defined by k linear inequalities.
Revised: Let f ( x ) = 12 x ⊤ Mx with λ | max | ( M ) ≤ L and let S = { x | Ax ≤ 0 } be defined by k linear inequalities.

Original: If 0 is a ( δ, r )-QCSP, then Algorithm 1 with probability at least 1 − ξ finds a point x ∈ S ∩ B ( r ) with f ( x ) < − Ω( δ ) using O (cid:16)
Revised: If 0 is a ( δ, r )-QCSP, then Algorithm 1 with probability at least 1 − ξ finds a point x ∈ S ∩ B ( r ) with f ( x ) < − Ω( δ ) using O (cid:16)

Original: Lr 2 k 2 k δ log 1 ξ (cid:17) deterministic gradient oracle calls.
Revised: Lr 2 k 2 k δ log 1 ξ (cid:17) deterministic gradient oracle calls.

Original: For the rest of the section, we assume that 0 is a ( δ, r ) -QCSP, i.e. min x ∈ S ∩B ( r ) f
Revised: For the rest of the section, we assume that 0 is a ( δ, r ) -QCSP, i.e. min x ∈ S ∩B ( r ) f

Original: ( x ) < − δ .
Revised: ( x ) < − δ .

Original: We consider two cases depending on whether 0 is a ( δ, r ) -boundary QCSP.
Revised: We consider two cases depending on whether 0 is a ( δ, r ) -boundary QCSP.

Original: Case 1: 0 is a ( δ, r ) -boundary QCSP.
Revised: Case 1: 0 is a ( δ, r ) -boundary QCSP.

Original: For a subset of inequality constraints I ⊆
Revised: For a subset of inequality constraints I ⊆

Original: [ k ] we define the subspace where these constraints are active:
Revised: [ k ] we define the subspace where these constraints are active:

Original: A I = { x |
Revised: A I = { x |

Original: A ⊤ i x = 0 for all i ∈ I} .
Revised: A ⊤ i x = 0 for all i ∈ I} .

Original: Let I be a maximal 4 subset of constraints such that min x ∈A I ∩B ( r ) f ( x ) < − δ .
Revised: Let I be a maximal 4 subset of constraints such that min x ∈A I ∩B ( r ) f ( x ) < − δ .

Original: If P is a projection operator on
Revised: If P is a projection operator on

Original: A I , it suffices to optimize g ( x ) :
Revised: A I , it suffices to optimize g ( x ) :

Original: = f ( Px ) = 12 x ⊤ ( PMP ) x .
Revised: = f ( Px ) = 12 x ⊤ ( PMP ) x .

Original: Therefore, we reduced the original problem to minimizing a different quadratic form in the same feasible set.
Revised: Therefore, we reduced the original problem to minimizing a different quadratic form in the same feasible set.

Original: For any i ∈ I , A ⊤ i Px ≤holds trivially, since A ⊤ i y = 0 for any y ∈ A , and hence constraints from I can be ignored.
Revised: For any i ∈ I , A ⊤ i Px ≤holds trivially, since A ⊤ i y = 0 for any y ∈ A , and hence constraints from I can be ignored.

Original: If a constraint not from I is active in Px , then f ( Px ) ≥ − δ , since I is a maximal subset of constraints with min x ∈A I ∩B ( r ) < − δ .
Revised: If a constraint not from I is active in Px , then f ( Px ) ≥ − δ , since I is a maximal subset of constraints with min x ∈A I ∩B ( r ) < − δ .

Original: Therefore, this reduces Case 1 to Case 2.
Revised: Therefore, this reduces Case 1 to Case 2.

Original: Case 2: 0 is not a ( δ, r ) -boundary QCSP.
Revised: Case 2: 0 is not a ( δ, r ) -boundary QCSP.

Original: In this case, we show that any x ∈ B ( r ) with f ( x ) < − δ must lie in S , and for f ( x ) =
Revised: In this case, we show that any x ∈ B ( r ) with f ( x ) < − δ must lie in S , and for f ( x ) =

Original: 12 x ⊤ Mx it suffices to find the eigenvector corresponding to the smallest eigenvalue of M .
Revised: 12 x ⊤ Mx it suffices to find the eigenvector corresponding to the smallest eigenvalue of M .

Original: We first show that there exists an eigenvector that improves the objective.
Revised: We first show that there exists an eigenvector that improves the objective.

Original: Lemma 14 Let f ( x ) = 12 x ⊤ Mx and S be a closed linear cone.
Revised: Lemma 14 Let f ( x ) = 12 x ⊤ Mx and S be a closed linear cone.

Original: If 0 is not a ( δ, r ) -boundary QCSP for δ, r > 0 , then the following statements are equivalent:
Revised: If 0 is not a ( δ, r ) -boundary QCSP for δ, r > 0 , then the following statements are equivalent:

Original: 1. 0 is ( δ, r ) -QCSP, i.e. min x ∈ S ∩B ( r ) f
Revised: 1. 0 is ( δ, r ) -QCSP, i.e. min x ∈ S ∩B ( r ) f

Original: ( x ) < − δ .
Revised: ( x ) < − δ .

Original: There exists an eigenvector e of M such that e ∈ Int S ∩ ∂ B ( r ) and f ( e ) < − δ .
Revised: There exists an eigenvector e of M such that e ∈ Int S ∩ ∂ B ( r ) and f ( e ) < − δ .

Original: Proof 2)
Revised: Proof 2)

Original: = ⇒ 1) follows trivially by definition of ( δ, r ) -QCSP.
Revised: = ⇒ 1) follows trivially by definition of ( δ, r ) -QCSP.

Original: 1) = ⇒ 2) .
Revised: 1) = ⇒ 2) .

Original: We show that minimizer x ⋆ lies in ∂ ( B ( r )) , and, since it’s a local minimum, by the method of Lagrangian multipliers, we show ∇ f ( x ⋆ ) =
Revised: We show that minimizer x ⋆ lies in ∂ ( B ( r )) , and, since it’s a local minimum, by the method of Lagrangian multipliers, we show ∇ f ( x ⋆ ) =

Original: λ x ⋆ for some λ , where ∇ f ( x ⋆ ) =
Revised: λ x ⋆ for some λ , where ∇ f ( x ⋆ ) =

Original: Mx ⋆ .
Revised: Mx ⋆ .

Original: Let x ⋆ be the minimizer of f on S ∩ B ( r ) .
Revised: Let x ⋆ be the minimizer of f on S ∩ B ( r ) .

Original: Then x ⋆ ∈ ∂ B ( r ) , since otherwise, we can rescale the minimizer to be on the boundary.
Revised: Then x ⋆ ∈ ∂ B ( r ) , since otherwise, we can rescale the minimizer to be on the boundary.

Original: I.e. for y = r ∥ x ⋆ ∥ x
Revised: I.e. for y = r ∥ x ⋆ ∥ x

Original: ⋆ we have f ( y ) =
Revised: ⋆ we have f ( y ) =

Original: r 2 ∥ x ⋆ ∥ 2 f ( x ⋆ ) <
Revised: r 2 ∥ x ⋆ ∥ 2 f ( x ⋆ ) <

Original: f ( x ⋆ ) .
Revised: f ( x ⋆ ) .

Original: Furthermore, y ∈ S since x ⋆ ∈ S and S is a linear cone, and
Revised: Furthermore, y ∈ S since x ⋆ ∈ S and S is a linear cone, and

Original: y ∈ ∂
Revised: y ∈ ∂

Original: B ( r ) since ∥ y ∥
Revised: B ( r ) since ∥ y ∥

Original: = r .
Revised: = r .

Original: From the above reasoning we have that x ⋆ ∈ S ∩ ∂ B ( r ) , and by the assumption of the lemma we have f ( x ⋆ ) < − δ .
Revised: From the above reasoning we have that x ⋆ ∈ S ∩ ∂ B ( r ) , and by the assumption of the lemma we have f ( x ⋆ ) < − δ .

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: , the minimizer x ⋆ lies in Int S ∩ ∂ B ( r ) .
Revised: , the minimizer x ⋆ lies in Int S ∩ ∂ B ( r ) .

Original: Since x ⋆ ∈ ∂ B ( r ) , it satisfies the constraint g ( x ⋆ ) =
Revised: Since x ⋆ ∈ ∂ B ( r ) , it satisfies the constraint g ( x ⋆ ) =

Original: ∥ x ⋆ ∥ 2 − r 2 = 0 , and by the method of Lagrangian multipliers we must have ∇ f ( x ⋆ )
Revised: ∥ x ⋆ ∥ 2 − r 2 = 0 , and by the method of Lagrangian multipliers we must have ∇ f ( x ⋆ )

Original: =
Revised: =

Original: λ ∇ g ( x ⋆ ) or equivalently Mx ⋆ = λ x ⋆ for some λ .
Revised: λ ∇ g ( x ⋆ ) or equivalently Mx ⋆ = λ x ⋆ for some λ .

Original: Therefore, x ⋆ is an eigenvector of M and f ( x ⋆ ) < − δ .
Revised: Therefore, x ⋆ is an eigenvector of M and f ( x ⋆ ) < − δ .

Original: Finding an exact eigenvector might be impossible.
Revised: Finding an exact eigenvector might be impossible.

Original: Hence, we show that finding x ∈ B ( r ) with f ( x ) < − δ suffices, since either x or − x are in S .
Revised: Hence, we show that finding x ∈ B ( r ) with f ( x ) < − δ suffices, since either x or − x are in S .

Original: Lemma 15 Let f ( x ) = 1 2 x ⊤ Mx and S be a closed linear cone.
Revised: Lemma 15 Let f ( x ) = 1 2 x ⊤ Mx and S be a closed linear cone.

Original: For δ, r > 0 and ˆ x ∈ ∂ B ( r ) , if the following conditions hold, then either ˆ x ∈ S or − ˆ x ∈ S :
Revised: For δ, r > 0 and ˆ x ∈ ∂ B ( r ) , if the following conditions hold, then either ˆ x ∈ S or − ˆ x ∈ S :

Original: 1. 0 is a ( δ, r ) -QCSP, i.e. min x ∈ S ∩B ( r ) f
Revised: 1. 0 is a ( δ, r ) -QCSP, i.e. min x ∈ S ∩B ( r ) f

Original: ( x ) < − δ , 2. 0 is not a ( δ, r ) -boundary QCSP, i.e. min x ∈ ∂S ∩B ( r ) f
Revised: ( x ) < − δ , 2. 0 is not a ( δ, r ) -boundary QCSP, i.e. min x ∈ ∂S ∩B ( r ) f

Original: ( x ) ≥ − δ , 3.
Revised: ( x ) ≥ − δ , 3.

Original: f (ˆ x ) < − δ .
Revised: f (ˆ x ) < − δ .

Original: Proof We know that there exists an eigenvector e ∈ S with f ( e ) < − δ .
Revised: Proof We know that there exists an eigenvector e ∈ S with f ( e ) < − δ .

Original: If both ˆ x and − ˆ x don’t belong to S , we consider an arc on ∂ B ( r ) connecting e and ˆ x (or − ˆ x , see below).
Revised: If both ˆ x and − ˆ x don’t belong to S , we consider an arc on ∂ B ( r ) connecting e and ˆ x (or − ˆ x , see below).

Original: We will show that for any point x on the arc, f ( x ) < − δ , and, since the arc intersects with ∂S , this contradicts assumption that 0 is not a ( δ, r ) -boundary QCSP.
Revised: We will show that for any point x on the arc, f ( x ) < − δ , and, since the arc intersects with ∂S , this contradicts assumption that 0 is not a ( δ, r ) -boundary QCSP.

Original: For contradiction, assume that both − ˆ x and ˆ x don’t belong to S .
Revised: For contradiction, assume that both − ˆ x and ˆ x don’t belong to S .

Original: From Lemma 14 we know that there exists an eigenvector e with f ( e ) < − δ and e ∈ S ∩ ∂ B ( r ) .
Revised: From Lemma 14 we know that there exists an eigenvector e with f ( e ) < − δ and e ∈ S ∩ ∂ B ( r ) .

Original: Either ˆ x ⊤ e ≥ 0 or − ˆ x ⊤ e ≥ 0 , and w.l.o.g.
Revised: Either ˆ x ⊤ e ≥ 0 or − ˆ x ⊤ e ≥ 0 , and w.l.o.g.

Original: we assume ˆ x ⊤ e ≥ 0 .
Revised: we assume ˆ x ⊤ e ≥ 0 .

Original: Consider an arc between ˆ x and e on ∂ B ( r ) :
Revised: Consider an arc between ˆ x and e on ∂ B ( r ) :

Original: [Equation]
Revised: [Equation]

Original: Since U is a connected set, ˆ x / ∈ S and e ∈ S , there exists x ∈ U ∩ ∂S .
Revised: Since U is a connected set, ˆ x / ∈ S and e ∈ S , there exists x ∈ U ∩ ∂S .

Original: Since x = α ˆ x + β e for some α, β ≥ 0 , we have:
Revised: Since x = α ˆ x + β e for some α, β ≥ 0 , we have:

Original: [Equation]
Revised: [Equation]

Original: By our assumption, 12 ˆ x ⊤ M ˆ x < − δ and 12 e ⊤ Me < − δ .
Revised: By our assumption, 12 ˆ x ⊤ M ˆ x < − δ and 12 e ⊤ Me < − δ .

Original: Since ˆ x ⊤ e ≥ 0 and e is an eigenvector of e with eigenvalue at most f ( e ) ∥ e ∥ 2 / 2 = − 2 δr 2 , we have
Revised: Since ˆ x ⊤ e ≥ 0 and e is an eigenvector of e with eigenvalue at most f ( e ) ∥ e ∥ 2 / 2 = − 2 δr 2 , we have

Original: [Equation]
Revised: [Equation]

Original: where we used that all terms are non-negative.
Revised: where we used that all terms are non-negative.

Original: Finally, using ∥ ˆ x ∥ = ∥ e ∥ =
Revised: Finally, using ∥ ˆ x ∥ = ∥ e ∥ =

Original: ∥
Revised: ∥

Original: α ˆ x + β e ∥ = r :
Revised: α ˆ x + β e ∥ = r :

Original: [Equation]
Revised: [Equation]

Original: contradicting our assumption that 0 is not a ( δ, r ) -boundary QCSP.
Revised: contradicting our assumption that 0 is not a ( δ, r ) -boundary QCSP.

Original: Finding x ∈ B ( r ) with f ( x ) < − δ might be difficult if min x ∈B ( r ) f ( x ) is very close to − δ .
Revised: Finding x ∈ B ( r ) with f ( x ) < − δ might be difficult if min x ∈B ( r ) f ( x ) is very close to − δ .

Original: Instead we aim to find x ∈ B ( r ) with f ( x ) < − (1 − ε )
Revised: Instead we aim to find x ∈ B ( r ) with f ( x ) < − (1 − ε )

Original: δ for a constant ε ∈ (0 , 1) .
Revised: δ for a constant ε ∈ (0 , 1) .

Original: As we show next, such a point can be found in polynomial time using the power method applied to the matrix
Revised: As we show next, such a point can be found in polynomial time using the power method applied to the matrix

Original: I − M L .
Revised: I − M L .

Original: Since λ | max | ( M ) ≤ L , all eigenvalues of I − M L are between 0 and 2 .
Revised: Since λ | max | ( M ) ≤ L , all eigenvalues of I − M L are between 0 and 2 .

Original: Since there exists an eigenvector e ∈ ∂ B ( r ) of M with 12 e ⊤ Me < − δ , we have λ min ( M ) <
Revised: Since there exists an eigenvector e ∈ ∂ B ( r ) of M with 12 e ⊤ Me < − δ , we have λ min ( M ) <

Original: − 2 δ ∥ e ∥ 2 = − 2 δ r 2 , and hence the largest eigenvalue of I − M L is at least 1 − λ min ( M ) L ≥ 1+ 2 δLr 2 .
Revised: − 2 δ ∥ e ∥ 2 = − 2 δ r 2 , and hence the largest eigenvalue of I − M L is at least 1 − λ min ( M ) L ≥ 1+ 2 δLr 2 .

Original: Finding x with 12 x ⊤ Mx < − (1 − ε ) δ is equivalent to finding x with x ⊤ ( I − M L ) x ≥ (1 + 2(1 − ε ) δ Lr 2 )
Revised: Finding x with 12 x ⊤ Mx < − (1 − ε ) δ is equivalent to finding x with x ⊤ ( I − M L ) x ≥ (1 + 2(1 − ε ) δ Lr 2 )

Original: ∥ x ∥ 2 , and the power method achieves this in O (log d + Lr 2 εδ ) iterations (see Lemma 16).
Revised: ∥ x ∥ 2 , and the power method achieves this in O (log d + Lr 2 εδ ) iterations (see Lemma 16).

Original: Finally, we show that Algorithm 2 emulates the power method on matrix I − M L .
Revised: Finally, we show that Algorithm 2 emulates the power method on matrix I − M L .

Original: Lemma 16 Let δ, r > 0 , x ∈ R f and ε ∈ (0 , 1) .
Revised: Lemma 16 Let δ, r > 0 , x ∈ R f and ε ∈ (0 , 1) .

Original: Let f ( y )
Revised: Let f ( y )

Original: =
Revised: =

Original: 12 ( y − x ) ⊤ M
Revised: 12 ( y − x ) ⊤ M

Original: ( y − x ) with λ | max | ( M ) ≤ L .
Revised: ( y − x ) with λ | max | ( M ) ≤ L .

Original: Let A be an affine subspace of R d such that x ∈ A .
Revised: Let A be an affine subspace of R d such that x ∈ A .

Original: If min y ∈A∩B ( x ,r ) f ( y ) < − δ , then Algorithm 2 with η = 1 L with probability 1 − O ( ξ ) finds y ∈ A∩
Revised: If min y ∈A∩B ( x ,r ) f ( y ) < − δ , then Algorithm 2 with η = 1 L with probability 1 − O ( ξ ) finds y ∈ A∩

Original: ∂ B ( x , r ) with f ( y ) ≤ − (1 − ε ) δ after T
Revised: ∂ B ( x , r ) with f ( y ) ≤ − (1 − ε ) δ after T

Original: = O (cid:16)
Revised: = O (cid:16)

Original: Lr 2 εδ log (cid:16) Lrdξεδ (cid:17)(cid:17) iterations 5 .
Revised: Lr 2 εδ log (cid:16) Lrdξεδ (cid:17)(cid:17) iterations 5 .

Original: Proof We show that Algorithm 2 performs a power iteration on matrix I − η PMP , where P is the projection operator on linear subspace corresponding to A .
Revised: Proof We show that Algorithm 2 performs a power iteration on matrix I − η PMP , where P is the projection operator on linear subspace corresponding to A .

Original: We show that λ min ( PMP ) corresponds to λ | max | ( I − η PMP ) .
Revised: We show that λ min ( PMP ) corresponds to λ | max | ( I − η PMP ) .

Original: Finally, we use Fact 10 to establish the convergence rate.
Revised: Finally, we use Fact 10 to establish the convergence rate.

Original: By shifting the coordinate system so that x becomes 0 , we instead optimize function g ( y ) =
Revised: By shifting the coordinate system so that x becomes 0 , we instead optimize function g ( y ) =

Original: 12 y ⊤ My on set B ( r ) ∩ A 0 ,
Revised: 12 y ⊤ My on set B ( r ) ∩ A 0 ,

Original: where A 0 =
Revised: where A 0 =

Original: { y − x | y ∈ A} is the linear subspace parallel to A .
Revised: { y − x | y ∈ A} is the linear subspace parallel to A .

Original: Let
Revised: Let

Original: P ∈ R d × d be the projection operator on A 0 .
Revised: P ∈ R d × d be the projection operator on A 0 .

Original: Then for any y ∈ A 

Revised: Then for any y ∈ A

Original: we have y =
Revised: we have y =

Original: Py and for any
Revised: Py and for any

Original: y ∈ R d we have Proj
Revised: y ∈ R d we have Proj

Original: A 

Revised: A

Original: ( y ) =
Revised: ( y ) =

Original: Py .
Revised: Py .

Original: Defining y t
Revised: Defining y t

Original: = x t − x , we have:
Revised: = x t − x , we have:

Original: [Equation]
Revised: [Equation]

Original: Definition of
Revised: Definition of

Original: [Equation]
Revised: [Equation]

Original: Py
Revised: Py

Original: [Equation]
Revised: [Equation]

Original: since, by induction, y t
Revised: since, by induction, y t

Original: [Equation]
Revised: [Equation]

Original: Therefore, Algorithm 2 performs a power iteration for matrix B =
Revised: Therefore, Algorithm 2 performs a power iteration for matrix B =

Original: I − η PMP .
Revised: I − η PMP .

Original: Since λ | max | ( PMP ) ≤ λ | max | ( M ) ≤ L and η = 1 L , all eigenvalues of B lie in [0 , 2] , and its largest eigenvalue is 1 − ηλ min ( PMP ) .
Revised: Since λ | max | ( PMP ) ≤ λ | max | ( M ) ≤ L and η = 1 L , all eigenvalues of B lie in [0 , 2] , and its largest eigenvalue is 1 − ηλ min ( PMP ) .

Original: Since there exists e with ∥ e ∥ = r and h ( e ) < − δ , we have
Revised: Since there exists e with ∥ e ∥ = r and h ( e ) < − δ , we have

Original: [Equation]
Revised: [Equation]

Original: which implies λ | max | ( B )
Revised: which implies λ | max | ( B )

Original: = 

Revised: =

Original: + 

Revised: +

Original: δLr 2 .
Revised: δLr 2 .

Original: Our goal is to find
Revised: Our goal is to find

Original: y ∈ ∂
Revised: y ∈ ∂

Original: B ( r ) with f ( y ) < − (1 − ε ) δ , meaning
Revised: B ( r ) with f ( y ) < − (1 − ε ) δ , meaning

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to y ⊤ By
Revised: which is equivalent to y ⊤ By

Original: [Equation]
Revised: [Equation]

Original: By Fact 10, with probability 1 − O ( ξ ) we find such y after the following number of iterations:
Revised: By Fact 10, with probability 1 − O ( ξ ) we find such y after the following number of iterations:

Original: [Equation]
Revised: [Equation]

Original: Rescaling y so that ∥ y ∥ =
Revised: Rescaling y so that ∥ y ∥ =

Original: r finishes the proof.
Revised: r finishes the proof.

Original: We now prove Theorem 7.
Revised: We now prove Theorem 7.

Original: First, by exhaustive search we guess a maximal subset of active constraints I such that subspace
Revised: First, by exhaustive search we guess a maximal subset of active constraints I such that subspace

Original: A I formed by these linear constraints has x ∈ B ( r ) ∩ S with f ( x ) < δ .
Revised: A I formed by these linear constraints has x ∈ B ( r ) ∩ S with f ( x ) < δ .

Original: Using Algorithm 2, we find y ∈ B ( r ) ∩
Revised: Using Algorithm 2, we find y ∈ B ( r ) ∩

Original: A I with f ( y ) <
Revised: A I with f ( y ) <

Original: − (1 − ε ) δ .
Revised: − (1 − ε ) δ .

Original: Then y ∈ S by
Revised: Then y ∈ S by

Original: Lemma 16, since I is a maximal subset of constraints with an escape direction.
Revised: Lemma 16, since I is a maximal subset of constraints with an escape direction.

Original: Definition 17 A set S is a linear cone if x ∈ S implies α x ∈ S for all α ≥ 0 .
Revised: Definition 17 A set S is a linear cone if x ∈ S implies α x ∈ S for all α ≥ 0 .

Original: If S is a linear cone, then scaling a vector preserves its belonging to S .
Revised: If S is a linear cone, then scaling a vector preserves its belonging to S .

Original: In particular, S = { x | Ax ≤ 0 } is a linear cone.
Revised: In particular, S = { x | Ax ≤ 0 } is a linear cone.

Original: Theorem 18 (Theorem 7 restated) Let δ, r > 0 .
Revised: Theorem 18 (Theorem 7 restated) Let δ, r > 0 .

Original: Let f ( x ) = 12 x ⊤ Mx with λ | max | ( M ) ≤ L and let S = { x | Ax ≤ 0 } be defined by k linear inequality constraints.
Revised: Let f ( x ) = 12 x ⊤ Mx with λ | max | ( M ) ≤ L and let S = { x | Ax ≤ 0 } be defined by k linear inequality constraints.

Original: If x is a ( δ, r )-QCSP, then
Revised: If x is a ( δ, r )-QCSP, then

Original: Algorithm 1 with probability at least 1 − ξ finds a point x ∈ S ∩ B ( r ) with f ( x ) < − Ω( δ ) using ˜ O (cid:16) Lr 2 k 2 k δ (cid:17) deterministic gradient oracle calls.
Revised: Algorithm 1 with probability at least 1 − ξ finds a point x ∈ S ∩ B ( r ) with f ( x ) < − Ω( δ ) using ˜ O (cid:16) Lr 2 k 2 k δ (cid:17) deterministic gradient oracle calls.

Original: Proof
Revised: Proof

Original: For I ⊆
Revised: For I ⊆

Original: [ k ] , we define A
Revised: [ k ] , we define A

Original: I = { x |
Revised: I = { x |

Original: A ⊤ i x = 0 , i ∈ I} .
Revised: A ⊤ i x = 0 , i ∈ I} .

Original: By induction, we’ll prove the following: for any I ∈ [ k ] , if Algorithm 1 didn’t find point x with f ( x ) < − (1 − |I| +

Revised: By induction, we’ll prove the following: for any I ∈ [ k ] , if Algorithm 1 didn’t find point x with f ( x ) < − (1 − |I| +

Original: k ε )
Revised: k ε )

Original: δ after executing
Revised: δ after executing

Original: Algorithm 2 on all sets A ˜
Revised: Algorithm 2 on all sets A ˜

Original: I such that I ⊆
Revised: I such that I ⊆

Original: ˜
Revised: ˜

Original: I ⊆
Revised: I ⊆

Original: [ k ] , then
Revised: [ k ] , then

Original: [Equation]
Revised: [Equation]

Original: Basically, we have a subroutine which can return 1 − ε approximation, and, since this approximation factor may accumulate with each additional active constraint, we instead use approximation 1 − εk for every set of constraints.
Revised: Basically, we have a subroutine which can return 1 − ε approximation, and, since this approximation factor may accumulate with each additional active constraint, we instead use approximation 1 − εk for every set of constraints.

Original: We fix I and assume that the statement holds for all ˜
Revised: We fix I and assume that the statement holds for all ˜

Original: I such that I ⊊ ˜
Revised: I such that I ⊊ ˜

Original: I ⊆
Revised: I ⊆

Original: [ k ] .
Revised: [ k ] .

Original: By induction hypothesis, if for all such ˜ I we didn’t find point x with f ( x ) < − (1 − |I| +2 k ε ) δ , then
Revised: By induction hypothesis, if for all such ˜ I we didn’t find point x with f ( x ) < − (1 − |I| +2 k ε ) δ , then

Original: [Equation]
Revised: [Equation]

Original: for all such ˜ I .
Revised: for all such ˜ I .

Original: If
Revised: If

Original: [Equation]
Revised: [Equation]

Original: then, since on the boundary we have
Revised: then, since on the boundary we have

Original: [Equation]
Revised: [Equation]

Original: by Lemma 15 and Lemma 16 we can find x ∈ A
Revised: by Lemma 15 and Lemma 16 we can find x ∈ A

Original: I ∩ S ∩ B ( r ) with f ( x ) < − (1 − |I| +

Revised: I ∩ S ∩ B ( r ) with f ( x ) < − (1 − |I| +

Original: k ε )
Revised: k ε )

Original: δ using ˜ O (cid:16)
Revised: δ using ˜ O (cid:16)

Original: Lkr 2 εδ (cid:17) gradient computations.
Revised: Lkr 2 εδ (cid:17) gradient computations.

Original: Hence, if the algorithm didn’t find such x , then, as required:
Revised: Hence, if the algorithm didn’t find such x , then, as required:

Original: [Equation]
Revised: [Equation]

Original: Since there 2 k possible subsets of constraints, the total number of gradient oracle calls is ˜ O (cid:16) Lkr 2 2 k εδ (cid:17) .
Revised: Since there 2 k possible subsets of constraints, the total number of gradient oracle calls is ˜ O (cid:16) Lkr 2 2 k εδ (cid:17) .

Original: Appendix C. Algorithm and Proofs for General Case
Revised: Appendix C. Algorithm and Proofs for General Case

Original: In this section, we present Algorithms 3 and 4 and show that they can escape from a saddle point.
Revised: In this section, we present Algorithms 3 and 4 and show that they can escape from a saddle point.

Original: Algorithm 3 computes a quadratic approximation of the function in all affine subspaces, defined by all possible intersections of active constraints.
Revised: Algorithm 3 computes a quadratic approximation of the function in all affine subspaces, defined by all possible intersections of active constraints.

Original: For each affine subspace, it calls Algorithm 4 as a subroutine, which tries to find an escape direction in this subspace.
Revised: For each affine subspace, it calls Algorithm 4 as a subroutine, which tries to find an escape direction in this subspace.

Original: C.1. Algorithm 3: Quadratic approximation
Revised: C.1. Algorithm 3: Quadratic approximation

Original: We first focus on the Lines 1-8 of Algorithm 3.
Revised: We first focus on the Lines 1-8 of Algorithm 3.

Original: The goal is to build a quadratic approximation of the objective with some additional properties (see below).
Revised: The goal is to build a quadratic approximation of the objective with some additional properties (see below).

Original: To simplify the presentation, w.l.o.g. by shifting the coordinate system, we assume that the current saddle point is 0 and consider the quadratic approximation of the function: f ( x ) =
Revised: To simplify the presentation, w.l.o.g. by shifting the coordinate system, we assume that the current saddle point is 0 and consider the quadratic approximation of the function: f ( x ) =

Original: 12 x ⊤ ∇ 2 f ( 0 ) x
Revised: 12 x ⊤ ∇ 2 f ( 0 ) x

Original: + x ⊤ ∇ f ( 0 ) .
Revised: + x ⊤ ∇ f ( 0 ) .

Original: Since f is ρ -Lipschitz and r = 3 (cid:112) δ / ρ , in B ( r ) the quadratic approximation deviates from f by at most δ 6 (see derivation before Definition 4).
Revised: Since f is ρ -Lipschitz and r = 3 (cid:112) δ / ρ , in B ( r ) the quadratic approximation deviates from f by at most δ 6 (see derivation before Definition 4).

Original: For a small value ξ (to be specified later), we instead analyze a noisy function f ′ ( x ) = 12 x ⊤ Mx + x ⊤ v , where (all lines refer to Algorithm 3):
Revised: For a small value ξ (to be specified later), we instead analyze a noisy function f ′ ( x ) = 12 x ⊤ Mx + x ⊤ v , where (all lines refer to Algorithm 3):

Original: M = H + γI (Line 8), where H is an approximation of the Hessian (Line 5) and γ is small uniform noise, which guarantees that M is non-degenerate with probability 1 .
Revised: M = H + γI (Line 8), where H is an approximation of the Hessian (Line 5) and γ is small uniform noise, which guarantees that M is non-degenerate with probability 1 .

Original: v = g
Revised: v = g

Original: + ζ (Line 7), where g is an approximation of the gradient and ζ is small Gaussian noise, which guarantees that all coordinates of v are sufficiently separated from 0 w.h.p.
Revised: + ζ (Line 7), where g is an approximation of the gradient and ζ is small Gaussian noise, which guarantees that all coordinates of v are sufficiently separated from 0 w.h.p.

Original: Additionally, with probability 1 , linear systems of the form ( M − µI ) x = v with rank( M − µI ) <
Revised: Additionally, with probability 1 , linear systems of the form ( M − µI ) x = v with rank( M − µI ) <

Original: d don’t have any solutions, simplifying the analysis.
Revised: d don’t have any solutions, simplifying the analysis.

Original: We show that f ′ is a good approximation of f , and first, we need to show that H and g are good approximations of the Hessian and the gradient respectively.
Revised: We show that f ′ is a good approximation of f , and first, we need to show that H and g are good approximations of the Hessian and the gradient respectively.

Original: For the gradient, we use algorithm
Revised: For the gradient, we use algorithm

Original: VRSG from Line 1, which w.h.p.
Revised: VRSG from Line 1, which w.h.p.

Original: estimates the gradient with precision ˜ σ using ˜ O ( σ 2 ˜ σ 2 ) stochastic gradient oracle calls (see Appendix C, Lemma 20).
Revised: estimates the gradient with precision ˜ σ using ˜ O ( σ 2 ˜ σ 2 ) stochastic gradient oracle calls (see Appendix C, Lemma 20).

Original: For the Hessian, since the i -th column of ∇ 2 f ( 0 ) is by definition lim
Revised: For the Hessian, since the i -th column of ∇ 2 f ( 0 ) is by definition lim

Original: τ → 0 ∇ f ( τ e i ) −∇
Revised: τ → 0 ∇ f ( τ e i ) −∇

Original: f
Revised: f

Original: τ , using a sufficiently small τ and approximating ∇ f ( τ e i ) and ∇ f ( 0 ) using VRSG , we find good approximation of ∇ 2 f ( x ) .
Revised: τ , using a sufficiently small τ and approximating ∇ f ( τ e i ) and ∇ f ( 0 ) using VRSG , we find good approximation of ∇ 2 f ( x ) .

Original: By selecting parameters as in
Revised: By selecting parameters as in

Original: Algorithm 3 and by guaranteeing that noises ζ and γ are bounded, we show the following result:
Revised: Algorithm 3 and by guaranteeing that noises ζ and γ are bounded, we show the following result:

Original: Algorithm
Revised: Algorithm

Original: Let VRSG( x , ˜ σ ) be an algorithm that returns g such that ∥ g (0) − ∇ f ( x ) ∥ ≤ ˜ σ w.h.p.
Revised: Let VRSG( x , ˜ σ ) be an algorithm that returns g such that ∥ g (0) − ∇ f ( x ) ∥ ≤ ˜ σ w.h.p.

Original: // Compute H - approximate Hessian θ ← Θ( r √ d ) , ˜ σ ← Θ( ρr 2 d ) g (0) ← VRSG( x , ˜ σ ) for e 1 , . . .
Revised: // Compute H - approximate Hessian θ ← Θ( r √ d ) , ˜ σ ← Θ( ρr 2 d ) g (0) ← VRSG( x , ˜ σ ) for e 1 , . . .

Original: , e d – the standard basis in R d do g i ← VRSG( x + θ e i , ˜ σ ) ;
Revised: , e d – the standard basis in R d do g i ← VRSG( x + θ e i , ˜ σ ) ;

Original: Let H ∈ R d × d whose i -th column is g i − g (0) θ //
Revised: Let H ∈ R d × d whose i -th column is g i − g (0) θ //

Original: f ( x + h )
Revised: f ( x + h )

Original: ≈ f
Revised: ≈ f

Original: ( x ) + h ⊤ v + 12 h
Revised: ( x ) + h ⊤ v + 12 h

Original: ⊤
Revised: ⊤

Original: Mh
Revised: Mh

Original: Sample ζ ∼ N ( 0 , ξ δ r √ d · I ) , γ ∼ U [ − ξ, ξ ] v ← VRSG( x , Θ( δ / r ))
Revised: Sample ζ ∼ N ( 0 , ξ δ r √ d · I ) , γ ∼ U [ − ξ, ξ ] v ← VRSG( x , Θ( δ / r ))

Original: + ζ
Revised: + ζ

Original: M ← H + γI for I – every subset of constraints do
Revised: M ← H + γI for I – every subset of constraints do

Original: A ← { y | A ⊤ i x = b i , i ∈ I} , where A i is the i -th row of A //
Revised: A ← { y | A ⊤ i x = b i , i ∈ I} , where A i is the i -th row of A //

Original: Optimize in A p ← Proj A ( x ) //
Revised: Optimize in A p ← Proj A ( x ) //

Original: Ball center in A if ∥ p − x ∥ > r then continue ; // Optimization of g ( y ) =
Revised: Ball center in A if ∥ p − x ∥ > r then continue ; // Optimization of g ( y ) =

Original: 12 y ⊤ M ⊥ y
Revised: 12 y ⊤ M ⊥ y

Original: + y ⊤ v ⊥ in
Revised: + y ⊤ v ⊥ in

Original: S
Revised: S

Original: ⊥ ∩
Revised: ⊥ ∩

Original: B dim A ( r ⊥ )
Revised: B dim A ( r ⊥ )

Original: r ⊥ ← (cid:112) r 2 − ∥ p − x ∥
Revised: r ⊥ ← (cid:112) r 2 − ∥ p − x ∥

Original: Let
Revised: Let

Original: O ∈ R
Revised: O ∈ R

Original: d × dim A be an orthonormal basis of A
Revised: d × dim A be an orthonormal basis of A

Original: S ⊥ ← { y | A ( p + Oy ) ≤ b }
Revised: S ⊥ ← { y | A ( p + Oy ) ≤ b }

Original: // y ∈ S ⊥ guarantees that p + Oy ∈ S
Revised: // y ∈ S ⊥ guarantees that p + Oy ∈ S

Original: M
Revised: M

Original: ← O ⊤
Revised: ← O ⊤

Original: MO
Revised: MO

Original: ( v + M ( p − x )
Revised: ( v + M ( p − x )

Original: M
Revised: M

Original: ( r
Revised: ( r

Original: S
Revised: S

Original: // Algorithm
Revised: // Algorithm

Original: u ̸ = ⊥ then return u ;
Revised: u ̸ = ⊥ then return u ;

Original: report that x is a δ -SOSP
Revised: report that x is a δ -SOSP

Original: Algorithm 4:
Revised: Algorithm 4:

Original: F IND I NSIDE ( x , δ, p , O , ( M ⊥ , v ⊥ ) , ( r ⊥ , S ⊥ ))
Revised: F IND I NSIDE ( x , δ, p , O , ( M ⊥ , v ⊥ ) , ( r ⊥ , S ⊥ ))

Original: g ( y ) =
Revised: g ( y ) =

Original: 12 y ⊤ M ⊥ y
Revised: 12 y ⊤ M ⊥ y

Original: + y v
Revised: + y v

Original: // Case 1 :
Revised: // Case 1 :

Original: Large gradient
Revised: Large gradient

Original: Let y ← argmin y ∈ S ⊥ ∩B ( r ⊥ )
Revised: Let y ← argmin y ∈ S ⊥ ∩B ( r ⊥ )

Original: y ⊤ v ⊥ if f ( p + Oy ) < f
Revised: y ⊤ v ⊥ if f ( p + Oy ) < f

Original: ( x ) − δ 3 then return p + Oy ; // Case 2 : Solution is in Int B ( r ⊥ )
Revised: ( x ) − δ 3 then return p + Oy ; // Case 2 : Solution is in Int B ( r ⊥ )

Original: Let y ← − M − 1 v if y ∈ S ⊥ ∩ B ( r ⊥ ) and f ( p + Oy ) < f ( x ) − δ 3 then return p + Oy ; // Case 3 :
Revised: Let y ← − M − 1 v if y ∈ S ⊥ ∩ B ( r ⊥ ) and f ( p + Oy ) < f ( x ) − δ 3 then return p + Oy ; // Case 3 :

Original: Solution is in
Revised: Solution is in

Original: ∂
Revised: ∂

Original: B
Revised: B

Original: ( r ⊥ )
Revised: ( r ⊥ )

Original: // Matrix diagonalization
Revised: // Matrix diagonalization

Original: Let Q be an orthogonal matrix and Λ = diag ( λ 1 , . . .
Revised: Let Q be an orthogonal matrix and Λ = diag ( λ 1 , . . .

Original: , λ dim A ) be such that
Revised: , λ dim A ) be such that

Original: [Equation]
Revised: [Equation]

Original: // Rotation of the linear term
Revised: // Rotation of the linear term

Original: Let ˜ v ← Qv ⊥ and ˜ v i be its i -th coordinate //
Revised: Let ˜ v ← Qv ⊥ and ˜ v i be its i -th coordinate //

Original: ˜
Revised: ˜

Original: y from Line 9 must have norm r
Revised: y from Line 9 must have norm r

Original: Let { µ } j be approximate real solutions of (cid:80) di = ˜ v 

Revised: Let { µ } j be approximate real solutions of (cid:80) di = ˜ v

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: λ i ) 2 = r 2 ⊥ for each µ j do
Revised: λ i ) 2 = r 2 ⊥ for each µ j do

Original: Let ˜ y be a vector such that its i -th coordinate is ˜ y
Revised: Let ˜ y be a vector such that its i -th coordinate is ˜ y

Original: i ← ˜ v
Revised: i ← ˜ v

Original: [Equation]
Revised: [Equation]

Original: µ j − λ
Revised: µ j − λ

Original: i
Revised: i

Original: // Correction of approximate solutions
Revised: // Correction of approximate solutions

Original: y ← Proj S ⊥ ∩B ( r ⊥ )
Revised: y ← Proj S ⊥ ∩B ( r ⊥ )

Original: ( Q ⊤ ˜ y )
Revised: ( Q ⊤ ˜ y )

Original: if f ( p + Oy ) < f ( x ) − δ 3 then return p + Oy ;
Revised: if f ( p + Oy ) < f ( x ) − δ 3 then return p + Oy ;

Original: return ⊥ //
Revised: return ⊥ //

Original: No solution in affine subspace
Revised: No solution in affine subspace

Original: A
Revised: A

Original: Lemma 19 Let f satisfy Assumptions 1 and 2.
Revised: Lemma 19 Let f satisfy Assumptions 1 and 2.

Original: Let f ′ ( x ) =
Revised: Let f ′ ( x ) =

Original: f ( 0 )
Revised: f ( 0 )

Original: + x ⊤ v + 12 x ⊤ Mx , where v and
Revised: + x ⊤ v + 12 x ⊤ Mx , where v and

Original: M are as in Algorithm 3.
Revised: M are as in Algorithm 3.

Original: For δ > 0 , r = 3 (cid:112) δ / ρ we have ∥ f ′ ( x ) − f ( x ) ∥ < δ 2 for all x ∈ B ( r ) w.h.p.
Revised: For δ > 0 , r = 3 (cid:112) δ / ρ we have ∥ f ′ ( x ) − f ( x ) ∥ < δ 2 for all x ∈ B ( r ) w.h.p.

Original: Reducing Case x ⋆ ∈ ∂S to Case x ⋆ ∈ Int S .
Revised: Reducing Case x ⋆ ∈ ∂S to Case x ⋆ ∈ Int S .

Original: We now elaborate on Lines 13-17 of Algorithm 3.
Revised: We now elaborate on Lines 13-17 of Algorithm 3.

Original: The goal is, as in Section B, to reduce the case x ⋆ ∈ ∂S to the case x ⋆ ∈ Int S .
Revised: The goal is, as in Section B, to reduce the case x ⋆ ∈ ∂S to the case x ⋆ ∈ Int S .

Original: If x ⋆ ∈ ∂S , then there exist a non-empty set I of constraints active at x ⋆ .
Revised: If x ⋆ ∈ ∂S , then there exist a non-empty set I of constraints active at x ⋆ .

Original: Then we work in affine subspace
Revised: Then we work in affine subspace

Original: A where constraints from I are active.
Revised: A where constraints from I are active.

Original: We parameterize A : if p = Proj A ( x ) and
Revised: We parameterize A : if p = Proj A ( x ) and

Original: O ∈ R d × dim A is an orthonormal basis of A , then any point in A can be represented as p + Oy for y ∈ R dim A .
Revised: O ∈ R d × dim A is an orthonormal basis of A , then any point in A can be represented as p + Oy for y ∈ R dim A .

Original: Defining g
Revised: Defining g

Original: ( y ) =
Revised: ( y ) =

Original: f ′ ( p + Oy ) , since
Revised: f ′ ( p + Oy ) , since

Original: [Equation]
Revised: [Equation]

Original: minimizing f ′ in A ∩ S ∩ B ( r ) is equivalent to minimizing g in S ⊥ ∩ B ( r ⊥ )
Revised: minimizing f ′ in A ∩ S ∩ B ( r ) is equivalent to minimizing g in S ⊥ ∩ B ( r ⊥ )

Original: (Lines 13-17), where: 1.
Revised: (Lines 13-17), where: 1.

Original: S ⊥ is a set of points
Revised: S ⊥ is a set of points

Original: y ∈ R dim A such that p + Oy ∈ S , namely S ⊥ =
Revised: y ∈ R dim A such that p + Oy ∈ S , namely S ⊥ =

Original: { y | A
Revised: { y | A

Original: i ( p + Oy ) ≤ b i , i / ∈ I} .
Revised: i ( p + Oy ) ≤ b i , i / ∈ I} .

Original: Hence, S ⊥ is defined by linear inequalities, similarly to S .
Revised: Hence, S ⊥ is defined by linear inequalities, similarly to S .

Original: r ⊥ is a radius such that condition y ∈ B ( r ⊥ ) is equivalent to p + Oy ∈ B ( r ) .
Revised: r ⊥ is a radius such that condition y ∈ B ( r ⊥ ) is equivalent to p + Oy ∈ B ( r ) .

Original: Since p is the projection of 0 on A , we have O ⊤ p = 0 , and hence ∥ p + Oy ∥ 2 = ∥
Revised: Since p is the projection of 0 on A , we have O ⊤ p = 0 , and hence ∥ p + Oy ∥ 2 = ∥

Original: p ∥ 2 + ∥ Oy ∥ 2 .
Revised: p ∥ 2 + ∥ Oy ∥ 2 .

Original: Since O is an orthonormal basis of A ,
Revised: Since O is an orthonormal basis of A ,

Original: ∥
Revised: ∥

Original: Oy ∥ =
Revised: Oy ∥ =

Original: ∥
Revised: ∥

Original: y ∥ , and hence r ⊥ =
Revised: y ∥ , and hence r ⊥ =

Original: (cid:112) r 2 − ∥ p ∥ 2 .
Revised: (cid:112) r 2 − ∥ p ∥ 2 .

Original: For y ⋆ such that x ⋆ = p + Oy ⋆ , no constraints from S ⊥ are active, and hence x ⋆ ∈ Int S ⊥ .
Revised: For y ⋆ such that x ⋆ = p + Oy ⋆ , no constraints from S ⊥ are active, and hence x ⋆ ∈ Int S ⊥ .

Original: C.2. Algorithm 4: Escaping when y ⋆ ∈ Int S ⊥
Revised: C.2. Algorithm 4: Escaping when y ⋆ ∈ Int S ⊥

Original: In this section, we find minimizer y ⋆ of function g ( y )
Revised: In this section, we find minimizer y ⋆ of function g ( y )

Original: =
Revised: =

Original: 12 y ⊤ M ⊥
Revised: 12 y ⊤ M ⊥

Original: y
Revised: y

Original: + y ⊤ v ⊥ + C in S ⊥ ∩ B ( r ⊥ ) , while assuming that y ⋆ ∈
Revised: + y ⊤ v ⊥ + C in S ⊥ ∩ B ( r ⊥ ) , while assuming that y ⋆ ∈

Original: Int S ⊥ .
Revised: Int S ⊥ .

Original: Since the solutions we find can be approximate, we have to guarantee that the objective is not too sensitive to the change of its argument.
Revised: Since the solutions we find can be approximate, we have to guarantee that the objective is not too sensitive to the change of its argument.

Original: It suffices to guarantee that ∥ v ∥ is bounded, since for any y ∈ B ( r ⊥ ) and perturbation h there exists τ ∈ [0 , 1] such that:
Revised: It suffices to guarantee that ∥ v ∥ is bounded, since for any y ∈ B ( r ⊥ ) and perturbation h there exists τ ∈ [0 , 1] such that:

Original: [Equation]
Revised: [Equation]

Original: where we used that the objective is L -smooth and hence ∥∇ g ( y + τ h ) ∥ ≤ ∥∇ g ( 0 ) ∥ + L ∥ y
Revised: where we used that the objective is L -smooth and hence ∥∇ g ( y + τ h ) ∥ ≤ ∥∇ g ( 0 ) ∥ + L ∥ y

Original: + τ h ∥ .
Revised: + τ h ∥ .

Original: We consider the situation when ∥ v ⊥ ∥ is large as a separate case.
Revised: We consider the situation when ∥ v ⊥ ∥ is large as a separate case.

Original: Otherwise, for y ⋆ , there are only two options:
Revised: Otherwise, for y ⋆ , there are only two options:

Original: either y ⋆ ∈
Revised: either y ⋆ ∈

Original: Int B ( r ⊥ ) or y ⋆ ∈
Revised: Int B ( r ⊥ ) or y ⋆ ∈

Original: ∂ B ( r ⊥ ) .
Revised: ∂ B ( r ⊥ ) .

Original: Algorithm 4 handles these cases, as well as the case when ∥ v ∥ is large, separately.
Revised: Algorithm 4 handles these cases, as well as the case when ∥ v ∥ is large, separately.

Original: Case 1: ∥ v ⊥ ∥ is large (Lines 1-2).
Revised: Case 1: ∥ v ⊥ ∥ is large (Lines 1-2).

Original: If ∥ v ⊥ ∥ is large and we can find y with small y ⊤ v ⊥ , the linear term alone suffices to improve the objective.
Revised: If ∥ v ⊥ ∥ is large and we can find y with small y ⊤ v ⊥ , the linear term alone suffices to improve the objective.

Original: We show that, if such y doesn’t exist, then g ( y ⋆ ) requires y ⋆ ∈ ∂S ⊥ , which contradicts that y ⋆ ∈
Revised: We show that, if such y doesn’t exist, then g ( y ⋆ ) requires y ⋆ ∈ ∂S ⊥ , which contradicts that y ⋆ ∈

Original: Int S ⊥ .
Revised: Int S ⊥ .

Original: Below we assume that ∥ v ⊥ ∥ is bounded.
Revised: Below we assume that ∥ v ⊥ ∥ is bounded.

Original: Case 2:
Revised: Case 2:

Original: y ∈
Revised: y ∈

Original: Int B ( r ⊥
Revised: Int B ( r ⊥

Original: ) (Lines 3-4).
Revised: ) (Lines 3-4).

Original: In this case, y ⋆ is an unconstrained critical point of g , and hence it must satisfy ∇ g ( y ) = 0 , implying M ⊥ y + v ⊥ = 0 which gives the unique solution y = − M − 1 ⊥ v ⊥ .
Revised: In this case, y ⋆ is an unconstrained critical point of g , and hence it must satisfy ∇ g ( y ) = 0 , implying M ⊥ y + v ⊥ = 0 which gives the unique solution y = − M − 1 ⊥ v ⊥ .

Original: since M is a perturbed matrix, so is M ⊥ , and hence M ⊥ is non-degenerate with probability 1 .
Revised: since M is a perturbed matrix, so is M ⊥ , and hence M ⊥ is non-degenerate with probability 1 .

Original: It remains to verify
Revised: It remains to verify

Original: that y ∈ B ( r ⊥ ) ∩ S ⊥ and y decreases the objective by Ω( δ ) .
Revised: that y ∈ B ( r ⊥ ) ∩ S ⊥ and y decreases the objective by Ω( δ ) .

Original: Case 3: y ∈ ∂ B ( r ⊥ )
Revised: Case 3: y ∈ ∂ B ( r ⊥ )

Original: (Lines 5-11).
Revised: (Lines 5-11).

Original: Since the only active constraint at y ⋆ is c ( y ) =
Revised: Since the only active constraint at y ⋆ is c ( y ) =

Original: 12 ( ∥ y ∥ 2 − r 2 ⊥ )
Revised: 12 ( ∥ y ∥ 2 − r 2 ⊥ )

Original: = 0 , by the KKT conditions, any critical point must satisfy ∇ g ( y ) =
Revised: = 0 , by the KKT conditions, any critical point must satisfy ∇ g ( y ) =

Original: µ ∇ c ( y ) for some µ ∈ R , which is equivalent to M ⊥ y
Revised: µ ∇ c ( y ) for some µ ∈ R , which is equivalent to M ⊥ y

Original: + v ⊥ = µ y .
Revised: + v ⊥ = µ y .

Original: Hence, for any fixed µ , y must be a solution of linear system ( M ⊥ − µI ) y
Revised: Hence, for any fixed µ , y must be a solution of linear system ( M ⊥ − µI ) y

Original: = − v ⊥ .
Revised: = − v ⊥ .

Original: When M ⊥ − µI is degenerate (i.e. µ is an eigenvalue of M ⊥ ), due to perturbation of v ⊥ , the system doesn’t have any solution with probability 1 .
Revised: When M ⊥ − µI is degenerate (i.e. µ is an eigenvalue of M ⊥ ), due to perturbation of v ⊥ , the system doesn’t have any solution with probability 1 .

Original: It leaves us with the case when M ⊥ − µI is non-degenerate, when there exists a unique solution y ( µ ) :
Revised: It leaves us with the case when M ⊥ − µI is non-degenerate, when there exists a unique solution y ( µ ) :

Original: = − ( M ⊥ − µI ) − 1 v ⊥ .
Revised: = − ( M ⊥ − µI ) − 1 v ⊥ .

Original: Diagonalization.
Revised: Diagonalization.

Original: Dependence of ( M ⊥ − µI ) − 1 on µ is non-trivial, but for a diagonal M ⊥ , the inverse can be found explicitly.
Revised: Dependence of ( M ⊥ − µI ) − 1 on µ is non-trivial, but for a diagonal M ⊥ , the inverse can be found explicitly.

Original: Hence, we perform diagonalization of M ⊥ : we find orthogonal
Revised: Hence, we perform diagonalization of M ⊥ : we find orthogonal

Original: Q and diagonal Λ such that ∥ M ⊥ − Q ⊤ ΛQ ∥ < ε (Line 5) in time O
Revised: Q and diagonal Λ such that ∥ M ⊥ − Q ⊤ ΛQ ∥ < ε (Line 5) in time O

Original: ( d 3 log 1 / ε ) [30].
Revised: ( d 3 log 1 / ε ) [30].

Original: Setting ε
Revised: Setting ε

Original: = O ( δ / r 2 ⊥ )
Revised: = O ( δ / r 2 ⊥ )

Original: , we guarantee that the function changes by at most O ( δ ) in B ( r ⊥ ) .
Revised: , we guarantee that the function changes by at most O ( δ ) in B ( r ⊥ ) .

Original: The function y ( µ ) = − ( Q ⊤ ΛQ − µI ) − 1 v ⊥ can be written as Qy ( µ ) =
Revised: The function y ( µ ) = − ( Q ⊤ ΛQ − µI ) − 1 v ⊥ can be written as Qy ( µ ) =

Original: − ( Λ − µI )
Revised: − ( Λ − µI )

Original: Qv ⊥ , and hence we work with rotated vectors ˜ y ( µ ) :
Revised: Qv ⊥ , and hence we work with rotated vectors ˜ y ( µ ) :

Original: =
Revised: =

Original: Qy ( µ ) and ˜ v :=
Revised: Qy ( µ ) and ˜ v :=

Original: Qv ⊥ (Line 6).
Revised: Qv ⊥ (Line 6).

Original: Finding candidate µ .
Revised: Finding candidate µ .

Original: Since ˜ y ( µ ) = − ( M ⊥ − µI ) − 1 ˜ v , for the i -th coordinate of ˜ y ( µ ) we have ˜ y
Revised: Since ˜ y ( µ ) = − ( M ⊥ − µI ) − 1 ˜ v , for the i -th coordinate of ˜ y ( µ ) we have ˜ y

Original: i ( µ ) =
Revised: i ( µ ) =

Original: ˜ v i µ − λ
Revised: ˜ v i µ − λ

Original: i (Line 9).
Revised: i (Line 9).

Original: Since we are only interested in y ( µ ) ∈ ∂ B ( r ⊥ ) and Q is an orthogonal matrix, we must have (Line 7): ∥
Revised: Since we are only interested in y ( µ ) ∈ ∂ B ( r ⊥ ) and Q is an orthogonal matrix, we must have (Line 7): ∥

Original: y ( µ ) ∥ 2 = ∥
Revised: y ( µ ) ∥ 2 = ∥

Original: ˜ y ( µ ) ∥ 

Revised: ˜ y ( µ ) ∥

Original: = (cid:80) di
Revised: = (cid:80) di

Original: =1 ˜ y
Revised: =1 ˜ y

Original: i ( µ ) 2 = r 2 ⊥ , and hence (cid:80) i ˜ v 2 i ( µ − λ i ) 2 = r 2 ⊥ .
Revised: i ( µ ) 2 = r 2 ⊥ , and hence (cid:80) i ˜ v 2 i ( µ − λ i ) 2 = r 2 ⊥ .

Original: After multiplying the equation by (cid:81) i ( µ − λ i ) 2 , we get an equation of the form p ( µ ) = 0 , where p is a polynomial of degree 2 d .
Revised: After multiplying the equation by (cid:81) i ( µ − λ i ) 2 , we get an equation of the form p ( µ ) = 0 , where p is a polynomial of degree 2 d .

Original: We find roots µ 1 , . . .
Revised: We find roots µ 1 , . . .

Original: , µ 2 d of the polynomial in time O ( d 2 log d · log log 1 / ε ) [29], where ε is the required root precision.
Revised: , µ 2 d of the polynomial in time O ( d 2 log d · log log 1 / ε ) [29], where ε is the required root precision.

Original: For each i , we compute y ( µ i ) and verify whether it lies in B ( r ⊥ )
Revised: For each i , we compute y ( µ i ) and verify whether it lies in B ( r ⊥ )

Original: ∩ S ⊥ and improves the objective by − Ω( δ ) .
Revised: ∩ S ⊥ and improves the objective by − Ω( δ ) .

Original: Precision.
Revised: Precision.

Original: Since the roots of the polynomial are approximate, when µ is close to λ i , even a small perturbation of µ can strongly affect y i ( µ ) =
Revised: Since the roots of the polynomial are approximate, when µ is close to λ i , even a small perturbation of µ can strongly affect y i ( µ ) =

Original: ˜ v i µ − λ i .
Revised: ˜ v i µ − λ i .

Original: We solve this as follows: since ∥ ˜ y ( µ ) ∥ =
Revised: We solve this as follows: since ∥ ˜ y ( µ ) ∥ =

Original: r ⊥ , for each i we must have | ˜
Revised: r ⊥ , for each i we must have | ˜

Original: y
Revised: y

Original: i ( µ ) | ≤ r ⊥ , implying | µ − λ i | ≥ | ˜ v i | r ⊥ .
Revised: i ( µ ) | ≤ r ⊥ , implying | µ − λ i | ≥ | ˜ v i | r ⊥ .

Original: Therefore, µ must be sufficiently far from any λ i , where the lower bound on the distance depends on r ⊥ ≤ 3 (cid:112) δ / ρ and on | ˜ v i | .
Revised: Therefore, µ must be sufficiently far from any λ i , where the lower bound on the distance depends on r ⊥ ≤ 3 (cid:112) δ / ρ and on | ˜ v i | .

Original: By adding noise to v (Line 7 of Algorithm 3), we guarantee that the noise is preserved in ˜ v so that each coordinate is sufficiently separated from 0 w.h.p.
Revised: By adding noise to v (Line 7 of Algorithm 3), we guarantee that the noise is preserved in ˜ v so that each coordinate is sufficiently separated from 0 w.h.p.

Original: This is formalized in Appendix C, Lemma 24.
Revised: This is formalized in Appendix C, Lemma 24.

Original: C.3. Proof of correctness
Revised: C.3. Proof of correctness

Original: Algorithm 5: V RSG ( x , ˜ σ ) :
Revised: Algorithm 5: V RSG ( x , ˜ σ ) :

Original: Variance Reduced Stochastic Gradient parameters: stochastic gradient descent variance σ , error probability ε output : g such that ∥ g − ∇ f ( x ) ∥ < ˜ σ with probability at least 1 − ε σ = return ∇ f ( x ) ;
Revised: Variance Reduced Stochastic Gradient parameters: stochastic gradient descent variance σ , error probability ε output : g such that ∥ g − ∇ f ( x ) ∥ < ˜ σ with probability at least 1 − ε σ = return ∇ f ( x ) ;

Original: [Equation]
Revised: [Equation]

Original: , M do
Revised: , M do

Original: Sample g i, 1 , . . .
Revised: Sample g i, 1 , . . .

Original: , g i,K – stochastic gradients f at m
Revised: , g i,K – stochastic gradients f at m

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Table]
Revised: [Table]

Original: g i,j return coordinate median
Revised: g i,j return coordinate median

Original: , m
Revised: , m

Original: Lemma 20 Let stochastic gradient oracle satisfy Assumption 2.
Revised: Lemma 20 Let stochastic gradient oracle satisfy Assumption 2.

Original: Then for any ˜ σ > 0 , Algorithmreturns a vector g with ∥ g −∇ f ( x ) ∥ <
Revised: Then for any ˜ σ > 0 , Algorithmreturns a vector g with ∥ g −∇ f ( x ) ∥ <

Original: ˜ σ with probability 1 − O ( ε ) using O (1+ σ 2 ˜ σ 2 log dε ) stochastic gradient oracle calls.
Revised: ˜ σ with probability 1 − O ( ε ) using O (1+ σ 2 ˜ σ 2 log dε ) stochastic gradient oracle calls.

Original: Proof Since the stochastic gradients are sampled independently and E
Revised: Proof Since the stochastic gradients are sampled independently and E

Original: ∥ g i,j − ∇ f ( x ) ∥ 2 ≤ σ 2 by
Revised: ∥ g i,j − ∇ f ( x ) ∥ 2 ≤ σ 2 by

Original: Assumption 2, for each i we have
Revised: Assumption 2, for each i we have

Original: [Equation]
Revised: [Equation]

Original: Applying Fact 11 to m 1 , . .
Revised: Applying Fact 11 to m 1 , . .

Original: , m M , we have ∥ g − ∇ f ( x ) ∥ <
Revised: , m M , we have ∥ g − ∇ f ( x ) ∥ <

Original: ˜ σ with probability
Revised: ˜ σ with probability

Original: [Equation]
Revised: [Equation]

Original: Lemma 21 (Lemma 19) Let f satisfy Assumptions 1 and 2.
Revised: Lemma 21 (Lemma 19) Let f satisfy Assumptions 1 and 2.

Original: Let f ′ ( x + h ) =
Revised: Let f ′ ( x + h ) =

Original: f ( x )+ h ⊤ v + 12 h ⊤ Mh , where v and M are as in Algorithm 3.
Revised: f ( x )+ h ⊤ v + 12 h ⊤ Mh , where v and M are as in Algorithm 3.

Original: For δ > 0 , r = 3 (cid:112) δ / ρ we have ∥ f ′ ( x + h ) − f ( x + h ) ∥ <
Revised: For δ > 0 , r = 3 (cid:112) δ / ρ we have ∥ f ′ ( x + h ) − f ( x + h ) ∥ <

Original: [Equation]
Revised: [Equation]

Original: for all h ∈ B ( r ) w.h.p.
Revised: for all h ∈ B ( r ) w.h.p.

Original: Proof Recall that v and M are noisy estimations of the gradient and the Hessian (below we reiterate how they are computed).
Revised: Proof Recall that v and M are noisy estimations of the gradient and the Hessian (below we reiterate how they are computed).

Original: By the Taylor expansion of f , we have
Revised: By the Taylor expansion of f , we have

Original: [Equation]
Revised: [Equation]

Original: where ϕ ( h ) =
Revised: where ϕ ( h ) =

Original: O ( ∥ h ∥ 3 ) .
Revised: O ( ∥ h ∥ 3 ) .

Original: To bound | f ′ ( x + h ) − f ( x + h )
Revised: To bound | f ′ ( x + h ) − f ( x + h )

Original: | , we analyze difference between f and f ′ for linear, quadratic and higher-order terms separately.
Revised: | , we analyze difference between f and f ′ for linear, quadratic and higher-order terms separately.

Original: Namely, we bound | h ⊤ ∇ f ( x ) − h ⊤ v | , | 12 h ⊤ ∇ 2 f ( x ) h −
Revised: Namely, we bound | h ⊤ ∇ f ( x ) − h ⊤ v | , | 12 h ⊤ ∇ 2 f ( x ) h −

Original: 12 h
Revised: 12 h

Original: ⊤ Mh | , and | ϕ ( h ) | with δ 6 each.
Revised: ⊤ Mh | , and | ϕ ( h ) | with δ 6 each.

Original: Higher-order terms
Revised: Higher-order terms

Original: Since f has a ρ -Lipschitz Hessian, we have | ϕ ( h ) | ≤ ρr 3 6 ≤ δ 6 for r = 3 (cid:112) δ / ρ .
Revised: Since f has a ρ -Lipschitz Hessian, we have | ϕ ( h ) | ≤ ρr 3 6 ≤ δ 6 for r = 3 (cid:112) δ / ρ .

Original: Linear terms
Revised: Linear terms

Original: Recall that v = g + ζ , where g = VRSG( x , O ( δ / r )) and ζ ∼ N ( 0 , ξ δ r √ d · I ) for some small ξ .
Revised: Recall that v = g + ζ , where g = VRSG( x , O ( δ / r )) and ζ ∼ N ( 0 , ξ δ r √ d · I ) for some small ξ .

Original: By Lemma 20, ∥ g − ∇ f ( x ) ∥ <
Revised: By Lemma 20, ∥ g − ∇ f ( x ) ∥ <

Original: O ( δ / r ) w.h.p.
Revised: O ( δ / r ) w.h.p.

Original: By choosing the appropriate constants, with probability 1 − O ( ε ) we have
Revised: By choosing the appropriate constants, with probability 1 − O ( ε ) we have

Original: [Equation]
Revised: [Equation]

Original: where we selected ξ ≤ 112 log − 1 1 ε .
Revised: where we selected ξ ≤ 112 log − 1 1 ε .

Original: Using this bound, for all h ∈ B ( r ) we have:
Revised: Using this bound, for all h ∈ B ( r ) we have:

Original: [Equation]
Revised: [Equation]

Original: Quadratic terms
Revised: Quadratic terms

Original: Recall that M = H + γI , where γ ∼ U [ − ξ, ξ ] and H is as in Algorithm 3: for { e i } – the standard basis in R d , θ = Θ( δρr 2 ) and ˜ σ =
Revised: Recall that M = H + γI , where γ ∼ U [ − ξ, ξ ] and H is as in Algorithm 3: for { e i } – the standard basis in R d , θ = Θ( δρr 2 ) and ˜ σ =

Original: Θ( δ 2 ρr 2 ) , the i -th column of H is g
Revised: Θ( δ 2 ρr 2 ) , the i -th column of H is g

Original: i − g (0)
Revised: i − g (0)

Original: θ with g i =
Revised: θ with g i =

Original: VRSG( x + θ e i , ˜ σ ) and g (0) =
Revised: VRSG( x + θ e i , ˜ σ ) and g (0) =

Original: VRSG( x + θ e i , ˜ σ ) .
Revised: VRSG( x + θ e i , ˜ σ ) .

Original: For all h ∈ B ( r ) we have:
Revised: For all h ∈ B ( r ) we have:

Original: [Equation]
Revised: [Equation]

Original: The second term is bounded by selecting a sufficiently small γ (i.e. a sufficiently small ξ ), and hence it remains to bound the second term.
Revised: The second term is bounded by selecting a sufficiently small γ (i.e. a sufficiently small ξ ), and hence it remains to bound the second term.

Original: Let w i be the difference between ∇ f ( x + θ e i ) −∇ f ( x ) θ and its stochastic estimate:
Revised: Let w i be the difference between ∇ f ( x + θ e i ) −∇ f ( x ) θ and its stochastic estimate:

Original: [Equation]
Revised: [Equation]

Original: Since g i and g (0) are within distance ˜ σ of the true gradients, we have ∥ w i ∥
Revised: Since g i and g (0) are within distance ˜ σ of the true gradients, we have ∥ w i ∥

Original: =
Revised: =

Original: O (cid:0) ˜ σθ (cid:1) .
Revised: O (cid:0) ˜ σθ (cid:1) .

Original: Since ∇ 2 f ( x ) e i is the i -th column of ∇ 2 f ( x ) , we have:
Revised: Since ∇ 2 f ( x ) e i is the i -th column of ∇ 2 f ( x ) , we have:

Original: [Equation]
Revised: [Equation]

Original: where ∥ u i ∥ =
Revised: where ∥ u i ∥ =

Original: O ( ρθ ) since ∇ 2 f is ρ -Lipschitz.
Revised: O ( ρθ ) since ∇ 2 f is ρ -Lipschitz.

Original: Therefore, for any h ∈ B ( r ) we have
Revised: Therefore, for any h ∈ B ( r ) we have

Original: [Equation]
Revised: [Equation]

Original: and hence:
Revised: and hence:

Original: [Equation]
Revised: [Equation]

Original: where the third line follows from ∥ w i ∥
Revised: where the third line follows from ∥ w i ∥

Original: =
Revised: =

Original: O ( ˜ σθ ) and ∥ u i ∥ =
Revised: O ( ˜ σθ ) and ∥ u i ∥ =

Original: O ( ρθ )) .
Revised: O ( ρθ )) .

Original: Hence, it suffices to choose θ so that:
Revised: Hence, it suffices to choose θ so that:

Original: [Equation]
Revised: [Equation]

Original: where we used r = 3 (cid:112) δ / ρ .
Revised: where we used r = 3 (cid:112) δ / ρ .

Original: Similarly, we choose ˜ σ so that:
Revised: Similarly, we choose ˜ σ so that:

Original: [Equation]
Revised: [Equation]

Original: Lemma 22 Let δ, r > 0 .
Revised: Lemma 22 Let δ, r > 0 .

Original: Let f ( x ) = 12 x ⊤ Mx + x ⊤ v + C , with the following conditions satisfied:
Revised: Let f ( x ) = 12 x ⊤ Mx + x ⊤ v + C , with the following conditions satisfied:

Original: • λ max ( M ) ≤ L • (Large linear term)
Revised: • λ max ( M ) ≤ L • (Large linear term)

Original: ∥ v ∥ ≥ 4( Ldr + ( C + δ ) d r ) • For x ⋆ = argmin x ∈ S ∩B ( r ) f
Revised: ∥ v ∥ ≥ 4( Ldr + ( C + δ ) d r ) • For x ⋆ = argmin x ∈ S ∩B ( r ) f

Original: ( x ) we have f ( x ⋆ ) <
Revised: ( x ) we have f ( x ⋆ ) <

Original: − δ
Revised: − δ

Original: Then either ( x ⋆ ) ⊤ v < − r 8 d ∥ v ∥ or x ⋆ ∈ ∂S .
Revised: Then either ( x ⋆ ) ⊤ v < − r 8 d ∥ v ∥ or x ⋆ ∈ ∂S .

Original: Remark 23 As we show in the proof of Lemma 24, if ∥ v ∥ ≥ Ldr + ( C + δ ) d r and x ⋆ / ∈ ∂S , the linear term alone is sufficient to sufficiently improve the function, and the quadratic term is negligible.
Revised: Remark 23 As we show in the proof of Lemma 24, if ∥ v ∥ ≥ Ldr + ( C + δ ) d r and x ⋆ / ∈ ∂S , the linear term alone is sufficient to sufficiently improve the function, and the quadratic term is negligible.

Original: Proof For contradiction, assume that ( x ⋆ ) ⊤ v ≥ − r 4 d ∥ v ∥ and x ⋆ ∈ Int S .
Revised: Proof For contradiction, assume that ( x ⋆ ) ⊤ v ≥ − r 4 d ∥ v ∥ and x ⋆ ∈ Int S .

Original: We show that we can perturb x ⋆ so that the point still lies in B ( r ) ∩ S and the objective decreases.
Revised: We show that we can perturb x ⋆ so that the point still lies in B ( r ) ∩ S and the objective decreases.

Original: We consider two cases: when x ⋆ ∈ Int B ( r ) , we can simply move in the direction of − v .
Revised: We consider two cases: when x ⋆ ∈ Int B ( r ) , we can simply move in the direction of − v .

Original: When x ⋆ ∈ ∂ B ( r ) , we rely on the fact that ( x ⋆ ) ⊤ v < − r 8 d ∥ v ∥ , which guarantees that small change in direction − v requires a small change in the orthogonal direction.
Revised: When x ⋆ ∈ ∂ B ( r ) , we rely on the fact that ( x ⋆ ) ⊤ v < − r 8 d ∥ v ∥ , which guarantees that small change in direction − v requires a small change in the orthogonal direction.

Original: Case 1: x ⋆ ∈ Int B ( r ) .
Revised: Case 1: x ⋆ ∈ Int B ( r ) .

Original: Let x ⋆ = α v + w , where w ⊥ v .
Revised: Let x ⋆ = α v + w , where w ⊥ v .

Original: Since x ⋆ ∈ Int( B ( r ) ∩ S ) , for a sufficiently small ε we have x ⋆ − ε v ∈ B ( r ) ∩ S , and then:
Revised: Since x ⋆ ∈ Int( B ( r ) ∩ S ) , for a sufficiently small ε we have x ⋆ − ε v ∈ B ( r ) ∩ S , and then:

Original: [Equation]
Revised: [Equation]

Original: contradicting the fact that x ⋆ is the minimizer.
Revised: contradicting the fact that x ⋆ is the minimizer.

Original: Case 2: x ⋆ ∈ ∂ B ( r ) .
Revised: Case 2: x ⋆ ∈ ∂ B ( r ) .

Original: Let x ⋆ = − α v + w , where w ⊥ v .
Revised: Let x ⋆ = − α v + w , where w ⊥ v .

Original: Since ( x ⋆ ) ⊤ v ≥ − r 8 d ∥ v ∥ , we have α ≤ r 8 d ∥ v ∥ , and hence ∥ w ∥ ≥ r 2 and α ∥ v ∥ ≤ ∥ w ∥ 4 d .
Revised: Since ( x ⋆ ) ⊤ v ≥ − r 8 d ∥ v ∥ , we have α ≤ r 8 d ∥ v ∥ , and hence ∥ w ∥ ≥ r 2 and α ∥ v ∥ ≤ ∥ w ∥ 4 d .

Original: We’ll construct x ′ such that x ′ ∈ B
Revised: We’ll construct x ′ such that x ′ ∈ B

Original: ( r ) , x ′ is arbitrarily close to x ⋆ , and, since x ⋆ ∈ Int S , we also have x ′
Revised: ( r ) , x ′ is arbitrarily close to x ⋆ , and, since x ⋆ ∈ Int S , we also have x ′

Original: ∈
Revised: ∈

Original: Int S .
Revised: Int S .

Original: Let x ′ =
Revised: Let x ′ =

Original: − ( α + ε ) v +(1 − θ ) w for ε > 0 .
Revised: − ( α + ε ) v +(1 − θ ) w for ε > 0 .

Original: To have x ′ ∈ B ( r ) , it suffices to guarantee that ∥ x ′ ∥ ≤ ∥ x ⋆ ∥ , meaning:
Revised: To have x ′ ∈ B ( r ) , it suffices to guarantee that ∥ x ′ ∥ ≤ ∥ x ⋆ ∥ , meaning:

Original: [Equation]
Revised: [Equation]

Original: and hence for x ⋆ ∈ ∂ B ( r ) it suffices to select θ = 2 αε ∥ v ∥ 2 ∥ w ∥ 2 .
Revised: and hence for x ⋆ ∈ ∂ B ( r ) it suffices to select θ = 2 αε ∥ v ∥ 2 ∥ w ∥ 2 .

Original: Since θ = O ( ε ) , ∥ x ′ − x ⋆ ∥ = O ( ε ) , and hence we can select x ′ arbitrarily close to x ⋆ , which guarantees x ′ ∈ S .
Revised: Since θ = O ( ε ) , ∥ x ′ − x ⋆ ∥ = O ( ε ) , and hence we can select x ′ arbitrarily close to x ⋆ , which guarantees x ′ ∈ S .

Original: It remains to show that f ( x ′ ) <
Revised: It remains to show that f ( x ′ ) <

Original: f ( x ⋆ ) .
Revised: f ( x ⋆ ) .

Original: Using θ = 2 αε ∥ v ∥ 2 ∥ w ∥
Revised: Using θ = 2 αε ∥ v ∥ 2 ∥ w ∥

Original: [Equation]
Revised: [Equation]

Original: Lemma 24 For δ, r, θ > 0 , let S =
Revised: Lemma 24 For δ, r, θ > 0 , let S =

Original: { x | Ay ≥ b } and let f ( x ) = 12 x ⊤ Mx + x ⊤ v + C satisfy the following conditions:
Revised: { x | Ay ≥ b } and let f ( x ) = 12 x ⊤ Mx + x ⊤ v + C satisfy the following conditions:

Original: λ | max | ( M ) ≤ L , 2.
Revised: λ | max | ( M ) ≤ L , 2.

Original: ( M − µI ) y
Revised: ( M − µI ) y

Original: = − v has no solutions for all µ such that rank( Λ − µI ) <
Revised: = − v has no solutions for all µ such that rank( Λ − µI ) <

Original: d , 3.
Revised: d , 3.

Original: | ˜ v i | ≥ θ for all i , where ˜ v =
Revised: | ˜ v i | ≥ θ for all i , where ˜ v =

Original: Qv and Q is an orthogonal matrix defined in Line 5 of Algorithm 4.
Revised: Qv and Q is an orthogonal matrix defined in Line 5 of Algorithm 4.

Original: Let x ⋆ = argmin y ∈ S ∩B ( r ) f ( x ) and assume that f ( x ⋆ ) <
Revised: Let x ⋆ = argmin y ∈ S ∩B ( r ) f ( x ) and assume that f ( x ⋆ ) <

Original: − δ and x ⋆ ∈ Int S .
Revised: − δ and x ⋆ ∈ Int S .

Original: Then Algorithmfinds x ∈ S ∩ B ( r ) such that f ( x ) < − (1 − ε ) δ in time ˜ O
Revised: Then Algorithmfinds x ∈ S ∩ B ( r ) such that f ( x ) < − (1 − ε ) δ in time ˜ O

Original: ( d 3 ) .
Revised: ( d 3 ) .

Original: Remark 

Revised: Remark

Original: In the statement above, we select θ sufficiently small so that adding random noise to ∇ f ( 0 ) guarantees that condition 3 is satisfied w.h.p.
Revised: In the statement above, we select θ sufficiently small so that adding random noise to ∇ f ( 0 ) guarantees that condition 3 is satisfied w.h.p.

Original: Condition 2 is not required, but it’s easy to satisfy and simplifies the analysis.
Revised: Condition 2 is not required, but it’s easy to satisfy and simplifies the analysis.

Original: Proof As outlined in Section 2.2, we analyze the cases from Algorithm 3 separately.
Revised: Proof As outlined in Section 2.2, we analyze the cases from Algorithm 3 separately.

Original: We consider the situation when ∥ v ∥ is large as a separate case, since in this case the function is sensitive to the change of its argument.
Revised: We consider the situation when ∥ v ∥ is large as a separate case, since in this case the function is sensitive to the change of its argument.

Original: Otherwise, for x ⋆ , there are only two options: either x ⋆ ∈ Int B ( r ) or x
Revised: Otherwise, for x ⋆ , there are only two options: either x ⋆ ∈ Int B ( r ) or x

Original: ⋆ ∈ ∂ B ( r ) .
Revised: ⋆ ∈ ∂ B ( r ) .

Original: Case 1: Large ∥ v ∥ .
Revised: Case 1: Large ∥ v ∥ .

Original: If ∥ v ∥ ≥ 4( Ldr + ( C + δ ) d r ) , then by Lemma 22 either ( x ⋆ ) ⊤ v < − r 4 d ∥ v ∥ or x ⋆ ∈ ∂S .
Revised: If ∥ v ∥ ≥ 4( Ldr + ( C + δ ) d r ) , then by Lemma 22 either ( x ⋆ ) ⊤ v < − r 4 d ∥ v ∥ or x ⋆ ∈ ∂S .

Original: Since we assume that x ⋆ ∈ Int S , it leaves us with the case ( x ⋆ )
Revised: Since we assume that x ⋆ ∈ Int S , it leaves us with the case ( x ⋆ )

Original: ⊤ v < − r 4 d ∥ v ∥ .
Revised: ⊤ v < − r 4 d ∥ v ∥ .

Original: Then
Revised: Then

Original: Algorithm 3 finds x with x ⊤ v < − rd ∥ v ∥ and hence
Revised: Algorithm 3 finds x with x ⊤ v < − rd ∥ v ∥ and hence

Original: [Equation]
Revised: [Equation]

Original: Case 2: x ⋆ ∈ Int S ∩ Int B ( r ) .
Revised: Case 2: x ⋆ ∈ Int S ∩ Int B ( r ) .

Original: In this case, x ⋆ is an unconstrained critical point, and so we must have ∇ f ( x ) =
Revised: In this case, x ⋆ is an unconstrained critical point, and so we must have ∇ f ( x ) =

Original: Mx + v = 0 .
Revised: Mx + v = 0 .

Original: If M is degenerate, by condition 2, there are no solutions.
Revised: If M is degenerate, by condition 2, there are no solutions.

Original: Otherwise, there exists a unique x such that Mx = − v .
Revised: Otherwise, there exists a unique x such that Mx = − v .

Original: We then check that x ∈ S ∩ B ( r ) and f ( x ) < f ( 0 ) − δ 2 .
Revised: We then check that x ∈ S ∩ B ( r ) and f ( x ) < f ( 0 ) − δ 2 .

Original: [Equation]
Revised: [Equation]

Original: By the method of Lagrangian multipliers, we must have Mx + v = µ x for some λ .
Revised: By the method of Lagrangian multipliers, we must have Mx + v = µ x for some λ .

Original: By condition 2, if rank( M − µI ) < d , then ( M − µI ) x = − v has no solutions.
Revised: By condition 2, if rank( M − µI ) < d , then ( M − µI ) x = − v has no solutions.

Original: Hence, it suffices to consider cases when µ is not an eigenvalue of M .
Revised: Hence, it suffices to consider cases when µ is not an eigenvalue of M .

Original: When µ is not an eigenvalue of M , ( M − µI ) x = − v has a unique solution.
Revised: When µ is not an eigenvalue of M , ( M − µI ) x = − v has a unique solution.

Original: Based on [30], we find orthogonal Q and diagonal Λ such that ∥ M − Q ⊤ ΛQ ∥ < ε , which can be done in time
Revised: Based on [30], we find orthogonal Q and diagonal Λ such that ∥ M − Q ⊤ ΛQ ∥ < ε , which can be done in time

Original: O ( d 3 log 1 / ε ) .
Revised: O ( d 3 log 1 / ε ) .

Original: By setting ε =
Revised: By setting ε =

Original: O ( δ / r 2 ) , we guarantee that the function changes by at most O ( δ ) in
Revised: O ( δ / r 2 ) , we guarantee that the function changes by at most O ( δ ) in

Original: B ( r ) .
Revised: B ( r ) .

Original: Then ( Q ⊤ ΛQ − µI ) x = − v is equivalent to
Revised: Then ( Q ⊤ ΛQ − µI ) x = − v is equivalent to

Original: [Equation]
Revised: [Equation]

Original: Let ˜ v =
Revised: Let ˜ v =

Original: Qv and ˜ x
Revised: Qv and ˜ x

Original: = Qx .
Revised: = Qx .

Original: Then the solutions for ˜ x have coordinates ˜ x
Revised: Then the solutions for ˜ x have coordinates ˜ x

Original: i = − ˜ v i µ − λ i for all i .
Revised: i = − ˜ v i µ − λ i for all i .

Original: Since we are only interested in solutions on ∂ B ( r ) and Q is an orthogonal matrix, we must have
Revised: Since we are only interested in solutions on ∂ B ( r ) and Q is an orthogonal matrix, we must have

Original: [Equation]
Revised: [Equation]

Original: By multiplying both sides by (cid:81) i ( µ − λ i ) 2 , we get an equation of form p ( λ ) = 0 , where p is the polynomial with degree 2 d and the highest degree coefficient being r 2 .
Revised: By multiplying both sides by (cid:81) i ( µ − λ i ) 2 , we get an equation of form p ( λ ) = 0 , where p is the polynomial with degree 2 d and the highest degree coefficient being r 2 .

Original: We can find all the roots of p with precision ν (to be specified later) in time O
Revised: We can find all the roots of p with precision ν (to be specified later) in time O

Original: ( d 2 log d · log log 1 ν ) [29].
Revised: ( d 2 log d · log log 1 ν ) [29].

Original: Since we are only interested in real roots, we take the real parts of the approximate roots (if the exact root is real, taking the real part only improves the accuracy).
Revised: Since we are only interested in real roots, we take the real parts of the approximate roots (if the exact root is real, taking the real part only improves the accuracy).

Original: Next, we estimate the required precision ν .
Revised: Next, we estimate the required precision ν .

Original: Since we have ∥ ˜ x ∥ = r , it requires, in particular, that | x i | ≤ r for all i , which means that λ i must satisfy
Revised: Since we have ∥ ˜ x ∥ = r , it requires, in particular, that | x i | ≤ r for all i , which means that λ i must satisfy

Original: [Equation]
Revised: [Equation]

Original: Since | v i | ≥ θ by our assumption, we have | µ − λ i | ≥ θr .
Revised: Since | v i | ≥ θ by our assumption, we have | µ − λ i | ≥ θr .

Original: Let µ be the exact root of the polynomial and µ ′ ∈ [ µ − ν, µ + ν ] be the corresponding approximate root found the algorithm.
Revised: Let µ be the exact root of the polynomial and µ ′ ∈ [ µ − ν, µ + ν ] be the corresponding approximate root found the algorithm.

Original: Let ˜ x and ˜ x
Revised: Let ˜ x and ˜ x

Original: ′ be the points in R d corresponding to µ and µ ′ .
Revised: ′ be the points in R d corresponding to µ and µ ′ .

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: where we assumed ν < θ 2 r .
Revised: where we assumed ν < θ 2 r .

Original: If x = Q ⊤ ˜ x and x ′ =
Revised: If x = Q ⊤ ˜ x and x ′ =

Original: Q ⊤ ˜ x , then ∥ x − x ′ ∥ =
Revised: Q ⊤ ˜ x , then ∥ x − x ′ ∥ =

Original: ∥ ˜ x − ˜ x ′ ∥ ≤√ dr 2 ν θ .
Revised: ∥ ˜ x − ˜ x ′ ∥ ≤√ dr 2 ν θ .

Original: Let x ′′ be the projection of x ′ on B ( r ) ∩ S .
Revised: Let x ′′ be the projection of x ′ on B ( r ) ∩ S .

Original: Then ∥ x ′′ − x ∥ ≤ 2 ∥ x ′ − x ∥ , and hence:
Revised: Then ∥ x ′′ − x ∥ ≤ 2 ∥ x ′ − x ∥ , and hence:

Original: [Equation]
Revised: [Equation]

Original: by selecting ν <
Revised: by selecting ν <

Original: [Equation]
Revised: [Equation]

Original: Theorem 26 (Theorem 8) Let S
Revised: Theorem 26 (Theorem 8) Let S

Original: =
Revised: =

Original: { x | Ax ≤ b } be a set defined by an intersection of k linear inequality constraints.
Revised: { x | Ax ≤ b } be a set defined by an intersection of k linear inequality constraints.

Original: Let f satisfy Assumptions 1 and 2 and let min x ∈ S
Revised: Let f satisfy Assumptions 1 and 2 and let min x ∈ S

Original: f ( x ) =
Revised: f ( x ) =

Original: f ⋆ .
Revised: f ⋆ .

Original: Then there exists an algorithm which for δ > 0 finds a δ -SOSP in ˜ O
Revised: Then there exists an algorithm which for δ > 0 finds a δ -SOSP in ˜ O

Original: ( f ( x 0 ) − f ⋆ δ
Revised: ( f ( x 0 ) − f ⋆ δ

Original: [Equation]
Revised: [Equation]

Original: ) time using ˜ O
Revised: ) time using ˜ O

Original: ( f ( x 0 ) − f ⋆ δ ( d + d
Revised: ( f ( x 0 ) − f ⋆ δ ( d + d

Original: [Equation]
Revised: [Equation]

Original: σ 2 δ 4 / 3 )) stochastic gradient oracle calls w.h.p.
Revised: σ 2 δ 4 / 3 )) stochastic gradient oracle calls w.h.p.

Original: Proof It suffices to show that if x is not a δ -SOSP, then Algorithm 3 decreases function value by Ω( δ ) , and hence there can’t be more that O
Revised: Proof It suffices to show that if x is not a δ -SOSP, then Algorithm 3 decreases function value by Ω( δ ) , and hence there can’t be more that O

Original: ( f ( x 0 )
Revised: ( f ( x 0 )

Original: − f ⋆ δ ) iterations.
Revised: − f ⋆ δ ) iterations.

Original: W.l.o.g.
Revised: W.l.o.g.

Original: we can assume that the current iterate is 0 and f ( 0 )
Revised: we can assume that the current iterate is 0 and f ( 0 )

Original: = 0 .
Revised: = 0 .

Original: As shown in Lemma 19, we can replace f with its quadratic approximation 12 x ⊤ Mx + x ⊤ v while increasing the function value by at most δ 2 .
Revised: As shown in Lemma 19, we can replace f with its quadratic approximation 12 x ⊤ Mx + x ⊤ v while increasing the function value by at most δ 2 .

Original: As shown in Section C.1, we can consider function g ( y )
Revised: As shown in Section C.1, we can consider function g ( y )

Original: =
Revised: =

Original: 12 y ⊤ M ⊥
Revised: 12 y ⊤ M ⊥

Original: y
Revised: y

Original: + y ⊤ v ⊥ + C in S ⊥ ∩B ( r ⊥ ) .
Revised: + y ⊤ v ⊥ + C in S ⊥ ∩B ( r ⊥ ) .

Original: By Lemma 24, we can find y with g ( y ) <
Revised: By Lemma 24, we can find y with g ( y ) <

Original: − 12 δ in ˜ O
Revised: − 12 δ in ˜ O

Original: ( d 3 ) time.
Revised: ( d 3 ) time.

Original: It remains to show that the conditions from Lemma 24 are satisfied.
Revised: It remains to show that the conditions from Lemma 24 are satisfied.

Original: Recall that for O ∈ R d × dim A – an orthonormal basis of A , we define the following:
Revised: Recall that for O ∈ R d × dim A – an orthonormal basis of A , we define the following:

Original: • S ⊥ ← { y |
Revised: • S ⊥ ← { y |

Original: A ( p + Oy ) ≤ b } • M ⊥ ← O ⊤ MO • v ⊥ ← O ⊤ ( v + M ( p − x ))
Revised: A ( p + Oy ) ≤ b } • M ⊥ ← O ⊤ MO • v ⊥ ← O ⊤ ( v + M ( p − x ))

Original: As required in Lemma 24, S ⊥ is the set defined by linear inequality constraints.
Revised: As required in Lemma 24, S ⊥ is the set defined by linear inequality constraints.

Original: We next check other requirements in Lemma 24.
Revised: We next check other requirements in Lemma 24.

Original: 1) λ | max | ( M ⊥ ) ≤
Revised: 1) λ | max | ( M ⊥ ) ≤

Original: L :
Revised: L :

Original: Since O defines an orthonormal basis, λ | max |
Revised: Since O defines an orthonormal basis, λ | max |

Original: M ⊥
Revised: M ⊥

Original: = λ | max |
Revised: = λ | max |

Original: O ⊤
Revised: O ⊤

Original: MO
Revised: MO

Original: ) ≤
Revised: ) ≤

Original: λ | max | ( M ) ≤ L . 2) ( M ⊥ − µI )
Revised: λ | max | ( M ) ≤ L . 2) ( M ⊥ − µI )

Original: y = − v ⊥ has
Revised: y = − v ⊥ has

Original: no solutions
Revised: no solutions

Original: for all
Revised: for all

Original: µ
Revised: µ

Original: such that rank( Λ − µI ) <
Revised: such that rank( Λ − µI ) <

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: where ζ ∼ N ( 0 , ξ δ r √ d · I ) .
Revised: where ζ ∼ N ( 0 , ξ δ r √ d · I ) .

Original: Since O is an orthonormal basis, we have O ⊤ ζ ∼ N ( 0 , ξ δ r √ d · I ) .
Revised: Since O is an orthonormal basis, we have O ⊤ ζ ∼ N ( 0 , ξ δ r √ d · I ) .

Original: This
Revised: This

Original: Gaussian noise guarantees that the v ⊥ has the probability 0 to lie in any fixed subspace, which guarantees that there are no solutions for degenerate systems of the form ( M ⊥ − µI ) y
Revised: Gaussian noise guarantees that the v ⊥ has the probability 0 to lie in any fixed subspace, which guarantees that there are no solutions for degenerate systems of the form ( M ⊥ − µI ) y

Original: = − v ⊥ .
Revised: = − v ⊥ .

Original: [Equation]
Revised: [Equation]

Original: all i
Revised: all i

Original: Recall that ˜ v =
Revised: Recall that ˜ v =

Original: Qv ⊥ as defined in Line 6 of Algorithm 4, where Q is an orthogonal matrix.
Revised: Qv ⊥ as defined in Line 6 of Algorithm 4, where Q is an orthogonal matrix.

Original: For a fixed i and θ , we consider Pr ζ [ | ˜ v i | < θ ] .
Revised: For a fixed i and θ , we consider Pr ζ [ | ˜ v i | < θ ] .

Original: Due to rotation invariance of
Revised: Due to rotation invariance of

Original: Gaussian noise, its projection on any linear subspace has distribution N ( 0 , ξ δ r √ d I ) .
Revised: Gaussian noise, its projection on any linear subspace has distribution N ( 0 , ξ δ r √ d I ) .

Original: By selecting θ = ε ξδd 2 r , we have that the for all i , | ˜ v i | >
Revised: By selecting θ = ε ξδd 2 r , we have that the for all i , | ˜ v i | >

Original: θ with probability at least 1 − ε .
Revised: θ with probability at least 1 − ε .

Original: By selecting ε such that taking the union bound over all O ( f ( x 0 ) − f ⋆ δ ) iterations, d coordinates and 2 k sets of constraints gives bounded error probability, we finish the proof.
Revised: By selecting ε such that taking the union bound over all O ( f ( x 0 ) − f ⋆ δ ) iterations, d coordinates and 2 k sets of constraints gives bounded error probability, we finish the proof.