Original: SP2 : A S ECOND O RDER S TOCHASTIC P OLYAK 
Revised: SP2 : A S ECOND O RDER S TOCHASTIC P OLYAK M ETHOD

Original: M ETHOD

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently the SP (Stochastic Polyak step size) method has emerged as a competitive adaptive method for setting the step sizes of SGD.
Revised: Recently the SP (Stochastic Polyak step size) method has emerged as a competitive adaptive method for setting the step sizes of SGD.

Original: SP can be interpreted as a method specialized to interpolated models, since it solves the interpolation equations .
Revised: SP can be interpreted as a method specialized to interpolated models, since it solves the interpolation equations .

Original: SP solves these equation by using local linearizations of the model.
Revised: SP solves these equation by using local linearizations of the model.

Original: We take a step further and develop a method for solving the interpolation equations that uses the local second-order approximation of the model.
Revised: We take a step further and develop a method for solving the interpolation equations that uses the local second-order approximation of the model.

Original: Our resulting method SP2 uses
Revised: Our resulting method SP2 uses

Original: Hessian-vector products to speed-up the convergence of SP .
Revised: Hessian-vector products to speed-up the convergence of SP .

Original: Furthermore, and rather uniquely among second-order methods, the design of SP2 in no way relies on positive deﬁnite Hessian matrices or convexity of the objective function.
Revised: Furthermore, and rather uniquely among second-order methods, the design of SP2 in no way relies on positive deﬁnite Hessian matrices or convexity of the objective function.

Original: We show SP2 is very competitive on matrix completion, non-convex test problems and logistic regression.
Revised: We show SP2 is very competitive on matrix completion, non-convex test problems and logistic regression.

Original: We also provide a convergence theory on sums-of-quadratics.
Revised: We also provide a convergence theory on sums-of-quadratics.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Consider the problem
Revised: Consider the problem

Original: [Equation]
Revised: [Equation]

Original: where f is twice continuously differentiable, and the set of minimizers is nonempty.
Revised: where f is twice continuously differentiable, and the set of minimizers is nonempty.

Original: Let the optimal value of equation 1 be f ∗ ∈ R , and w 0 be a given initial point.
Revised: Let the optimal value of equation 1 be f ∗ ∈ R , and w 0 be a given initial point.

Original: Here each f i ( w ) is the loss of a model parametrized in w ∈ R d over an i -th data point.
Revised: Here each f i ( w ) is the loss of a model parametrized in w ∈ R d over an i -th data point.

Original: Our discussion, and forth coming results, also hold for a loss given as an expectation f ( w ) =
Revised: Our discussion, and forth coming results, also hold for a loss given as an expectation f ( w ) =

Original: E ξ ∼D [ f ξ ( w )] , where ξ ∼ D is the data generating process and f ξ ( w ) the loss over this sampled data point.
Revised: E ξ ∼D [ f ξ ( w )] , where ξ ∼ D is the data generating process and f ξ ( w ) the loss over this sampled data point.

Original: But for simplicity we use the f i ( w ) notation.
Revised: But for simplicity we use the f i ( w ) notation.

Original: Contrary to classic statistical modeling, there is now a growing trend of using overparametrized models that are able to interpolate the data Ma et al.
Revised: Contrary to classic statistical modeling, there is now a growing trend of using overparametrized models that are able to interpolate the data Ma et al.

Original: (2018); that is, models for which the loss is minimized over every data point as described in the following assumption.
Revised: (2018); that is, models for which the loss is minimized over every data point as described in the following assumption.

Original: Assumption 1.
Revised: Assumption 1.

Original: We say that the interpolation condition holds when the loss is nonnegative, f i ( w ) ≥ 0 , and [Equation] such that f ( w ∗ )
Revised: We say that the interpolation condition holds when the loss is nonnegative, f i ( w ) ≥ 0 , and there exists w ∗ ∈ R d such that f ( w ∗ )

Original: = 0 .
Revised: = 0 .

Original: [Equation]
Revised: [Equation]

Original: Consequently, f i ( w ∗ ) = 0 for i = 1 , . . . , n .
Revised: Consequently, f i ( w ∗ ) = 0 for i = 1 , . . . , n .

Original: Overparameterized deep neural networks are the most notorious example of models that satisfy Assumption 1.
Revised: Overparameterized deep neural networks are the most notorious example of models that satisfy Assumption 1.

Original: Indeed, with sufﬁciently more parameters than data points, we are able to simultaneously minimize the loss over all data points.
Revised: Indeed, with sufﬁciently more parameters than data points, we are able to simultaneously minimize the loss over all data points.

Original: If we admit that our model can interpolate the data, then we have that our optimization problem equation 1 is equivalent to solving the system of nonlinear equations
Revised: If we admit that our model can interpolate the data, then we have that our optimization problem equation 1 is equivalent to solving the system of nonlinear equations

Original: [Equation]
Revised: [Equation]

Original: Since we assume f i ( w ) ≥ 

Revised: Since we assume f i ( w ) ≥

Original: any solution to the above is a solution to our original problem.
Revised: any solution to the above is a solution to our original problem.

Original: Recently, it was shown in  Gower et al.
Revised: Recently, it was shown in Berrada et al. ; Gower et al.

Original: (2021b)
Revised: (2021b)

Original: that the Stochastic Polyak step size ( SP ) methodBerrada et al. (2020); Loizou et al.
Revised: that the Stochastic Polyak step size ( SP ) method Loizou et al.

Original: ; Polyak (1987)
Revised: ; Polyak (1987)

Original: [Equation]
Revised: [Equation]

Original: directly solves the interpolation equations.
Revised: directly solves the interpolation equations.

Original: Indeed, at each iteration SP samples a single i -th equation from equation 3, then projects the current iterate w t onto the linearization of this constraint, that is
Revised: Indeed, at each iteration SP samples a single i -th equation from equation 3, then projects the current iterate w t onto the linearization of this constraint, that is

Original: [Equation]
Revised: [Equation]

Original: Here we take one step further, and instead of projecting onto the linearization of f i ( w ) we use the local quadratic expansion.
Revised: Here we take one step further, and instead of projecting onto the linearization of f i ( w ) we use the local quadratic expansion.

Original: That is, as a proxy of setting f i ( w ) = 

Revised: That is, as a proxy of setting f i ( w ) =

Original: we set the quadratic expansion of f i ( w ) around w t to zero
Revised: we set the quadratic expansion of f i ( w ) around w t to zero

Original: [Equation]
Revised: [Equation]

Original: The above quadratic constraint could have inﬁnite solutions, a unique solution or no solution at all 1 .
Revised: The above quadratic constraint could have inﬁnite solutions, a unique solution or no solution at all 1 .

Original: Indeed, for example if ∇ 2 f i ( w t ) is positive deﬁnite, there may exist no solution, which occurs when f i is convex, and is the most studied setting for second order methods.
Revised: Indeed, for example if ∇ 2 f i ( w t ) is positive deﬁnite, there may exist no solution, which occurs when f i is convex, and is the most studied setting for second order methods.

Original: But if the loss is positive f i and the Hessian has at least one negative eigenvalue, then equation 6 always has a solution.
Revised: But if the loss is positive f i and the Hessian has at least one negative eigenvalue, then equation 6 always has a solution.

Original: If equation 6 has solutions, then analogously to the SP method, we can choose one using a projection step
Revised: If equation 6 has solutions, then analogously to the SP method, we can choose one using a projection step

Original: [Equation]
Revised: [Equation]

Original: We refer to equation 7 as the SP2 method.
Revised: We refer to equation 7 as the SP2 method.

Original: Using a quadratic expansion has several advantages.
Revised: Using a quadratic expansion has several advantages.

Original: First, quadratic expansions are more accurate than linearizations, which will allow us to take larger steps.
Revised: First, quadratic expansions are more accurate than linearizations, which will allow us to take larger steps.

Original: Furthermore, using the quadratic expansion will lead to convergence rates which are independent on how well conditioned the Hessian matrices are, as we show later in Proposition 1.
Revised: Furthermore, using the quadratic expansion will lead to convergence rates which are independent on how well conditioned the Hessian matrices are, as we show later in Proposition 1.

Original: Our SP2 method occupies a unique position in the literature of stochastic second order method since it is incremental and in no way relies on convexity or positive semi-deﬁnite Hessian matrices.
Revised: Our SP2 method occupies a unique position in the literature of stochastic second order method since it is incremental and in no way relies on convexity or positive semi-deﬁnite Hessian matrices.

Original: Indeed, as we will show in our non-convex experiments in 6.1 and matrix completition B, the SP2 excels at minimizing non-convex problems that satisfy interpolation.
Revised: Indeed, as we will show in our non-convex experiments in 6.1 and matrix completition B, the SP2 excels at minimizing non-convex problems that satisfy interpolation.

Original: In contrast, Newton based methods often converge to stationary points other than the global minima.
Revised: In contrast, Newton based methods often converge to stationary points other than the global minima.

Original: We also relax the interpolation assumption, and develop analogous quadratic methods for ﬁnding w and the smallest possible s ∈ R such that
Revised: We also relax the interpolation assumption, and develop analogous quadratic methods for ﬁnding w and the smallest possible s ∈ R such that

Original: [Equation]
Revised: [Equation]

Original: We refer to this as the slack interpolation equations, which were introduced in Crammer et al.
Revised: We refer to this as the slack interpolation equations, which were introduced in Crammer et al.

Original: (2006) for linear models.
Revised: (2006) for linear models.

Original: If the interpolation assumption holds then s = 0 and the above is equivalent to solving equation 3.
Revised: If the interpolation assumption holds then s = 0 and the above is equivalent to solving equation 3.

Original: When interpolation does not hold, then equation 8 is still a upper approximation of equation 1, as detailed in Gower et al.
Revised: When interpolation does not hold, then equation 8 is still a upper approximation of equation 1, as detailed in Gower et al.

Original: The rest of this paper is organized as follows.
Revised: The rest of this paper is organized as follows.

Original: We introduce some related work in Section 2.
Revised: We introduce some related work in Section 2.

Original: We present the proposed SP2 methods in Section 3 and corresponding convergence analysis in Section 4.
Revised: We present the proposed SP2 methods in Section 3 and corresponding convergence analysis in Section 4.

Original: In Section 5, we relax the interpolation condition and develop a variety of quadratic methods to solve the slack version of this problem.
Revised: In Section 5, we relax the interpolation condition and develop a variety of quadratic methods to solve the slack version of this problem.

Original: We test the proposed methods with a series of experiments in
Revised: We test the proposed methods with a series of experiments in

Original: Section 6.
Revised: Section 6.

Original: Finally, we conclude our work and discuss future directions in Section 7.
Revised: Finally, we conclude our work and discuss future directions in Section 7.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Since it became clear that Stochastic Gradient Descent (SGD), with appropriate step size tuning, was an efﬁcient method for solving the training problem equation 1, there has been a search for an efﬁcient second order counter part.
Revised: Since it became clear that Stochastic Gradient Descent (SGD), with appropriate step size tuning, was an efﬁcient method for solving the training problem equation 1, there has been a search for an efﬁcient second order counter part.

Original: The hope being, and our objective here, is to ﬁnd a second order stochastic method that is incremental ; that is, it can work with mini-batches, requires little to no tuning since it would depend less on how well scaled or conditioned the data is, and ﬁnally, would also apply to non-convex problems.
Revised: The hope being, and our objective here, is to ﬁnd a second order stochastic method that is incremental ; that is, it can work with mini-batches, requires little to no tuning since it would depend less on how well scaled or conditioned the data is, and ﬁnally, would also apply to non-convex problems.

Original: To date there is a vast literature on stochastic second order methods, yet none that achieve all of the above.
Revised: To date there is a vast literature on stochastic second order methods, yet none that achieve all of the above.

Original: The subsampled Newton methods such as (Roosta-Khorasani & Mahoney, 2019; Bollapragada et al., 2018; Liu & Roosta, 2021; Erdogdu & Montanari, 2015; Kohler & Lucchi, 2017; Jahani et al., 2017) require large batch sizes in order to guarantee that the subsampled Newton direction is close to the full Newton direction in high probability.
Revised: The subsampled Newton methods such as (Roosta-Khorasani & Mahoney, 2019; Bollapragada et al., 2018; Liu & Roosta, 2021; Erdogdu & Montanari, 2015; Kohler & Lucchi, 2017; Jahani et al., 2017) require large batch sizes in order to guarantee that the subsampled Newton direction is close to the full Newton direction in high probability.

Original: As such are not incremental.
Revised: As such are not incremental.

Original: Other examples of large sampled based methods include the Stochastic quasi-Newton methods (Byrd et al., 2011; Mokhtari & Ribeiro, 2015; Moritz et al., 2016;
Revised: Other examples of large sampled based methods include the Stochastic quasi-Newton methods (Byrd et al., 2011; Mokhtari & Ribeiro, 2015; Moritz et al., 2016;

Original: Gower et al., 2016; Wang et al., 2017; Berahas et al., 2016), stochastic cubic Newton Tripuraneni et al.
Revised: Gower et al., 2016; Wang et al., 2017; Berahas et al., 2016), stochastic cubic Newton Tripuraneni et al.

Original: , SDNA (Qu et al., 2016), Newton sketch (Pilanci &
Revised: , SDNA (Qu et al., 2016), Newton sketch (Pilanci &

Original: Wainwright, 2017) and Lissa (Agarwal et al., 2017), since these require a large mini-batch or full gradient evaluations.
Revised: Wainwright, 2017) and Lissa (Agarwal et al., 2017), since these require a large mini-batch or full gradient evaluations.

Original: The only incremental second order methods we aware of are IQN (Incremental QuasiNewton)
Revised: The only incremental second order methods we aware of are IQN (Incremental QuasiNewton)

Original: (Mokhtari et al., 2018), SNM (Stochastic Newton Method) (Kovalev et al., 2019;
Revised: (Mokhtari et al., 2018), SNM (Stochastic Newton Method) (Kovalev et al., 2019;

Original: Rodomanov & Kropotov, 2016) and very recently SAN (Stochastic Average Newton)
Revised: Rodomanov & Kropotov, 2016) and very recently SAN (Stochastic Average Newton)

Original: Chen et al.
Revised: Chen et al.

Original: IQN and
Revised: IQN and

Original: SNM enjoy a fast local convergence, but their computational and memory costs per iteration, is of
Revised: SNM enjoy a fast local convergence, but their computational and memory costs per iteration, is of

Original: O ( d 2 ) making them prohibitive in large dimensions.
Revised: O ( d 2 ) making them prohibitive in large dimensions.

Original: Handling non-convexity in second order methods is particularly challenging because most second order methods rely on convexity in their design.
Revised: Handling non-convexity in second order methods is particularly challenging because most second order methods rely on convexity in their design.

Original: For instance, the classic Newton iteration is the minima of the local quadratic approximation if this approximation is convex.
Revised: For instance, the classic Newton iteration is the minima of the local quadratic approximation if this approximation is convex.

Original: If it is not convex, the Newton step can be meaningless, or worse, a step uphill.
Revised: If it is not convex, the Newton step can be meaningless, or worse, a step uphill.

Original: Quasi-Newton methods maintain positive deﬁnite approximation of the Hessian matrix, and thus are also problematic when applied to non-convex problems Wang et al.
Revised: Quasi-Newton methods maintain positive deﬁnite approximation of the Hessian matrix, and thus are also problematic when applied to non-convex problems Wang et al.

Original: (2017) for which the Hessian is typically indeﬁnite.
Revised: (2017) for which the Hessian is typically indeﬁnite.

Original: Furthermore the incremental Newton methods IQN, SNM and SAN methods rely on the convexity of f i in their design.
Revised: Furthermore the incremental Newton methods IQN, SNM and SAN methods rely on the convexity of f i in their design.

Original: Indeed, without convexity, the iterates of IQN, SNM and SAN are not well deﬁned.
Revised: Indeed, without convexity, the iterates of IQN, SNM and SAN are not well deﬁned.

Original: In contrast, our approach of ﬁnding roots of the local quadratic approximation equation 7 in no way relies on convexity, and relies solely on the fact that the local quadratic approximation around w t is good if we are not far from w t .
Revised: In contrast, our approach of ﬁnding roots of the local quadratic approximation equation 7 in no way relies on convexity, and relies solely on the fact that the local quadratic approximation around w t is good if we are not far from w t .

Original: But our approach does introduce a new problem: the need to solve a system of quadratic equations.
Revised: But our approach does introduce a new problem: the need to solve a system of quadratic equations.

Original: We propose a series of methods to solve this in Sections 3 and 5.
Revised: We propose a series of methods to solve this in Sections 3 and 5.

Original: Solving quadratic equations has been heavily studied.
Revised: Solving quadratic equations has been heavily studied.

Original: There are even dedicated methods for solving
Revised: There are even dedicated methods for solving

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for a given ¯ w , where H is a nonzero symmetric ( not necessarily PSD ) matrix, and the level set { w : Q ( w ) = 0 } is nonempty.
Revised: for a given ¯ w , where H is a nonzero symmetric ( not necessarily PSD ) matrix, and the level set { w : Q ( w ) = 0 } is nonempty.

Original: Note that since Q ( w ) is a quadratic function, the problem equationis nonconvex.
Revised: Note that since Q ( w ) is a quadratic function, the problem equationis nonconvex.

Original: Yet despite this non-convexity, so long as there exists a feasible point, the projection equation 9 can be solved in polynomial time by re-writing the projection as a semi-deﬁnite program, or by using the S-procedure, which involves computing the eigenvalue decomposition of H and using a line search as proposed in Park & Boyd (2017), and detailed here in Section A.
Revised: Yet despite this non-convexity, so long as there exists a feasible point, the projection equation 9 can be solved in polynomial time by re-writing the projection as a semi-deﬁnite program, or by using the S-procedure, which involves computing the eigenvalue decomposition of H and using a line search as proposed in Park & Boyd (2017), and detailed here in Section A.

Original: But this approach is too costly when the dimension d is large.
Revised: But this approach is too costly when the dimension d is large.

Original: An alternative iterative method is proposed in Sosa & MP Raupp (2020), but only asymptotic convergence is guaranteed.
Revised: An alternative iterative method is proposed in Sosa & MP Raupp (2020), but only asymptotic convergence is guaranteed.

Original: In Dai (2006), the authors consider a similar problem by projecting a point onto a general ellipsoid, which is again a problem of solving quadratic equations.
Revised: In Dai (2006), the authors consider a similar problem by projecting a point onto a general ellipsoid, which is again a problem of solving quadratic equations.

Original: However, they require the matrix H to be a positive deﬁnite matrix.
Revised: However, they require the matrix H to be a positive deﬁnite matrix.

Original: The problem equation 7 and equation 9 are also an instance of a quadratic constrained quadratic program (QCQP).
Revised: The problem equation 7 and equation 9 are also an instance of a quadratic constrained quadratic program (QCQP).

Original: Although the QCQP in equation 7 has no closed form solution in general, we show in the next section that there is a closed form solution for Generalized linear models (GLMs), that holds for convex and non-convex GLMs alike.
Revised: Although the QCQP in equation 7 has no closed form solution in general, we show in the next section that there is a closed form solution for Generalized linear models (GLMs), that holds for convex and non-convex GLMs alike.

Original: For general non-linear models we propose in
Revised: For general non-linear models we propose in

Original: Section 3.2 an approximate solution to equation 7 by iteratively linearizing the quadratic constraint and projecting onto the linearization.
Revised: Section 3.2 an approximate solution to equation 7 by iteratively linearizing the quadratic constraint and projecting onto the linearization.

Original:  T HE SP2 M ETHOD
Revised:  T HE SP2 M ETHOD

Original: Next we give a closed form solution to equation 7 for GLMs.
Revised: Next we give a closed form solution to equation 7 for GLMs.

Original: We then provide an approximate solution to equation 7 for more general models.
Revised: We then provide an approximate solution to equation 7 for more general models.

Original: 3. SP2 - G ENERALIZED L INEAR M ODELS
Revised: 3. SP2 - G ENERALIZED L INEAR M ODELS

Original: Consider when f i is the loss over a linear model with
Revised: Consider when f i is the loss over a linear model with

Original: [Equation]
Revised: [Equation]

Original: (10) where φ i : R → R is a loss function, and ( x i , y i )
Revised: (10) where φ i : R → R is a loss function, and ( x i , y i )

Original: ∈
Revised: ∈

Original: R
Revised: R

Original: d +1 is an input-output pair.
Revised: d +1 is an input-output pair.

Original: Consequently
Revised: Consequently

Original: [Equation]
Revised: [Equation]

Original: The quadratic constraint problem equation 7 can be solved exactly for GLMs equation 10 as we show next.
Revised: The quadratic constraint problem equation 7 can be solved exactly for GLMs equation 10 as we show next.

Original: Lemma 1.
Revised: Lemma 1.

Original: (SP2) Assume f i ( w ) is the loss of a generalized linear model equation 10 and is nonnegative.
Revised: (SP2) Assume f i ( w ) is the loss of a generalized linear model equation 10 and is nonnegative.

Original: Let f i
Revised: Let f i

Original: = f
Revised: = f

Original: i ( w t ) for short.
Revised: i ( w t ) for short.

Original: Let a i :=
Revised: Let a i :=

Original: φ (cid:48)
Revised: φ (cid:48)

Original: i ( x (cid:62) i
Revised: i ( x (cid:62) i

Original: w − y i ) and h i := φ (cid:48)(cid:48)
Revised: w − y i ) and h i := φ (cid:48)(cid:48)

Original: i ( x (cid:62) i
Revised: i ( x (cid:62) i

Original: w − y i ) .
Revised: w − y i ) .

Original: If
Revised: If

Revised: [Equation]

Revised: i

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (12) then the optimal solution of equation 7 is as follows
Revised: (12) then the optimal solution of equation 7 is as follows

Original: [Equation]
Revised: [Equation]

Original: Alternatively if equation 12 does not hold, since f i ≥ 

Revised: Alternatively if equation 12 does not hold, since f i ≥

Original: we have necessarily that h i > 0 , and consequently a Newton step will give the minima of the local quadratic, that is
Revised: we have necessarily that h i > 0 , and consequently a Newton step will give the minima of the local quadratic, that is

Original: [Equation]
Revised: [Equation]

Original: The proof to the above lemma, and all subsequent missing proofs can be found in the appendix.
Revised: The proof to the above lemma, and all subsequent missing proofs can be found in the appendix.

Original: Lemma 1 establishes a condition equation 12 under which we should not take a full Newton step.
Revised: Lemma 1 establishes a condition equation 12 under which we should not take a full Newton step.

Original: Interestingly, this condition equation 12 holds when the square root of the loss function has negative curvature, as we show in the next lemma.
Revised: Interestingly, this condition equation 12 holds when the square root of the loss function has negative curvature, as we show in the next lemma.

Original: Lemma 2.
Revised: Lemma 2.

Original: Let φ be a non-negative function which is twice differentiable at all t with φ ( t ) (cid:54) = 0 .
Revised: Let φ be a non-negative function which is twice differentiable at all t with φ ( t ) (cid:54) = 0 .

Original: The condition equation 12, in other words
Revised: The condition equation 12, in other words

Original: [Equation]
Revised: [Equation]

Original: holds when (cid:112) φ ( t ) is concave away from its roots, i.e. when
Revised: holds when (cid:112) φ ( t ) is concave away from its roots, i.e. when

Original: [Equation]
Revised: [Equation]

Original: φ [Equation] t ) ≤ 0 for all t with φ ( t ) (cid:54) = 0 .
Revised: t ) ≤ 0 for all t with φ ( t ) (cid:54) = 0 .

Original: Examples of loss functions include φ ( t ) =
Revised: Examples of loss functions include φ ( t ) =

Original: tanh 2 ( t ) and φ ( t ) =
Revised: tanh 2 ( t ) and φ ( t ) =

Original: t p with 0 ≤ p ≤ 2 (see Figure (a) ).
Revised: t p with 0 ≤ p ≤ 2 (see Figure (a) ).

Original: 2 t 0.
Revised: 2 t 0.

Original: concave_funcs tanh| t | 1/ | t | 1/
Revised: concave_funcs tanh| t | 1/ | t | 1/

Original: In conclusion to this section, SP2 has a closed form solution for GLMs, and this closed form solution includes many non-convex loss functions.
Revised: In conclusion to this section, SP2 has a closed form solution for GLMs, and this closed form solution includes many non-convex loss functions.

Original: 3. SP2 + - L INEARIZING AND P ROJECTING
Revised: 3. SP2 + - L INEARIZING AND P ROJECTING

Original: In general, there is no closed form solution to equation 7.
Revised: In general, there is no closed form solution to equation 7.

Original: Indeed, there may not even exist a solution.
Revised: Indeed, there may not even exist a solution.

Original: Inspired by the fact that computing a Hessian-vector product can be done with a single backpropagation 3 at the same cost as computing a gradient Christianson (1992), we will make use of the cheap Hessian-vector product to derive an approximate solution to equation 7.
Revised: Inspired by the fact that computing a Hessian-vector product can be done with a single backpropagation 3 at the same cost as computing a gradient Christianson (1992), we will make use of the cheap Hessian-vector product to derive an approximate solution to equation 7.

Original: Instead of solving equation 7 exactly, here we propose to take two steps towards solving equationby projecting onto the linearized constraints.
Revised: Instead of solving equation 7 exactly, here we propose to take two steps towards solving equationby projecting onto the linearized constraints.

Original: To describe this method let
Revised: To describe this method let

Original: [Equation]
Revised: [Equation]

Original: In the ﬁrst step we linearize the quadratic constraint equation 15 around w t and project onto this linearization:
Revised: In the ﬁrst step we linearize the quadratic constraint equation 15 around w t and project onto this linearization:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The closed form update of this ﬁrst step is given by
Revised: The closed form update of this ﬁrst step is given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: which is a Stochastic Polyak step equation 4.
Revised: which is a Stochastic Polyak step equation 4.

Original: For the second step, we once again linearize the quadratic constraint equation 15, but this time around the point w t +1 / 2 and set this linearization to zero, that is
Revised: For the second step, we once again linearize the quadratic constraint equation 15, but this time around the point w t +1 / 2 and set this linearization to zero, that is

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The closed form update of this second step is given by
Revised: The closed form update of this second step is given by

Original: [Equation]
Revised: [Equation]

Original: We refer to the resulting proposed method as the SP2 + method, summarized in the following.
Revised: We refer to the resulting proposed method as the SP2 + method, summarized in the following.

Original: Lemma 3.
Revised: Lemma 3.

Original: (SP2 + ) Let g t ≡ ∇ f i ( w t ) and H t ≡ ∇
Revised: (SP2 + ) Let g t ≡ ∇ f i ( w t ) and H t ≡ ∇

Original: 2 f i ( w t ) .
Revised: 2 f i ( w t ) .

Original: Update equation 17 and equation 19 is given by
Revised: Update equation 17 and equation 19 is given by

Original: [Equation]
Revised: [Equation]

Revised: t +1 ,

Original: [Equation]
Revised: [Equation]

Original: +
Revised: v t +1 = (cid:16) I − H t f i ( w t [Equation]

Original: [Equation]
Revised: [Equation]

Original: In equation 20 we can see that SP2 + applies a second order correction to the SP step.
Revised: In equation 20 we can see that SP2 + applies a second order correction to the SP step.

Original: SP2 + is equivalent to two steps of a Newton Raphson method applied to ﬁnding a root of q ( w ) .
Revised: SP2 + is equivalent to two steps of a Newton Raphson method applied to ﬁnding a root of q ( w ) .

Original: If we apply multiple steps of the Newton Raphson method, as opposed to two, the resulting method converges to the root of q , see Theorem 2 in the appendix.
Revised: If we apply multiple steps of the Newton Raphson method, as opposed to two, the resulting method converges to the root of q , see Theorem 2 in the appendix.

Original: Theorem 2 shows that this multi-step version of SP2 + converges when q belongs to a large class of non-convex functions known as the star-convex functions.
Revised: Theorem 2 shows that this multi-step version of SP2 + converges when q belongs to a large class of non-convex functions known as the star-convex functions.

Original: Star-convexity, which is a generalization of convexity, includes several non-convex loss functions Hinder et al.
Revised: Star-convexity, which is a generalization of convexity, includes several non-convex loss functions Hinder et al.

Original:  C ONVERGENCE T HEORY
Revised:  C ONVERGENCE T HEORY

Original: Here we provide a convergence theory for SP2 and SP2 + for when f ( w ) is an average of quadratic functions.
Revised: Here we provide a convergence theory for SP2 and SP2 + for when f ( w ) is an average of quadratic functions.

Original: Fix w ∗ ∈ R d and let the loss over the i -th data point be given by
Revised: Fix w ∗ ∈ R d and let the loss over the i -th data point be given by

Original: [Equation]
Revised: [Equation]

Original: where H i ∈ R d × d is a symmetric positive semi-deﬁnite matrix for i = 1 , . .
Revised: where H i ∈ R d × d is a symmetric positive semi-deﬁnite matrix for i = 1 , . .

Original: . , n. Note that we assume that our algorithms have access to the second order terms H i , but not to the minimizer w ∗ .
Revised: . , n. Note that we assume that our algorithms have access to the second order terms H i , but not to the minimizer w ∗ .

Original: Consequently f i ( w ∗ ) = 

Revised: Consequently f i ( w ∗ ) =

Original: = f ( w ∗ ) =
Revised: = f ( w ∗ ) =

Original: min w ∈ R d f ( w ) , thus the interpolation condition holds.
Revised: min w ∈ R d f ( w ) , thus the interpolation condition holds.

Original: Proposition 1.
Revised: Proposition 1.

Original: Consider the loss functions given in equation 21.
Revised: Consider the loss functions given in equation 21.

Original: The SP2 method equationconverges linearly
Revised: The SP2 method equationconverges linearly

Original: [Equation]
Revised: [Equation]

Original: The rate of convergence of SP2 in equation 22 can be orders of magnitude better than SGD.
Revised: The rate of convergence of SP2 in equation 22 can be orders of magnitude better than SGD.

Original: Indeed, since equation 21 is convex, smooth and interpolation holds, we have from Needell et al.
Revised: Indeed, since equation 21 is convex, smooth and interpolation holds, we have from Needell et al.

Original: that
Revised: that

Original: SGD converges at a rate of
Revised: SGD converges at a rate of

Original: [Equation]
Revised: [Equation]

Original: To compare equation 23 to ρ rate in Proposition 1, consider the case where all H i are invertible.
Revised: To compare equation 23 to ρ rate in Proposition 1, consider the case where all H i are invertible.

Original: In this case H i H
Revised: In this case H i H

Original: + i
Revised: + i

Original: =
Revised: =

Original: I and thus ρ = 0 and SP2 converges in one step.
Revised: I and thus ρ = 0 and SP2 converges in one step.

Original: Indeed, even if a single H i is invertible, after sampling i the SP2 method will reach w ∗ , solving the system exactly.
Revised: Indeed, even if a single H i is invertible, after sampling i the SP2 method will reach w ∗ , solving the system exactly.

Original: Of course the method is more interesting when the H i ’s are low rank .
Revised: Of course the method is more interesting when the H i ’s are low rank .

Original: In contrast, the SGD method is still at the mercy of the spectra of the H i matrices and depend on how well conditioned these matrices are.
Revised: In contrast, the SGD method is still at the mercy of the spectra of the H i matrices and depend on how well conditioned these matrices are.

Original: Even in the extreme case where all H i are well conditioned, for example H i =
Revised: Even in the extreme case where all H i are well conditioned, for example H i =

Original: i × I , the rate of convergence of SGD can be very slow, for instance in this case we have ρ SGD = 1 − 12 n 2 .
Revised: i × I , the rate of convergence of SGD can be very slow, for instance in this case we have ρ SGD = 1 − 12 n 2 .

Original: Proposition 2.
Revised: Proposition 2.

Original: Consider the loss functions in equation 21.
Revised: Consider the loss functions in equation 21.

Original: The SP2 + method equation 20 converges linearly
Revised: The SP2 + method equation 20 converges linearly

Original: [Equation]
Revised: [Equation]

Original: The proof of Proposition 2 follows from Corollary 5.7 in Gower et al.
Revised: The proof of Proposition 2 follows from Corollary 5.7 in Gower et al.

Original: (2021a).
Revised: (2021a).

Original: The rate of convergence of SP2 + now depends on the average condition number of the H i matrix.
Revised: The rate of convergence of SP2 + now depends on the average condition number of the H i matrix.

Original: The rate in equation 24 can be greater or smaller than the rate of SGD in equation 23.
Revised: The rate in equation 24 can be greater or smaller than the rate of SGD in equation 23.

Original: For instance, if one H j has a condition number that is much greater, then equation 24 will be smaller than equation 23.
Revised: For instance, if one H j has a condition number that is much greater, then equation 24 will be smaller than equation 23.

Original: On the other hand, if all the H j ’s are such that max i =1 ,...,n λ max ( H i ) =
Revised: On the other hand, if all the H j ’s are such that max i =1 ,...,n λ max ( H i ) =

Original: λ max ( H j ) , for j = 1 , . . . , n, then equation 23 is a smaller rate as compared to equation 24.
Revised: λ max ( H j ) , for j = 1 , . . . , n, then equation 23 is a smaller rate as compared to equation 24.

Original: Note also that the rate of SP2 + appears squared in equation 24 and the rate ρ SGD of SGD is not squared.
Revised: Note also that the rate of SP2 + appears squared in equation 24 and the rate ρ SGD of SGD is not squared.

Original: But this difference accounts for the fact that each step of SP2 + is at least twice the cost of
Revised: But this difference accounts for the fact that each step of SP2 + is at least twice the cost of

Original: SGD, since each step of SP2 + is comprised of two gradient steps, see equation 17 and equation 19.
Revised: SGD, since each step of SP2 + is comprised of two gradient steps, see equation 17 and equation 19.

Original: Thus we can neglect the apparent advantage of the rate ρ SP 2+ being squared.
Revised: Thus we can neglect the apparent advantage of the rate ρ SP 2+ being squared.

Original:  Q UADRATIC WITH S LACK
Revised:  Q UADRATIC WITH S LACK

Original: Here we depart from the interpolation Assumption 1 and design a variant of SP2 + that can be applied to models that are close to interpolation.
Revised: Here we depart from the interpolation Assumption 1 and design a variant of SP2 + that can be applied to models that are close to interpolation.

Original: Instead of trying to set all the losses to zero, we now will try to ﬁnd the smallest slack variable s > 0 for which
Revised: Instead of trying to set all the losses to zero, we now will try to ﬁnd the smallest slack variable s > 0 for which

Original: [Equation]
Revised: [Equation]

Original: If interpolation holds, then s = 0 is a solution.
Revised: If interpolation holds, then s = 0 is a solution.

Original: Outside of interpolation, s may be non-zero.
Revised: Outside of interpolation, s may be non-zero.

Original: There are two natural ways of translating the slack constraint above into a penalty term.
Revised: There are two natural ways of translating the slack constraint above into a penalty term.

Original: Namely one can consider either the associated (cid:96) 2 or (cid:96) 1 loss.
Revised: Namely one can consider either the associated (cid:96) 2 or (cid:96) 1 loss.

Original: We explore both of these in turn.
Revised: We explore both of these in turn.

Original: 5. L2 SLACK FORMULATION
Revised: 5. L2 SLACK FORMULATION

Original: To make s as small as possible, we can solve the following problem
Revised: To make s as small as possible, we consider the following problem

Original: [Equation]
Revised: [Equation]

Original: which is called the L2 slack formulation .
Revised: which is called the L2 slack formulation .

Original: This type of slack problem was introduced in Crammer et al.
Revised: This type of slack problem was introduced in Crammer et al.

Original: (2006) to derive variants of the passive-aggressive method that could be applied to linear models on non-separable data, in other words, that do not interpolate the data.
Revised: (2006) to derive variants of the passive-aggressive method that apply to linear models on non-separable data; in other words, when the models cannot interpolate the data.

Original: To solve equation 25 we will again project onto a local quadratic approximation of the constraint.
Revised: To solve equation 25 we will again project onto a local quadratic approximation of the constraint.

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original:  and let ∆ t = (cid:107) w − w t (cid:107) 

Revised: ) and let ∆ t = (cid:107) w − w t (cid:107)

Original: + ( s − s t ) 2 .
Revised: + ( s − s t ) 2 .

Original: Consider the iterative method given by
Revised: Consider the iterative method given by

Original: [Equation]
Revised: [Equation]

Original: where λ ∈ [0 , 1] is a regularization parameter that trades off between having a small s , and using the previous iterates as a regularizor.
Revised: where λ ∈ [0 , 1] is a regularization parameter that trades off between having a small s , and using the previous iterates as a regularizer.

Original: The resulting projection problem in equation 27 has a quadratic inequality, and thus in most cases has no closed form solution, despite always being feasible 5 .
Revised: The resulting projection problem in equation 27 has a quadratic inequality, and thus in most cases has no closed form solution, despite always being feasible 5 .

Original: So instead of solving equation 27 exactly [Equation] we propose an approximate solution by iteratively linearizing and projecting onto the constraints.
Revised: So instead of solving equation 27 exactly, we propose an approximate solution by iteratively linearizing and projecting onto the constraints.

Original: Our approximate solution has two steps, the ﬁrst step being
Revised: Our approximate solution has two steps, the ﬁrst step being

Original: [Equation]
Revised: [Equation]

Original: The second step is given by projecting w t +1 / 2 onto the linearization around w t +1 / 2 as follows
Revised: The second step is given by projecting w t +1 / 2 onto the linearization around w t +1 / 2 as follows

Original: [Equation]
Revised: [Equation]

Original: The closed form solution to our two step method is given in Lemma 7 of Appendix C.7.
Revised: The closed form solution to our two step method is given in Lemma 7 of Appendix C.7.

Original: We refer to this method as SP2L2 + .
Revised: We refer to this method as SP2L2 + .

Original: 5. L1 SLACK FORMULATION
Revised: 5. L1 SLACK FORMULATION

Original: To make s as small as possible, we can also solve the following L1 slack formulation
Revised: To make s as small as possible, we can also solve the following L1 slack formulation

Original: [Equation]
Revised: [Equation]

Original: Similarly, we can again project onto a local quadratic approximation of the constraint and consider the iterative method given by
Revised: We can again project onto a local quadratic approximation of the constraint. That is, let λ ∈ [0 , 1] be a regularization parameter that trades off between having a small s , and using the previous iterates as and s ) is feasible a regularizer, and consider the iterative method given by

Original: [Equation]
Revised: [Equation]

Original: where λ ∈ [0 , 1] is a regularization parameter that trades off between having a small s , and using the previous iterates as a regularizor.

Original: To approximately solve equation 30, we again propose an approximate two step method similar to equation 28 and equation 29.
Revised: To approximately solve equation 30, we again propose an approximate two step method similar to equation 28 and equation 29.

Original: The closed form solution to the two step method is given in Lemmaof Appendix C.8.
Revised: The closed form solution to the two step method is given in Lemmaof Appendix C.8.

Original: We refer to this method as SP2L1 + .
Revised: We refer to this method as SP2L1 + .

Original: 5. D ROPPING THE S LACK R EGULARIZATION
Revised: 5. D ROPPING THE S LACK R EGULARIZATION

Original: Note that the objective function in equation 30 contains a regularization term ( s − s t ) 2 , which forces s to be close to s t .
Revised: Note that the objective function in equation 30 contains a regularization term ( s − s t ) 2 , which forces s to be close to s t .

Original: If we allow s to be far from s t , we can instead solve the following unregularized problem
Revised: If we allow s to be far from s t , we can instead solve the following unregularized problem

Original: [Equation]
Revised: [Equation]

Original: [Equation] where λ ∈ [0 , 1] is again a regularization parameter that trades off between having a small s , and using the previous iterates as a regularizor.
Revised: where λ ∈ [0 , 1] is again a regularization parameter that trades off between having a small s , and using the previous iterates as a regularizer.

Original: We call the resulting method in equation 31 the SP2max method since it is a second order variant of the SPmax method Loizou et al.
Revised: We call the resulting method in equation 31 the SP2max method since it is a second order variant of the SPmax method Loizou et al.

Original: Gower et al.
Revised: Gower et al.

Original: The advantage of SP2max is that it has a closed form solution for GLMs  equation 10 as shown in the following lemma which is proved in Appendix C.10.
Revised: The advantage of SP2max is that it has a closed form solution for GLMs in the form of equation 10. We give the closed form in the following lemma which is proved in Appendix C.10.

Original: Lemma 4.
Revised: Lemma 4.

Original: ( SP2max ) Consider the GLM model given in equation 10 and equation 11.
Revised: ( SP2max ) Consider the GLM model given in equation 10 and equation 11.

Original: If the loss f i = f i ( w t ) is non-negative, then the iterates of equation 31 have a closed form solution given byw t +1 = w t + c (cid:63) x i ,
Revised: If the loss f i = f i ( w t ) is non-negative, then the iterates of equation 31 have a closed form solution given by

Original: [Equation]
Revised: [Equation]

Original: 1 = max { (cid:101) s, 0 } , c (cid:63) =   0 , if f i =− (cid:101) λa i 1+ (cid:101) λh i (cid:96) , if f i > 0 and (cid:101) s ≥ 0 , − a
Revised: =   0 , if f i =− (cid:101) λa i 1+ (cid:101) λh i (cid:96) , if f i > 0 and (cid:101) s ≥ 0 , − a

Original: i + √ a 2 i − 2 h
Revised: i + √ a 2 i − 2 h

Original: i f i h i (cid:96) , otherwise.
Revised: i f i h i (cid:96) , otherwise.

Original: and where (cid:101) s =
Revised: and where (cid:101) s =

Original: f i − (cid:101) λa 2 i (cid:96) 1+ (cid:101) λh i (cid:96) + h i (cid:101) λ 2 a 2 i (cid:96) 2 2(1+ (cid:101) λh i (cid:96) ) 2 , and (cid:96) =
Revised: f i − (cid:101) λa 2 i (cid:96) 1+ (cid:101) λh i (cid:96) + h i (cid:101) λ 2 a 2 i (cid:96) 2 2(1+ (cid:101) λh i (cid:96) ) 2 , and (cid:96) =

Original: (cid:107) x
Revised: (cid:107) x

Original: i (cid:107) 2 , (cid:101) λ
Revised: i (cid:107) 2 , (cid:101) λ

Original: =
Revised: =

Original: λ 2(1 − λ ) .
Revised: λ 2(1 − λ ) .

Original: To approximately solve equation 31 in general, we again propose an approximate two step method.
Revised: To approximately solve equation 31 in general, we again propose an approximate two step method.

Original: The closed form solution to the two step method is given in Lemma 10 of Appendix C.9.
Revised: The closed form solution to the two step method is given in Lemma 10 of Appendix C.9.

Original: We refer to this method as SP2max + .
Revised: We refer to this method as SP2max + .

Original:  E XPERIMENTS 6. N ON - CONVEX PROBLEMS
Revised:  E XPERIMENTS 6. N ON - CONVEX PROBLEMS

Original: To emphasize how our new SP2 methods can handle non-convexity, we have tested SP2 equation(blue curve) , SP2 + equation 20 (green curve) on the non-convex problems PermD β + , Rastrigin and
Revised: To emphasize how our new SP2 methods can handle non-convexity, we have tested SP2 equation(blue curve) , SP2 + equation 20 (green curve) on the non-convex problems PermD β + , Rastrigin and

Original: Levy N. 13, Rosenbrock Jamil & Yang (2013) 6 , see Figures 1, 2 in the main text, and Figures 6 andin the appendix.
Revised: Levy N. 13, Rosenbrock Jamil & Yang (2013) 6 , see Figures 1, 2 in the main text, and Figures 6 andin the appendix.

Original: For a baseline we compared against SGD (yellow ), Adam (pink) and Newton (red).
Revised: For a baseline we compared against SGD (yellow ), Adam (pink) and Newton (red).

Original: The two experiments with the function Levy N. 13 and Rosenbrock are detailed in the appendix in
Revised: The two experiments with the function Levy N. 13 and Rosenbrock are detailed in the appendix in

Original: Section D.1.2.
Revised: Section D.1.2.

Original: All of these functions are 2D sums-of-terms of the format equation 1 and satisfy the interpolation
Revised: All of these functions are 2D sums-of-terms of the format equation 1 and satisfy the interpolation

Original: Assumption 1.
Revised: Assumption 1.

Original: To compute the SP2 update we used ten steps of Newton’s Raphson method as detailed in Section C.4.
Revised: To compute the SP2 update we used ten steps of Newton’s Raphson method as detailed in Section C.4.

Original: We consistently ﬁnd across these non-convex problems that SP2 and SP2 + are very competitive, with SP2 converging in under 20 epochs.
Revised: We consistently ﬁnd across these non-convex problems that SP2 and SP2 + are very competitive, with SP2 converging in under 20 epochs.

Original: SP2 also converges faster in terms of time taken (see middle of Figures 1 and 2).
Revised: SP2 also converges faster in terms of time taken (see middle of Figures 1 and 2).

Original: Here we can clearly see that SP2 converges to a high precision solution (like most second order methods), and different than other second order methods is not attracted to local maxima or saddle points.
Revised: Here we can clearly see that SP2 converges to a high precision solution (like most second order methods), and different than other second order methods is not attracted to local maxima or saddle points.

Original: In contrast, Newtons method converges to a local maxima on all problems excluding the Rosenbrock function in Figure 7 in the appendix.
Revised: In contrast, Newtons method converges to a local maxima on all problems excluding the Rosenbrock function in Figure 7 in the appendix.

Original: For instance on the right of Figure 2 we can see the red dot of Newton stuck on a local maxima. 
Revised: For instance on the right of Figure 2 we can see the red dot of Newton stuck on a local maxima. [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Revised: t

Original: [Figure]
Revised: [Figure]

Revised: o n v

Original: [Figure]
Revised: [Figure]

Original: When increasing the dimension to 100 , these non-convex problems become very challenging, see
Revised: When increasing the dimension to 100 , these non-convex problems become very challenging, see

Original: Figure 3 7 , and often only SP2 , SP2 + and SP are able to ﬁnd the global minima.
Revised: Figure 3 7 , and often only SP2 , SP2 + and SP are able to ﬁnd the global minima.

Original: Note that we no longer compare to Newton’s method because it was exceeding our maximum allocated time.
Revised: Note that we no longer compare to Newton’s method because it was exceeding our maximum allocated time.

Original: 6. M ATRIX C OMPLETION
Revised: 6. M ATRIX C OMPLETION

Original: Assume a set of known values { a i,j } ( i,
Revised: Assume a set of known values { a i,j } ( i,

Original: j ) ∈
Revised: j ) ∈

Original: Ω where Ω is a set of known elements of the matrix, and we want to determine the missing elements.
Revised: Ω where Ω is a set of known elements of the matrix, and we want to determine the missing elements.

Original: One approach is solving the matrix completion problem
Revised: One approach is solving the matrix completion problem

Original: [Equation]
Revised: [Equation]

Original: (32) where
Revised: (32) where

Original: A =
Revised: A =

Original: [ a i,j ] ∈ R m × n , U =
Revised: [ a i,j ] ∈ R m × n , U =

Original: [ u i ] i =1 ,...,m ∈ R r × m and V =
Revised: [ u i ] i =1 ,...,m ∈ R r × m and V =

Original: [ v j ] j =1 ,...,n ∈ R r × n .
Revised: [ v j ] j =1 ,...,n ∈ R r × n .

Original: After solving equation 32, we then use the matrix U (cid:62) V as an approximation to the complete matrix A.
Revised: After solving equation 32, we then use the matrix U (cid:62) V as an approximation to the complete matrix A.

Original: Despite equation 32 being a non-convex problem, if there exists an interpolating solution to equation 32, one where u Ti v j = a i,j , for ( i, j ) ∈ Ω , then the SP2 method can solve equation 32.
Revised: Despite equation 32 being a non-convex problem, if there exists an interpolating solution to equation 32, one where u Ti v j = a i,j , for ( i, j ) ∈ Ω , then the SP2 method can solve equation 32.

Original: Indeed, SP2 can be applied to equation 32 by sampling a single pair ( i, j ) ∈
Revised: Indeed, SP2 can be applied to equation 32 by sampling a single pair ( i, j ) ∈

Original: Ω uniformly, then projecting onto the quadratic (the solution to which is detailed in Theorem 1 in Appendix B): 
Revised: Ω uniformly, then projecting onto the quadratic (the solution to which is detailed in Theorem 1 in Appendix B): [Equation] to

Original: [Equation]
Revised: [Equation]

Original: We compared our method 33 to a specialized variant of SGD for online matrix completion described in Jin et al.
Revised: We compared our method 33 to a specialized variant of SGD for online matrix completion described in Jin et al.

Original: (2016), see Figure 4.
Revised: (2016), see Figure 4.

Original: To compare the two methods we generated a rank r = 2 matrix
Revised: To compare the two methods we generated a rank r = 2 matrix

Original: A ∈ R 100 × 50 .
Revised: A ∈ R 100 × 50 .

Original: We selected a subset entries with probability p = 0 .
Revised: We selected a subset entries with probability p = 0 .

Original: 2 or 0 .
Revised: 2 or 0 .

Original: 3 to form our set Ω init that was used to obtain an initial estimate U 0 , V 0 using rank-k SVD method as described in Jin et al.
Revised: 3 to form our set Ω init that was used to obtain an initial estimate U 0 , V 0 using rank-k SVD method as described in Jin et al.

Original: We extensively tuned the step size of SGD using a grid search, and the method labelled
Revised: We extensively tuned the step size of SGD using a grid search, and the method labelled

Original: Non-convex SGD is the resulting run of SGD with the best step size.
Revised: Non-convex SGD is the resulting run of SGD with the best step size.

Original: We also show how sensitive SGD is to this step size, by including the run of SGD with step sizes that were only a factor of 2 to 4 away from the optimal, which greatly degrades the performance of SGD .
Revised: We also show how sensitive SGD is to this step size, by including the run of SGD with step sizes that were only a factor of 2 to 4 away from the optimal, which greatly degrades the performance of SGD .

Original: In contrast, SP2 worked with no tuning, and matches the performance of SGD with the optimal step size in the p = 0 .
Revised: In contrast, SP2 worked with no tuning, and matches the performance of SGD with the optimal step size in the p = 0 .

Original: 1 experiment, and outperforms SGD when more measurements are available in the p = 0 .
Revised: 1 experiment, and outperforms SGD when more measurements are available in the p = 0 .

Original: 2 and p = 0 .
Revised: 2 and p = 0 .

Original: 3 ﬁgures.
Revised: 3 ﬁgures.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: C ONVEX CLASSIFICATION
Revised: C ONVEX CLASSIFICATION

Original: Here we compare our proposed methods on a logistic regression problem to SGD, SP, and
Revised: Here we compare our proposed methods on a logistic regression problem to SGD, SP, and

Original: ADAM.
Revised: ADAM.

Original: In particular, we consider the problem of minimizing the following loss function f ( w ) = 1 n (cid:80) ni
Revised: In particular, we consider the problem of minimizing the following loss function f ( w ) = 1 n (cid:80) ni

Original: =

Revised: =

Original: f i ( w ) + σ 2 (cid:107) w (cid:107) 22 ,
Revised: f i ( w ) + σ 2 (cid:107) w (cid:107) 22 ,

Original: where f i ( w ) =
Revised: where f i ( w ) =

Original: φ i ( x (cid:62) i w ) with φ i ( t ) =
Revised: φ i ( x (cid:62) i w ) with φ i ( t ) =

Original: ln(1 + e − y i t ) .
Revised: ln(1 + e − y i t ) .

Original: Here, { ( x i , y i ) ∈ R d +1 } ni
Revised: Here, { ( x i , y i ) ∈ R d +1 } ni

Original: =1 stands for the feature-label pairs and σ > 0 is the regularization parameter.
Revised: =1 stands for the feature-label pairs and σ > 0 is the regularization parameter.

Original: We control how far each problem is from interpolation by increasing σ .
Revised: We control how far each problem is from interpolation by increasing σ .

Original: When σ > 0 the problem cannot interpolate, and thus we expect to see a beneﬁt of the slack methods in Section 5 over SP2 + .
Revised: When σ > 0 the problem cannot interpolate, and thus we expect to see a beneﬁt of the slack methods in Section 5 over SP2 + .

Original: We used two data sets: colon-cancer ( ( n, d ) =
Revised: We used two data sets: colon-cancer ( ( n, d ) =

Original: Alon et al.
Revised: Alon et al.

Original: (1999) and mushrooms ( ( n, d ) =
Revised: (1999) and mushrooms ( ( n, d ) =

Original: West et al.
Revised: West et al.

Original: (2001), both of which interpolate when σ = 0 .
Revised: (2001), both of which interpolate when σ = 0 .

Original: We compare the proposed methods SP2 + equation 20, SP2L2 + (Lemma 7), SP2L1 + (Lemma 9), and SP2max + (Lemma 10) with SGD , SP equation 4, and ADAM on both data sets with three regularizations σ ∈ { 0 , 0 .
Revised: We compare the proposed methods SP2 + equation 20, SP2L2 + (Lemma 7), SP2L1 + (Lemma 9), and SP2max + (Lemma 10) with SGD , SP equation 4, and ADAM on both data sets with three regularizations σ ∈ { 0 , 0 .

Original: 001 , 0 . 008 } and with momentum set to 0 .
Revised: 001 , 0 . 008 } and with momentum set to 0 .

Original: For the SGD method, we use a learning rate L max / √ t in the t -th iteration, where L max = 14 max i =1 ,...,n (cid:107) x i (cid:107) 2 denotes the smoothness constant of the loss function.
Revised: For the SGD method, we use a learning rate L max / √ t in the t -th iteration, where L max = 14 max i =1 ,...,n (cid:107) x i (cid:107) 2 denotes the smoothness constant of the loss function.

Original: We chose λ for SP2L2 + , SP2L1 + , and SP2max + using a grid search of λ ∈ { 0 . 1 , 0 . 2 , . .
Revised: We chose λ for SP2L2 + , SP2L1 + , and SP2max + using a grid search of λ ∈ { 0 . 1 , 0 . 2 , . .

Original: , 0 . 9 } , the details are in Section D.
Revised: , 0 . 9 } , the details are in Section D.

Original: The gradient norm evaluated at each epoch is presented in Figures 5 and 16 (see Appendix D).
Revised: The gradient norm evaluated at each epoch is presented in Figures 5 and 16 (see Appendix D).

Original: We see that SP2 methods converge much faster than classical methods (e.g., SGD , SP , ADAM ) and need fewer epochs to achieve the tolerance when σ is small (left and middle plots).
Revised: We see that SP2 methods converge much faster than classical methods (e.g., SGD , SP , ADAM ) and need fewer epochs to achieve the tolerance when σ is small (left and middle plots).

Original: However, they can all fail when the problem is far from interpolation, e.g., when σ = 8 × 10 − 3 .
Revised: However, they can all fail when the problem is far from interpolation, e.g., when σ = 8 × 10 − 3 .

Original: The running time used for each algorithm to achieve the tolerance for both data sets is presented in Figure 17 (see Appendix D).
Revised: The running time used for each algorithm to achieve the tolerance for both data sets is presented in Figure 17 (see Appendix D).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Here we have proposed newincremental second order methods that exploit models that  interpolate  the data, or are close to interpolation.
Revised: We have proposed new second order methods aimed at overparameterized models that can interpolate (or nearly interpolate) the data.

Original: What sets our methods apart from most previous incremental second order methods is that they do not rely on convexity in their design.
Revised: In contrast to previous incremental second order methods, ours do not rely on convexity.

Original: Quite the opposite, the SP method can beneﬁt from the Hessian having at least one negative eigenvalue.
Revised: Quite the opposite, the SP2 method can beneﬁt from the Hessian having at least one negative eigenvalue.

Original: Consequently the SP method excels at minimizing non-convex models that interpolate, as can be seen in Sections 6.1 and B. 
Revised: Consequently the SP2 method excels at optimizing non-convex models that interpolate, as we demonstrated in Sections 6.1 and B. We also provided a convergence result in

Original: We also provided an indicative convergence in Theorem 2 that shows that SP2 enjoys a signiﬁcantly faster rate of convergence as compared to SGD for sums-of-quadratics. 
Revised: Theorem 2 showing that SP2 can converge signiﬁcantly faster than SGD for sums-of-quadratics. In

Original: We then developed second order methods thatcan solve a relaxed version of the interpolation equations that allows for some slack in Section 5, and showed that these methods still perform well on problems that are close to interpolation in Section 6.3.
Revised: Section 5, we then developed second order methods that solve a relaxed version of the interpolation equations by allowing some slack. We showed that these methods still perform well on problems that are close to interpolation in Section 6.3.

Original: In future work, it would be interesting to develop specialized variants of SP2 for optimizing(DNNs) Deep Neural Networks.
Revised: In future work, it would be interesting to develop specialized variants of SP2 for optimizing Deep Neural Networks (DNNs).

Original: DNNs are particularly well suited since they can interpolate, are non-convex and since gradients and Hessian vector products can be computed efﬁciently using back-propagation.
Revised: DNNs are particularly well suited since modern deep models often interpolate, are non-convex, and enjoy fast gradient and Hessian-vector computations via back-propagation.

Original: A P ROJECTING ONTO Q UADRATIC
Revised: A P ROJECTING ONTO Q UADRATIC

Original: This following projection lemma is based on Section B in Park & Boyd (2017).
Revised: This following projection lemma is based on Section B in Park & Boyd (2017).

Original: What we do in addition to Park & Boyd (2017) is to clarify how to compute the resulting projection, and add further details on the proof.
Revised: What we do in addition to Park & Boyd (2017) is to clarify how to compute the resulting projection, and add further details on the proof.

Original: Lemma 
Revised: Lemma 5.

Original: Let w ∈ R d and let P ∈ R
Revised: Let w ∈ R d and let P ∈ R

Original: d × d is a symmetric matrix
Revised: d × d is a symmetric matrix.

Original: Consider the projection
Revised: Consider the projection

Original: [Equation]
Revised: [Equation]

Original: Let λ 1 ≤ λ 2 ≤ · · · ≤ λ d be the eigenvalues of P and let Q Λ Q (cid:62)
Revised: Let λ 1 ≤ λ 2 ≤ · · · ≤ λ d be the eigenvalues of P and let Q Λ Q (cid:62)

Original: =
Revised: =

Original: P be the eigenvalue decomposition of P , where Λ = diag
Revised: P be the eigenvalue decomposition of P , where Λ = diag

Original: ( λ i ) and QQ (cid:62) =
Revised: ( λ i ) and QQ (cid:62) =

Original: I. Let ˆ q = Q (cid:62) q .
Revised: I. Let ˆ q = Q (cid:62) q .

Original: If the quadratic constraint in equation 34 is feasible, then there exists a solution to equation 34.
Revised: If the quadratic constraint in equation 34 is feasible, then there exists a solution to equation 34.

Original: Now we give the three candidate solutions.
Revised: Now we give the three candidate solutions.

Original: If r = 0 , then the solution is given byw = z.
Revised: If r = 0 , then the solution is given by

Revised: [Equation]

Original: Now assuming r (cid:54) = 0 .
Revised: Now assuming r (cid:54) = 0 .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: If
Revised: If

Original: [Equation]
Revised: [Equation]

Original: 39) then the solution is given by
Revised: 39) then the solution is given by

Original: [Equation]
Revised: [Equation]

Original: where n ∈ R d and
Revised: where n ∈ R d and

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: ˆ q
Revised: ˆ q

Original: [Equation]
Revised: [Equation]

Original: i ∈ N n i = 0 , for i (cid:54)∈ N.
Revised: i ∈ N n i = 0 , for i (cid:54)∈ N.

Original: Alternatively if equation 39 does not hold, then the solution is given by
Revised: Alternatively if equation 39 does not hold, then the solution is given by

Original: [Equation]
Revised: [Equation]

Original: where ν is the solution to the nonlinear equation
Revised: where ν is the solution to the nonlinear equation

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First note that there exists a solution to equation 34 since the constraint is a closed feasible set.
Revised: First note that there exists a solution to equation 34 since the constraint is a closed feasible set.

Original: Let Q Λ Q (cid:62) =
Revised: Let Q Λ Q (cid:62) =

Original: P be the SVD of P , where QQ (cid:62) =
Revised: P be the SVD of P , where QQ (cid:62) =

Original: I.
Revised: I.

Original: By changing variables x = Q (cid:62) ( w − z ) we have that equation 34 is equivalent to
Revised: By changing variables x = Q (cid:62) ( w − z ) we have that equation 34 is equivalent to

Original: [Equation]
Revised: [Equation]

Original: where ˆ x = Q (cid:62) ( w − z ) and ˆ q = Q (cid:62) q.
Revised: where ˆ x = Q (cid:62) ( w − z ) and ˆ q = Q (cid:62) q.

Original: The Lagrangian of equation 43 is given by
Revised: The Lagrangian of equation 43 is given by

Original: [Equation]
Revised: [Equation]

Original: Thus the KKT conditions are given by
Revised: Thus the KKT conditions are given by

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since we are guaranteed that the projection has a solution, we have that as a necessary condition that the solution satisﬁes
Revised: Since we are guaranteed that the projection has a solution, we have that as a necessary condition that the solution satisﬁes

Original: [Equation]
Revised: [Equation]

Original: see Theorem 12.5 in Wright & Nocedal (1999).
Revised: see Theorem 12.5 in Wright & Nocedal (1999).

Original: Consequently either ( I + ν Λ) (cid:31) 0 or ( I + ν Λ) has a zero eigenvalue.
Revised: Consequently either ( I + ν Λ) (cid:31) 0 or ( I + ν Λ) has a zero eigenvalue.

Original: Consider the case where ( I + ν Λ) (cid:31) 0 .
Revised: Consider the case where ( I + ν Λ) (cid:31) 0 .

Original: From equation 45 we have that
Revised: From equation 45 we have that

Original: [Equation]
Revised: [Equation]

Original: Now note that if ν = 0 then x = 0 and by the constraint we must have r = 0 .
Revised: Now note that if ν = 0 then x = 0 and by the constraint we must have r = 0 .

Original: Otherwise, if r (cid:54) = 0 , then ν (cid:54) = 0 .
Revised: Otherwise, if r (cid:54) = 0 , then ν (cid:54) = 0 .

Original: Assume now ν (cid:54) = 0 and substituting the above into equation 46 and letting Λ = diag
Revised: Assume now ν (cid:54) = 0 and substituting the above into equation 46 and letting Λ = diag

Original: ( λ i ) gives
Revised: ( λ i ) gives

Original: [Equation]
Revised: [Equation]

Original: Using equation
Revised: Using equation

Original: [Equation]
Revised: [Equation]

Original: Using equation
Revised: Using equation

Original: [Equation]
Revised: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: Upon ﬁnding the solution ν to the above, we have that our ﬁnal solution is given by w (cid:48)
Revised: Upon ﬁnding the solution ν to the above, we have that our ﬁnal solution is given by w (cid:48)

Original: =
Revised: =

Original: z + Qx, that is
Revised: z + Qx, that is

Original: [Equation]
Revised: [Equation]

Original: Alternatively, suppose that ( I + ν Λ) (cid:23) 0 is non-singular.
Revised: Alternatively, suppose that ( I + ν Λ) (cid:23) 0 is non-singular.

Original: The positive deﬁniteness implies thatν
Revised: The positive deﬁniteness implies that

Original: [Equation]
Revised: [Equation]

Original: For ( I + ν Λ) to be non-singular, at least one of the above inequalities will hold to equality.
Revised: For ( I + ν Λ) to be non-singular, at least one of the above inequalities will hold to equality.

Original: To ease notation, let us arrange the eigenvalues in increasing order so that
Revised: To ease notation, let us arrange the eigenvalues in increasing order so that

Original: [Equation]
Revised: [Equation]

Original: For one of the equation 50 inequalities to hold to equality we need that
Revised: For one of the equation 50 inequalities to hold to equality we need that

Original: [Equation]
Revised: [Equation]

Original: Since ( I + ν Λ) is now singular with this ν , we have that the solution to equation 45 is given by
Revised: Since ( I + ν Λ) is now singular with this ν , we have that the solution to equation 45 is given by

Original: [Equation]
Revised: [Equation]

Original: where (cid:104) x ∗ , n (cid:105)
Revised: where (cid:104) x ∗ , n (cid:105)

Original: = 0 , (51) where † denotes the pseudo-inverse and where n is in the kernel of ( I + ν Λ) , in other words ( I + ν Λ) n = 0 .
Revised: = 0 , (51) where † denotes the pseudo-inverse and where n is in the kernel of ( I + ν Λ) , in other words ( I + ν Λ) n = 0 .

Original: It remains to determine n , which we can do with equation 46.
Revised: It remains to determine n , which we can do with equation 46.

Original: Indeed, substituting equation 51 into equation 46 gives
Revised: Indeed, substituting equation 51 into equation 46 gives

Original: [Equation]
Revised: [Equation]

Original: Using equation
Revised: Using equation

Original: [Equation]
Revised: [Equation]

Original: Using equation 51 .
Revised: Using equation 51 .

Original: Setting to zero and completing the squares in n we have that
Revised: Setting to zero and completing the squares in n we have that

Original: [Equation]
Revised: [Equation]

Original: To characterize the solutions in n of the above
Revised: To characterize the solutions in n of the above

Original: [Equation]
Revised: [Equation]

Original: ﬁrst note that n will only have a few non-zero elements.
Revised: ﬁrst note that n will only have a few non-zero elements.

Original: To see this
Revised: To see this

Original: [Equation]
Revised: [Equation]

Original: let i ∗ ∈ arg max i : λ i (cid:54) =0 (cid:110) − 1 λ 1 , − 1 λ d (cid:111) , and note that ( I + ν Λ) has as many zeros on the diagonal as the multiplicity of the eigenvalue
Revised: let i ∗ ∈ arg max i : λ i (cid:54) =0 (cid:110) − 1 λ 1 , − 1 λ d (cid:111) , and note that ( I + ν Λ) has as many zeros on the diagonal as the multiplicity of the eigenvalue

Original: λ i ∗ .
Revised: λ i ∗ .

Original: That is, it has zeros elements on the indices in
Revised: That is, it has zeros elements on the indices in

Original: [Equation]
Revised: [Equation]

Original: Thus the non-zero elements of n are in the set
Revised: Thus the non-zero elements of n are in the set

Original: [Equation]
Revised: [Equation]

Original: Because of this observation we further re-write equation 52 as
Revised: Because of this observation we further re-write equation 52 as

Original: [Equation]
Revised: [Equation]

Original: Consequently, if the above is positive
Revised: Consequently, if the above is positive

Original: [Equation]
Revised: [Equation]

Original: then there exists solutions to the above of which
Revised: then there exists solutions to the above of which

Original: [Equation]
Revised: [Equation]

Original: for i

Original: [Equation]

Original: N.

Original: (54) is one.
Revised: (54) is one.

Original: Consequently, the ﬁnal solution is given by w = z
Revised: Consequently, the ﬁnal solution is given by w = z

Original: + Q ( x ∗ + n ) where x ∗ is given in by equation 51.
Revised: + Q ( x ∗ + n ) where x ∗ is given in by equation 51.

Original: Corollary 1.
Revised: Corollary 1.

Original: If r > 0 and P has at least one negative eigenvalue, there always exists a solution to the projection equation 34.
Revised: If r > 0 and P has at least one negative eigenvalue, there always exists a solution to the projection equation 34.

Original: Proof.
Revised: Proof.

Original: We only need to prove that there exists a solution to the quadratic equation in equation 34, after which Lemma 5 guarantees the existance of a solution.
Revised: We only need to prove that there exists a solution to the quadratic equation in equation 34, after which Lemma 5 guarantees the existance of a solution.

Original: B M ATRIX C OMPLETION
Revised: B M ATRIX C OMPLETION

Original: The projection equation 33 can be solved as we shown in the following theorem.
Revised: The projection equation 33 can be solved as we shown in the following theorem.

Original: Theorem 1.
Revised: Theorem 1.

Original: The solution to equation 33 is given by one of the following cases.
Revised: The solution to equation 33 is given by one of the following cases.

Original: If ( u kj ) (cid:62) v kj = a i,j then u = u ki and v = v kj . 2.
Revised: If ( u kj ) (cid:62) v kj = a i,j then u = u ki and v = v kj . 2.

Original: Alternatively if u kj = v kj and
Revised: Alternatively if u kj = v kj and

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: Finally, if none of the above holds then
Revised: Finally, if none of the above holds then

Original: [Equation]
Revised: [Equation]

Original: where γ ∈ ( − 1 , 1) and is the solution to the depressed quartic equation
Revised: where γ ∈ ( − 1 , 1) and is the solution to the depressed quartic equation

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The Lagrangian of equation 33 is given by
Revised: The Lagrangian of equation 33 is given by

Original: [Equation]
Revised: [Equation]

Original: where γ ∈ R is the unknown Lagrangian Multiplier.
Revised: where γ ∈ R is the unknown Lagrangian Multiplier.

Original: Thus the KKT equations are
Revised: Thus the KKT equations are

Original: [Equation]
Revised: [Equation]

Original: Subtracting γ times the the second equation from the ﬁrst equation, and analogously, subtracting the ﬁrst equation from the second gives
Revised: Subtracting γ times the the second equation from the ﬁrst equation, and analogously, subtracting the ﬁrst equation from the second gives

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: If γ = 1 , then necessarily u ki = v kj and furthermore from the ﬁrst equation in equation 62 we have that
Revised: If γ = 1 , then necessarily u ki = v kj and furthermore from the ﬁrst equation in equation 62 we have that

Original: [Equation]
Revised: [Equation]

Original: Substituting u out in the original projection problem equation 33 we have that
Revised: Substituting u out in the original projection problem equation 33 we have that

Original: [Equation]
Revised: [Equation]

Original: Consequently, for every v that satisﬁes the constraint we have that the objective value is invariant and equal to − a i,j .
Revised: Consequently, for every v that satisﬁes the constraint we have that the objective value is invariant and equal to − a i,j .

Original: Consequently there are inﬁnite solutions.
Revised: Consequently there are inﬁnite solutions.

Original: To ﬁnd one such solution, we complete the squares of the constraint and ﬁnd
Revised: To ﬁnd one such solution, we complete the squares of the constraint and ﬁnd

Original: [Equation]
Revised: [Equation]

Original: The above only has solutions if 1 4 (cid:13)(cid:13) v k j (cid:13)(cid:13) 2 − a i,j ≥ 0 .
Revised: The above only has solutions if 1 4 (cid:13)(cid:13) v k j (cid:13)(cid:13) 2 − a i,j ≥ 0 .

Original: One solution to the above is given by equation 55.
Revised: One solution to the above is given by equation 55.

Original: Alternatively, if γ (cid:54) = 1 then by isolating u and v in equation 63 and equation 64, respectively, gives
Revised: Alternatively, if γ (cid:54) = 1 then by isolating u and v in equation 63 and equation 64, respectively, gives

Original: [Equation]
Revised: [Equation]

Original: To ﬁgure out γ
Revised: To ﬁgure out γ

Original: [Equation]
Revised: [Equation]

Original: we use the third constraint in equation 62 and the above two equations
Revised: we use the third constraint in equation 62 and the above two equations

Original: [Equation]
Revised: [Equation]

Original: which gives
Revised: which gives

Original: [Equation]
Revised: [Equation]

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Can we now ﬁnd an interval which will contain the solution in γ ?
Revised: Can we now ﬁnd an interval which will contain the solution in γ ?

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Thus it sufﬁces to search for γ ∈ ( − 1 , 1) , which can be done efﬁciently with bisection.
Revised: Thus it sufﬁces to search for γ ∈ ( − 1 , 1) , which can be done efﬁciently with bisection.

Original: C P ROOFS OF I MPORTANT L EMMAS C. P ROOF OF L EMMA
Revised: C P ROOFS OF I MPORTANT L EMMAS C. P ROOF OF L EMMA

Original: Let us ﬁrst describe the set of solutions for given constraint.
Revised: Let us ﬁrst describe the set of solutions for given constraint.

Original: We need to have
Revised: We need to have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ∆ = w − w t is unknown.
Revised: where ∆ = w − w t is unknown.

Original: If we denote by τ i =
Revised: If we denote by τ i =

Original: x Ti ∆ then equation 70 will reduce to
Revised: x Ti ∆ then equation 70 will reduce to

Original: [Equation]
Revised: [Equation]

Original: This quadratic equation equation 71 has solution if
Revised: This quadratic equation equation 71 has solution if

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: If the condition above holds then we have that the solution for τ is in this set
Revised: If the condition above holds then we have that the solution for τ is in this set

Original: [Equation]
Revised: [Equation]

Original: Recall that the problem equation 7 now reduces into
Revised: Recall that the problem equation 7 now reduces into

Original: [Equation]
Revised: [Equation]

Original: , such that x Ti ∆ ∈ T ∗ (cid:9) .
Revised: , such that x Ti ∆ ∈ T ∗ (cid:9) .

Original: Note that because we want to minimize (cid:107) ∆ (cid:107) 2 , we want to choose the constraint with smallest possible absolute value, hence the problem equation 74 is equivalent to
Revised: Note that because we want to minimize (cid:107) ∆ (cid:107) 2 , we want to choose the constraint with smallest possible absolute value, hence the problem equation 74 is equivalent to

Original: [Equation]
Revised: [Equation]

Original:  , such that 
Revised: 2 , such that x

Revised: [Equation]

Revised: ∆ = τ

Original: [Equation]
Revised: [Equation]

Original:  (cid:9) , (75) where
Revised: i (cid:9) , (75) where

Original: [Equation]
Revised: [Equation]

Original: otherwise .

Original: In other words,
Revised: In other words,

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal solution is hence
Revised: The ﬁnal solution is hence

Original: [Equation]
Revised: [Equation]

Original: and therefore
Revised: and therefore

Original: [Equation]
Revised: [Equation]

Original: In case when equation 72 is not satisﬁed, and because we assumed that the loss function is nonnegative, we necessary have h i > 0 .
Revised: In case when equation 72 is not satisﬁed [Equation] and because we assumed that the loss function is nonnegative [Equation]

Revised: we necessary have h i > 0 .

Original: Then natural choice of τ i is the one that would minimize the
Revised: Then natural choice of τ i is the one that would minimize the

Original: [Equation]
Revised: [Equation]

Original: From ﬁrst order optimality conditions we obtain that
Revised: From ﬁrst order optimality conditions we obtain that

Original: [Equation]
Revised: [Equation]

Original: which leads to equation 14.
Revised: which leads to equation 14.

Original: C. P ROOF OF L EMMA
Revised: C. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: If φ ( t ) = 

Revised: If φ ( t ) =

Original: then the condition holds trivially.
Revised: then the condition holds trivially.

Original: For t such that φ ( t ) (cid:54) = 0 , (cid:112) φ ( t ) is differentiable, and we have
Revised: For t such that φ ( t ) (cid:54) = 0 , (cid:112) φ ( t ) is differentiable, and we have

Original: [Equation]
Revised: [Equation]

Original: , which is negative precisely when φ (cid:48)
Revised: , which is negative precisely when φ (cid:48)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: C. P ROOF OF L EMMA
Revised: C. P ROOF OF L EMMA

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Thus the second step equation 19 is given by
Revised: Thus the second step equation 19 is given by

Original: [Equation]
Revised: [Equation]

Original: Putting the ﬁrst equation 17 and second equation 76 updates together gives equation 20.
Revised: Putting the ﬁrst equation 17 and second equation 76 updates together gives equation 20.

Original: This gives a second order correction of the Polyak step that only requires computing a single Hessianvector product that can be done efﬁciently using an additional backwards pass of the function.
Revised: This gives a second order correction of the Polyak step that only requires computing a single Hessianvector product that can be done efﬁciently using an additional backwards pass of the function.

Original: We call this method SP2 .
Revised: We call this method SP2 .

Original: C. C ONVERGENCE OF MULTI - STEP SP2 +
Revised: C. C ONVERGENCE OF MULTI - STEP SP2 +

Original: If we apply multiple steps of the SP2 + , as opposed to two steps, the method converges to the solution of equation 15.
Revised: If we apply multiple steps of the SP2 + , as opposed to two steps, the method converges to the solution of equation 15.

Original: This follows because each step of SP2 + is a step of NR Newton Raphson
Revised: This follows because each step of SP2 + is a step of NR Newton Raphson

Original: ’s method applied to solving the nonlinear equation
Revised: ’s method applied to solving the nonlinear equation

Original: [Equation]
Revised: [Equation]

Original: Indeed, starting from w 0 = w t , the iterates of the NR (Newton Raphson) method are given by
Revised: Indeed, starting from w 0 = w t , the iterates of the NR (Newton Raphson) method are given by

Original: [Equation]
Revised: [Equation]

Original: where M † denotes the pseudo-inverse of the matrix M .
Revised: where M † denotes the pseudo-inverse of the matrix M .

Original: The NR iterates in equation 77 can also be written in a variational form given by
Revised: The NR iterates in equation 77 can also be written in a variational form given by

Original: [Equation]
Revised: [Equation]

Original: Comparing the above to the ﬁrst equation 16 and second step equation 18 are indeed two steps of the
Revised: Comparing the above to the ﬁrst equation 16 and second step equation 18 are indeed two steps of the

Original: NR method.
Revised: NR method.

Original: Further, we can see that equation 78 is indeed the multi-step version of SP2 + .
Revised: Further, we can see that equation 78 is indeed the multi-step version of SP2 + .

Original: This method equation 77 is also known as gradient descent with a Polyak Step step size, or SP for short.
Revised: This method equation 77 is also known as gradient descent with a Polyak Step step size, or SP for short.

Original: It is this connection we will use to prove the convergence of equation 77 to a root of q ( w ) .
Revised: It is this connection we will use to prove the convergence of equation 77 to a root of q ( w ) .

Original: We assume that q ( w ) has at least one root.
Revised: We assume that q ( w ) has at least one root.

Original: Let w ∗ q ∈ R
Revised: Let w ∗ q ∈ R

Original: d be a least norm root of q ( w ) , that is
Revised: d be a least norm root of q ( w ) , that is

Original: [Equation]
Revised: [Equation]

Original: subject toq ( w ) = 0 .
Revised: subject to

Original: [Equation]
Revised: [Equation]

Original: It follows from Theorem 3.2 of Sosa & MP Raupp (2020) that the above optimization equationhas solution if and only if the following matrix
Revised: It follows from Theorem 3.2 of Sosa & MP Raupp (2020) that the above optimization equationhas solution if and only if the following matrix

Original: [Equation]
Revised: [Equation]

Original: has at least a non-negative eigenvalue.
Revised: has at least a non-negative eigenvalue.

Original: Theorem 2.
Revised: Theorem 2.

Original: Assume that the matrix B deﬁned in equation 80 has at least a non-negative eigenvalue.
Revised: Assume that the matrix B deﬁned in equation 80 has at least a non-negative eigenvalue.

Original: If q ( w ) is star-convex with respect to w ∗ q , that is if
Revised: If q ( w ) is star-convex with respect to w ∗ q , that is if

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: then it follows that
Revised: then it follows that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof follows by applying the convergence Theorem 4.4 in Gower et al.
Revised: The proof follows by applying the convergence Theorem 4.4 in Gower et al.

Original: (2020) or equivalently Corollary D.3 in Gower et al.
Revised: (2020) or equivalently Corollary D.3 in Gower et al.

Original: (2021b).
Revised: (2021b).

Original: This result ﬁrst appeared in Theorem 4.
Revised: This result ﬁrst appeared in Theorem 4.

Original: in Gower et al.
Revised: in Gower et al.

Original: but we apply Corollary D.3 in Gower et al.
Revised: but we apply Corollary D.3 in Gower et al.

Original: (2021b) since it is a bit simpler.
Revised: (2021b) since it is a bit simpler.

Original: To apply this Corollary D.3 in Gower et al.
Revised: To apply this Corollary D.3 in Gower et al.

Original: (2021b), we need to verify that q is an L –smooth function and star-convex.
Revised: (2021b), we need to verify that q is an L –smooth function and star-convex.

Original: To verify if it is smooth, we need to ﬁnd L > 0 such that
Revised: To verify if it is smooth, we need to ﬁnd L > 0 such that

Original: [Equation]
Revised: [Equation]

Original: which holds with L = σ max ( ∇ 2 q ( y ))
Revised: which holds with L = σ max ( ∇ 2 q ( y ))

Original: =
Revised: =

Original: σ max ( ∇ 2 f i ( w t ) since q is a quadratic function.
Revised: σ max ( ∇ 2 f i ( w t ) since q is a quadratic function.

Original: Furthermore, for q to be star-convex along the iterates w i , we need to verify if
Revised: Furthermore, for q to be star-convex along the iterates w i , we need to verify if

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Since q is a quadratic, we have that
Revised: Since q is a quadratic, we have that

Original: [Equation]
Revised: [Equation]

Original: Using this in equation 85 gives that
Revised: Using this in equation 85 gives that

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to our assumption equation 81.
Revised: which is equivalent to our assumption equation 81.

Original: We can now apply the result in Corollary D. in Gower et al.
Revised: We can now apply the result in Corollary D. in Gower et al.

Original: (2021b) which states that
Revised: (2021b) which states that

Original: [Equation]
Revised: [Equation]

Original: Finally using q ( w ∗ q ) = 0 and that L = σ max ( ∇ 2 f i ( w t )) gives the result.
Revised: Finally using q ( w ∗ q ) = 0 and that L = σ max ( ∇ 2 f i ( w t )) gives the result.

Original: To simplify notation, we will omit the dependency on w t and denote c
Revised: To simplify notation, we will omit the dependency on w t and denote c

Original: = f i ( w t ) , g = ∇ f i ( w t ) and
Revised: = f i ( w t ) , g = ∇ f i ( w t ) and

Original: H = ∇ 2 f i ( w t ) , thus
Revised: H = ∇ 2 f i ( w t ) , thus

Original: [Equation]
Revised: [Equation]

Original: Lemma 6.
Revised: Lemma 6.

Original: If g ∈ Range( H ) and w 0 ∈
Revised: If g ∈ Range( H ) and w 0 ∈

Original: Range( H ) then w i , ∇ q ( w i ) ∈
Revised: Range( H ) then w i , ∇ q ( w i ) ∈

Original: Range( H ) for all i and w ∗ q ∈ Range( H ) .
Revised: Range( H ) for all i and w ∗ q ∈ Range( H ) .

Original: Proof.
Revised: Proof.

Original: First, note that since g ∈ Range( H ) and since ∇ q ( w ) =
Revised: First, note that since g ∈ Range( H ) and since ∇ q ( w ) =

Original: q + H ( w − w t ) (see equation 86)
Revised: q + H ( w − w t ) (see equation 86)

Original: we have that ∇ q ( w ) ∈
Revised: we have that ∇ q ( w ) ∈

Original: Range( H ) for all w .
Revised: Range( H ) for all w .

Original: Consequently by induction if w i ∈
Revised: Consequently by induction if w i ∈

Original: Range( H ) then by equation 77 we have that
Revised: Range( H ) then by equation 77 we have that

Original: w i +1 ∈
Revised: w i +1 ∈

Original: Range( H ) since it is a combination of ∇ q ( w i ) and w i .
Revised: Range( H ) since it is a combination of ∇ q ( w i ) and w i .

Original: Finally, let w ∗ q = w t + w H + w ⊥ H where w H ∈ Range( H ) and
Revised: Finally, let w ∗ q = w t + w H + w ⊥ H where w H ∈ Range( H ) and

Original: w ⊥ H
Revised: w ⊥ H

Original: ∈
Revised: ∈

Original: Range( H ) ⊥ .
Revised: Range( H ) ⊥ .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Furthermore, by orthogonality and Pythagoras’ Theorem
Revised: Furthermore, by orthogonality and Pythagoras’ Theorem

Original: [Equation]
Revised: [Equation]

Original: Consequently, since w ∗ q is the least norm solution, we must have that w ⊥ H = 0 and thus w ∗ q ∈
Revised: Consequently, since w ∗ q is the least norm solution, we must have that w ⊥ H = 0 and thus w ∗ q ∈

Original: Range( H ) .
Revised: Range( H ) .

Original: C. P ROOF OF P ROPOSITION
Revised: C. P ROOF OF P ROPOSITION

Original: First we repeat the proposition for ease of reference.
Revised: First we repeat the proposition for ease of reference.

Original: Proposition 3.
Revised: Proposition 3.

Original: Consider the loss functions given in equation 21.
Revised: Consider the loss functions given in equation 21.

Original: The SP2 method equationconverges according to
Revised: The SP2 method equationconverges according to

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: First consider the ﬁrst iterate of SP2 which applied to equation 21 are given by
Revised: First consider the ﬁrst iterate of SP2 which applied to equation 21 are given by

Original: [Equation]
Revised: [Equation]

Original: Thus every solution to the constraint set must satisfy
Revised: Thus every solution to the constraint set must satisfy

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: N i ∈ R
Revised: N i ∈ R

Original: d × d is a basis for the null space of H i .
Revised: d × d is a basis for the null space of H i .

Original: where α ∈ R d .
Revised: where α ∈ R d .

Original: Substituting into the objective we have the resulting linear least squares problem given by
Revised: Substituting into the objective we have the resulting linear least squares problem given by

Original: [Equation]
Revised: [Equation]

Original: The minimal norm solution in α is thus
Revised: The minimal norm solution in α is thus

Original: [Equation]
Revised: [Equation]

Original: which when substituted into equation 89 gives
Revised: which when substituted into equation 89 gives

Original: [Equation]
Revised: [Equation]

Original: Note that P i :=
Revised: Note that P i :=

Original: N i N
Revised: N i N

Original: + i is the orthogonal projector onto Null( H i ) .
Revised: + i is the orthogonal projector onto Null( H i ) .

Original: Subtracting w ∗ from both sides of equation 90 and applying the squared norm we have that
Revised: Subtracting w ∗ from both sides of equation 90 and applying the squared norm we have that

Original: [Equation]
Revised: [Equation]

Original: where we used that P i P i
Revised: where we used that P i P i

Original: = P i because it is a projection matrix.
Revised: = P i because it is a projection matrix.

Original: Now taking expectation conditioned on w t we have
Revised: Now taking expectation conditioned on w t we have

Original: [Equation]
Revised: [Equation]

Original: Since the null space is orthogonal to the range of adjoint, we have that
Revised: Since the null space is orthogonal to the range of adjoint, we have that

Original: [Equation]
Revised: [Equation]

Original: Thus taking expectation again gives the result equation 87.
Revised: Thus taking expectation again gives the result equation 87.

Original: Finally, the rate of convergence ρ in equation 88 is always smaller than one because, due Jensen’s inequality and that λ max is convex over positive deﬁnite matrices we have that
Revised: Finally, the rate of convergence ρ in equation 88 is always smaller than one because, due Jensen’s inequality and that λ max is convex over positive deﬁnite matrices we have that

Original: [Equation]
Revised: [Equation]

Original: where the greater than zero follows since there must exist H i (cid:54)
Revised: where the greater than zero follows since there must exist H i (cid:54)

Original: = 0 , otherwise the result still holds and the method converges in one step (with ρ = 0 ).
Revised: = 0 , otherwise the result still holds and the method converges in one step (with ρ = 0 ).

Original: Now multiplying equation 92 by − 1 then addinggives
Revised: Now multiplying equation 92 by − 1 then addinggives

Original: [Equation]
Revised: [Equation]

Original: Remark 1 (Connection to Block Kaczmarz proof) .
Revised: Remark 1 (Connection to Block Kaczmarz proof) .

Original: The proof of Proposition 3 follows the same proof technique of the Block Kaczmarz method Needell & Tropp (2014) 8 .
Revised: The proof of Proposition 3 follows the same proof technique of the Block Kaczmarz method Needell & Tropp (2014) 8 .

Original: To see this, ﬁrst note that since interpolation holds, to ﬁnd the minima we can solve the equations
Revised: To see this, ﬁrst note that since interpolation holds, to ﬁnd the minima we can solve the equations

Original: [Equation]
Revised: [Equation]

Original: Since each f i ( w ) function is a quadratic, this is equivalent to
Revised: Since each f i ( w ) function is a quadratic, this is equivalent to

Original: [Equation]
Revised: [Equation]

Original: We can now interpret the above as one big linear system that we need to solve.
Revised: We can now interpret the above as one big linear system that we need to solve.

Original: If we sample one of the block of rows, say H i ( w − w ∗ )
Revised: If we sample one of the block of rows, say H i ( w − w ∗ )

Original: = 

Revised: =

Original: for one i , and project w k onto this smaller system, the resulting method is the block Kaczmarz method equation 95.
Revised: for one i , and project w k onto this smaller system, the resulting method is the block Kaczmarz method equation 95.

Original: This also coincides with the SP2 method.
Revised: This also coincides with the SP2 method.

Original: Furthermore, our convergence analysis is now equivalent to applying the convergence analysis of block Kaczmarz to solving equation 95.
Revised: Furthermore, our convergence analysis is now equivalent to applying the convergence analysis of block Kaczmarz to solving equation 95.

Original: C. P ROOF OF P ROPOSITION
Revised: C. P ROOF OF P ROPOSITION

Original: For convenience we repeat the statement of the proposition here.
Revised: For convenience we repeat the statement of the proposition here.

Original: Proposition 4.
Revised: Proposition 4.

Original: Consider the loss functions given in equation 21.
Revised: Consider the loss functions given in equation 21.

Original: The SP2 + method equationconverges according to
Revised: The SP2 + method equationconverges according to

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof follows simply by observing that for quadratic function the SP2 + is equivalent to applying two steps of the SP method equation 5.
Revised: The proof follows simply by observing that for quadratic function the SP2 + is equivalent to applying two steps of the SP method equation 5.

Original: Indeed in Section 3.2 the SP2 + applies two steps of the SP method to the local quadratic approximation of the function we wish to minimize.
Revised: Indeed in Section 3.2 the SP2 + applies two steps of the SP method to the local quadratic approximation of the function we wish to minimize.

Original: But in this case, since our function is quadratic, it is itself equal to it’s local quadratic.
Revised: But in this case, since our function is quadratic, it is itself equal to it’s local quadratic.

Original: Consequently we can apply the convergence theory of SP for smooth, strongly convex functions that satisfy the interpolation condition, such as Corollary 5.7.I in Gower et al.
Revised: Consequently we can apply the convergence theory of SP for smooth, strongly convex functions that satisfy the interpolation condition, such as Corollary 5.7.I in Gower et al.

Original: (2021b), which states that
Revised: (2021b), which states that

Original: SP converges at a rate of equation
Revised: SP converges at a rate of equation

Original: C. P ROOF OF L EMMA
Revised: C. P ROOF OF L EMMA

Original: The following lemma gives the two step update for SP2L2 + .
Revised: The following lemma gives the two step update for SP2L2 + .

Original: Lemma 7.
Revised: Lemma 7.

Original: ( SP2L2 + )
Revised: ( SP2L2 + )

Original: The w t +1 and s t +1 update of equation 28–equation 29 is given by
Revised: The w t +1 and s t +1 update of equation 28–equation 29 is given by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Revised: :=

Original:  where we denote ( x ) + = (cid:26) x if x ≥ otherwise.
Revised: [Equation] + [Equation] where we denote ( x ) + = (cid:26) x if x ≥ otherwise.

Original: We will use the following lemma to prove Lemma 7, which has been proven in Lemma C.2 of Gower et al.
Revised: We will use the following lemma to prove Lemma 7, which has been proven in Lemma C.2 of Gower et al.

Original: Lemma 8 (L2 Unidimensional Inequality Constraint) .
Revised: Lemma 8 (L2 Unidimensional Inequality Constraint) .

Original: Let δ > 0 , c ∈ R and w, w 0 , a ∈ R d .
Revised: Let δ > 0 , c ∈ R and w, w 0 , a ∈ R d .

Original: The closed form solution to
Revised: The closed form solution to

Original: [Equation]
Revised: [Equation]

Original: is given by
Revised: is given by

Original: [Equation]
Revised: [Equation]

Original: where we denote ( x ) + = (cid:26) x if x ≥ otherwise .
Revised: where we denote ( x ) + = (cid:26) x if x ≥ otherwise .

Original: We are now in the position to prove Lemma 7.
Revised: We are now in the position to prove Lemma 7.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Consequently equation 28 is equivalent to
Revised: Consequently equation 28 is equivalent to

Original: [Equation]
Revised: [Equation]

Original: It follows from Lemma 8 that the closed form solution is
Revised: It follows from Lemma 8 that the closed form solution is

Original: [Equation]
Revised: [Equation]

Original: where we denote( x
Revised: where we denote

Original: [Equation]
Revised: [Equation]

Original: otherwise .

Original: Note that q i,t ( w t ) =
Revised: Note that q i,t ( w t ) =

Original: f i ( w t ) and ∇ q i,t ( w t ) = ∇ f i ( w t ) .
Revised: f i ( w t ) and ∇ q i,t ( w t ) = ∇ f i ( w t ) .

Original: To simplify the notation, we also denote
Revised: To simplify the notation, we also denote

Original: [Equation]
Revised: [Equation]

Original: With this notation we have that
Revised: With this notation we have that

Original: [Equation]
Revised: [Equation]

Original: In a completely analogous way
Revised: In a completely analogous way

Original: [Equation]
Revised: [Equation]

Original: the closed form solution to equation 29 is
Revised: the closed form solution to equation 29 is

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: ∇ f i ( w t ) (cid:11) =
Revised: f i ( w t ) (cid:11) =

Original: f i ( w t ) − Γ 1 (cid:13)(cid:13) ∇ f i ( w t ) (cid:13)(cid:13) 

Revised: f i ( w t ) − Γ 1 (cid:13)(cid:13) ∇ f i ( w t ) (cid:13)(cid:13)

Original: + 1 2 Γ 21 (cid:10) ∇ 2 f i ( w t ) ∇ f i ( w t ) , ∇ f i ( w t ) (cid:11) and ∇ q i,t ( w t +1 /2 ) = ∇ f i ( w t ) + ∇
Revised: + 1 2 Γ 21 (cid:10) ∇ 2 f i ( w t ) ∇ f i ( w t ) , ∇ f i ( w t ) (cid:11) and ∇ q i,t ( w t +1 /

Original: [Equation]
Revised: [Equation]

Original:  f i ( wt )( w t +1 /
Revised: = ∇ f i ( w

Original: [Equation]
Revised: [Equation]

Original: w t ) = ∇ f
Revised: +

Original: [Equation]
Revised: [Equation]

Original:  ( w t)
Revised: f i ( w t

Original: [Equation]
Revised: [Equation]

Original: f i (
Revised: [Equation] t [Equation]

Original: [Equation]
Revised: [Equation]

Original: t ) 
Revised: t ) =

Original: [Equation]
Revised: [Equation]

Original:  fi (
Revised: [Equation] f

Original: [Equation]
Revised: [Equation]

Original: t ) .
Revised: [Equation]

Original: Denoting Γ 2 as in the statement of the lemma we conclude that
Revised: Denoting Γ 2 as in the statement of the lemma we conclude that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: C. P ROOF OF L EMMA
Revised: C. P ROOF OF L EMMA

Original: The following Lemma gives a closed form for the two-step update for SP2L1 + .
Revised: The following Lemma gives a closed form for the two-step update for SP2L1 + .

Original: Lemma 9.
Revised: Lemma 9.

Original: ( SP2L1 + )
Revised: ( SP2L1 + )

Original: The w t +1 and s t +1 update is given by
Revised: The w t + 1 and s t +1 update is given by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: To solve equation
Revised: To solve equation

Original: [Equation]
Revised: [Equation]

Original: we consider the following two-step method similar to equation 28 and equation 29:
Revised: we consider the following two-step method similar to equation 28 and equation 29:

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Then, equation 112 is equivalent to solving
Revised: Then [Equation] equation 112 is equivalent to solving

Original: [Equation]
Revised: [Equation]

Original: It follows from Lemma C.4 in Gower et al.
Revised: It follows from Lemma C.4 in Gower et al.

Original: (2022) that the closed form solution to equation 112 is
Revised: (2022) that the closed form solution to equation 112 is

Original: [Equation]
Revised: [Equation]

Original: where we denote ( x ) + = (cid:26) x if x ≥ otherwise.
Revised: where we denote ( x ) + = (cid:26) x if x ≥ otherwise.

Original: Note that q i,t ( w t ) =
Revised: Note that q i,t ( w t ) =

Original: f i ( w t ) and ∇ q i,t ( w t ) = ∇ f i ( w t ) .
Revised: f i ( w t ) and ∇ q i,t ( w t ) = ∇ f i ( w t ) .

Original: To simplify the notation, denote
Revised: To simplify the notation, denote

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Then, we have
Revised: Then, we have

Original: [Equation]
Revised: [Equation]

Original: In a similar way
Revised: In a similar way

Original: [Equation]
Revised: [Equation]

Original: we can get the closed form solution to equation
Revised: we can get the closed form solution to equation

Original: [Equation]
Revised: [Equation]

Original: which is given as w
Revised: which is given as w

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Again
Revised: Again

Original: [Equation]
Revised: [Equation]

Original: to simplify the notation
Revised: to simplify the notation

Original: [Equation]
Revised: [Equation]

Original: we denote
Revised: we denote

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: C. P ROOF OF L EMMA
Revised: C. P ROOF OF L EMMA

Original: The following lemma gives a closed form for two step method SP2max + .
Revised: The following lemma gives a closed form for two step method SP2max + .

Original: Lemma 10.
Revised: Lemma 10.

Original: ( SP2max + )
Revised: ( SP2max + )

Original: The w t +1 and s t +1 update is given by
Revised: The w t +1 and s t +1 update is given by

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: To solve equation
Revised: To solve equation

Original: [Equation]
Revised: [Equation]

Original: we again consider a two step method similar to equation 112 and equation 113:
Revised: we again consider a two step method similar to equation 112 and equation 113:

Original: [Equation]
Revised: [Equation]

Original: Note that equation 115 is equivalent to solving
Revised: Note that equation 115 is equivalent to solving

Original: [Equation]
Revised: [Equation]

Original: It follows from Lemma D.2 in Gower et al.
Revised: It follows from Lemma D.2 in Gower et al.

Original: (2022) that the closed form solution to equation 117 is
Revised: (2022) that the closed form solution to equation 117 is

Original: [Equation]
Revised: [Equation]

Original: where we denote
Revised: where we denote

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: we have the closed form solution to equation 116 given as
Revised: we have the closed form solution to equation 116 given as

Original: [Equation]
Revised: [Equation]

Original: C. P ROOF OF L EMMA
Revised: C. P ROOF OF L EMMA

Original: In GLMs, the unregularized problem equation 31 becomes
Revised: In GLMs, the unregularized problem equation 31 becomes

Original: [Equation]
Revised: [Equation]

Original:  where (cid:101) λ :=
Revised: ) where (cid:101) λ :=

Original: λ 2(1 − λ ) , and we denote f i
Revised: λ 2(1 − λ ) , and we denote f i

Original: = f
Revised: = f

Original: i ( w t ) , a i := φ (cid:48)
Revised: i ( w t ) , a i := φ (cid:48)

Original: i ( x (cid:62) i w − y i ) , h i :=
Revised: i ( x (cid:62) i w − y i ) , h i :=

Original: φ (cid:48)(cid:48) i
Revised: φ (cid:48)(cid:48) i

Original: ( x (cid:62) i
Revised: ( x (cid:62) i

Original: w − y i ) for short.
Revised: w − y i ) for short.

Original: Denote (cid:52) :=
Revised: Denote (cid:52) :=

Original: w − w t .
Revised: w − w t .

Original: Then, problem equation 118 reduces to
Revised: Then, problem equation 118 reduces to

Original: [Equation]
Revised: [Equation]

Original: Note that we want to minimize (cid:107)(cid:52)(cid:107) 2 .
Revised: Note that we want to minimize (cid:107)(cid:52)(cid:107) 2 .

Original: Together with the above constraint, we can conclude that (cid:52) must be a multiple of x i since any other component will not help satisfy the constraint but increase (cid:107)(cid:52)(cid:107) 2 .
Revised: Together with the above constraint, we can conclude that (cid:52) must be a multiple of x i since any other component will not help satisfy the constraint but increase (cid:107)(cid:52)(cid:107) 2 .

Original: Let (cid:52) =
Revised: Let (cid:52) =

Original: cx i and (cid:96) =
Revised: cx i and (cid:96) =

Original: (cid:107) x
Revised: (cid:107) x

Original: i (cid:107) 2 , then problem equation 119 becomes
Revised: i (cid:107) 2 , then problem equation 119 becomes

Original: [Equation]
Revised: [Equation]

Original: The corresponding Lagrangian function is then given as
Revised: The corresponding Lagrangian function is then given as

Original: [Equation]
Revised: [Equation]

Original: where ν 1 , ν 2 ≥ 0 are the Lagrangian multipliers.
Revised: where ν 1 , ν 2 ≥ 0 are the Lagrangian multipliers.

Original: The KKT conditions are thus
Revised: The KKT conditions are thus

Original: [Equation]
Revised: [Equation]

Original: By checking the complementary conditions, the solution to the above KKT equations has three cases, which are summarized below.
Revised: By checking the complementary conditions, the solution to the above KKT equations has three cases, which are summarized below.

Original: Case I: The Lagragian multiplier ν 2 = 0 .
Revised: Case I: The Lagragian multiplier ν 2 = 0 .

Original: In which case ν
Revised: In which case ν

Original: [Equation]
Revised: [Equation]

Original: λh i (cid:96) , and s
Revised: λh i (cid:96) , and s

Original: [Equation]
Revised: [Equation]

Original: which is feasible if s (cid:63) ≥ 0 .
Revised: which is feasible if s (cid:63) ≥ 0 .

Original: The resulting objective function is 12 c (cid:63) 2 (cid:96) + (cid:101) λs (cid:63) , which is ≥ 0 .
Revised: The resulting objective function is 12 c (cid:63) 2 (cid:96) + (cid:101) λs (cid:63) , which is ≥ 0 .

Original: Case II: The Lagragian multiplier ν 1 = 0 .
Revised: Case II: The Lagragian multiplier ν 1 = 0 .

Original: In which case ν (cid:63) 1 = 0 , ν (cid:63) 2 = (cid:101) λ , c (cid:63) = 0 , s (cid:63) = 0 , which is feasible if f i = 0 .
Revised: In which case ν (cid:63) 1 = 0 , ν (cid:63) 2 = (cid:101) λ , c (cid:63) = 0 , s (cid:63) = 0 , which is feasible if f i = 0 .

Original: The objective function is 0 in this case and the variable w is unchanged since w − w t
Revised: The objective function is 0 in this case and the variable w is unchanged since w − w t

Original: = c (cid:63) x i = 0 .
Revised: = c (cid:63) x i = 0 .

Original: Case III:
Revised: Case III:

Original: Neither Lagragian multiplier is zero.
Revised: Neither Lagragian multiplier is zero.

Original: In which case there are two possible solutions for c given by c
Revised: In which case there are two possible solutions for c given by c

Original: [Equation]
Revised: [Equation]

Revised: =

Revised: [Equation]

Original: , s (cid:63) = 0 .
Revised: , s (cid:63) = 0 .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Consequently to guarantee that the Lagrangian multipliers ν 

Revised: Consequently to guarantee that the Lagrangian multipliers ν

Original: and ν 2 are non-negative, we must have
Revised: and ν 2 are non-negative, we must have

Original: [Equation]
Revised: [Equation]

Original: and in this case the objective function equals
Revised: and in this case the objective function equals

Original: [Equation]
Revised: [Equation]

Original: (cid:96) ≥ 0 .
Revised: ≥ 0 .

Original: As a summary, if f i = 0 , Case II is the optimal solution.
Revised: As a summary, if f i = 0 , Case II is the optimal solution.

Original: Alternatively if f i > 0 and if
Revised: Alternatively if f i > 0 and if

Original: [Equation]
Revised: [Equation]

Original: is non-negative then Case I is the optimal solution.
Revised: is non-negative then Case I is the optimal solution.

Original: Otherwise, Case III with c (cid:63) = − a i + √ a 2 i − 2 h
Revised: Otherwise, Case III with c (cid:63) = − a i + √ a 2 i − 2 h

Original: i f i h i (cid:96) is the optimal solution.
Revised: i f i h i (cid:96) is the optimal solution.

Original: Therefore, the optimal solution to equation 118 is then
Revised: Therefore, the optimal solution to equation 118 is then

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: if [Equation] i > 0 and
Revised: if f i > 0 and

Original: [Equation]
Revised: [Equation]

Original:  otherwise.
Revised: , otherwise.

Original: D A DDITIONAL N UMERICAL E XPERIMENTS D. N ON - CONVEX PROBLEMS
Revised: D A DDITIONAL N UMERICAL E XPERIMENTS D. N ON - CONVEX PROBLEMS

Original: For the non-convex experiements, we used the Python Package pybenchfunction available on github Python_Benchmark_Test_Optimization_Function_Single_Objective .
Revised: For the non-convex experiements, we used the Python Package pybenchfunction available on github Python_Benchmark_Test_Optimization_Function_Single_Objective .

Original: We compared SP2 equation 7 (blue curve) , SP2 + equation 20 (green curve) on the non-convex problems PermD β + , Rastrigin and Levy N. 13, Rosenbrock Jamil & Yang (2013) 9 .
Revised: We compared SP2 equation 7 (blue curve) , SP2 + equation 20 (green curve) on the non-convex problems PermD β + , Rastrigin and Levy N. 13, Rosenbrock Jamil & Yang (2013) 9 .

Original: For a baseline we compared against SGD (yellow ), Adam (pink) and Newton (red).
Revised: For a baseline we compared against SGD (yellow ), Adam (pink) and Newton (red).

Original: For the implementation of Newton ’s method, we found that the regularized Newton methods was best suited for these non-convex experiments where
Revised: For the implementation of Newton ’s method, we found that the regularized Newton methods was best suited for these non-convex experiments where

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) is the regularization parameter which we set to (cid:15)
Revised: where (cid:15) is the regularization parameter which we set to (cid:15)

Original: = 10 − 8 , and γ is a learning rate which we tuned.
Revised: = 10 − 8 , and γ is a learning rate which we tuned.

Original: Speciﬁcally, for all the experiments in this section we set a learning for all methods to 0 .
Revised: Speciﬁcally, for all the experiments in this section we set a learning for all methods to 0 .

Original: 25 except for
Revised: 25 except for

Original: SGD and ADAM, both which had to be tuned by doing a grid search on the ﬁrst 100 epochs over the grid [0 . 0001 , 0 . 0005 , 0 . 001 , 0 . 005 , 0 . 01 , , 0 . 05 , 0 . 25 , 1] proving much harder to tune.
Revised: SGD and ADAM, both which had to be tuned by doing a grid search on the ﬁrst 100 epochs over the grid [0 . 0001 , 0 . 0005 , 0 . 001 , 0 . 005 , 0 . 01 , , 0 . 05 , 0 . 25 , 1] proving much harder to tune.

Original: D.1. P ERM D β + IS AN INCORRECT IMPLEMENTATION OF P ERM D β .
Revised: D.1. P ERM D β + IS AN INCORRECT IMPLEMENTATION OF P ERM D β .

Original: We note here that the PermD β + implemented is given by
Revised: We note here that the PermD β + implemented is given by

Original: [Equation]
Revised: [Equation]

Original: which is different than the standard PermD β function which is given by
Revised: which is different than the standard PermD β function which is given by

Original: [Equation]
Revised: [Equation]

Original: =   d (cid:88)
Revised:   d (cid:88)

Original: j = ( j i + β ) (cid:32)(cid:18) x
Revised: j = ( j i + β ) (cid:32)(cid:18) x

Original: j j (cid:19) i −(cid:33)  .
Revised: j j (cid:19) i −(cid:33)  .

Original: We believe this is a small mistake, which is why we have introduced the plus in PermD β
Revised: We believe this is a small mistake, which is why we have introduced the plus in PermD β

Original: + to distinguish this function from the standard PermD β function.
Revised: + to distinguish this function from the standard PermD β function.

Original: Yet, the PermD β + is still an interesting non-convex problem, and thus we have used it in our experiments despite this small alteration.
Revised: Yet, the PermD β + is still an interesting non-convex problem, and thus we have used it in our experiments despite this small alteration.

Original: D.1. T HE L EVY N. 13 AND R OSENBROCK PROBLEMS
Revised: D.1. T HE L EVY N. 13 AND R OSENBROCK PROBLEMS

Original: Here we provide two additional experiments on the non-convex function Levy N. 13 and Rosenbrock that complement the ﬁndings in 6.1.
Revised: Here we provide two additional experiments on the non-convex function Levy N. 13 and Rosenbrock that complement the ﬁndings in 6.1.

Original: For the Levy N. 13 function in Figure 6 we have that again SP2 converges in 10 epochs to the global minima.
Revised: For the Levy N. 13 function in Figure 6 we have that again SP2 converges in 10 epochs to the global minima.

Original: In contrast Newton’s method converges immediately to a local maxima, that can be easily seen on the surface plot of the right of Figure 6.
Revised: In contrast Newton’s method converges immediately to a local maxima, that can be easily seen on the surface plot of the right of Figure 6.

Original: The one problem where SP2 was not the fastest was on the Rosenbrock function, see Figure 7.
Revised: The one problem where SP2 was not the fastest was on the Rosenbrock function, see Figure 7.

Original: Here
Revised: Here

Original: Newton was equally fast.
Revised: Newton was equally fast.

Original: But note, this problem was designed to emphasize the advantages of
Revised: But note, this problem was designed to emphasize the advantages of

Original: Newton over gradient descent.
Revised: Newton over gradient descent.

Original: For completeness, we also give the function value versus time plot in
Revised: For completeness, we also give the function value versus time plot in

Original: Figure 8.
Revised: Figure 8.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: f u n c t
Revised: f u n c t

Original: i
Revised: i

Original: o
Revised: o

Original: n v a l u e 1e
Revised: n v a l u e 1e

Original: X 4 2 0 2 4 6 

Revised: X 4 2 0 2 4 6

Original: X1 4 20 261 e0.0 0.2 0.
Revised: X1 4 20 261 e0.0 0.2 0.

Original: f un c t i o n v a l u e 1e
Revised: f un c t i o n v a l u e 1e

Original: D.1. L ARGER DIMENSIONS
Revised: D.

Revised: [Figure] L ARGER DIMENSIONS

Original: Here we increase the dimension of the problems to appraise the effect dimensions has on the convergence of the methods, see Figure 9 for a comparison in terms of epochs, and Figure 3 in the main text for a comparison in terms of time taken.
Revised: Here we increase the dimension of the problems to appraise the effect dimensions has on the convergence of the methods, see Figure 9 for a comparison in terms of epochs, and Figure 3 in the main text for a comparison in terms of time taken.

Original: We no longer compare to Newton’s method because it was exceeding our maximum allocated time.
Revised: We no longer compare to Newton’s method because it was exceeding our maximum allocated time.

Original: We ﬁnd that when increasing the dimension, these problems become extremely challenging.
Revised: We ﬁnd that when increasing the dimension, these problems become extremely challenging.

Original: To select the learning rate of each method, we used a grid search over the ﬁrst 100 epochs.
Revised: To select the learning rate of each method, we used a grid search over the ﬁrst 100 epochs.

Original: For the SP2 , SP2 + and SP methods we used the grid [0 . 005 , 0 . 01 , 0 .
Revised: For the SP2 , SP2 + and SP methods we used the grid [0 . 005 , 0 . 01 , 0 .

Original: For Adam and SGD we used the larger grid[
Revised: For Adam and SGD we used the larger grid

Original: [Equation]
Revised: [Equation]

Original: Thus we gave a competitive advantage to SGD and Adam.
Revised: Thus we gave a competitive advantage to SGD and Adam.

Original: D. A DDITIONAL C ONVEX E XPERIMENTS
Revised: D. A DDITIONAL C ONVEX E XPERIMENTS

Original: We set the desired tolerance for each algorithm to 0 .
Revised: We set the desired tolerance for each algorithm to 0 .

Original: 01 , and set the maximum number of epochs for each algorithm to 200 in colon-cancer and 30 in mushrooms.
Revised: 01 , and set the maximum number of epochs for each algorithm to 200 in colon-cancer and 30 in mushrooms.

Original: To choose an optimal slack parameter λ for SP2L2 + , SP2L1 + , and SP2max + , we test these three methods on a uniform grid λ ∈ { 0 . 1 , 0 . 2 , . . .
Revised: To choose an optimal slack parameter λ for SP2L2 + , SP2L1 + , and SP2max + , we test these three methods on a uniform grid λ ∈ { 0 . 1 , 0 . 2 , . . .

Original: , 0 . 9 } with σ = 0 .
Revised: , 0 . 9 } with σ = 0 .

Original: The gradient norm and loss evaluated at each epoch are presented in Figures 10-13 (see Appendix D).
Revised: The gradient norm and loss evaluated at each epoch are presented in Figures 10-13 (see Appendix D).

Original: It can be seen that SP2L2 + performs best when λ = 0 .
Revised: It can be seen that SP2L2 + performs best when λ = 0 .

Original: 9 in colon-cancer and λ = 0 . 1 in mushrooms, SP2L1 + and SP2max
Revised: 9 in colon-cancer and λ = 0 . 1 in mushrooms, SP2L1 + and SP2max

Original: + perform best when λ = 0 .
Revised: + perform best when λ = 0 .

Original: 1 in both data sets.
Revised: 1 in both data sets.

Original: Therefore, we set λ = 0 .
Revised: Therefore, we set λ = 0 .

Original: 9 for SP2L2 + in colon-cancer and ﬁx λ = 0 .
Revised: 9 for SP2L2 + in colon-cancer and ﬁx λ = 0 .

Original: 1 in other cases.
Revised: 1 in other cases.

Original: Under the same setting as in Section 6.3, we also compare the SP2max and SP2max + methods on a grid λ = [0 . 001 0 .
Revised: Under the same setting as in Section 6.3, we also compare the SP2max and SP2max + methods on a grid λ = [0 . 001 0 .

Original: with σ = 0 .
Revised: with σ = 0 .

Original: The gradient norm and loss evaluated at each epoch are presented in Figures 14-15 (see Appendix D).
Revised: The gradient norm and loss evaluated at each epoch are presented in Figures 14-15 (see Appendix D).

Original: As we observe, SP2max + always outperforms the
Revised: As we observe, SP2max + always outperforms the

Original: SP2max method.
Revised: SP2max method.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]