Original: N EW P ERSPECTIVE ON THE G LOBAL C ONVERGENCE OF F INITE -S UM O PTIMIZATION
Revised: N EW P ERSPECTIVE ON THE G LOBAL C ONVERGENCE OF F INITE -S UM O PTIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks (DNNs) have shown great success in many machine learning tasks.
Revised: Deep neural networks (DNNs) have shown great success in many machine learning tasks.

Original: Their training is challenging since the loss surface of the network architecture is generally non-convex, or even non-smooth.
Revised: Their training is challenging since the loss surface of the network architecture is generally non-convex, or even non-smooth.

Original: How and under what assumptions is guaranteed convergence to a global minimum possible?
Revised: How and under what assumptions is guaranteed convergence to a global minimum possible?

Original: We propose a reformulation of the minimization problem allowing for a new recursive algorithmic framework.
Revised: We propose a reformulation of the minimization problem allowing for a new recursive algorithmic framework.

Original: By using bounded style assumptions , we prove convergence to an ε -(global) minimum using ˜ O
Revised: By using bounded style assumptions, we prove convergence to an ε -(global) minimum using ˜ O

Original: (1 /ε 2 ) gradient computations.
Revised: (1 /ε 3 ) gradient computations.

Original: Our theoretical foundation motivates further study, implementation, and optimization of the new algorithmic framework and further investigation of its non-standard bounded style assumptions.
Revised: Our theoretical foundation motivates further study, implementation, and optimization of the new algorithmic framework and further investigation of its non-standard bounded style assumptions.

Original: This new direction broadens our understanding of why and under what circumstances training of a DNN converges to a global minimum.
Revised: This new direction broadens our understanding of why and under what circumstances training of a DNN converges to a global minimum.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, deep neural networks (DNNs) have shown a great success in many machine learning tasks.
Revised: In recent years, deep neural networks (DNNs) have shown a great success in many machine learning tasks.

Original: However, training these neural networks is challenging since the loss surface of network architecture is generally non-convex, or even non-smooth.
Revised: However, training these neural networks is challenging since the loss surface of network architecture is generally non-convex, or even non-smooth.

Original: Thus, there have been a long-standing question on how optimization algorithms may converge to a global minimum.
Revised: Thus, there have been a long-standing question on how optimization algorithms may converge to a global minimum.

Original: Many previous work have investigated Gradient Descent algorithm and its stochastic version for over-parameterized setting (Arora et al., 2018; Soudry et al., 2018; Allen-Zhu et al., 2019; Du et al., 2019a; Zou & Gu, 2019).
Revised: Many previous work have investigated Gradient Descent algorithm and its stochastic version for over-parameterized setting (Arora et al., 2018; Soudry et al., 2018; Allen-Zhu et al., 2019; Du et al., 2019a; Zou & Gu, 2019).

Original: Although these works have shown promising convergence results under certain assumptions, there is still a lack of new efﬁcient methods that can guarantee global convergence for machine learning optimization.
Revised: Although these works have shown promising convergence results under certain assumptions, there is still a lack of new efﬁcient methods that can guarantee global convergence for machine learning optimization.

Original: In this paper, we address this problem using a different perspective.
Revised: In this paper, we address this problem using a different perspective.

Original: Instead of analyzing the traditional ﬁnite-sum formulation, we adopt a new composite formulation that exactly depicts the structure of machine learning where a data set is used to learn a common classiﬁer.
Revised: Instead of analyzing the traditional ﬁnite-sum formulation, we adopt a new composite formulation that exactly depicts the structure of machine learning where a data set is used to learn a common classiﬁer.

Original: Representation.
Revised: Representation.

Original: Let (cid:8) ( x ( i ) , y ( i ) )
Revised: Let (cid:8) ( x ( i ) , y ( i ) )

Original: (cid:9) n i
Revised: (cid:9) n i

Original: =1 be a given training set with x ( i ) ∈ R m , y ( i
Revised: =1 be a given training set with x ( i ) ∈ R m , y ( i

Original: ) ∈ R c , we investigate the following novel representation for deep learning tasks:
Revised: ) ∈ R c , we investigate the following novel representation for deep learning tasks:

Original: [Equation]
Revised: [Equation]

Original: where h ( · ; i ) : R d → R c , i ∈ [ n ] = { 1 , . . . , n } , is the classiﬁer for each input data x ( i ) ; and φ i : R c
Revised: where h ( · ; i ) : R d → R c , i ∈ [ n ] = { 1 , . . . , n } , is the classiﬁer for each input data x ( i ) ; and φ i : R c

Original: → R , i ∈ [ n ] , is the loss function corresponding to each output data
Revised: → R , i ∈ [ n ] , is the loss function corresponding to each output data

Original: y
Revised: y

Original: ( i ) .
Revised: ( i ) .

Original: Our composite formulation (1) is a special case of the ﬁnite-sum problem min w ∈ R d (cid:8) F ( w ) = 1 n (cid:80) ni
Revised: Our composite formulation (1) is a special case of the ﬁnite-sum problem min w ∈ R d (cid:8) F ( w ) = 1 n (cid:80) ni

Original: =1 f ( w ; i ) (cid:9) where each individual function f ( · ; i ) is a composition of the loss function φ i and the classiﬁer h ( · ; i ) .
Revised: =1 f ( w ; i ) (cid:9) where each individual function f ( · ; i ) is a composition of the loss function φ i and the classiﬁer h ( · ; i ) .

Original: This problem covers various important applications in machine learning, including logistic regression and neural networks.
Revised: This problem covers various important applications in machine learning, including logistic regression and neural networks.

Original: The most common approach for the ﬁnite-sum problem is using ﬁrst-order methods such as (stochastic) gradient algorithms and making assumptions on the component functions f ( · ; i ) .
Revised: The most common approach for the ﬁnite-sum problem is using ﬁrst-order methods such as (stochastic) gradient algorithms and making assumptions on the component functions f ( · ; i ) .

Original: As an alternative, we further investigate the structure of the loss function φ i and narrow our assumption on the classiﬁer h ( · ; i ) .
Revised: As an alternative, we further investigate the structure of the loss function φ i and narrow our assumption on the classiﬁer h ( · ; i ) .

Original: For the purpose of this work, we ﬁrst consider convex and
Revised: For the purpose of this work, we ﬁrst consider convex and

Original: Lipschitz-smooth loss functions while the classiﬁers can be non-convex.
Revised: Lipschitz-smooth loss functions while the classiﬁers can be non-convex.

Original: Using this representation, we propose a new framework followed by two algorithms that guarantee global convergence for the minimization problem.
Revised: Using this representation, we propose a new framework followed by two algorithms that guarantee global convergence for the minimization problem.

Original: Algorithmic Framework.
Revised: Algorithmic Framework.

Original: Representation (1) admits a new perspective.
Revised: Representation (1) admits a new perspective.

Original: Our key insight is to (A) deﬁne z ( t )
Revised: Our key insight is to (A) deﬁne z ( t )

Original: i =
Revised: i =

Original: h ( w ( t ) ; i ) , where t is an iteration count of the outer loop in our algorithmic framework.
Revised: h ( w ( t ) ; i ) , where t is an iteration count of the outer loop in our algorithmic framework.

Original: Next (B), we want to approximate the change z
Revised: Next (B), we want to approximate the change z

Original: ( t +1)
Revised: ( t +1)

Original: i − z
Revised: i − z

Original: ( t ) i in terms of a step size times the gradient
Revised: ( t ) i in terms of a step size times the gradient

Original: [Equation]
Revised: [Equation]

Original: and (C) we approximate the change h ( w ( t +1) ; i ) − h ( w ( t ) ; i ) in terms of the ﬁrst order derivative
Revised: and (C) we approximate the change h ( w ( t +1) ; i ) − h ( w ( t ) ; i ) in terms of the ﬁrst order derivative

Original: [Equation]
Revised: [Equation]

Original: Finally, we combine (A), (B), and (C) to equate the approximations of z ( t +1)
Revised: Finally, we combine (A), (B), and (C) to equate the approximations of z ( t +1)

Original: i − z
Revised: i − z

Original: ( t ) i and h ( w ( t +1) ; i ) − h ( w ( t ) ; i ) .
Revised: ( t ) i and h ( w ( t +1) ; i ) − h ( w ( t ) ; i ) .

Original: This leads to a recurrence on w ( t ) of the form w ( t +1) =
Revised: This leads to a recurrence on w ( t ) of the form w ( t +1) =

Original: w ( t ) − η ( t ) v ( t ) , where η ( t ) is a step size and which involves computing v ( t ) by solving a convex quadratic subproblem, see the details in Section 4.
Revised: w ( t ) − η ( t ) v ( t ) , where η ( t ) is a step size and which involves computing v ( t ) by solving a convex quadratic subproblem, see the details in Section 4.

Original: We explain two methods for approximating a solution for the derived subproblem.
Revised: We explain two methods for approximating a solution for the derived subproblem.

Original: We show how to approximate the subproblem by transforming it into a strongly convex problem by adding a regularizer which can be solved in closed form.
Revised: We show how to approximate the subproblem by transforming it into a strongly convex problem by adding a regularizer which can be solved in closed form.

Original: And we show how to use Gradient Descent (GD) on the subproblem to ﬁnd an approximation v ( t ) of its solution.
Revised: And we show how to use Gradient Descent (GD) on the subproblem to ﬁnd an approximation v ( t ) of its solution.

Original: Convergence Analysis.
Revised: Convergence Analysis.

Original: Our analysis introduces non-standard bounded style assumptions.
Revised: Our analysis introduces non-standard bounded style assumptions.

Original: Intuitively, we assume that our convex and quadratic subproblem has a bounded solution.
Revised: Intuitively, we assume that our convex and quadratic subproblem has a bounded solution.

Original: This allows us to prove a total complexity of ˜ O
Revised: This allows us to prove a total complexity of ˜ O

Original: ( 1 ε 2 ) to ﬁnd an ε -(global) solution that satisﬁes F ( ˆ w ) − F ∗ ≤ ε , where F ∗ is the global minimizer of F .
Revised: ( 1 ε 3 ) to ﬁnd an ε -(global) solution that satisﬁes F ( ˆ w ) − F ∗ ≤ ε , where F ∗ is the global minimizer of F .

Original: Our analysis applies to a wide range of applications in machine learning: Our results hold for squared loss and softmax cross-entropy loss and applicable for a range of activation functions in DNN as we only assume that the h ( · ; i ) are twice continuously differentiable and their Hessian matrices (second order derivatives) as well as their gradients (ﬁrst order derivatives) are bounded.
Revised: Our analysis applies to a wide range of applications in machine learning: Our results hold for squared loss and softmax cross-entropy loss and applicable for a range of activation functions in DNN as we only assume that the h ( · ; i ) are twice continuously differentiable and their Hessian matrices (second order derivatives) as well as their gradients (ﬁrst order derivatives) are bounded.

Original: Our problem is generally non-convex and can also be extended to non-smooth settings (by relaxing the twice continuously differentiability).

Original: Contributions and Outline.
Revised: Contributions and Outline.

Original: Our contributions in this paper can be summarized as follows
Revised: Our contributions in this paper can be summarized as follows

Original: • We propose a new representation (1) for analyzing the machine learning minimization problem.
Revised: • We propose a new representation (1) for analyzing the machine learning minimization problem.

Original: Our formulation utilizes the structure of machine learning tasks where a training data set of inputs and outputs is used to learn a common classiﬁer.
Revised: Our formulation utilizes the structure of machine learning tasks where a training data set of inputs and outputs is used to learn a common classiﬁer.

Original: Related work in Sectionshows how (1) is different from the classical ﬁnite-sum problem.
Revised: Related work in Sectionshows how (1) is different from the classical ﬁnite-sum problem.

Original: • Based on the new representation we propose a novel algorithm framework.
Revised: • Based on the new representation we propose a novel algorithm framework.

Original: The algorithmic framework approximates a solution to a subproblem for which we show two distinct approaches.
Revised: The algorithmic framework approximates a solution to a subproblem for which we show two distinct approaches.

Original: • For general DNNs and based on bounded style assumptions, we prove a total complexity of ˜ O
Revised: • For general DNNs and based on bounded style assumptions, we prove a total complexity of ˜ O

Original: ( 1 ε 2 ) to ﬁnd an ε -(global) solution that satisﬁes F ( ˆ w ) − F ∗ ≤ ε , where F ∗ is the global minimizer of F .
Revised: ( 1 ε 3 ) to ﬁnd an ε -(global) solution that satisﬁes F ( ˆ w ) − F ∗ ≤ ε , where F ∗ is the global minimizer of F .

Original: We emphasize that our focus is on developing a new theoretical foundation and that a translation to a practical implementation with empirical results is for future work.
Revised: We emphasize that our focus is on developing a new theoretical foundation and that a translation to a practical implementation with empirical results is for future work.

Original: Our theoretical foundation motivates further study, implementation, and optimization of the new algorithmic framework and further investigation of its non-standard bounded style assumptions.
Revised: Our theoretical foundation motivates further study, implementation, and optimization of the new algorithmic framework and further investigation of its non-standard bounded style assumptions.

Original: This new direction broadens our understanding of why and under what circumstances training of a DNN converges to a global minimum.
Revised: This new direction broadens our understanding of why and under what circumstances training of a DNN converges to a global minimum.

Original: The rest of this paper is organized as follows.
Revised: The rest of this paper is organized as follows.

Original: Section 2 discusses related work.
Revised: Section 2 discusses related work.

Original: Section 3 describes our setting and deep learning representation.
Revised: Section 3 describes our setting and deep learning representation.

Original: Section 4 explains our key insight and derives our
Revised: Section 4 explains our key insight and derives our

Original: Framework 1.
Revised: Framework 1.

Original: Section 5 presents our algorithms and their global convergence.
Revised: Section 5 presents our algorithms and their global convergence.

Original: All technical proofs are deferred to the Appendix.
Revised: All technical proofs are deferred to the Appendix.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Formulation for Machine Learning Problems.
Revised: Formulation for Machine Learning Problems.

Original: The ﬁnite-sum problem is one of the most important and fundamental problems in machine learning.
Revised: The ﬁnite-sum problem is one of the most important and fundamental problems in machine learning.

Original: Analyzing this model is the most popular approach in the machine learning literature and it has been studied intensively throughout the years (Bottou et al., 2018; Reddi et al., 2016; Duchi et al., 2011b).
Revised: Analyzing this model is the most popular approach in the machine learning literature and it has been studied intensively throughout the years (Bottou et al., 2018; Reddi et al., 2016; Duchi et al., 2011b).

Original: Our new formulation (1) is a special case of the ﬁnite-sum problem, however, it is much more complicated than the previous model since it involves the data index
Revised: Our new formulation (1) is a special case of the ﬁnite-sum problem, however, it is much more complicated than the previous model since it involves the data index

Original: i both inside the classiﬁers h ( · ; i ) and the loss functions φ i .
Revised: i both inside the classiﬁers h ( · ; i ) and the loss functions φ i .

Original: For a comparison, previous works only consider a common loss function l
Revised: For a comparison, previous works only consider a common loss function l

Original: (ˆ y, y ) for the predicted value ˆ y and output data y
Revised: (ˆ y, y ) for the predicted value ˆ y and output data y

Original: (Zou et al., 2018;
Revised: (Zou et al., 2018;

Original: Soudry et al., 2018).
Revised: Soudry et al., 2018).

Revised: Our modiﬁed version of loss function φ i is a natural setting for machine learning.

Revised: We note that when h ( w ; i ) is the output produced by a model, our goal is to match this output with the corresponding target y ( i ) .

Revised: For that reason, the loss function for each output has a dependence on the output data y

Original:  This complication may potentially bring a challenge to theoretical analysis.
Revised: ( i ) , and is denoted by φ i . This fact reﬂects the natural setting of machine learning where the outputs are designed to ﬁt different targets, and the optimization process depends on both outer function φ i and inner functions h ( · ; i ) . This complication may potentially bring a challenge to theoretical analysis.

Original: However, with separate loss functions, we believe this model will help to exploit better the structure of machine learning problems and gain more insights on the neural network architecture.
Revised: However, with separate loss functions, we believe this model will help to exploit better the structure of machine learning problems and gain more insights on the neural network architecture.

Original: Other related composite optimization models are also investigated thoroughly in (Zhang & Xiao, 2019; Tran-Dinh et al., 2020).
Revised: Other related composite optimization models are also investigated thoroughly in (Lewis & Wright, 2016; Zhang & Xiao, 2019; Tran-Dinh et al., 2020).

Original: Our model is different from these works as it does not have a common function wrapping outside the ﬁnite-sum term.
Revised: Our model is different from these works as it does not have a common function wrapping outside the ﬁnite-sum term, as in (Lewis & Wright, 2016).

Revised: Note that a broad class of variance reduction algorithms (e.g. SAG (Le Roux et al., 2012),

Revised: SAGA (Defazio et al., 2014), SVRG (Johnson & Zhang, 2013), SARAH (Nguyen et al., 2017)) is designed specifically for the ﬁnite-sum formulation and is known to have certain beneﬁts over Gradient Descent.

Original: In addition, the multilevel composite problem considered in (Zhang & Xiao, 2021) also covers empirical risk minimization problem.
Revised: In addition, the multilevel composite problem considered in (Zhang & Xiao, 2021) also covers empirical risk minimization problem.

Original: However our formulation does not match their work since our inner function h ( w ; i ) is not an independent expectation over some data distribution, but a speciﬁc function that depends on the current data.
Revised: However our formulation does not match their work since our inner function h ( w ; i ) is not an independent expectation over some data distribution, but a speciﬁc function that depends on the current data.

Original: Global Convergence for Neural Networks.
Revised: Global Convergence for Neural Networks.

Original: A recent popular line of research is studying the dynamics of optimization methods on some speciﬁc neural network architectures.
Revised: A recent popular line of research is studying the dynamics of optimization methods on some speciﬁc neural network architectures.

Original: There are some early works that show the global convergence of Gradient Descent (GD) for simple linear network and two-layer network (Brutzkus et al., 2018; Soudry et al., 2018; Arora et al., 2019; Du et al., 2019b).
Revised: There are some early works that show the global convergence of Gradient Descent (GD) for simple linear network and two-layer network (Brutzkus et al., 2018; Soudry et al., 2018; Arora et al., 2019; Du et al., 2019b).

Original: Some further works extend these results to deep learning architectures (Allen-Zhu et al., 2019; Du et al., 2019a;
Revised: Some further works extend these results to deep learning architectures (Allen-Zhu et al., 2019; Du et al., 2019a;

Original: Zou & Gu, 2019).
Revised: Zou & Gu, 2019).

Original: These theoretical guarantees are generally proved for the case when the last output layer is ﬁxed, which is not standard in practice.
Revised: These theoretical guarantees are generally proved for the case when the last output layer is ﬁxed, which is not standard in practice.

Original: A recent work (Nguyen & Mondelli, 2020) prove the global convergence for GD when all layers are trained with some initial conditions.
Revised: A recent work (Nguyen & Mondelli, 2020) prove the global convergence for GD when all layers are trained with some initial conditions.

Original: However, these results are for neural networks without bias neurons and it is unclear how these analyses can be extended to handle the bias terms of deep networks with different activations.
Revised: However, these results are for neural networks without bias neurons and it is unclear how these analyses can be extended to handle the bias terms of deep networks with different activations.

Original: Our novel framework and algorithms do not exclude learning bias layers as in (Nguyen & Mondelli, 2020).
Revised: Our novel framework and algorithms do not exclude learning bias layers as in (Nguyen & Mondelli, 2020).

Original: Using a different algorithm, Brutzkus et al.
Revised: Using a different algorithm, Brutzkus et al.

Original: (2018) investigate Stochastic Gradient Descent (SGD) for two-layer networks in a restricted linearly separable data setting.
Revised: (2018) investigate Stochastic Gradient Descent (SGD) for two-layer networks in a restricted linearly separable data setting.

Original: They prove that SGD converges to a global minimum using Leaky ReLU activations.

Original: However, when ReLU activations are used, their results hold with high probability only.

Original: This line of research continues with the works from Allen-Zhu et al.
Revised: This line of research continues with the works from Allen-Zhu et al.

Original: ; Zou et al.
Revised: ; Zou et al.

Original: (2018) and later with Zou & Gu (2019).
Revised: (2018) and later with Zou & Gu (2019).

Original: They justify the global convergence of SGD for deep neural networks for some probability depending on the number of input data and the initialization process.
Revised: They justify the global convergence of SGD for deep neural networks for some probability depending on the number of input data and the initialization process.

Original: Over-Paramaterized Settings and other Assumptions for Machine Learning.
Revised: Over-Paramaterized Settings and other Assumptions for Machine Learning.

Original: Most of the modern learning architectures are over-parameterized, which means that the number of parameters are very large and often far more than the number of input data.
Revised: Most of the modern learning architectures are over-parameterized, which means that the number of parameters are very large and often far more than the number of input data.

Original: Some recent works prove the global convergence of Gradient Descent when the number of neurons are extensively large, e.g. (Zou & Gu, 2019) requires Ω( n 8 ) neurons for every hidden layer, and (Nguyen & Mondelli, 2020) improves this number to Ω( n 3 ) .
Revised: Some recent works prove the global convergence of Gradient Descent when the number of neurons are extensively large, e.g. (Zou & Gu, 2019) requires Ω( n 8 ) neurons for every hidden layer, and (Nguyen & Mondelli, 2020) improves this number to Ω( n 3 ) .

Original: If the initial point satisﬁes some special conditions, then they can show a better dependence of Ω( n ) . 
Revised: If the initial point satisﬁes some special conditions, then they can show a better dependence of Ω( n ) . In Allen-Zhu et al.

Revised: (2019), the authors initialize the weights using a random Gaussian distribution where the variance depends on the dimension of the problem.

Revised: In non-convex setting, they prove the convergence of SGD using the assumption that the dimension depends inversely on the tolerance (cid:15) .

Original: We will discuss how these over-paramaterized settings might be a necessary condition to develop our theory.
Revised: We will discuss how these over-paramaterized settings might be a necessary condition to develop our theory.

Original: Other standard assumptions for machine learning include the bounded gradient assumption (Nemirovski et al., 2009; Shalev-Shwartz et al., 2007; Reddi et al., 2016; Tran et al., 2021).
Revised: Other standard assumptions for machine learning include the bounded gradient assumption (Nemirovski et al., 2009; Shalev-Shwartz et al., 2007; Reddi et al., 2016; Tran et al., 2021).

Original: It is also common to assume all the iterations of an algorithm stays in a bounded domain (Duchi et al., 2011a;
Revised: It is also common to assume all the iterations of an algorithm stays in a bounded domain (Duchi et al., 2011a;

Original: Levy et al., 2018; G¨urb¨uzbalaban et al., 2019; Reddi et al., 2018; Vaswani et al., 2021).
Revised: Levy et al., 2018; G¨urb¨uzbalaban et al., 2019; Reddi et al., 2018; Vaswani et al., 2021).

Original: Since we are analyzing a new composite formulation , it is understandable that our assumptions may also not be standard.
Revised: Since we are analyzing a new composite formulation , it is understandable that our assumptions may also not be standard.

Original: However, we believe that there is a strong connection between our assumptions and the traditional setting of machine learning.
Revised: However, we believe that there is a strong connection between our assumptions and the traditional setting of machine learning.

Original: We will discuss this point more clearly in Section 4.
Revised: We will discuss this point more clearly in Section 4.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section, we discuss our formulation and notations in detail.
Revised: In this section, we discuss our formulation and notations in detail.

Original: Although this paper focus on deep neural networks, our framework and theoretical analysis are general and applicable for other learning architectures.
Revised: Although this paper focuses on deep neural networks, our framework and theoretical analysis are general and applicable for other learning architectures.

Original: Deep Learning Representation.
Revised: Deep Learning Representation.

Original: Let { ( x ( i ) , y ( i ) ) }
Revised: Let { ( x ( i ) , y ( i ) ) }

Original: ni
Revised: ni

Original: =1 be a training data set where x ( i ) ∈
Revised: =1 be a training data set where x ( i ) ∈

Original: R m is a training input and y ( i
Revised: R m is a training input and y ( i

Original: ) ∈ R c is a training output.
Revised: ) ∈ R c is a training output.

Original: We consider a fully-connected neural network with
Revised: We consider a fully-connected neural network with

Original: L layers, where the l -th layer, l ∈ { 0 , 1 , . . .
Revised: L layers, where the l -th layer, l ∈ { 0 , 1 , . . .

Original: , L } , has n l neurons.
Revised: , L } , has n l neurons.

Original: We represent layer 0 -th and L -th layer as input and output layers, respectively, that is, n
Revised: We represent layer 0 -th and L -th layer as input and output layers, respectively, that is, n

Original: 0 = d and n L = c .
Revised: 0 = d and n L = c .

Original: For l ∈ { 1 , . . .
Revised: For l ∈ { 1 , . . .

Original: , L } , let
Revised: , L } , let

Original: W ( l ) ∈ R n l − 1 × n l and b ( l ) ∈ R n l , where { ( W ( l ) , b ( l ) )
Revised: W ( l ) ∈ R n l − 1 × n l and b ( l ) ∈ R n l , where { ( W ( l ) , b ( l ) )

Original: Ll =1 } represent the parameters of the neural network.
Revised: Ll =1 } represent the parameters of the neural network.

Original: A classiﬁer h ( w ; i ) is formulated as 
Revised: A classiﬁer h ( w ; i ) is formulated as [Equation] [Equation] +

Original: [Equation]
Revised: [Equation]

Original:  where w = vec ( { W (1) , b (1) , . .
Revised: , where w = vec ( { W (1) , b (1) , . .

Original: , W ( L ) , b ( L ) } )
Revised: , W ( L ) , b ( L ) } )

Original: ∈ R
Revised: ∈ R

Original: d is the vectorized weight and { σ l } L − 1 l =1 are some activation functions.
Revised: d is the vectorized weight and { σ l } L − 1 l =1 are some activation functions.

Revised: The most common choices for machine learning are ReLU, sigmoid, hyperbolic tangent and softplus.

Original: For j ∈ [ c ] , h j ( · ; i ) : R d → R denotes the component function of the output h ( · ; i ) , for each data i ∈ [ n ] respectively.
Revised: For j ∈ [ c ] , h j ( · ; i ) : R d → R denotes the component function of the output h ( · ; i ) , for each data i ∈ [ n ] respectively.

Original: Moreover, we deﬁne h ∗ i =
Revised: Moreover, we deﬁne h ∗ i =

Original: arg min z ∈ R c φ
Revised: arg min z ∈ R c φ

Original: i ( z ) , i ∈ [ n ] .
Revised: i ( z ) , i ∈ [ n ] .

Original: Activation Functions.
Revised: Loss Functions.

Original: We summarize well-known choices of activation functions in Table 1. [Table] [Equation] [Table]
Revised: The well-known loss functions in neural networks for solving classiﬁcation and regression problems are softmax cross-entropy loss and square loss , respectively: (Softmax) Cross-Entropy Loss: F ( w ) =

Original: log( [Table]
Revised: 1 n (cid:80) ni

Original: + e x [Equation] Loss Functions. The well-known loss functions in neural networks for solving classiﬁcation and regression problems are softmax cross-entropy loss and square loss , respectively: (Softmax) Cross-Entropy Loss:
Revised: =1 f

Original: [Equation]
Revised: [Equation]

Original: where I ( y ( i ) ) is class of y ( i ) (output y ( i

Original: ) ∈ R c contains 1 at the I ( y ( i ) ) -th element and 0 otherwise). Hence, F ( w ) = 1 n (cid:80) ni =1 f ( w ;i ) with
Revised: w ;

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation]
Revised: [Equation]

Original: Squared Loss : F ( w ) = 1 n (cid:80) ni =1 f ( w ; i ) with
Revised: Squared Loss : F ( w ) = 1 n (cid:80) ni =1 f ( w ; i ) with

Original: [Equation]
Revised: [Equation]

Original: We provide some basic deﬁnitions in optimization theory to support our theory.
Revised: We provide some basic deﬁnitions in optimization theory to support our theory.

Original: Deﬁnition 1 ( L -smooth) .
Revised: Deﬁnition 1 ( L -smooth) .

Original: Function φ : R c → R is L φ -smooth if there exists a constant L φ >such that, ∀ x 1 , x 2 ∈ R c ,
Revised: Function φ : R c → R is L φ -smooth if there exists a constant L φ >such that, ∀ x 1 , x 2 ∈ R c ,

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 2 (Convex) .
Revised: Deﬁnition 2 (Convex) .

Original: Function φ : R c → R is convex if ∀ x 1 , x 2 ∈ R c ,
Revised: Function φ : R c → R is convex if ∀ x 1 , x 2 ∈ R c ,

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 3 (Twice continuously differentiablity) .

Original: Function g : R d → R is twice continuously differentiable if all of its second-order partial derivatives exist and are continuous.

Original: A multivariate function g : R d → R c is twice continuously differentiable if all of its scalar components g j : R d → R , j ∈ [ c ] are twice continuously differentiable.

Original: The following lemmas provide key tools for our results.

Original: Their proofs are in the Appendix.

Original: Lemma 1 (Squared loss) .

Original: Let b ∈ R c and deﬁne φ ( z ) =

Original: 12 (cid:107) z − b (cid:107) 2 for z ∈ R c .

Original: Then φ is convex and L φ -smooth with L φ = 1 .

Original: Lemma 2 (Softmax cross-entropy loss) . Let index a ∈ [ c ] and deﬁne [Equation] for z = ( z 1 , . . .
Revised: The following corollary shows the properties of softmax cross-entropy loss (2) and squared loss (3).

Original: , z c ) (cid:62) ∈ R c , where w k = e k − e a with e i representing the i -th unit vector (containing 1 at the i -th position and 0 elsewhere).

Original: Then φ is convex and L φ -smooth with L φ = 1 .

Original: Corollary 1.
Revised: Corollary 1.

Original: For softmax cross-entropy loss (2) and squared loss (3), there exist functions h ( · ; i ) :
Revised: For softmax cross-entropy loss (2) and squared loss (3), there exist functions h ( · ; i ) :

Original: R d → R c and φ i : R c
Revised: R d → R c and φ i : R c

Original: → R such that, for i ∈ [ n ] , φ i ( z ) is convex and L φ -smooth with L φ = 1 , and
Revised: → R such that, for i ∈ [ n ] , φ i ( z ) is convex and L φ -smooth with L φ = 1 , and

Original: [Equation]
Revised: [Equation]

Original:  N EW A LGORITHM F RAMEWORK 4.K EY I NSIGHT 
Revised:  N EW A LGORITHM F RAMEWORK 4.

Revised: [Equation] NSIGHT

Original: We assume f ( w ; i ) =
Revised: We assume f ( w ; i ) =

Original: φ i ( h ( w ; i )) with φ i convex and L φ -smooth.
Revised: φ i ( h ( w ; i )) with φ i convex and L φ -smooth.

Original: Our goal is to utilize the convexity of the outer function φ i .
Revised: Our goal is to utilize the convexity of the outer function φ i .

Original: In order to simplify notation, we write ∇ z φ i ( h ( w ( t ) ; i )) instead of ∇ z φ i ( z ) (cid:12)(cid:12)
Revised: In order to simplify notation, we write ∇ z φ i ( h ( w ( t ) ; i )) instead of ∇ z φ i ( z ) (cid:12)(cid:12)

Original: z = h ( w ( t ) ; i ) and denote z ( t ) i
Revised: z = h ( w ( t ) ; i ) and denote z ( t ) i

Original: = h ( w ( t ) ; i ) .
Revised: = h ( w ( t ) ; i ) .

Original: Starting from the current weight w ( t ) , we would like to ﬁnd the next point w ( t +1) that satisﬁes the following approximation for all i ∈ [ n ] :
Revised: Starting from the current weight w ( t ) , we would like to ﬁnd the next point w ( t +1) that satisﬁes the following approximation for all i ∈ [ n ] :

Original: [Equation]
Revised: [Equation]

Original: Apparently this approximation is a “noisy” version of a gradient descent update for every function φ i , simultaneously for all i ∈ [ n ] .
Revised: We can see that this approximation is a “noisy” version of a gradient descent update for every function φ i , simultaneously for all i ∈ [ n ] .

Original: In order to do this, we use the following update
Revised: In order to do this, we use the following update

Original: [Equation]
Revised: [Equation]

Original: where η ( t ) > 0 is a learning rate and v ( t ) is a search direction that helps us approximate equation (7).
Revised: where η ( t ) > 0 is a learning rate and v ( t ) is a search direction that helps us approximate equation (7).

Original: If the update term η ( t ) v ( t ) is small enough, and if h ( · ; i ) has some nice smooth properties, then from basic calculus we have the following approximation:
Revised: If the update term η ( t ) v ( t ) is small enough, and if h ( · ; i ) has some nice smooth properties, then from basic calculus we have the following approximation:

Original: [Equation]
Revised: [Equation]

Original: where H ( t ) i is a matrix in R c × d with ﬁrst-order derivatives.
Revised: where H ( t ) i is a matrix in R c × d with ﬁrst-order derivatives.

Original: Motivated by approximations (7) and (9), we consider the following optimization problem:
Revised: Motivated by approximations (7) and (9), we consider the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Hence, by solving for the solution v ( t ) ∗ of problem (10) we are able to ﬁnd a search direction for the key approximation (7).
Revised: Hence, by solving for the solution v ( t ) ∗ of problem (10) we are able to ﬁnd a search direction for the key approximation (7).

Original: This yields our new algorithmic Framework 1, see below.
Revised: This yields our new algorithmic Framework 1, see below.

Original: Framework 1 New Algorithm Framework
Revised: Framework 1 New Algorithm Framework

Original: Initialization: Choose an initial point w (0) ∈
Revised: Initialization: Choose an initial point w (0) ∈

Original: R
Revised: R

Original: d ; for t = 0 , 1 , · · · , T − 1 do
Revised: d ; for t = 0 , 1 , · · · , T − 1 do

Original: Solve for an approximation v ( t ) of the solution v ( t ) ∗ of the problem in (10) v ( 
Revised: Solve for an approximation v ( t ) of the solution v ( t ) ∗ of the problem in (10) [Equation] ( t )

Original: [Equation]
Revised: [Equation]

Revised: w

Original: [Equation]
Revised: [Equation]

Original: = arg
Revised: t

Original: [Table]
Revised: [Equation]

Original: end for
Revised: i [Equation] Update w ( t +1) = w ( t ) − η ( t ) v ( t )

Original: 4. T ECHNICAL A SSUMPTIONS
Revised: 4. T ECHNICAL A SSUMPTIONS

Original: Assumption 1.
Revised: Assumption 1.

Original: The loss function φ i is convex and L φ -smooth for i ∈ [ n ] .
Revised: The loss function φ i is convex and L φ -smooth for i ∈ [ n ] .

Original: Moreover, we assume that it is lower bounded, i.e. inf z ∈ R c
Revised: Moreover, we assume that it is lower bounded, i.e. inf z ∈ R c

Original: φ i ( z ) >
Revised: φ i ( z ) >

Original: −∞ for i ∈ [ n ] .
Revised: −∞ for i ∈ [ n ] .

Original: We have shown the convexity and smoothness of squared loss and softmax cross-entropy loss in
Revised: We have shown the convexity and smoothness of squared loss and softmax cross-entropy loss in

Original: Section 3.
Revised: Section 3.

Original: The bounded property of φ i is required in any algorithm for the well-deﬁnedness of (1).
Revised: The bounded property of φ i is required in any algorithm for the well-deﬁnedness of (1).

Original: Now, in order to use the Taylor series approximation, we need the following assumption on the neural network architecture h :
Revised: Now, in order to use the Taylor series approximation, we need the following assumption on the neural network architecture h :

Original: Assumption 2.
Revised: Assumption 2.

Original: We assume that h ( · ; i ) is twice continuously differentiable for all i ∈ [ n ] (i.e. the second-order partial derivatives of all scalars h j ( · ; i ) are continuous for all j ∈ [ c ] and i ∈ [ n ] ), and that their Hessian matrices are bounded, that is, there exists a G > 0 such that for all w ∈ R d , i ∈
Revised: We assume that h ( · ; i ) is twice continuously differentiable for all i ∈ [ n ] (i.e. the second-order partial derivatives of all scalars h j ( · ; i ) are continuous for all j ∈ [ c ] and i ∈ [ n ] ), and that their Hessian matrices are bounded, that is, there exists a G > 0 such that for all w ∈ R d , i ∈

Original: [ n ] and j ∈ [ c ] ,
Revised: [ n ] and j ∈ [ c ] ,

Original: [Equation]
Revised: [Equation]

Original: where J w denotes the Jacobian 1 .
Revised: where J w denotes the Jacobian 1 .

Original: Remark 1 (Non-smooth activations) .
Revised: Remark 1 (Relation to second-order methods) .

Original: Apparently when the activation function is smooth and all the iterations w ( t ) stay in a bounded region for every t then Assumption 2 is satisﬁed. However this fact may not be true for some other architectures with non-smooth activations, e.g. ReLU function. A common solution to  this problem is choosing a smooth version of ReLU toreplace the non-smooth activation (Nguyen & Mondelli, 2020).
Revised: Although our analysis requires an assumption on the Hessian matrices of h ( w ; i ) , our algorithms do not use any second order information or try to approximate this information. Our theoretical analysis focused on the approximation of the classiﬁer and the gradient information, therefore is not related to the second order type algorithms.

Original: An alternative approach is  to approximate h ( · ; i ) by some nice smooth function k ( · ; i ) that satisﬁes Assumption 2. In the Appendix we prove that our analysis still holds if the approximations satisfy | k j ( w ; i ) − h j ( w ; i ) | ≤ ε for all w ∈ R d , i ∈ [ n ] and j ∈ [ c ] , where ε is from our theorems.
Revised: It is currently unclear how to apply second order methods into our problem, however, this is an interesting research question to expand the scope of this work.

Original: Assumption 2 allows us to apply a Taylor approximation of each function h j ( · ; i ) with which we prove the following Lemma that bounds the error in equation (9):
Revised: Assumption 2 allows us to apply a Taylor approximation of each function h j ( · ; i ) with which we prove the following Lemma that bounds the error in equation (9):

Original: Lemma 3.
Revised: Lemma 1.

Original: Suppose that Assumption 2 holds for the classiﬁer h .
Revised: Suppose that Assumption 2 holds for the classiﬁer h .

Original: Then for all i ∈ [ n ] and 0 ≤ t < T ,
Revised: Then for all i ∈ [ n ] and 0 ≤ t < T ,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: is deﬁned as the Jacobian matrix of h ( w ; i ) at w ( t ) and entries (cid:15) ( t ) i,j , j ∈ [ c ] , of vector (cid:15) ( t ) i satisfy
Revised: is deﬁned as the Jacobian matrix of h ( w ; i ) at w ( t ) and entries (cid:15) ( t ) i,j , j ∈ [ c ] , of vector (cid:15) ( t ) i satisfy

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In order to approximate (7) combined with (9), that is, to make sure the right hand sides of (7) and (9) are close to one another, we consider the optimization problem (10):
Revised: In order to approximate (7) combined with (9), that is, to make sure the right hand sides of (7) and (9) are close to one another, we consider the optimization problem (10):

Original: [Equation]
Revised: [Equation]

Original: The optimal value of problem (10) is equal to 0 if there exists a vector v ( t ) ∗ satisfying H ( t ) i η ( t )  v ( t ) ∗ =
Revised: The optimal value of problem (10) is equal to 0 if there exists a vector v ( t ) ∗ satisfying η ( t ) H ( t ) i v ( t ) ∗ =

Original: α ( t ) i ∇ z φ i ( h ( w ( t ) ; i )) for every i ∈ [ n ] .
Revised: α ( t ) i ∇ z φ i ( h ( w ( t ) ; i )) for every i ∈ [ n ] .

Original: Since the solution v ( t ) ∗ is in R d and ∇ z φ i ( h ( w ( t ) ; i )) is in R c , this condition is equivalent to a linear system with n · c constraints and d variables.
Revised: Since the solution v ( t ) ∗ is in R d and ∇ z φ i ( h ( w ( t ) ; i )) is in R c , this condition is equivalent to a linear system with n · c constraints and d variables.

Original: In the overparameterized setting where dimension d is sufﬁciently large ( d (cid:29) n · c ) and there are no identical data, there exists almost surely a vector v ( t ) ∗ that interpolates all the training set, see the Appendix for details.
Revised: In the overparameterized setting where dimension d is sufﬁciently large ( d (cid:29) n · c ) and there are no identical data, there exists almost surely a vector v ( t ) ∗ that interpolates all the training set, see the Appendix for details.

Original: Let us note that an approximation of v ( t ) ∗ serves as the search direction for Framework 1.
Revised: Let us note that an approximation of v ( t ) ∗ serves as the search direction for Framework 1.

Original: For this reason, the solution v ( t ) ∗ of problem (10) plays a similar role as a gradient in the search direction of (stochastic) gradient descent method.
Revised: For this reason, the solution v ( t ) ∗ of problem (10) plays a similar role as a gradient in the search direction of (stochastic) gradient descent method.

Original: It is standard to assume a bounded gradient in the machine learning literature (Nemirovski et al., 2009; Shalev-Shwartz et al., 2007; Reddi et al., 2016).
Revised: It is standard to assume a bounded gradient in the machine learning literature (Nemirovski et al., 2009; Shalev-Shwartz et al., 2007; Reddi et al., 2016).

Original: Motivated by these facts, we assume the following Assumption 3, which implies the existence of a near-optimal bounded solution of (10):
Revised: Motivated by these facts, we assume the following Assumption 3, which implies the existence of a near-optimal bounded solution of (10):

Original: Assumption 3.
Revised: Assumption 3.

Revised: We consider an over-parameterized setting where dimension d is sufﬁciently large enough to interpolate all the data and the tolerance ε .

Original: There exists a bound V > 0 such that for ε > 0 and 0 ≤ t < T as in Framework 1, there exists a vector ˆ v ( t ) ∗  with (cid:107) ˆ v ( t ) ∗  (cid:107) 2 ≤ V so that
Revised: We assume that there exists a bound V >such that for ε > 0 and 0 ≤ t < T as in Framework 1, there exists a vector ˆ v ( t ) ∗ ε with (cid:107) ˆ v ( t ) ∗ ε (cid:107) 2 ≤ V so that

Original: [Equation]
Revised: [Equation]

Revised: Our Assumption 3 requires a nice dependency on the tolerance ε for the gradient matrices H ( t ) i and ∇ z φ i ( h ( w ( t ) ; i )) .

Revised: We note that at the starting point t = 0 , these matrices may depend on ε due to the initialization process and the dependence of d on ε .

Revised: This setting is similar to previous works, e.g.

Original:   N EW A LGORITHMS AND C ONVERGENCE R ESULTS 5. A PPROXIMATING THE SOLUTION USING REGULARIZER
Revised: Allen-Zhu et al.  N EW A LGORITHMS AND C ONVERGENCE R ESULTS 5. A PPROXIMATING THE SOLUTION USING REGULARIZER

Original: Since problem (10) is convex and quadratic
Revised: Since problem (10) is convex and quadratic

Original: [Equation]
Revised: [Equation]

Original: we consider the following regularized problem:
Revised: we consider the following regularized problem:

Original: [Equation]
Revised: [Equation]

Original: for some small ε > 0 and t ≥ 0 .
Revised: for some small ε > 0 and t ≥ 0 .

Original: It is widely known that problem (15) is strongly convex [Equation] and has a unique minimizer v ( t ) ∗ reg .
Revised: It is widely known that problem (15) is strongly convex, and has a unique minimizer v ( t ) ∗ reg .

Original: The global minimizer satisﬁes ∇ v Ψ( v ( t ) ∗ reg )
Revised: The global minimizer satisﬁes ∇ v Ψ( v ( t ) ∗ reg )

Original: = 0 .
Revised: = 0 .

Original: We have∇
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: If ε 2 is small enough, then v ( t ) ∗ reg is a close approximation of the solution v ( t ) ∗ for problem (10).
Revised: If ε 2 is small enough, then v ( t ) ∗ reg is a close approximation of the solution v ( t ) ∗ for problem (10).

Original: Our ﬁrst algorithm updates Framework 1 based on this approximation.
Revised: Our ﬁrst algorithm updates Framework 1 based on this approximation.

Original: Algorithm 1 Solve for the exact solution of the regularized problem
Revised: Algorithm 1 Solve for the exact solution of the regularized problem

Original: Initialization: Choose an initial point w (0) ∈ R d , tolerance ε > 0 ; for t = 0 , 1 , · · · , T − 1 do
Revised: Initialization: Choose an initial point w (0) ∈ R d , tolerance ε > 0 ; for t = 0 , 1 , · · · , T − 1 do

Original: Update the search direction v ( t ) as the solution v ( t ) ∗ reg of problem in (15): v ( t )
Revised: Update the search direction v ( t ) as the solution v ( t ) ∗ reg of problem in (15): v ( t )

Original: = v ( t ) ∗ reg =
Revised: = v ( t ) [Equation] reg [Equation]

Original: (cid:32)
Revised: (cid:32)

Original: [Equation]
Revised: [Equation]

Original: n 
Revised: n n (cid:88) [Equation] ( t )

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: ( t )

Original: [Equation]
Revised: [Equation]

Revised: (cid:62)

Revised: [Equation]

Revised: ( t )

Original: =
Revised: [Equation] ( t ) [Equation] (cid:33) − 1 (cid:32)

Original: [Equation]
Revised: [Equation]

Revised: n n (cid:88) i

Revised: =

Revised: α ( t ) i η ( t ) H ( t ) i (cid:62) ∇ z φ

Revised: i ( h ( w ( t )

Original: [Equation]
Revised: [Equation]

Revised: i ))

Original: (cid:33)
Revised: (cid:33)

Original: Update w ( t +1) =
Revised: Update w ( t +1) =

Original: w ( t ) − η ( t ) v ( t ) end for
Revised: w ( t ) − η ( t ) v ( t ) end for

Original: The following Lemma shows the relation between the regularized solution v ( t ) ∗ reg and the optimal solution of the original convex problem  v ( t ) ∗  .
Revised: The following Lemma shows the relation between the regularized solution v ( t ) ∗ reg and the optimal solution of the original convex problem ˆ v ( t ) ∗ ε .

Original: Lemma 4.
Revised: Lemma 2.

Original: For given ε > 0 , suppose that Assumption 3 holds for bound V > 0 .
Revised: For given ε > 0 , suppose that Assumption 3 holds for bound V > 0 .

Original: Then, for iteration 0 ≤ t < T , the optimal solution v ( t ) ∗ reg of problem (15) satisﬁes (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V and[Equation]
Revised: Then, for iteration 0 ≤ t < T , the optimal solution v ( t ) ∗ reg of problem (15) satisﬁes (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V and

Original: [Equation]
Revised: [Equation]

Original: Based on Lemma 4, we guarantee the global convergence of Algorithm 1 and prove our ﬁrst theorem.
Revised: Based on Lemma 2, we guarantee the global convergence of Algorithm 1 and prove our ﬁrst theorem.

Original: Since it is currently expensive to solve for the exact solution of problem (15), our algorithm serves as a theoretical method to obtain the global convergence for the ﬁnite-sum minimization.
Revised: Since it is currently expensive to solve for the exact solution of problem (15), our algorithm serves as a theoretical method to obtain the global convergence for the ﬁnite-sum minimization.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let w ( t ) be generated by Algorithm 1 where we use the closed form solution for the search direction.
Revised: Let w ( t ) be generated by Algorithm 1 where we use the closed form solution for the search direction.

Original: We execute Algorithm 1 for T =
Revised: We execute Algorithm 1 for T =

Original: βε outer loops for some constant β > 0 .
Revised: βε outer loops for some constant β > 0 .

Original: We assume Assumption 1 holds.
Revised: We assume Assumption 1 holds.

Original: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .
Revised: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .

Original: We set the step size equal to η ( t ) =
Revised: We set the step size equal to η ( t ) =

Original: D √ ε for some D > 0 and choose a learning rate α ( t ) i
Revised: D √ ε for some D > 0 and choose a learning rate α ( t ) i

Original: = (1 + ε ) α
Revised: = (1 + ε ) α

Original: ( t − 1)
Revised: ( t − 1)

Original: i =
Revised: i =

Original: (1 + ε ) t α (0) i .
Revised: (1 + ε ) t α (0) i .

Original: Based on β , we deﬁne α (0) 
Revised: Based on β , we deﬁne α (0) i

Original: i
Revised: =

Original: [Equation]

Original:  L φ with α ∈ (0 , 13 ) .
Revised: α e β L φ with α ∈ (0 , 13 ) .

Original: Let F ∗ be the global minimizer of F , and h ∗ i =
Revised: Let F ∗ be the global minimizer of F , and h ∗ i =

Original: arg
Revised: arg

Original: min z ∈ R c φ
Revised: min z ∈ R c φ

Original: i ( z ) , i ∈ [ n ] .
Revised: i ( z ) , i ∈ [ n ] .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Revised: We note that β is a constant for the purpose of choosing the number of iterations T .

Revised: The analysis can be simpliﬁed by choosing β = 1 with T = 1 ε .

Revised: Notice that the common convergence criteria for ﬁnding a stationary point for non-convex problems is 1 T (cid:80)

Revised: Tt =1 ||∇ F ( w t )

Revised: ||

Revised: 2 ≤ O ( ε ) .

Revised: This criteria has been widely used in the existing literature for non-convex optimization problems.

Original: {+Our convergence criteria 1 T (cid:80) Tt =
+}
Revised: {+Our convergence criteria 1 T (cid:80) Tt =
+}

Revised: [ F ( w t ) − F ∗ ] ≤ O ( ε ) is slightly different, in order to ﬁnd a global solution for non-convex problems.

Original: Our proof for Theorem 1 is novel and insightful.
Revised: Our proof for Theorem 1 is novel and insightful.

Original: It is originally motivated by the Gradient Descent update (7) and the convexity of the loss functions φ i .
Revised: It is originally motivated by the Gradient Descent update (7) and the convexity of the loss functions φ i .

Original: For this reason it may not be a surprise that
Revised: For this reason it may not be a surprise that

Original: Algorithm 1 can ﬁnd an ε -global solution after O (cid:0) 1 ε (cid:1) iterations.
Revised: Algorithm 1 can ﬁnd an ε -global solution after O (cid:0) 1 ε (cid:1) iterations.

Original: However, computing the exact solution in every iteration might be extremely challenging, especially when the number of samples n is large.
Revised: However, computing the exact solution in every iteration might be extremely challenging, especially when the number of samples n is large.

Original: Therefore, we present a different approach to this problem in the following section.
Revised: Therefore, we present a different approach to this problem in the following section.

Original: 5. A PPROXIMATION USING G RADIENT D ESCENT
Revised: 5. A PPROXIMATION USING G RADIENT D ESCENT

Original: In this section, we use Gradient Descent (GD) algorithm to solve the strongly convex problem (15).
Revised: In this section, we use Gradient Descent (GD) algorithm to solve the strongly convex problem (15).

Original: It is well-known that if ψ ( x ) − µ 2 (cid:107) x (cid:107) 2 is convex for ∀ x ∈ R c , then ψ ( x ) is µ -strongly convex (see e.g. Nesterov (2004)).
Revised: It is well-known that if ψ ( x ) − µ 2 (cid:107) x (cid:107) 2 is convex for ∀ x ∈ R c , then ψ ( x ) is µ -strongly convex (see e.g. Nesterov (2004)).

Original: Hence Ψ( · ) is ε 2 -strongly convex.
Revised: Hence Ψ( · ) is ε 2 -strongly convex.

Original: For each iteration t , we use GD to ﬁnd a search direction v ( t ) which is sufﬁciently close to the optimal solution v ( t ) ∗ reg in that(cid:107)
Revised: For each iteration t , we use GD to ﬁnd a search direction v ( t ) which is sufﬁciently close to the optimal solution v ( t ) ∗ reg in that

Original: [Equation]
Revised: [Equation]

Original: Our Algorithm 2 is described as follows.
Revised: Our Algorithm 2 is described as follows.

Original: Algorithm 2 Solve the regularized problem using Gradient Descent
Revised: Algorithm 2 Solve the regularized problem using Gradient Descent

Original: Initialization: Choose an initial point w (0) ∈ R d , tolerance ε > 0 ; for t = 0 , 1 , · · · , T − 1 do
Revised: Initialization: Choose an initial point w (0) ∈ R d , tolerance ε > 0 ; for t = 0 , 1 , · · · , T − 1 do

Original: Use Gradient Descent algorithm to solve Problem (15) and ﬁnd a solution v ( t ) that satisﬁes 
Revised: Use Gradient Descent algorithm to solve Problem (15) and ﬁnd a solution v ( t ) that satisﬁes (cid:107) v ( t

Original: [Equation]
Revised: [Equation]

Original: Update w ( t +1) =
Revised: Update w ( [Equation] +

Original:  w ( t ) − η ( t ) v ( t ) end for
Revised: [Equation] = w ( [Equation] [Equation]

Original: Since Algorithm 2 can only approximate a solution within some ε -preciseness, we need a supplemental assumption for the analysis of our next Theorem 2:
Revised: Since Algorithm 2 can only approximate a solution within some ε -preciseness, we need a supplemental assumption for the analysis of our next Theorem 2:

Original: Assumption 4.
Revised: Assumption 4.

Original: Let H ( t ) i be the Jacobian matrix deﬁned in Lemma 3.
Revised: Let H ( t ) i be the Jacobian matrix deﬁned in Lemma 1.

Original: We assume that there exists some  H > 0 such that, for i ∈ [ n ]  and 0 ≤ t < T as in Algorithm 2,
Revised: We assume that there exists some constant H > 0 such that, for i ∈ [ n ] , ε > 0 , and 0 ≤ t < T as in Algorithm 2,

Original: [Equation]
Revised: [Equation]

Original: Assumption 4 bounds the  Jacobian matrix H ( t ) i  , hence, if all the iterations w ( t ) stay in a bounded region, then this assumption is satisﬁed.
Revised: Assumption 4 requires a mild condition on the bounded Jacobian of h ( w ; i ) , and the upper bound may depend on ε .

Original: This bounded region assumption is widely used in machine learning literature (Duchi et al., 2011a; Levy et al., 2018;
Revised: This ﬂexibility allows us to accommodate a good dependence of ε for the theoretical analysis.

Original: Reddi et al., 2018; G ¨ urb ¨ uzbalaban et al., 2019).

Original: We are now ready to present our convergence theorem for Algorithm 2.
Revised: We are now ready to present our convergence theorem for Algorithm 2.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let w ( t ) be generated by Algorithm 2 where v ( t ) satisﬁes (19).
Revised: Let w ( t ) be generated by Algorithm 2 where v ( t ) satisﬁes (19).

Original: We execute Algorithm 2 for T =
Revised: We execute Algorithm 2 for T =

Original: βε outer loops for some constant β > 0 .
Revised: βε outer loops for some constant β > 0 .

Original: We assume Assumption 1 holds.
Revised: We assume Assumption 1 holds.

Original: Suppose that Assumption 2 holds for G > 0 , Assumption 3 holds for V > 0 and Assumption 4 holds for
Revised: Suppose that Assumption 2 holds for G > 0 , Assumption 3 holds for V > 0 and Assumption 4 holds for

Original: H > 0 .We set the step size equal to η ( t ) =
Revised: H > 0 . We set the step size equal to η ( t ) =

Original: D √ ε for some D > 0 and choose a learning rate 
Revised: D √ ε for some D > 0 and choose a learning rate α ( t ) i

Original: [Equation] = 
Revised: = (1 + ε ) α

Original: [Equation]
Revised: ( t − 1) i =

Original: [Equation] )  i .
Revised: (1 + ε ) t α (0) i .

Original: Based on β , we deﬁne α (0) i
Revised: Based on β , we deﬁne α (0) i

Original: =
Revised: =

Original: α e β L φ with α ∈ (0 , 14 ) .
Revised: α e β L φ with α ∈ (0 , 14 ) .

Original: Let F ∗ be the global minimizer of F , and h ∗ i =
Revised: Let F ∗ be the global minimizer of F , and h ∗ i =

Original: arg
Revised: arg

Original: min z ∈ R c φ
Revised: min z ∈ R c φ

Original: i ( z ) , i ∈ [ n ] .
Revised: i ( z ) , i ∈ [ n ] .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Revised: h

Revised: [Equation]

Revised: [Equation]

Revised: + e

Revised: [Equation]

Revised: L

Revised: [Equation]

Revised: (4 ε + 3)

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: H

Revised: [Equation]

Revised: ε

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original:  Theorem 2 implies Corollary 2 which provides the computational complexity for Algoritm 2.
Revised: + [Equation] · ε. Theorem 2 implies Corollary 2 which provides the computational complexity for Algorithm 2.

Original: Note that for (Stochastic) Gradient Descent, we derive the complexity in terms of component gradient calculations for the ﬁnite-sum problem (1).
Revised: Note that for (Stochastic) Gradient Descent, we derive the complexity in terms of component gradient calculations for the ﬁnite-sum problem (1).

Original: As an alternative, for Algorithm 2 we compare the number of component gradients in problem (15).
Revised: As an alternative, for Algorithm 2 we compare the number of component gradients in problem (15).

Original: Such individual gradient has the following form:
Revised: Such individual gradient has the following form:

Original: [Equation]
Revised: [Equation]

Original: In machine learning applications, the gradient of f ( · ; i ) is calculated using automatic differentiation (i.e. backpropagation).
Revised: In machine learning applications, the gradient of f ( · ; i ) is calculated using automatic differentiation (i.e. backpropagation).

Original: Since f ( · ; i ) is the composition of the network structure h ( · ; i ) and loss function φ i ( · ) , this process also computes the Jacobian matrix H ( t ) i and the gradient ∇ z φ
Revised: Since f ( · ; i ) is the composition of the network structure h ( · ; i ) and loss function φ i ( · ) , this process also computes the Jacobian matrix H ( t ) i and the gradient ∇ z φ

Original: i ( h ( w ( t ) ; i )) at a speciﬁc weight w ( t ) .
Revised: i ( h ( w ( t ) ; i )) at a speciﬁc weight w ( t ) .

Original: Since matrix-vector multiplication computation is not expensive, the cost for computing the component gradient of problem (15) is similar to problem (1).
Revised: Since matrix-vector multiplication computation is not expensive, the cost for computing the component gradient of problem (15) is similar to problem (1).

Original: Corollary 2.
Revised: Corollary 2.

Original: Suppose that the conditions in Theorem 2 hold with η ( t ) =
Revised: Suppose that the conditions in Theorem 2 hold with η ( t ) =

Original: D √ ˆ ε √ N for some D > 0 and 0 < ˆ ε ≤ N (that is, we set ε
Revised: D √ ˆ ε √ N for some D > 0 and 0 < ˆ ε ≤ N (that is, we set ε

Original: = ˆ ε/N ), where
Revised: = ˆ ε/N ), where

Original: [Equation]
Revised: [Equation]

Original: Then, the total complexity to guarantee min 0 ≤ t ≤ T − 1 [ F ( 
Revised: Then, the total complexity to guarantee min 0 ≤ t ≤ T − 1 [ F ( w ( t ) ) − F ∗ ] ≤ 1 T [Equation] [ F ( w

Original: [Equation]
Revised: [Equation]

Original:  ˆ ε is 
Revised: ) ) − F [Equation] ] ≤ ˆ ε is O (cid:16)

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: Corollary 2 shows that O (1 / ˆ ε ) outer loop iterations are needed in order to reach an ˆ ε -global solution, and it proves that each iteration needs the equivalent of O (cid:0) n ˆ ε  log( 1ˆ ε ) (cid:1) gradient computations for computing an approximate solution.
Revised: Corollary 2 shows that O (1 / ˆ ε ) outer loop iterations are needed in order to reach an ˆ ε -global solution, and it proves that each iteration needs the equivalent of O (cid:0) n ˆ ε 2 log( 1ˆ ε ) (cid:1) gradient computations for computing an approximate solution.

Original: In total, Algorithm 2 has total complexity
Revised: In total, Algorithm 2 has total complexity

Original: O (cid:0) n ˆ ε 2 log( 1ˆ ε ) (cid:1) for ﬁnding an ˆ ε -global solution.
Revised: O (cid:0) n ˆ ε 3 log( 1ˆ ε ) (cid:1) for ﬁnding an ˆ ε -global solution.

Original: For a comparison, Stochastic Gradient Descent uses a total of O ( 1 ε 2 ) gradient computations to ﬁnd a stationary point satisfying  (cid:107)∇ F ( ˆ w ) (cid:107) 2  ≤ ε for non-convex problem (Ghadimi & Lan, 2013).
Revised: For a comparison, Stochastic Gradient Descent uses a total of O ( 1 ε 2 ) gradient computations to ﬁnd a stationary point satisfying E [ (cid:107)∇ F ( ˆ w ) (cid:107) 2 ] ≤ ε for non-convex problems (Ghadimi & Lan, 2013).

Original: Gradient Descent has a better complexity in terms of ε , i.e. O ( nε )  (Nesterov, 2004).
Revised: Gradient Descent has a better complexity in terms of ε , i.e. O ( nε ) such that (cid:107)∇ F ( ˆ w ) (cid:107) 2 ≤ ε (Nesterov, 2004).

Original: However, both methods may not be able to reach a global solution of (1).
Revised: However, both methods may not be able to reach a global solution of (1).

Original: In order to guarantee global convergence for nonconvex settings, one may resort to use Polyak-Lojasiewicz (PL) inequality (Karimi et al., 2016; Gower et al., 2021).
Revised: In order to guarantee global convergence for nonconvex settings, one may resort to use Polyak-Lojasiewicz (PL) inequality (Karimi et al., 2016; Gower et al., 2021).

Original: This assumption is widely known to be strong, which implies that every stationary point is also a global minimizer.
Revised: This assumption is widely known to be strong, which implies that every stationary point is also a global minimizer.

Original:  F URTHER D ISCUSSION AND C ONCLUSIONS
Revised:  F URTHER D ISCUSSION AND C ONCLUSIONS

Original: This paper presents an alternative composite formulation for solving the ﬁnite-sum optimization problem.
Revised: This paper presents an alternative composite formulation for solving the ﬁnite-sum optimization problem.

Original: Our formulation allows a new way of exploiting the structure of machine learning problems and the convexity of squared loss and softmax cross entropy loss, and leads to a novel algorithmic framework that guarantees global convergence (when the outer loss functions are convex and
Revised: Our formulation allows a new way of exploiting the structure of machine learning problems and the convexity of squared loss and softmax cross entropy loss, and leads to a novel algorithmic framework that guarantees global convergence (when the outer loss functions are convex and

Original: Lipschitz-smooth).
Revised: Lipschitz-smooth).

Original: Our analysis is general and can be applied to various different learning architectures, in particular, our analysis and assumptions match practical neural networks; in recent years, there has been a great interest in the structure of deep learning architectures for over-parameterized settings (Arora et al., 2018; Allen-Zhu et al., 2019; Nguyen & Mondelli, 2020).
Revised: Our analysis is general and can be applied to various different learning architectures, in particular, our analysis and assumptions match practical neural networks; in recent years, there has been a great interest in the structure of deep learning architectures for over-parameterized settings (Arora et al., 2018; Allen-Zhu et al., 2019; Nguyen & Mondelli, 2020).

Revised: Algorithm 2 demonstrates a gradient method to solve the regularized problem, however, other methods can be applied to our framework (e.g. conjugate gradient descent).

Original: Our theoretical foundation motivates further study, implementation, and optimization of the new algorithmic framework and further investigation of its non-standard bounded style assumptions.
Revised: Our theoretical foundation motivates further study, implementation, and optimization of the new algorithmic framework and further investigation of its non-standard bounded style assumptions.

Original: Possible research directions include more practical algorithm designs based on our Framework 1, and different related methods to solve the regularized problem and approximate the solution.
Revised: Possible research directions include more practical algorithm designs based on our Framework 1, and different related methods to solve the regularized problem and approximate the solution.

Original: This potentially leads to a new class of efﬁcient algorithms for machine learning problems.
Revised: This potentially leads to a new class of efﬁcient algorithms for machine learning problems.

Original: This paper presents a new perspective to the research community.
Revised: This paper presents a new perspective to the research community.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: This paper does not contain ethics concerns.
Revised: This paper does not contain ethics concerns.

Original: A PPENDIX A T ABLE OF NOTATIONS
Revised: A PPENDIX A T ABLE OF NOTATIONS

Original: Notation
Revised: Notation

Original: Meaning
Revised: Meaning

Original: F ∗
Revised: F [Equation]

Original: Global minimization function of F in (1)
Revised: Global minimization function of F in (1)

Original: [Equation]
Revised: [Equation]

Original: Solution of the convex problem in (10) min 
Revised: Solution of the convex problem in (10) min v ∈

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: v (
Revised: i ))

Original: [Equation]
Revised: [Equation]

Original:  An approximation of v ( t ) ∗ which is used as the search direction in Framework
Revised: v ( t ) An approximation of v ( t ) ∗ which is used as the search direction in Framework

Original: [Equation]
Revised: [Equation]

Original: A vector that satisﬁes
Revised: A vector that satisﬁes

Original: [Equation]
Revised: [Equation]

Original: for some ε > 0 and (cid:107) ˆ v ( t ) ∗  (cid:107) 2 ≤ V , for some V > 0 .
Revised: for some ε > 0 and (cid:107) ˆ v ( t ) ∗ ε (cid:107) 2 ≤ V , for some V > 0 .

Original: v ( t ) ∗ reg
Revised: v ( t ) ∗ reg

Original: Solution of the strongly convex problem in (15) 
Revised: Solution of the strongly convex problem in (15) min

Original: [Equation]
Revised: [Equation]

Original: B U SEFUL R ESULTS
Revised: B U SEFUL R ESULTS

Original: The following lemma is a standard result in (Nesterov, 2004).
Revised: The following lemmas provide key tools for our results. Lemma 3 (Squared loss) . Let b ∈ R c and deﬁne φ ( z ) = 12 (cid:107) z − b (cid:107) 2 for z ∈ R c .

Revised: Then φ is convex and L φ -smooth with L φ = 1 .

Revised: Lemma 4 (Softmax cross-entropy loss) .

Revised: Let index a ∈ [ c ] and deﬁne φ ( z ) = log (cid:34)

Revised: [Equation]

Revised: for z = ( z 1 , . . .

Revised: , z c ) (cid:62) ∈ R c , where w k = e k − e a with e i representing the i -th unit vector (containing 1 at the i -th position and 0 elsewhere).

Revised: Then φ is convex and L φ -smooth with L φ = 1 .

Revised: The following lemma is a standard result in (Nesterov, 2004).

Original: Lemma 5 ((Nesterov, 2004)) .
Revised: Lemma 5 ((Nesterov, 2004)) .

Original: If φ is L φ -smooth and convex, then for ∀ z ∈ R c ,
Revised: If φ is L φ -smooth and convex, then for ∀ z ∈ R c ,

Original: [Equation]
Revised: [Equation]

Original: (21) where z ∗ = arg min z φ ( z ) .
Revised: where z ∗ = arg min z φ ( z ) .

Original: The following useful derivations could be used later in our theoretical analysis.
Revised: The following useful derivations could be used later in our theoretical analysis.

Original: Since φ i is convex, by Deﬁnition 2 we have
Revised: Since φ i is convex, by Deﬁnition 2 we have

Original: [Equation]
Revised: [Equation]

Original: If φ i is convex and L φ -smooth
Revised: If φ i is convex and L φ -smooth, then by Lemma

Original: [Equation]
Revised: [Equation]

Original: then by Lemma

Original: [Equation]
Revised: [Equation]

Original: where h ∗ i = arg
Revised: where h ∗ i = arg

Original: min z ∈ R c φ
Revised: min z ∈ R c φ

Original: i ( z ) 
Revised: i ( z ) .

Original: [Equation]

Original: We compute gradients of f ( w ; i ) in term of φ i ( h ( w ; i )) .
Revised: We compute gradients of f ( w ; i ) in term of φ i ( h ( w ; i )) .

Original: Gradient of softmax cross

Original: -entropy loss

Original: [Equation]
Revised: [Equation]

Original: where for j ∈ [ c ] ,
Revised: where for j ∈ [ c ] ,

Original: [Equation]
Revised: [Equation]

Revised: • Gradient of squared loss:

Original: [Equation]
Revised: [Equation]

Original: C A DDITIONAL D ISCUSSION C. A BOUT A SSUMPTION
Revised: C A DDITIONAL D ISCUSSION C. A BOUT A SSUMPTION

Original: We make a formal assumption for the case h ( · ; i ) is closely approximated by k ( · ; i ) .
Revised: We make a formal assumption for the case h ( · ; i ) is closely approximated by k ( · ; i ) .

Original: Assumption 5.
Revised: Assumption 5.

Original: We assume that for all i ∈ [ n ] there exists some approximations k ( w ; i ) :
Revised: We assume that for all i ∈ [ n ] there exists some approximations k ( w ; i ) :

Original: R d → R c such that
Revised: R d → R c such that

Original: [Equation]
Revised: [Equation]

Original: where k ( · ; i ) are twice continuously differentiable (i
Revised: where k ( · ; i ) are twice continuously differentiable (i

Original: [Equation]
Revised: [Equation]

Original: e. the second-order partial derivatives of all scalars k j ( · ; i ) are continuous for all i ∈
Revised: e. the second-order partial derivatives of all scalars k j ( · ; i ) are continuous for all i ∈

Original: [ n ] ), and that their Hessian matrices are bounded:
Revised: [ n ] ) [Equation] and that their Hessian matrices are bounded:

Original: [Equation]
Revised: [Equation]

Original: Assumption 5 allows us to prove the following Lemma that bound the error in equation (9):
Revised: Assumption 5 allows us to prove the following Lemma that bound the error in equation (9):

Original: Lemma 6.
Revised: Lemma 6.

Original: Suppose that Assumption 5 holds for the classiﬁer h .
Revised: Suppose that Assumption 5 holds for the classiﬁer h .

Original: Then for all i ∈ [ n ] and 0 ≤ t < T , we have:
Revised: Then for all i ∈ [ n ] and 0 ≤ t < T , we have:

Original: [Equation]
Revised: [Equation]

Original: where H ( t ) i is deﬁned to be the Jacobian matrix of the approximation k ( w ; i ) at w ( t ) :
Revised: where H ( t ) i is deﬁned to be the Jacobian matrix of the approximation k ( w ; i ) at w ( t ) :

Original: [Equation]
Revised: [Equation]

Original: Additionally we have,
Revised: Additionally we have

Original: [Equation]
Revised: [Equation]

Original: Note that these result recover the case when h ( · ; i ) is itself smooth.
Revised: Note that these result recover the case when h ( · ; i ) is itself smooth.

Original: Hence we analyze our algorithms using the result of Lemma 6, which generalizes the result from Lemma 3.
Revised: Hence we analyze our algorithms using the result of Lemma 6, which generalizes the result from Lemma 1.

Original: C. A BOUT A SSUMPTION
Revised: C. A BOUT A SSUMPTION

Original: In this section [Equation] we justify the existence of the search direction in Assumption 3 (almost surely).
Revised: In this section, we justify the existence of the search direction in Assumption 3 (almost surely).

Original: We argue that there exists a vector ˆ v ( t ) ∗  satisfying
Revised: We argue that there exists a vector ˆ v ( t ) ∗ ε satisfying

Original: [Equation]
Revised: [Equation]

Original: Apparently we only need to guarantee the existence of a vector v such that
Revised: It is sufﬁcient to ﬁnd a vector v satisfying that

Original: [Equation]
Revised: [Equation]

Original: Since the solution v is in R d and ∇ z φ i ( h ( w ( t ) ; i )) is in R c [Equation] this condition is equivalent to a linear system with n · c constraints and d variables.
Revised: Since the solution v is in R d and ∇ z φ i ( h ( w ( t ) ; i )) is in R c , this condition is equivalent to a linear system with n · c constraints and d variables.

Original: Let A and b be the following stacked matrix and vector:
Revised: Let A and b be the following stacked matrix and vector:

Original: [Equation]
Revised: [Equation]

Original: then the problem reduce to ﬁnding the solution of the equation Av = b .
Revised: then the problem reduce to ﬁnding the solution of the equation Av = b .

Original: In the over-parameterized setting where dimension d is sufﬁciently large ( d (cid:29) n · c ), then rank A = n · c almost surely and there exists almost surely a vector v that interpolates all the training set.
Revised: In the over-parameterized setting where dimension d is sufﬁciently large ( d (cid:29) n · c ), then rank A = n · c almost surely and there exists almost surely a vector v that interpolates all the training set.

Original: To demonstrate this fact easier, we consider a simple neural network where the classiﬁer h ( w ; i ) is formulated as
Revised: To demonstrate this fact easier, we consider a simple neural network where the classiﬁer h ( w ; i ) is formulated as

Original: [Equation]
Revised: [Equation]

Original: where c = 1 , W (1) ∈
Revised: where c = 1 , W (1) ∈

Original: R m × l and W (2) ∈ R l × 1 , w = vec ( { W (1) , W (2) } )
Revised: R m × l and W (2) ∈ R l × 1 , w = vec ( { W (1) , W (2) } )

Original: ∈ R
Revised: ∈ R

Original: d is the vectorized weight where d = l ( m + 1) and σ is sigmoid activation function.
Revised: d is the vectorized weight where d = l ( m + 1) and σ is sigmoid activation function.

Original: H ( t ) i is deﬁned to be the Jacobian matrix of h ( w ; i ) at w ( t ) : 
Revised: H ( t ) i is deﬁned to be the Jacobian matrix of h ( w ; i ) at w ( t ) : [Equation] [Equation] w ; i ) ∂w

Revised: [Equation]

Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) w = w ( t ) ∈ R

Original: [Equation]
Revised: [Equation]

Original:  thenA
Revised: d , then

Original: [Equation]
Revised: [Equation]

Original: We want to show that A has full rank, almost surely.
Revised: We want to show that A has full rank, almost surely.

Original: We consider the over-parameterized setting where the last layer has at least n neuron (i.e. l = n and the simple version when c = 1 .
Revised: We consider the over-parameterized setting where the last layer has at least n neuron (i.e. l = n and the simple version when c = 1 .

Original: We argue that rank of matrix A is greater than or equal to rank of the submatrix B created by the weights of the last layer W (2) ∈ R n :
Revised: We argue that rank of matrix A is greater than or equal to rank of the submatrix B created by the weights of the last layer W (2) ∈ R n :

Original: [Equation]
Revised: [Equation]

Original: Note that h ( · , i ) is a linear function of the last weight layers (in this simple case W (2) ∈ R n and σ ( W (1) (cid:62)
Revised: Note that h ( · , i ) is a linear function of the last weight layers (in this simple case W (2) ∈ R n and σ ( W (1) (cid:62)

Original: x
Revised: x

Original: ( i ) )
Revised: ( i ) )

Original: ∈ R n ), we can compute the partial derivatives as follows:
Revised: ∈ R n ), we can compute the partial derivatives as follows:

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Assuming that there are no identical data, and
Revised: Assuming that there are no identical data, and

Original: σ is the sigmoid activation, the set of weights W (1) that make matrix B degenerate has measure zero.
Revised: σ is the sigmoid activation, the set of weights W (1) that make matrix B degenerate has measure zero.

Original: Hence B has full rank almost surely, and we have the same conclusion for A .
Revised: Hence B has full rank almost surely, and we have the same conclusion for A .

Original: Therefore we are able to prove the almost surely existence of a solution v of the linear equation Av = b for simple two layers network.
Revised: Therefore we are able to prove the almost surely existence of a solution v of the linear equation Av = b for simple two layers network.

Original: Using the same argument, this result can be generalize for larger neural networks where the dimension d is sufﬁciently large ( d (cid:29) nc ). 
Revised: Using the same argument, this result can be generalized for larger neural networks where the dimension d is sufﬁciently large ( d (cid:29) nc ). C. I NITIALIZATION EXAMPLE

Original:  D P ROOF FOR L EMMAS IN M AIN T EXT P ROOF OF L EMMA
Revised: Our Assumption 3 requires a nice dependency on the tolerance ε for the gradient matrices H (0) i and ∇ z φ i ( h ( w (0) ; i )) . We note that at the starting point t = 0 , these matrices may depend on ε due to the initialization process and the dependence of d on ε . In order to accommodate the choice of learning rate η (0) = D √ ε in our theorems, in this section we describe a network initialization that satisﬁes (cid:107) H (0)

Revised: i (cid:107) =

Revised: Θ (cid:16) 1 √ ε (cid:17) where the gradient norm (cid:107)∇ z φ i ( h ( w (0) ; i ))

Revised: (cid:107) is at most constant order with respect to ε .

Revised: To simplify the problem, we only consider small-dimension data and networks without activation.

Revised: About the target vector: We choose φ i to be the softmax cross-entropy loss.

Revised: By Lemma 7 (see below), we have that the gradient norm is upper bounded by a constant c , where c is the output dimension of the problem and is not dependent on ε .

Revised: Note that when we stack all gradients for n data points, then the size of new vector is still not dependent on ε .

Revised: About the network architecture:

Revised: For simplicity, we consider the following classiﬁcation problem where

Revised: • The input data is in R 2 .

Revised: There are only two data points { x (1) , x (2) } .

Revised: Input data is bounded and non-degenerate (we will clarify this property later).

Revised: • The output data is (categorical) in R 2 : { y (1) =

Revised: (1 , 0) , y (2) =

Revised: We want to have an over-parameterized setting where the dimension of weight vector is at least nc = 4 .

Revised: We consider a simple network with two layers, no biases and no activation functions.

Revised: Let the number of neurons in the hidden layer be m .

Revised: The ﬂow of this network is (in) R 2 → R m → R 2 (out).

Revised: First, we consider the case where m = 1 .

Revised: • The ﬁrst layer has 2 parameters ( w 1 , w 2 ) and only 1 neuron that outputs z ( i ) =

Revised: w 1 x ( i ) 1 + w 2 x ( i ) 2 (the subscript is for the coordinate of input data x ( i ) ).

Revised: • The second layer has 2 parameters ( w 3 , w 4 ) .

Revised: The ﬁnal output is

Revised: [Equation]

Revised: with w =

Revised: [ w 1 , w 2 , w 3 , w 4 ] (cid:62) ∈ R 4 .

Revised: This network satisﬁes that the Hessian matrices of h ( w ; i ) are bounded.

Revised: Let Q and b be the following stacked matrix and vector:

Revised: [Equation]

Revised: Then we have the following:

Revised: [Equation]

Revised: 2) w 1 x (2)1 + w 2 x (2)  w 4 x (2)w 4 x (2) w 1 x (2) 1 + w 2 x (2)   .

Revised: The determinant of this matrix is a polynomial of the weight w and the input data.

Revised: Under some mild non-degenerate condition of the input data, we can choose some base point w (cid:48) that made this matrix invertible (note that if this condition is not satisﬁed, we can rescale/add a very small noise to the data - which is the common procedure in machine learning).

Revised: Hence the system Qu =

Revised: b always has a solution.

Revised: Now we consider the following two initializations: 1.

Revised: We choose to initialize the starting point at w (0)

Revised: = 1 √ ε w (cid:48) and note that Q ( w ) is a linear function of w and Q ( w (cid:48) ) is independent of ε .

Revised: Then the norm of matrix Q ( w (0) ) has the same scale with 1 √ ε .

Revised: Instead of choosing m = 1 , we consider an over-parameterized network where m = 1 ε (recall that m is the number of neurons in the hidden layer).

Revised: The hidden layer in this case is:

Revised: [Equation]

Revised: The output layer is:

Revised: [Equation]

Revised: with w =

Revised: [ w (1)

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: w

Revised: [Equation]

Revised: [Equation]

Revised: w

Revised: [Equation]

Revised: [Equation]

Revised: w (2)1 , 1 , w (2)1 , 2 , . . .

Revised: , w (2) m, 1 , w (2) m, 2 ] (cid:62) ∈

Revised: R 4 m .

Revised: Hence,

Revised: Q ( w ) =

Revised:   w (2)1 , 1 x (1) . . .

Revised: w (2) m, 1 x (1) w (2)1 , 1 x (1) . . .

Revised: w (2) m, 1 x (1) z

Revised: z (1) m w (2)1 , 2 x (1) . . .

Revised: w (2) m, 2 x (1) w (2)1 , 2 x (1) . . .

Revised: w (2) m, 2 x (1)  z (1) . .

Revised: .z

Revised: (1) m w (2)1 , 1 x (2) . . .

Revised: w (2) m, 1 x (2) w (2)1 , 1 x (2) . . .

Revised: w (2) m, 1 x (2) z

Revised: z (2) m w (2)1 , 2 x (2) . . .

Revised: w (2) m, 2 x (2) w (2)1 , 2 x (2) . . .

Revised: w (2) m, 2 x (2)  z (2) . .

Revised: .z

Revised: (2) m 

Revised:  .

Revised: Hence, the number of (possibly) non-zero elements in each row is 3 m = 3 ε .

Revised: For matrix A of rank r , we have (cid:107) A (cid:107) 2 ≤ (cid:107) A (cid:107) F ≤ √ r (cid:107) A (cid:107) 2 .

Revised: Since the rank of Q ( w ) is at most( nc = 4 , independent of ε ), we only need to ﬁnd the Frobenius norm of Q ( w ) .

Revised: We have (cid:107) Q ( w ) (cid:107) F = (cid:118)(cid:117)(cid:117)(cid:116) 4 (cid:88) i = 4 m (cid:88)

Revised: j = | q ij | 2 .

Revised: Let q min and q max be the element with smallest/largest magnitude of Q ( w ) .

Revised: Suppose that x ( i ) (cid:54) =

Original: {+(0 , 0) and choose w (cid:54) = 
+}
Revised: {+(0 , 0) and choose w (cid:54) = 
+}

Revised: such that z (cid:54) = 0 , q min > 0 and independent of ε .

Revised: Hence, √ 8 √ ε | q min | ≤ (cid:107) Q ( w ) (cid:107) F ≤ √ 12 √ ε | q max | .

Revised: Hence, (cid:107) Q ( w ) (cid:107) =

Revised: Θ (cid:16) 1 √ ε (cid:17) .

Revised: Therefore this simple network initialization supports the dependence on ε for our Assumption 3.

Revised: We note that a similar setting is found in (Allen-Zhu et al., 2019), where the authors initialize the weights using a random Gaussian distribution with a variance depending on the dimension of the problem.

Revised: In non-convex setting, they prove the convergence of SGD using the assumption that the number of neurons m depends inversely on the tolerance (cid:15) .

Revised: Lemma 7.

Revised: For softmax cross-entropy loss, and x = h ( w ; i ) ∈ R c , for ∀ w ∈ R d and i ∈ [ n ] , we have

Revised: [Equation]

Revised: [Equation]

Revised: Proof.

Revised: By (24), we have for i = 1 , . . .

Revised: , n ,

Revised: [Table]

Revised: [Equation]

Revised: w ;

Revised: i

Revised: [Equation]

Revised: Hence, for i = 1 , . . . , n ,

Revised: [Equation]

Revised: This completes the proof.

Revised: D P ROOFS OF L EMMAS AND C OROLLARYP ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Revised: Since h ( · ; i ) are twice continuously differentiable for all i ∈ [ n ] , we have the following

Revised: Taylor approximation for each component outputs h j ( · ; i ) where j ∈ [ c ] and i ∈ [ n ] :

Revised: [Equation]

Revised: (32) where M i,j ( ˜ w ( t ) ) is the Hessian matrices of h j ( · ; i ) at ˜ w ( t ) and ˜ w ( t ) =

Revised: αw ( t ) + (1 − α ) w ( t +1) for some α ∈ [0 , 1] .

Revised: This leads to our desired statement:

Revised: [Equation]

Revised: where

Revised: [Equation]

Revised: Hence we get the ﬁnal bound:

Revised: [Equation]

Revised: P ROOF OF L EMMA

Revised: Proof.

Revised: From Assumption 3, we know that there exists ˆ v ( t ) ∗ ε so that

Revised: [Equation]

Revised: and (cid:107) ˆ v ( t ) ∗ ε (cid:107) 2 ≤ V , for some V > 0 .

Revised: Hence,

Revised: [Equation]

Revised: Since v ( t ) ∗ reg is the optimal solution of the problem in (15) for 0 ≤ t < T

Revised: [Equation]

Revised: we have

Revised: [Equation]

Revised: Therefore, we have (17) and (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V for 0 ≤ t < T .

Revised: P ROOF OF L EMMA

Original:  We want to show that for any α ∈ [0 , 1]
Revised: Proof. We want to show that for any α ∈ [0 , 1]

Original: [Equation]
Revised: [Equation]

Original: (31) in order to have the convexity of φ with respect to z (see (Nesterov, 2004)).
Revised: in order to have the convexity of φ with respect to z (see (Nesterov, 2004)).

Original: For any α ∈ [0 , 1] , we have for ∀ z 1 , z 2 ∈ R c ,
Revised: For any α ∈ [0 , 1] , we have for ∀ z 1 , z 2 ∈ R c ,

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality follows according to Cauchy-Schwarz inequality (cid:104) 
Revised: where the ﬁrst inequality follows according to Cauchy-Schwarz inequality (cid:104) a, b

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: a (cid:107)·(cid:107) b (cid:107) .
Revised: a [Equation] b (cid:107) .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Therefore, (31) implies the convexity of φ with respect to z .
Revised: Therefore, (33) implies the convexity of φ with respect to z .

Original: We want to show that ∃ L φ > 0 such that
Revised: We want to show that ∃ L φ > 0 such that

Original: [Equation]
Revised: [Equation]

Original: Notice that ∇ φ ( z ) =
Revised: Notice that ∇ φ ( z ) =

Original: z − b , then clearly ∀
Revised: z − b , then clearly ∀

Original: z 1 , z 2 ∈ R c ,
Revised: z 1 , z 2 ∈ R c ,

Original: [Equation]
Revised: [Equation]

Original: ( z [Equation] ∇ φ ( z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: (cid:107) = (cid:107) z

Original: 1 − z 2 (cid:107) .

Original: Therefore, (32) implies the L φ -smoothness of φ with respect to z with L φ = 1 .
Revised: Therefore, (34) implies the L φ -smoothness of φ with respect to z with L φ = 1 .

Original: P ROOF OF L EMMA
Revised: P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: For ∀ z 1 , z 2 ∈ R c and 1 ≤ k ≤ c , denote u k, 1 = exp( w (cid:62) k
Revised: For ∀ z 1 , z 2 ∈ R c and 1 ≤ k ≤ c , denote u k, 1 = exp( w (cid:62) k

Original: z 1 ) and u k, 2 = exp( w (cid:62) k
Revised: z 1 ) and u k, 2 = exp( w (cid:62) k

Original: z 2 ) and using Holder inequality
Revised: z 2 ) and using Holder inequality

Original: [Equation]
Revised: [Equation]

Original: , where 1 p + 1 q = 1 , (33) we have
Revised: , where 1 p + 1 q = 1 , [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality since log ( x ) is an increasing function for ∀ x > 0 and exp( v ) > 0 for ∀ v ∈ R .
Revised: where the ﬁrst inequality since log ( x ) is an increasing function for ∀ x > 0 and exp( v ) > 0 for ∀ v ∈ R .

Original: Therefore, (31) implies the convexity of φ with respect to z .
Revised: Therefore, (33) implies the convexity of φ with respect to z .

Original: Note that (cid:107)∇ 2 φ ( z ) (cid:107) ≤ L φ if and only if φ ( z ) is L φ
Revised: Note that (cid:107)∇ 2 φ ( z ) (cid:107) ≤ L φ if and only if φ ( z ) is L φ

Original: -smooth
Revised: -smooth

Original: (see (Nesterov, 2004)).
Revised: (see (Nesterov, 2004)).

Original: First, we compute gradient of φ ( z ) :
Revised: First, we compute gradient of φ ( z ) :

Original: • For i (cid:54)
Revised: • For i (cid:54)

Original: =
Revised: =

Original: a :
Revised: a :

Original: [Equation]
Revised: [Equation]

Revised: ( z )

Revised: [Equation]

Original: • For i = a :
Revised: • For i = a :

Original: [Equation]
Revised: [Equation]

Original: We then calculate
Revised: We then calculate

Original: [Equation]
Revised: [Equation]

Original: • For i =
Revised: For i =

Original: j :
Revised: j ∂

Original: [Equation]
Revised: [Equation]

Revised: • For i (cid:54) =

Revised: j : ∂ 2 φ ( z ) ∂z j ∂z

Revised: i

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: • For i (cid:54) =

Original: j
Revised: z

Original: [Equation]
Revised: [Equation]

Original: Denote that y i =
Revised: Denote that y i =

Original: exp( z i − z a ) ≥ 0 , i ∈
Revised: exp( z i − z a ) ≥ 0 , i ∈

Original: [ c ]
Revised: [ c ]

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: • For i =
Revised: • For i =

Original: j : (cid:12)(cid:12)(cid:12)(cid:12) ∂
Revised: j : (cid:12) (cid:12) (cid:12) (cid:12) ∂

Original: 2 φ ( z ) ∂z j ∂zi (cid:12)(cid:12)(cid:12)(cid:12) = (cid:12)(cid:12)(cid:12)(cid:12) y i ( (cid:80) ck
Revised: [Equation] ( z ) ∂z j ∂z

Original: [-=
 y k − y-] i) ( (cid:80) ck =1 y k )(cid:12)(cid:12)(cid:12)(cid:12) .
Revised: [-=
 y k − y-] i

Original: • For i (cid:54) = j : (cid:12)(cid:12)(cid:12)(cid:12) ∂ 2 φ ( z ) ∂z j ∂z i (cid:12)(cid:12)(cid:12)(cid:12) = |
Revised: [Equation]

Original: y i y j | ( (cid:80) ck =1 y k
Revised: (cid:12)(cid:12)(cid:12) =

Original: Recall that for matrix A

Original: = ( a ij ) ∈ R c × c : (cid:107) A (cid:107) 2 ≤ (cid:107) A (cid:107) 2 F = (cid:80) ci =1 (cid:80) cj =1 | a ij | 2 .
Revised: (cid:12)

Original: We have: c (cid:88) j = (cid:12)(cid:12)(cid:12)(cid:12) ∂ 2 φ ( z ) ∂z j ∂z i (cid:12)(cid:12)(cid:12)(cid:12) ≤ ( (cid:80) ck =1 y k )
Revised: [Equation]

Original:  

Original: y2 i (  c(cid:88) k 
Revised: y i ( (cid:80) c k =

Original: = y k − y i ) + (cid:88) j (cid:54) = i ( y i y j )  =
Revised: [Equation]

Original: ( (cid:80) ck =1 y k )   y 2 i ( c (cid:88) k
Revised: • For i j :

Original: [-= y k ) 2 − 
 y 
 i c (cid:88) k = y k .y-]
Revised: [-= y k ) 2 − 
 y 
 i c (cid:88) k = y k .y-][Equation]

Original: i + y 4 i + (cid:88) j (cid:54) = i ( y i y j )  =
Revised: [Table]

Original: ( (cid:80) ck

Original: =1 y k )(cid:34) y 2 i ( c (cid:88) k = y k ) 2 − 2 y 3 i c (cid:88) k = y k
Revised: Therefore,

Original: + y 2 i c (cid:88) k

Original: = y 2 k (cid:35) Therefore, (cid:107)∇ 2 φ ( z ) (cid:107) 2 ≤ c (cid:88) i = c (cid:88) j = (cid:12) (cid:12)(cid:12)(cid:12) ∂
Revised: [Table]

Original: 2 φ ( z ) ∂z j ∂z i (cid:12) (cid:12)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds since
Revised: where the last inequality holds since

Original: [Equation]
Revised: [Equation]

Original: which follows by the application of Holder inequality (33) with p = 2 , q = 2 , a k = y 3 / 2 k , and
Revised: which follows by the application of Holder inequality (35) with p = 2 , q = 2 , a k = y 3 / 2 k , and

Original: b k = y 1 / 2 k
Revised: b k = y 1 / 2 k

Original: (Note that y k ≥ 0 , k ∈ [ c ] ).
Revised: (Note that y k ≥ 0 , k ∈ [ c ] ).

Original: Hence, (cid:107)∇ 2 φ ( z ) (cid:107) ≤ L φ with L φ = 1 which is equivalent to
Revised: Hence, (cid:107)∇ 2 φ ( z ) (cid:107) ≤ L φ with L φ = 1 which is equivalent to

Original: L φ -smoothness of φ .
Revised: L φ -smoothness of φ .

Original: P ROOF OF L EMMA
Revised: P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: Since h ( · ; i ) are twice continuously differentiable for all i ∈ [ n ] , we have the following
Revised: Since k ( · ; i ) are twice continuously differentiable for all i ∈ [ n ] , we have the following

Original: Taylor approximation for each component outputs h j ( · ; i ) where j ∈ [ c ] and i ∈ [ n ] :[Equation] [Equation]
Revised: Taylor approximation for each component outputs k j ( · ; i ) where j ∈ [ c ] and i ∈ [ n ] :

Original: where M i,j ( ˜ w ( t ) ) is the Hessian matrices of h j ( · ; i ) at ˜ w ( t ) and ˜ w ( t ) =

Original: αw ( t ) + (1 − α ) w ( t +1) for some α ∈ [0 , 1] .

Original: This leads to our desired statement:

Original: [Equation]

Original: where

Original: [Equation]
Revised: [Equation]

Original: Hence we get the ﬁnal bound:

Original: [Equation]
Revised: [Equation]

Original: P ROOF OF L EMMA

Original: Proof.

Original: From Assumption 3, we know that there exists ˆ v ( t )∗ so that [Equation] 2 ≤ ε 2 , and (cid:107) ˆ v ( t ) ∗ (cid:107) 2 ≤ V , for some V > 0 . Hence,
Revised: where M i,j ( ˜ w ( t ) ) is the Hessian matrices of k j ( · ; i ) at ˜ w ( t ) and ˜ w ( t ) =

Original: [Equation] Since v ( t ) ∗ reg is the optimal solution of the problem in (15) for 0 ≤ t < T [Equation] we have
Revised: αw ( t ) + (1 − α ) w ( t +1) for some α ∈ [0 , 1] .

Original: [Equation]

Original: Therefore, we have (17) and (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V for 0 ≤ t < T .

Original: Proof. Since k ( · ; i ) are twice continuously differentiable for all i ∈ [ n ] , we have the following Taylor approximation for each component outputs k j ( · ; i ) where j ∈ [ c ] and i ∈ [
Revised: Shifting this back to the original function h j ( · ; i ) we have:

Original: [Equation]
Revised: [Equation]

Original: where M i,j ( ˜ w ( t ) ) is the Hessian matrices of k j ( · ; i ) at ˜ w ( t ) and ˜ w ( t ) =
Revised: which leads to our desired statement:

Original: αw ( t ) + (1 − α ) w ( t +1) for some α ∈ [0 , 1] .
Revised: [Equation]

Original: Shifting this back to the original function h j ( · ; i )we have:
Revised: i )

Original: [Equation]
Revised: [Equation]

Original: which leads to our desired statement:
Revised: h

Original: [Equation]
Revised: [Equation]

Revised: t )

Revised: [Equation]

Revised: η

Revised: [Equation]

Revised: t )

Revised: [Equation]

Revised: t )

Revised: [Equation]

Revised: i )

Revised: [Equation]

Revised: t )

Revised: [Equation]

Revised: i )

Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: t )

Revised: [Equation]

Revised: t )

Revised: [Equation]

Revised: t )

Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Hence we get the ﬁnal bound:
Revised: Hence we get the ﬁnal bound:

Original: [Equation]
Revised: [Equation]

Revised: P ROOF OF C OROLLARY

Revised: Proof.

Revised: The proof of this corollary follows directly by the applications of Lemmas 3 and 4.

Original: E T ECHNICAL P ROOFS FOR T HEOREM
Revised: E T ECHNICAL P ROOFS FOR T HEOREM

Original: Lemma 7.
Revised: Lemma 8.

Original: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > [Equation] and v ( t ) = v ( t ) ∗ reg .
Revised: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 , and v ( t ) = v ( t ) ∗ reg .

Original: Consider η ( t )
Revised: Consider η ( t )

Original: =
Revised: =

Original: D √ ε for some D > 0 and ε > 0 .
Revised: D √ ε for some D > 0 and ε > 0 .

Original: For i ∈ [ n ] and 0 ≤ t < T , we have
Revised: For i ∈ [ n ] and 0 ≤ t < T , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From (14), for i ∈ [ n ] , j ∈ [ c ] , and for 0 ≤ t < T , by Lemma 3 and Lemma 6 we have
Revised: From (14), for i ∈ [ n ] , j ∈ [ c ] , and for 0 ≤ t < T , by Lemma 1 and Lemma 6 we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows by the fact (cid:107) v ( t ) (cid:107)
Revised: where the last inequality follows by the fact (cid:107) v ( t ) (cid:107)

Original: 2 = (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V of Lemma 4 and η ( t ) =
Revised: 2 = (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V of Lemma 2 and η ( t ) =

Original: D √ ε .
Revised: D √ ε .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Lemma 8.
Revised: Lemma 9.

Original: Let w ( t ) be generated by Algorithm 1 where we use the closed form solution for the search direction.
Revised: Let w ( t ) be generated by Algorithm 1 where we use the closed form solution for the search direction.

Original: We execute Algorithm 1 for T =
Revised: We execute Algorithm 1 for T =

Original: βε outer loops for some constant β > 0 .
Revised: βε outer loops for some constant β > 0 .

Original: We assume Assumption 1 holds.
Revised: We assume Assumption 1 holds.

Original: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .
Revised: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .

Original: We set the step size equal to η ( t ) =
Revised: We set the step size equal to η ( t ) =

Original: D √ ε for some D > 0 and choose a learning rate α ( t ) i ≤ αL φ , for some α ∈ (0 , 13 ) .
Revised: D √ ε for some D > 0 and choose a learning rate α ( t ) i ≤ αL φ , for some α ∈ (0 , 13 ) .

Original: For i ∈ [ n ] and 0 ≤ t < T , we have
Revised: For i ∈ [ n ] and 0 ≤ t < T , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that we have the optimal solution v ( t ) ∗ reg for the optimization problem (15) for 0 ≤ t <
Revised: Note that we have the optimal solution v ( t ) ∗ reg for the optimization problem (15) for 0 ≤ t <

Original: T .
Revised: T .

Original: From (12), we have, for i ∈ [ n ] , h( w ( t +1) ; i ) = h ( w ( t )
Revised: From (12), we have, for i ∈ [ n ] , h

Original: [Equation]
Revised: [Equation]

Original: η ( t ) [Equation] ( t ) ∗ reg ; i )
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: i )
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: h ( w [Equation]
Revised: where we expand the square term.

Original: t ) ; i )
Revised: Now applying Young’s inequalities: 2 |

Original: [Equation]
Revised: [Equation]

Original: Hence, we have
Revised: u

Original: [Equation]
Revised: [Equation]

Original: where we expand the square term.
Revised: for ε > 0 and 2 |

Original: Now applying Young’s inequalities: 2 |(cid:104) u, v (cid:105)| ≤ (cid:107) u (cid:107) 2 ε/ 2 +( ε/ 2) (cid:107) v (cid:107)for ε > 0 and 2 |(cid:104) u, v (cid:105)| ≤[-(cid:107) u (cid:107) 
-]
Revised: [Equation] u, v [Equation] | ≤[-(cid:107) u (cid:107) 
-]

Revised: [Equation]

Revised: u

Original:  + (cid:107) v (cid:107) 2 we have:
Revised: [Equation] 2 + [Equation] v (cid:107) 2 we have:

Original: [Equation]
Revised: [Equation]

Original: t ) ∗ reg −  α ( t ) i ∇ z φ 
Revised: − 2 α ( t [Equation] i [ φ i [Equation]

Original: i ( h ( w ( t) ; i ))
Revised: h [Equation] w ( t

Original: (cid:107)+ 2( α ( t ) i ) 2 (cid:107)∇ z φ i ( h ( w ( t ) ; i ))
Revised: [Equation] ; i [Equation]

Original: (cid:107) 2 + 2 (cid:107) (cid:15) ( t ) i (cid:107)+ 2 (cid:107) H ( t ) i η ( t ) v ( t ) ∗ reg − α ( t ) i ∇ z φ i( h ( w ( t ) ; i )) (cid:107)(22) ≤ (1 + ε ) (cid:107) h ( w ( t ) ; i ) − h ∗
Revised: φ i

Original: i (cid:107) 2 + 3( α ( t ) i ) 2 (cid:107)∇ z φ i ( h ( w ( t ) ; i ))

Original: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: h ∗ i )]

Original: [Table] [Equation] Note [Table]
Revised: Note that from (23) we get that (cid:107)∇ z φ

Original: Applying this and using the fact that α ( t )  i ≤ αL φ [Table] for some α ∈ (0 , 13 ) [Table]
Revised: i ( h ( w ( t ) ; i ))

Original: we are able to derive: [Equation] [Table] [Equation]
Revised: (cid:107) 2 ≤ 2 L φ [ φ i ( h ( w ( t ) ; i ))

Original: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: − φ

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: i

Original: [Table] [Equation] [Table] [Equation]
Revised: ( h ∗ i )] .

Original: [Table] [Equation] [Table] [Equation]
Revised: Applying this and using the fact that α ( t ) i ≤ αL φ , for some α ∈ (0 , 13 ) , we are able to derive:

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows by Lemma 7.
Revised: where the last inequality follows by Lemma 8.

Original: Lemma 9.
Revised: Lemma 10.

Original: Let w ( t ) be generated by Algorithm 1 where we use the closed form solution for the search direction.
Revised: Let w ( t ) be generated by Algorithm 1 where we use the closed form solution for the search direction.

Original: We execute Algorithm 1 for T =
Revised: We execute Algorithm 1 for T =

Original: βε outer loops for some constant β > 0 .
Revised: βε outer loops for some constant β > 0 .

Original: We assume Assumption 1 holds.
Revised: We assume Assumption 1 holds.

Original: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .
Revised: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .

Original: We set the step size equal to η ( t ) =
Revised: We set the step size equal to η ( t ) =

Original: D √ ε for some D > 0 and choose a learning rate α ( t 
Revised: D √ ε for some D > 0 and choose a learning rate α ( t ) i

Original: [Equation] i = (1 + 
Revised: = (1 + ε ) α

Original: [Equation] ) α ( t − [Equation]
Revised: ( t − 1)

Original: i 
Revised: i =

Original: [Equation] [Equation] i .
Revised: (1 + ε ) t α (0) i .

Original: Based on β , we deﬁne α (0) i=
Revised: Based on β , we deﬁne α (0) i

Original: [Equation] with α ∈ (0 , 13 ) .
Revised: = α e β L φ with α ∈ (0 , 13 ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Rearranging the terms in Lemma [Equation] we have [Equation]
Revised: Rearranging the terms in Lemma 9, we have φ i ( h ( w ( t ) ; i ))

Original: + [Equation]
Revised: − φ

Original: i
Revised: i

Original: [Equation] i 
Revised: ( h ∗ i ) ≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The last inequality follows because the learning rate satisﬁes α (0) i
Revised: The last inequality follows because the learning rate satisﬁes α (0) i

Original: = α e β L φ ≤ αL φ and for t = 1 , . .
Revised: [Equation] β L φ ≤ αL φ and for t = 1 , . .

Original: , T =
Revised: , T =

Original: βε for some β >α
Revised: βε for some β >

Original: [Equation]
Revised: [Equation]

Original: since (1 + x ) 1 /x ≤ e , x > 0 .
Revised: since (1 + x ) 1 /x ≤ e , x > 0 .

Original: Moreover, we have 1 α ( t )
Revised: Moreover, we have 1 α ( t )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original:  Taking the average sum from t = 0 , . .
Revised: T [Equation] [Equation] Taking the average sum from t = 0 , . .

Original: , T − 1 , we have
Revised: , T − 1 , we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Taking the average sum from i = 1 , . .
Revised: Taking the average sum from i = [Equation] [Equation]

Original: . , n , we have
Revised: n , we have

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Therefore, applying (40) to (39), we have
Revised: Therefore, applying (41) to (40) [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: which is our desired result.
Revised: which is our desired result

Revised: [Equation]

Original: P ROOF OF T HEOREM
Revised: P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Revised: (cid:88) i =

Revised: φ i ( h ∗ i ) .

Original: Hence F ∗ − 1 n (cid:80) ni
Revised: Hence F ∗ − 1 n (cid:80) ni

Original: =

Revised: =

Original: φ i ( h ∗ i ) ≥ 0 .
Revised: φ i ( h ∗ i ) ≥ 0 .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Revised: ≤

Revised: T

Revised: T (cid:88) t =  n n (cid:88) i =

Revised: [ f ( w ( t ) ; i ) − φ

Revised: i ( h ∗ i )] (39) ≤ e β L φ (1 + ε ) 2(1 − 3 α ) αβ · n n (cid:88)

Revised: i =

Revised: (cid:107) h

Revised: ( w (0)

Revised: ; i ) − h ∗

Revised: i (cid:107) 2 · ε + e β L φ (3 ε + 2) 8 α (1 − 3 α ) (cid:2) c (4 + ( V + 2) GD 2 ) 2 + 8 + 4 V (cid:3) · ε.

Original: F T ECHNICAL P ROOFS FOR T HEOREM
Revised: F T ECHNICAL P ROOFS FOR T HEOREM

Original: Lemma 10.
Revised: Lemma 11.

Original: For 0 ≤ t < T , suppose that Assumption 3 holds for V ≥ 0 and v ( t ) satisﬁes (19).
Revised: For 0 ≤ t < T , suppose that Assumption 3 holds for V ≥ 0 and v ( t ) satisﬁes (19).

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From (cid:107) v ( t ) − v ( t ) ∗ reg (cid:107)2 ≤ ε .
Revised: From (cid:107) v ( t ) − v ( t ) ∗ reg (cid:107) ≤ ε .

Original: Using (cid:107) a (cid:107) 2 ≤ 2 (cid:107) a − b (cid:107) 2 + 2 (cid:107) b (cid:107) 2 , we have
Revised: Using (cid:107) a (cid:107) 2 ≤ 2 (cid:107) a − b (cid:107) 2 + 2 (cid:107) b (cid:107) 2 , we have

Original: [Equation]
Revised: [Equation]

Original:  where the last inequality follows since (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V for some V > 0 in Lemma 4.
Revised: + 2 V. where the last inequality follows since (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V for some V > 0 in Lemma 2.

Original: Lemma 11.
Revised: Lemma 12.

Original: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .
Revised: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .

Original: Consider η ( t )
Revised: Consider η ( t )

Original: =
Revised: =

Original: D √ ε for some D > 0 and ε > 0 .
Revised: D √ ε for some D > 0 and ε > 0 .

Original: For i ∈ [ n ] and 0 ≤ t < T , we have
Revised: For i ∈ [ n ] and 0 ≤ t < T , we have

Original: [Equation]
Revised: [Equation]

Original: 2 ) 2 ε 2 .
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From (14), for i ∈ [ n ] , j ∈ [ c ] , and for 0 ≤ t < T , by Lemma 3 and Lemma 6 we have
Revised: From (14), for i ∈ [ n ] , j ∈ [ c ] , and for 0 ≤ t < T , by Lemma 1 and Lemma 6 we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows by the application of Lemma 10 and η ( t )
Revised: where the last inequality follows by the application of Lemma 11 and η ( t )

Original: =
Revised: =

Original: D √ ε .
Revised: D √ ε .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Lemma 12.
Revised: Lemma 13.

Original: Let w ( t ) be generated by Algorithm 2 where v ( t ) satisﬁes (19).
Revised: Let w ( t ) be generated by Algorithm 2 where v ( t ) satisﬁes (19).

Original: We execute Algorithm 2 for T =
Revised: We execute Algorithm 2 for T =

Original: βε outer loops for some constant β > 0 .
Revised: βε outer loops for some constant β > 0 .

Original: We assume Assumption 1 holds.
Revised: We assume Assumption 1 holds.

Original: Suppose that
Revised: Suppose that

Original: Assumption 2 holds for G > 0 , Assumption 3 holds for V > 0 and Assumption 4 holds for H > 0 .We set the step size equal to η ( t ) =
Revised: Assumption 2 holds for G > 0 , Assumption 3 holds for V > 0 and Assumption 4 holds for H > 0 . We set the step size equal to η ( t ) =

Original: D √ ε for some D > 0 and choose a learning rate α ( t ) i ≤ αL φ , for some α ∈ (0 , 14 ) .
Revised: D √ ε for some D > 0 and choose a learning rate α ( t ) i ≤ αL φ , for some α ∈ (0 , 14 ) .

Original: For i ∈ [ n ] and 0 ≤ t < T , we have 
Revised: For i ∈ [ n ] and 0 ≤ t < T , we have (cid:107) [Equation] i [Equation]

Revised: [Equation]

Revised: w

Original: [Equation]
Revised: [Equation]

Revised: t

Revised: [Equation]

Revised: i ))

Revised: − φ

Revised: i

Revised: ( h

Revised: [Equation]

Revised: i )]

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that v ( t ) is obtained from the optimization problem (15) for 0 ≤ t < T .
Revised: Note that v ( t ) is obtained from the optimization problem (15) for 0 ≤ t < T .

Original: From (9), we have, for i ∈ [ n ] ,
Revised: From (9), we have, for i ∈ [ n ] ,

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence, we have [Equation] [Table] +

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: + 2 ·

Revised: [Table]

Revised: η

Revised: [Table]

Revised: H

Revised: [Table]

Revised: v

Revised: [Table]

Revised: − v

Revised: [Table]

Revised: [Table]

Revised: v

Revised: [Table]

Revised: −

Revised: [Table]

Revised: ∇

Revised: [Table]

Revised: φ

Original: [Equation]
Revised: [Table] [Table] − 2 · [Table]

Revised: α

Revised: [Table]

Revised: t

Revised: [Table]

Revised: ∇

Revised: [Table]

Revised: φ

Revised: [Table]

Original: we have
Revised: w [Table] [Table] +

Revised: [Table]

Revised: ∇

Revised: [Table]

Revised: φ

Revised: [Table]

Revised: w

Revised: [Table]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where we expand the square term.
Revised: where we expand the square term

Original:  Now applying Young’s inequalities: 2 |(cid:104) u, v (cid:105)| ≤(cid:107) u (cid:107) 2 ε/ 3 +( ε/ 3) (cid:107) v (cid:107)for ε > 0 and 2 |(cid:104) u, v (cid:105)| ≤ (cid:107) u (cid:107) 2 +
Revised: [Equation] Now applying Young’s inequalities: 2 |(cid:104) u, v (cid:105)| ≤

Original: [Equation]
Revised: [Equation]

Original: v
Revised: u

Original: [Equation]
Revised: [Equation]

Original: {+2 ε/ 3 +( ε/ 3) (cid:107) v (cid:107)for ε > 0 and 2 |(cid:104) u, v (cid:105)| ≤ (cid:107) u (cid:107) 
 + (cid:107) v (cid:107) 2+} we have:
Revised: {+2 ε/ 3 +( ε/ 3) (cid:107) v (cid:107)for ε > 0 and 2 |(cid:104) u, v (cid:105)| ≤ (cid:107) u (cid:107) 
 + (cid:107) v (cid:107) 2+} we have:

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Note that from (23) we get that (cid:107)∇ z φ
Revised: Note that from (23) we get that (cid:107)∇ z φ

Original: i ( h ( w ( t ) ; i ))
Revised: i ( h ( w ( t ) ; i ))

Original: (cid:107) 2 ≤ 2 L φ [ φ i ( h ( w ( t ) ; i ))
Revised: (cid:107) 2 ≤ 2 L φ [ φ i ( h ( w ( t ) ; i ))

Original: − φ
Revised: − φ

Original: i
Revised: i

Original: ( h ∗ i )] .
Revised: ( h ∗ i )] .

Original: Applying this and using the fact that α ( t ) i ≤ αL φ , for some α ∈ (0 , 14 ) , we are able to derive:
Revised: Applying this and using the fact that α ( t ) i ≤ αL φ , for some α ∈ (0 , 14 ) , we are able to derive:

Original: [Equation]
Revised: [Equation]

Original: where ( a ) follows by using matrix vector inequality (cid:107) Hv (cid:107) ≤ (cid:107) H (cid:107)(cid:107) v (cid:107) , where H ∈ R c × d and v ∈ R d and Assumption 4 in (20) and η ( t )
Revised: where ( a ) follows by using matrix vector inequality (cid:107) Hv (cid:107) ≤ (cid:107) H (cid:107)(cid:107) v (cid:107) , where H ∈ R c × d and v ∈ R d and Assumption 4 in (20) and η ( t )

Original: =
Revised: =

Original: D √ ε for some D > 0 and ε > 0 ; ( b ) follows by the fact that (cid:107) v ( t ) − v ( t ) ∗ reg (cid:107) 2 ≤ ε  in (19) and Lemma 11.
Revised: D √ ε for some D > 0 and ε > 0 ; ( b ) follows by the fact that (cid:107) v ( t ) − v ( t ) ∗ reg (cid:107) 2 ≤ ε 2 in (19) and Lemma 12.

Original: Lemma 13.
Revised: Lemma 14.

Original: Let w ( t ) be generated by Algorithm 2 where v ( t ) satisﬁes (19).
Revised: Let w ( t ) be generated by Algorithm 2 where v ( t ) satisﬁes (19).

Original: We execute Algorithm 2 for T =
Revised: We execute Algorithm 2 for T =

Original: βε outer loops for some constant β > 0 .
Revised: βε outer loops for some constant β > 0 .

Original: We assume Assumption 1 holds.
Revised: We assume Assumption 1 holds.

Original: Suppose that Assumption 2 holds for G > 0 , Assumption 3 holds for V > 0 and Assumption 4 holds for
Revised: Suppose that Assumption 2 holds for G > 0 , Assumption 3 holds for V > 0 and Assumption 4 holds for

Original: H > 0 .We set the step size equal to η ( t ) =
Revised: H > 0 .We set the step size equal to η ( t ) =

Original: D √ ε for some D > 0 and choose a learning rate α ( t ) i
Revised: D √ ε for some D > 0 and choose a learning rate α ( t ) i

Original: = (1 + ε ) α
Revised: = (1 + ε ) α

Original: ( t − 1)
Revised: ( t − 1)

Original: i = 
Revised: i = (

Original: (1 + ε ) t α (0) i .
Revised: [Equation] α (0) i .

Original: Based on β , we deﬁne α (0) i 
Revised: Based on β , we deﬁne α (0) i =

Original: = α e β L φ with α ∈ (0 , 1 4 ) .
Revised: [Equation] e β L φ with α ∈ (0 , 14 ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Rearranging the terms in Lemma 
Revised: Rearranging the terms in Lemma 13, we have φ i ( [Equation] [Equation] ( t

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: i )) − φ [Equation] ( h

Original: [Equation]
Revised: [Equation]

Revised: i ) ≤

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: The last inequality follows because the learning rate satisﬁes α (0) i 
Revised: The last inequality follows because the learning rate satisﬁes α (0) i =

Original: [Equation]
Revised: [Equation]

Original:  and for t = 1 , . ..
Revised: β L φ ≤ αL φ and for t = 1 , . .

Original: , T =
Revised: , T =

Original: βε for some β >
Revised: βε for some β >

Original: [Equation]
Revised: [Equation]

Original: since (1 + x ) 1 /x ≤ e , x > 0 .
Revised: since (1 + x ) 1 /x ≤ e , x > 0 .

Original: Moreover, we have 1 α ( t ) i≤
Revised: Moreover, we have 1 α ( t ) i

Original: [Equation]
Revised: [Equation]

Original: α [Equation] = 
Revised: Taking the average sum from t = 0 , . . , T − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Taking the average sum from t = 0 , . .
Revised: Taking the average sum from i = 1 , . .

Original:  , T − 1 , we have 
Revised: . , n , we have T

Original: T
Revised: T

Original: T − 1 (cid:88) t =
Revised: [Equation]

Original: [ φ i ( h ( w ( t ) ; i )) − φ
Revised: [Equation] t =  n n [Equation]

Original: i 
Revised: i =

Original:  ( h ∗ i )] ≤
Revised: [ φ i ( h (

Original: [Equation]
Revised: [Equation]

Revised: ( t )

Original: [Equation]
Revised: [Equation]

Original: 2 ) 2 (cid:3) + e
Revised: i ))

Original: [Equation]
Revised: [Equation]

Revised: i ( h

Original: [Equation]
Revised: [Equation]

Revised: i )]

Original: [Equation]
Revised: [Equation]

Original: ε 
Revised: ε )

Original: [Equation]
Revised: [Equation]

Original: T −
Revised: n

Original: [Equation]
Revised: [Equation]

Original: t = (cid:107) H ( t ) i η ( t ) v ( t ) ∗ reg − α ( t ) i ∇ z φ i( h ( w ( t ); i )) (cid:107)
Revised: i )

Original: [Equation]  [Equation]
Revised: [Equation] + [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: i
Revised: n

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation] [Equation] [Equation] ( t )
Revised: =

Original: [Equation]
Revised: [Equation]

Original: η
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: t )

Original: [Equation]
Revised: [Equation]

Original: ( t )

Original: [Equation]
Revised: [Equation]

Original: ( t )

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation]
Revised: [Equation]

Original: t )

Original: [Equation]
Revised: [Equation]

Original: i ) [Equation] Taking the average sum from i =
Revised: n

Original: [Equation]
Revised: [Equation]

Original: . , n

Original: [Equation]
Revised: [Equation]

Original: we have

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: T [Equation] n n (cid:88) i = [ f ( w ( t ) ; i )
Revised: Therefore, applying (48) to (47)

Original: [Equation]
Revised: [Equation]

Original: φ i ( h ∗ i )] . Therefore, applying (47) to (46), we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: P ROOF OF T HEOREM
Revised: P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: From (41) we have F ∗ − 1 n (cid:80) ni
Revised: From (42) we have F ∗ − 1 n (cid:80) ni

Original: =

Revised: =

Original: φ i ( h ∗ i ) ≥ 0 .
Revised: φ i ( h ∗ i ) ≥ 0 .

Original: This leads to
Revised: This leads to

Original: [Equation] n [Equation] i
Revised: T

Original: [Equation]

Original: i

Original: [Equation] φ i [Equation] h ∗
Revised: T (cid:88) t =

Original: i

Original: [Equation]
Revised: [Equation]

Original:  t [Equation] n [Equation]
Revised: (cid:88) t = (cid:32)n n (cid:88)

Original: i

Original: [Equation]

Original: [ f (

Original: [Equation]

Original: ( t

Original: [Equation]

Original: i

Original: [Equation]
Revised: [Equation]

Original: h

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: P ROOF OF C OROLLARY
Revised: P ROOF OF C OROLLARY

Original: Proof.
Revised: Proof.

Original: For each iteration 0 ≤ t < T , we need to ﬁnd v ( t ) satisfying the following criteria:
Revised: For each iteration 0 ≤ t < T , we need to ﬁnd v ( t ) satisfying the following criteria:

Original: [Equation]
Revised: [Equation]

Original: for some ε > 0 .
Revised: for some ε > 0 .

Original: Using Gradient Descent we need O ( n Lµ log( 1 ε  )) number of gradient evaluations (Nesterov, 2004), where L and µ = ε 2 are the smooth and strongly convex constants, respectively, of Ψ .
Revised: Using Gradient Descent we need O ( n Lµ log( 1 ε 2 )) = O (2 n Lµ log( 1 ε )) number of gradient evaluations (Nesterov, 2004), where L and µ = ε 2 are the smooth and strongly convex constants, respectively, of Ψ .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Then, for any v ∈ R c
Revised: Then, for any v ∈ R c

Original: [Equation]
Revised: [Equation]

Revised: t ) ; i ))] , i ∈ [ n ] .

Original: Consider η ( t )
Revised: Consider η ( t )

Original: =
Revised: =

Original: D √ ε for some D > 0 and ε > 0 , we have for i ∈ [ n ] and 0 ≤ t < T
Revised: D √ ε for some D > 0 and ε > 0 , we have for i ∈ [ n ] and 0 ≤ t < T

Original: [Equation]
Revised: [Equation]

Original: Hence, (cid:107)∇2 v Φ
Revised: Hence, (cid:107)∇

Original: [Equation]
Revised: [Equation]

Original: v )
Revised: ψ i (

Original: [Equation]
Revised: [Equation]

Original: ≤ (cid:107)∇ 2 v ψ i ( v ) (cid:107) + ε 2 for any v ∈ R c which implies that L = D 2 εH 2 + ε(Nesterov (2004)) and Lµ = D 2 H 2ε + ε 2 ε= D 2 H 2 + ε ε .
Revised: (cid:107) + ε 2 for any v ∈ R c which implies that L = D 2 H 2 + ε 2 (Nesterov (2004)) and Lµ = D 2 H 2 + ε 2 ε.

Original: Therefore, the complexity to ﬁnd v ( t ) for each iteration t is O ( n D 2 H 2 + ε ε log( 1 ε )) .
Revised: Therefore, the complexity to ﬁnd v ( t ) for each iteration t is O (2 n D 2 H 2 + ε 2 εlog( 1 ε )) .

Original: Let us choose 0 < ε ≤ 1 .
Revised: Let us choose 0 < ε ≤ 1 .

Original: From (48), we have
Revised: From (49), we have

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: where

Original: N = 
Revised: [Equation] = [Equation] +

Revised: [Equation]

Revised: +

Original: [Equation]
Revised: [Equation]

Revised: + V

Revised: [Equation]

Original: Let ˆ ε =
Revised: Let ˆ ε =

Original: Nε with 0 < ˆ ε ≤ N .
Revised: Nε with 0 < ˆ ε ≤ N .

Original: Then, we need T =
Revised: Then, we need T =

Original: Nβ ˆ ε for some β > 0 to guarantee 
Revised: Nβ ˆ ε for some β > 0 to guarantee min

Original: [Equation]
Revised: [Equation]

Original: [Equation] F 
Revised: [ F ( w ( t ) )

Original: [Equation]
Revised: [Equation]

Original: {+F ∗ ] ≤ 1 T (cid:80) T − 1 t =
 [ F ( w ( t ) ) − F ∗+} ] ≤ ˆ ε .
Revised: {+F ∗ ] ≤ 1 T (cid:80) T − 1 t =
 [ F ( w ( t ) ) − F ∗+} ] ≤ ˆ ε .

Original: Hence, the total complexity is 
Revised: Hence, the total complexity is O

Original: [Equation]
Revised: [Equation]