Original: N EW P ERSPECTIVE ON THE G LOBAL C ONVERGENCE OF F INITE -S UM O PTIMIZATION
Revised: N EW P ERSPECTIVE ON THE G LOBAL C ONVERGENCE OF F INITE -S UM O PTIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks (DNNs) have shown great success in many machine learning tasks.
Revised: Deep neural networks (DNNs) have shown great success in many machine learning tasks.

Original: Their training is challenging since the loss surface of the network architecture is generally non-convex, or even non-smooth.
Revised: Their training is challenging since the loss surface of the network architecture is generally non-convex, or even non-smooth.

Original: How and under what assumptions is guaranteed convergence to a global minimum possible?
Revised: How and under what assumptions is guaranteed convergence to a global minimum possible?

Original: We propose a reformulation of the minimization problem allowing for a new recursive algorithmic framework.
Revised: We propose a reformulation of the minimization problem allowing for a new recursive algorithmic framework.

Original: By using bounded style assumptions, we prove convergence to an ε -(global) minimum using ˜ O
Revised: By using bounded style assumptions, we prove convergence to an ε -(global) minimum using ˜ O

Original: (1 /ε 3 ) gradient computations.
Revised: (1 /ε 3 ) gradient computations.

Original: Our theoretical foundation motivates further study, implementation, and optimization of the new algorithmic framework and further investigation of its non-standard bounded style assumptions.
Revised: Our theoretical foundation motivates further study, implementation, and optimization of the new algorithmic framework and further investigation of its non-standard bounded style assumptions.

Original: This new direction broadens our understanding of why and under what circumstances training of a DNN converges to a global minimum.
Revised: This new direction broadens our understanding of why and under what circumstances training of a DNN converges to a global minimum.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, deep neural networks (DNNs) have shown a great success in many machine learning tasks.
Revised: In recent years, deep neural networks (DNNs) have shown a great success in many machine learning tasks.

Original: However, training these neural networks is challenging since the loss surface of network architecture is generally non-convex, or even non-smooth.
Revised: However, training these neural networks is challenging since the loss surface of network architecture is generally non-convex, or even non-smooth.

Original: Thus, there have been a long-standing question on how optimization algorithms may converge to a global minimum.
Revised: Thus, there have been a long-standing question on how optimization algorithms may converge to a global minimum.

Original: Many previous work have investigated Gradient Descent algorithm and its stochastic version for over-parameterized setting (Arora et al., 2018; Soudry et al., 2018; Allen-Zhu et al., 2019; Du et al., 2019a; Zou & Gu, 2019).
Revised: Many previous work have investigated Gradient Descent algorithm and its stochastic version for over-parameterized setting (Arora et al., 2018; Soudry et al., 2018; Allen-Zhu et al., 2019; Du et al., 2019a; Zou & Gu, 2019).

Original: Although these works have shown promising convergence results under certain assumptions, there is still a lack of new efﬁcient methods that can guarantee global convergence for machine learning optimization.
Revised: Although these works have shown promising convergence results under certain assumptions, there is still a lack of new efﬁcient methods that can guarantee global convergence for machine learning optimization.

Original: In this paper, we address this problem using a different perspective.
Revised: In this paper, we address this problem using a different perspective.

Original: Instead of analyzing the traditional ﬁnite-sum formulation, we adopt a new composite formulation that exactly depicts the structure of machine learning where a data set is used to learn a common classiﬁer.
Revised: Instead of analyzing the traditional ﬁnite-sum formulation, we adopt a new composite formulation that exactly depicts the structure of machine learning where a data set is used to learn a common classiﬁer.

Original: Representation.
Revised: Representation.

Original: Let (cid:8) ( x ( i ) , y ( i ) )
Revised: Let (cid:8) ( x ( i ) , y ( i ) )

Original: (cid:9) n i
Revised: (cid:9) n i

Original: =1 be a given training set with x ( i ) ∈ R m , y ( i
Revised: =1 be a given training set with x ( i ) ∈ R m , y ( i

Original: ) ∈ R c , we investigate the following novel representation for deep learning tasks:
Revised: ) ∈ R c , we investigate the following novel representation for deep learning tasks:

Original: [Equation]
Revised: [Equation]

Original: where h ( · ; i ) : R d → R c , i ∈ [ n ] = { 1 , . . . , n } , is the classiﬁer for each input data x ( i ) ; and φ i : R c
Revised: where h ( · ; i ) : R d → R c , i ∈ [ n ] = { 1 , . . . , n } , is the classiﬁer for each input data x ( i ) ; and φ i : R c

Original: → R , i ∈ [ n ] , is the loss function corresponding to each output data
Revised: → R , i ∈ [ n ] , is the loss function corresponding to each output data

Original: y
Revised: y

Original: ( i ) .
Revised: ( i ) .

Original: Our composite formulation (1) is a special case of the ﬁnite-sum problem min w ∈ R d (cid:8) F ( w ) = 1 n (cid:80) ni
Revised: Our composite formulation (1) is a special case of the ﬁnite-sum problem min w ∈ R d (cid:8) F ( w ) = 1 n (cid:80) ni

Original: =1 f ( w ; i ) (cid:9) where each individual function f ( · ; i ) is a composition of the loss function φ i and the classiﬁer h ( · ; i ) .
Revised: =1 f ( w ; i ) (cid:9) where each individual function f ( · ; i ) is a composition of the loss function φ i and the classiﬁer h ( · ; i ) .

Original: This problem covers various important applications in machine learning, including logistic regression and neural networks.
Revised: This problem covers various important applications in machine learning, including logistic regression and neural networks.

Original: The most common approach for the ﬁnite-sum problem is using ﬁrst-order methods such as (stochastic) gradient algorithms and making assumptions on the component functions f ( · ; i ) .
Revised: The most common approach for the ﬁnite-sum problem is using ﬁrst-order methods such as (stochastic) gradient algorithms and making assumptions on the component functions f ( · ; i ) .

Original: As an alternative, we further investigate the structure of the loss function φ i and narrow our assumption on the classiﬁer h ( · ; i ) .
Revised: As an alternative, we further investigate the structure of the loss function φ i and narrow our assumption on the classiﬁer h ( · ; i ) .

Original: For the purpose of this work, we ﬁrst consider convex and
Revised: For the purpose of this work, we ﬁrst consider convex and

Original: Lipschitz-smooth loss functions while the classiﬁers can be non-convex.
Revised: Lipschitz-smooth loss functions while the classiﬁers can be non-convex.

Original: Using this representation, we propose a new framework followed by two algorithms that guarantee global convergence for the minimization problem.
Revised: Using this representation, we propose a new framework followed by two algorithms that guarantee global convergence for the minimization problem.

Original: Algorithmic Framework.
Revised: Algorithmic Framework.

Original: Representation (1) admits a new perspective.
Revised: Representation (1) admits a new perspective.

Original: Our key insight is to (A) deﬁne z ( t )
Revised: Our key insight is to (A) deﬁne z ( t )

Original: i =
Revised: i =

Original: h ( w ( t ) ; i ) , where t is an iteration count of the outer loop in our algorithmic framework.
Revised: h ( w ( t ) ; i ) , where t is an iteration count of the outer loop in our algorithmic framework.

Original: Next (B), we want to approximate the change z
Revised: Next (B), we want to approximate the change z

Original: ( t +1)
Revised: ( t +1)

Original: i − z
Revised: i − z

Original: ( t ) i in terms of a step size times the gradient
Revised: ( t ) i in terms of a step size times the gradient

Original: [Equation]
Revised: [Equation]

Original: and (C) we approximate the change h ( w ( t +1) ; i ) − h ( w ( t ) ; i ) in terms of the ﬁrst order derivative
Revised: and (C) we approximate the change h ( w ( t +1) ; i ) − h ( w ( t ) ; i ) in terms of the ﬁrst order derivative

Original: [Equation]
Revised: [Equation]

Original: Finally, we combine (A), (B), and (C) to equate the approximations of z ( t +1)
Revised: Finally, we combine (A), (B), and (C) to equate the approximations of z ( t +1)

Original: i − z
Revised: i − z

Original: ( t ) i and h ( w ( t +1) ; i ) − h ( w ( t ) ; i ) .
Revised: ( t ) i and h ( w ( t +1) ; i ) − h ( w ( t ) ; i ) .

Original: This leads to a recurrence on w ( t ) of the form w ( t +1) =
Revised: This leads to a recurrence on w ( t ) of the form w ( t +1) =

Original: w ( t ) − η ( t ) v ( t ) , where η ( t ) is a step size and which involves computing v ( t ) by solving a convex quadratic subproblem, see the details in Section 4.
Revised: w ( t ) − η ( t ) v ( t ) , where η ( t ) is a step size and which involves computing v ( t ) by solving a convex quadratic subproblem, see the details in Section 4.

Original: We explain two methods for approximating a solution for the derived subproblem.
Revised: We explain two methods for approximating a solution for the derived subproblem.

Original: We show how to approximate the subproblem by transforming it into a strongly convex problem by adding a regularizer which can be solved in closed form.
Revised: We show how to approximate the subproblem by transforming it into a strongly convex problem by adding a regularizer which can be solved in closed form.

Original: And we show how to use Gradient Descent (GD) on the subproblem to ﬁnd an approximation v ( t ) of its solution.
Revised: And we show how to use Gradient Descent (GD) on the subproblem to ﬁnd an approximation v ( t ) of its solution.

Original: Convergence Analysis.
Revised: Convergence Analysis.

Original: Our analysis introduces non-standard bounded style assumptions.
Revised: Our analysis introduces non-standard bounded style assumptions.

Original: Intuitively, we assume that our convex and quadratic subproblem has a bounded solution.
Revised: Intuitively, we assume that our convex and quadratic subproblem has a bounded solution.

Original: This allows us to prove a total complexity of ˜ O
Revised: This allows us to prove a total complexity of ˜ O

Original: ( 1 ε 3 ) to ﬁnd an ε -(global) solution that satisﬁes F ( ˆ w ) − F ∗ ≤ ε , where F ∗ is the global minimizer of F .
Revised: ( 1 ε 3 ) to ﬁnd an ε -(global) solution that satisﬁes F ( ˆ w ) − F ∗ ≤ ε , where F ∗ is the global minimizer of F .

Original: Our analysis applies to a wide range of applications in machine learning: Our results hold for
Revised: Our analysis applies to a wide range of applications in machine learning: Our results hold for

Original: squared loss and softmax cross-entropy loss and applicable for a range of activation functions in DNN as we only assume that the h ( · ; i ) are twice continuously differentiable and their Hessian matrices (second order derivatives) as well as their gradients (ﬁrst order derivatives) are bounded.
Revised: squared loss and softmax cross-entropy loss and applicable for a range of activation functions in DNN as we only assume that the h ( · ; i ) are twice continuously differentiable and their Hessian matrices (second order derivatives) as well as their gradients (ﬁrst order derivatives) are bounded.

Original: Contributions and Outline.
Revised: Contributions and Outline.

Original: Our contributions in this paper can be summarized as follows
Revised: Our contributions in this paper can be summarized as follows

Original: • We propose a new representation (1) for analyzing the machine learning minimization problem.
Revised: • We propose a new representation (1) for analyzing the machine learning minimization problem.

Original: Our formulation utilizes the structure of machine learning tasks where a training data set of inputs and outputs is used to learn a common classiﬁer.
Revised: Our formulation utilizes the structure of machine learning tasks where a training data set of inputs and outputs is used to learn a common classiﬁer.

Original: Related work in Sectionshows how (1) is different from the classical ﬁnite-sum problem.
Revised: Related work in Sectionshows how (1) is different from the classical ﬁnite-sum problem.

Original: • Based on the new representation we propose a novel algorithm framework.
Revised: • Based on the new representation we propose a novel algorithm framework.

Original: The algorithmic framework approximates a solution to a subproblem for which we show two distinct approaches.
Revised: The algorithmic framework approximates a solution to a subproblem for which we show two distinct approaches.

Original: • For general DNNs and based on bounded style assumptions, we prove a total complexity of ˜ O
Revised: • For general DNNs and based on bounded style assumptions, we prove a total complexity of ˜ O

Original: ( 1 ε 3 ) to ﬁnd an ε -(global) solution that satisﬁes F ( ˆ w ) − F ∗ ≤ ε , where F ∗ is the global minimizer of F .
Revised: ( 1 ε 3 ) to ﬁnd an ε -(global) solution that satisﬁes F ( ˆ w ) − F ∗ ≤ ε , where F ∗ is the global minimizer of F .

Original: We emphasize that our focus is on developing a new theoretical foundation and that a translation to a practical implementation with empirical results is for future work.
Revised: We emphasize that our focus is on developing a new theoretical foundation and that a translation to a practical implementation with empirical results is for future work.

Original: Our theoretical foundation motivates further study, implementation, and optimization of the new algorithmic framework and further investigation of its non-standard bounded style assumptions.
Revised: Our theoretical foundation motivates further study, implementation, and optimization of the new algorithmic framework and further investigation of its non-standard bounded style assumptions.

Original: This new direction broadens our understanding of why and under what circumstances training of a DNN converges to a global minimum.
Revised: This new direction broadens our understanding of why and under what circumstances training of a DNN converges to a global minimum.

Original: The rest of this paper is organized as follows.
Revised: The rest of this paper is organized as follows.

Original: Section 2 discusses related work.
Revised: Section 2 discusses related work.

Original: Section 3 describes our setting and deep learning representation.
Revised: Section 3 describes our setting and deep learning representation.

Original: Section 4 explains our key insight and derives our
Revised: Section 4 explains our key insight and derives our

Original: Framework 1.
Revised: Framework 1.

Original: Section 5 presents our algorithms and their global convergence.
Revised: Section 5 presents our algorithms and their global convergence.

Original: All technical proofs are deferred to the Appendix.
Revised: All technical proofs are deferred to the Appendix.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Formulation for Machine Learning Problems.
Revised: Formulation for Machine Learning Problems.

Original: The ﬁnite-sum problem is one of the most important and fundamental problems in machine learning.
Revised: The ﬁnite-sum problem is one of the most important and fundamental problems in machine learning.

Original: Analyzing this model is the most popular approach in the machine learning literature and it has been studied intensively throughout the years (Bottou et al., 2018; Reddi et al., 2016; Duchi et al., 2011b).
Revised: Analyzing this model is the most popular approach in the machine learning literature and it has been studied intensively throughout the years (Bottou et al., 2018; Reddi et al., 2016; Duchi et al., 2011b).

Original: Our new formulation (1) is a special case of the ﬁnite-sum problem, however, it is much more complicated than the previous model since it involves the data index
Revised: Our new formulation (1) is a special case of the ﬁnite-sum problem, however, it is much more complicated than the previous model since it involves the data index

Original: i both inside the classiﬁers h ( · ; i ) and the loss functions φ i .
Revised: i both inside the classiﬁers h ( · ; i ) and the loss functions φ i .

Original: For a comparison, previous works only consider a common loss function l
Revised: For a comparison, previous works only consider a common loss function l

Original: (ˆ y, y ) for the predicted value ˆ y and output data y
Revised: (ˆ y, y ) for the predicted value ˆ y and output data y

Original: (Zou et al., 2018;
Revised: (Zou et al., 2018;

Original: Soudry et al., 2018).
Revised: Soudry et al., 2018).

Original: Our modiﬁed version of loss function φ i is a natural setting for machine learning.
Revised: Our modiﬁed version of loss function φ i is a natural setting for machine learning.

Original: We note that when h ( w ; i ) is the output produced by a model, our goal is to match this output with the corresponding target y ( i ) .
Revised: We note that when h ( w ; i ) is the output produced by a model, our goal is to match this output with the corresponding target y ( i ) .

Original: For that reason, the loss function for each output has a dependence on the output data y
Revised: For that reason, the loss function for each output has a dependence on the output data y

Original: ( i ) ,
Revised: ( i ) ,

Original: and is denoted by φ i .
Revised: and is denoted by φ i .

Original: This fact reﬂects the natural setting of machine learning where the outputs are designed to ﬁt different targets, and the optimization process depends on both outer function φ i and inner functions h ( · ; i ) .
Revised: This fact reﬂects the natural setting of machine learning where the outputs are designed to ﬁt different targets, and the optimization process depends on both outer function φ i and inner functions h ( · ; i ) .

Original: This complication may potentially bring a challenge to theoretical analysis.
Revised: This complication may potentially bring a challenge to theoretical analysis.

Original: However, with separate loss functions, we believe this model will help to exploit better the structure of machine learning problems and gain more insights on the neural network architecture.
Revised: However, with separate loss functions, we believe this model will help to exploit better the structure of machine learning problems and gain more insights on the neural network architecture.

Original: Other related composite optimization models are also investigated thoroughly in (Lewis & Wright, 2016; Zhang & Xiao, 2019; Tran-Dinh et al., 2020).
Revised: Other related composite optimization models are also investigated thoroughly in (Lewis & Wright, 2016; Zhang & Xiao, 2019; Tran-Dinh et al., 2020).

Original: Our model is different from these works as it does not have a common function wrapping outside the ﬁnite-sum term, as in (Lewis & Wright, 2016).
Revised: Our model is different from these works as it does not have a common function wrapping outside the ﬁnite-sum term, as in (Lewis & Wright, 2016).

Original: Note that a broad class of variance reduction algorithms (e.g. SAG (Le Roux et al., 2012),
Revised: Note that a broad class of variance reduction algorithms (e.g. SAG (Le Roux et al., 2012),

Original: SAGA (Defazio et al., 2014), SVRG (Johnson & Zhang, 2013), SARAH (Nguyen et al., 2017)) is designed specifically for the ﬁnite-sum formulation and is known to have certain beneﬁts over Gradient Descent.
Revised: SAGA (Defazio et al., 2014), SVRG (Johnson & Zhang, 2013), SARAH (Nguyen et al., 2017)) is designed speciﬁcally for the ﬁnite-sum formulation and is known to have certain beneﬁts over Gradient Descent.

Original: In addition, the multilevel composite problem considered in (Zhang & Xiao, 2021) also covers empirical risk minimization problem.
Revised: In addition, the multilevel composite problem considered in (Zhang & Xiao, 2021) also covers empirical risk minimization problem.

Original: However our formulation does not match their work since our inner function h ( w ; i ) is not an independent expectation over some data distribution, but a speciﬁc function that depends on the current data.
Revised: However our formulation does not match their work since our inner function h ( w ; i ) is not an independent expectation over some data distribution, but a speciﬁc function that depends on the current data.

Original: Global Convergence for Neural Networks.
Revised: Global Convergence for Neural Networks.

Original: A recent popular line of research is studying the dynamics of optimization methods on some speciﬁc neural network architectures.
Revised: A recent popular line of research is studying the dynamics of optimization methods on some speciﬁc neural network architectures.

Original: There are some early works that show the global convergence of Gradient Descent (GD) for simple linear network and two-layer network (Brutzkus et al., 2018; Soudry et al., 2018; Arora et al., 2019; Du et al., 2019b).
Revised: There are some early works that show the global convergence of Gradient Descent (GD) for simple linear network and two-layer network (Brutzkus et al., 2018; Soudry et al., 2018; Arora et al., 2019; Du et al., 2019b).

Original: Some further works extend these results to deep learning architectures (Allen-Zhu et al., 2019; Du et al., 2019a;
Revised: Some further works extend these results to deep learning architectures (Allen-Zhu et al., 2019; Du et al., 2019a;

Original: Zou & Gu, 2019).
Revised: Zou & Gu, 2019).

Original: These theoretical guarantees are generally proved for the case when the last output layer is ﬁxed, which is not standard in practice.
Revised: These theoretical guarantees are generally proved for the case when the last output layer is ﬁxed, which is not standard in practice.

Original: A recent work (Nguyen & Mondelli, 2020) prove the global convergence for GD when all layers are trained with some initial conditions.
Revised: A recent work (Nguyen & Mondelli, 2020) prove the global convergence for GD when all layers are trained with some initial conditions.

Original: However, these results are for neural networks without bias neurons and it is unclear how these analyses can be extended to handle the bias terms of deep networks with different activations.
Revised: However, these results are for neural networks without bias neurons and it is unclear how these analyses can be extended to handle the bias terms of deep networks with different activations.

Original: Our novel framework and algorithms do not exclude learning bias layers as in (Nguyen & Mondelli, 2020).
Revised: Our novel framework and algorithms do not exclude learning bias layers as in (Nguyen & Mondelli, 2020).

Original: Using a different algorithm, Brutzkus et al.
Revised: Using a different algorithm, Brutzkus et al.

Original: (2018) investigate Stochastic Gradient Descent (SGD) for two-layer networks in a restricted linearly separable data setting.
Revised: (2018) investigate Stochastic Gradient Descent (SGD) for two-layer networks in a restricted linearly separable data setting.

Original: This line of research continues with the works from Allen-Zhu et al.
Revised: This line of research continues with the works from Allen-Zhu et al.

Original: ; Zou et al.
Revised: ; Zou et al.

Original: (2018) and later with Zou & Gu (2019).
Revised: (2018) and later with Zou & Gu (2019).

Original: They justify the global convergence of SGD for deep neural networks for some probability depending on the number of input data and the initialization process.
Revised: They justify the global convergence of SGD for deep neural networks for some probability depending on the number of input data and the initialization process.

Original: Over-Paramaterized Settings and other Assumptions for Machine Learning.
Revised: Over-Paramaterized Settings and other Assumptions for Machine Learning.

Original: Most of the modern learning architectures are over-parameterized, which means that the number of parameters are very large and often far more than the number of input data.
Revised: Most of the modern learning architectures are over-parameterized, which means that the number of parameters are very large and often far more than the number of input data.

Original: Some recent works prove the global convergence of Gradient Descent when the number of neurons are extensively large, e.g. (Zou &
Revised: Some recent works prove the global convergence of Gradient Descent when the number of neurons are extensively large, e.g. (Zou &

Original: Gu, 2019) requires Ω( n 8 ) neurons for every hidden layer, and (Nguyen & Mondelli, 2020) improves this number to Ω( n 3 ) .
Revised: Gu, 2019) requires Ω( n 8 ) neurons for every hidden layer, and (Nguyen & Mondelli, 2020) improves this number to Ω( n 3 ) .

Original: If the initial point satisﬁes some special conditions, then they can show a better dependence of Ω( n ) .
Revised: If the initial point satisﬁes some special conditions, then they can show a better dependence of Ω( n ) .

Original: In Allen-Zhu et al.
Revised: In Allen-Zhu et al.

Original: (2019), the authors initialize the weights using a random Gaussian distribution where the variance depends on the dimension of the problem.
Revised: (2019), the authors initialize the weights using a random Gaussian distribution where the variance depends on the dimension of the problem.

Original: In non-convex setting, they prove the convergence of SGD using the assumption that the dimension depends inversely on the tolerance (cid:15) .
Revised: In non-convex setting, they prove the convergence of SGD using the assumption that the dimension depends inversely on the tolerance (cid:15) .

Original: We will discuss how these over-paramaterized settings might be a necessary condition to develop our theory.
Revised: We will discuss how these over-paramaterized settings might be a necessary condition to develop our theory.

Original: Other standard assumptions for machine learning include the bounded gradient assumption (Nemirovski et al., 2009; Shalev-Shwartz et al., 2007; Reddi et al., 2016; Tran et al., 2021).
Revised: Other standard assumptions for machine learning include the bounded gradient assumption (Nemirovski et al., 2009; Shalev-Shwartz et al., 2007; Reddi et al., 2016; Tran et al., 2021).

Original: It is also common to assume all the iterations of an algorithm stays in a bounded domain (Duchi et al., 2011a;
Revised: It is also common to assume all the iterations of an algorithm stays in a bounded domain (Duchi et al., 2011a;

Original: Levy et al., 2018; G¨urb¨uzbalaban et al., 2019; Reddi et al., 2018; Vaswani et al., 2021).
Revised: Levy et al., 2018; G¨urb¨uzbalaban et al., 2019; Reddi et al., 2018; Vaswani et al., 2021).

Original: Since we are analyzing a new composite formulation , it is understandable that our assumptions may also not be standard.
Revised: Since we are analyzing a new composite formulation , it is understandable that our assumptions may also not be standard.

Original: However, we believe that there is a strong connection between our assumptions and the traditional setting of machine learning.
Revised: However, we believe that there is a strong connection between our assumptions and the traditional setting of machine learning.

Original: We will discuss this point more clearly in Section 4.
Revised: We will discuss this point more clearly in Section 4.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: In this section, we discuss our formulation and notations in detail.
Revised: In this section, we discuss our formulation and notations in detail.

Original: Although this paper focuses on deep neural networks, our framework and theoretical analysis are general and applicable for other learning architectures.
Revised: Although this paper focuses on deep neural networks, our framework and theoretical analysis are general and applicable for other learning architectures.

Original: Deep Learning Representation.
Revised: Deep Learning Representation.

Original: Let { ( x ( i ) , y ( i ) ) }
Revised: Let { ( x ( i ) , y ( i ) ) }

Original: ni
Revised: ni

Original: =1 be a training data set where x ( i ) ∈
Revised: =1 be a training data set where x ( i ) ∈

Original: R m is a training input and y ( i
Revised: R m is a training input and y ( i

Original: ) ∈ R c is a training output.
Revised: ) ∈ R c is a training output.

Original: We consider a fully-connected neural network with
Revised: We consider a fully-connected neural network with

Original: L layers, where the l -th layer, l ∈ { 0 , 1 , . . .
Revised: L layers, where the l -th layer, l ∈ { 0 , 1 , . . .

Original: , L } , has n l neurons.
Revised: , L } , has n l neurons.

Original: We represent layer 0 -th and L -th layer as input and output layers, respectively, that is, n
Revised: We represent layer 0 -th and L -th layer as input and output layers, respectively, that is, n

Original: 0 = d and n L = c .
Revised: 0 = d and n L = c .

Original: For l ∈ { 1 , . . .
Revised: For l ∈ { 1 , . . .

Original: , L } , let
Revised: , L } , let

Original: W ( l ) ∈ R n l − 1 × n l and b ( l ) ∈ R n l , where { ( W ( l ) , b ( l ) )
Revised: W ( l ) ∈ R n l − 1 × n l and b ( l ) ∈ R n l , where { ( W ( l ) , b ( l ) )

Original: Ll =1 } represent the parameters of the neural network.
Revised: Ll =1 } represent the parameters of the neural network.

Original: A classiﬁer h ( w ; i ) is formulated as
Revised: A classiﬁer h ( w ; i ) is formulated as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: , where w = vec ( { W (1) , b (1) , . .
Revised: , where w = vec ( { W (1) , b (1) , . .

Original: , W ( L ) , b ( L ) } )
Revised: , W ( L ) , b ( L ) } )

Original: ∈ R
Revised: ∈ R

Original: d is the vectorized weight and { σ l } L − 1 l =1 are some activation functions.
Revised: d is the vectorized weight and { σ l } L − 1 l =1 are some activation functions.

Original: The most common choices for machine learning are ReLU, sigmoid, hyperbolic tangent and softplus.
Revised: The most common choices for machine learning are ReLU, sigmoid, hyperbolic tangent and softplus.

Original: For j ∈ [ c ] , h j ( · ; i ) : R d → R denotes the component function of the output h ( · ; i ) , for each data i ∈ [ n ] respectively.
Revised: For j ∈ [ c ] , h j ( · ; i ) : R d → R denotes the component function of the output h ( · ; i ) , for each data i ∈ [ n ] respectively.

Original: Moreover, we deﬁne h ∗ i =
Revised: Moreover, we deﬁne h ∗ i =

Original: arg min z ∈ R c φ
Revised: arg min z ∈ R c φ

Original: i ( z ) , i ∈ [ n ] .
Revised: i ( z ) , i ∈ [ n ] .

Original: Loss Functions.
Revised: Loss Functions.

Original: The well-known loss functions in neural networks for solving classiﬁcation and regression problems are softmax cross-entropy loss and square loss , respectively: (Softmax) Cross-Entropy Loss: F ( w ) =
Revised: The well-known loss functions in neural networks for solving classiﬁcation and regression problems are softmax cross-entropy loss and square loss , respectively: (Softmax) Cross-Entropy Loss: F ( w ) =

Original: 1 n (cid:80) ni
Revised: 1 n (cid:80) ni

Original: =1 f
Revised: =1 f

Original: [Equation]
Revised: [Equation]

Original: w ;
Revised: w ;

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Squared Loss : F ( w ) = 1 n (cid:80) ni =1 f ( w ; i ) with
Revised: Squared Loss : F ( w ) = 1 n (cid:80) ni =1 f ( w ; i ) with

Original: [Equation]
Revised: [Equation]

Original: We provide some basic deﬁnitions in optimization theory to support our theory.
Revised: We provide some basic deﬁnitions in optimization theory to support our theory.

Original: Deﬁnition 1 ( L -smooth) .
Revised: Deﬁnition 1 ( L -smooth) .

Original: Function φ : R c → R is L φ -smooth if there exists a constant L φ >such that, ∀ x 1 , x 2 ∈ R c ,
Revised: Function φ : R c → R is L φ -smooth if there exists a constant L φ >such that, ∀ x 1 , x 2 ∈ R c ,

Original: [Equation]
Revised: [Equation]

Original: Deﬁnition 2 (Convex) .
Revised: Deﬁnition 2 (Convex) .

Original: Function φ : R c → R is convex if ∀ x 1 , x 2 ∈ R c ,
Revised: Function φ : R c → R is convex if ∀ x 1 , x 2 ∈ R c ,

Original: [Equation]
Revised: [Equation]

Original: The following corollary shows the properties of softmax cross-entropy loss (2) and squared loss (3).
Revised: The following corollary shows the properties of softmax cross-entropy loss (2) and squared loss (3).

Original: Corollary 1.
Revised: Corollary 1.

Original: For softmax cross-entropy loss (2) and squared loss (3), there exist functions h ( · ; i ) :
Revised: For softmax cross-entropy loss (2) and squared loss (3), there exist functions h ( · ; i ) :

Original: R d → R c and φ i : R c
Revised: R d → R c and φ i : R c

Original: → R such that, for i ∈ [ n ] , φ i ( z ) is convex and L φ -smooth with L φ = 1 , and
Revised: → R such that, for i ∈ [ n ] , φ i ( z ) is convex and L φ -smooth with L φ = 1 , and

Original: [Equation]
Revised: [Equation]

Original:  N EW A LGORITHM F RAMEWORK 4. 
Revised:  N EW A LGORITHM F RAMEWORK 4.

Original: [Equation]
Revised: [Equation]

Original: NSIGHT
Revised: NSIGHT

Original: We assume f ( w ; i ) =
Revised: We assume f ( w ; i ) =

Original: φ i ( h ( w ; i )) with φ i convex and L φ -smooth.
Revised: φ i ( h ( w ; i )) with φ i convex and L φ -smooth.

Original: Our goal is to utilize the convexity of the outer function φ i .
Revised: Our goal is to utilize the convexity of the outer function φ i .

Original: In order to simplify notation, we write ∇ z φ i ( h ( w ( t ) ; i )) instead of ∇ z φ i ( z ) (cid:12)(cid:12)
Revised: In order to simplify notation, we write ∇ z φ i ( h ( w ( t ) ; i )) instead of ∇ z φ i ( z ) (cid:12)(cid:12)

Original: z = h ( w ( t ) ; i ) and denote z ( t ) i
Revised: z = h ( w ( t ) ; i ) and denote z ( t ) i

Original: = h ( w ( t ) ; i ) .
Revised: = h ( w ( t ) ; i ) .

Original: Starting from the current weight w ( t ) , we would like to ﬁnd the next point w ( t +1) that satisﬁes the following approximation for all i ∈ [ n ] :
Revised: Starting from the current weight w ( t ) , we would like to ﬁnd the next point w ( t +1) that satisﬁes the following approximation for all i ∈ [ n ] :

Original: [Equation]
Revised: [Equation]

Original: We can see that this approximation is a “noisy” version of a gradient descent update for every function φ i , simultaneously for all i ∈ [ n ] .
Revised: We can see that this approximation is a “noisy” version of a gradient descent update for every function φ i , simultaneously for all i ∈ [ n ] .

Original: In order to do this, we use the following update
Revised: In order to do this, we use the following update

Original: [Equation]
Revised: [Equation]

Original: where η ( t ) > 0 is a learning rate and v ( t ) is a search direction that helps us approximate equation (7).
Revised: where η ( t ) > 0 is a learning rate and v ( t ) is a search direction that helps us approximate equation (7).

Original: If the update term η ( t ) v ( t ) is small enough, and if h ( · ; i ) has some nice smooth properties, then from basic calculus we have the following approximation:
Revised: If the update term η ( t ) v ( t ) is small enough, and if h ( · ; i ) has some nice smooth properties, then from basic calculus we have the following approximation:

Original: [Equation]
Revised: [Equation]

Original: where H ( t ) i is a matrix in R c × d with ﬁrst-order derivatives.
Revised: where H ( t ) i is a matrix in R c × d with ﬁrst-order derivatives.

Original: Motivated by approximations (7) and (9), we consider the following optimization problem:
Revised: Motivated by approximations (7) and (9), we consider the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Hence, by solving for the solution v ( t ) ∗ of problem (10) we are able to ﬁnd a search direction for the key approximation (7).
Revised: Hence, by solving for the solution v ( t ) ∗ of problem (10) we are able to ﬁnd a search direction for the key approximation (7).

Original: This yields our new algorithmic Framework 1, see below.
Revised: This yields our new algorithmic Framework 1, see below.

Original: Framework 1 New Algorithm Framework
Revised: Framework 1 New Algorithm Framework

Original: Initialization: Choose an initial point w (0) ∈
Revised: Initialization: Choose an initial point w (0) ∈

Original: R
Revised: R

Original: d ; for t = 0 , 1 , · · · , T − 1 do
Revised: d ; for t = 0 , 1 , · · · , T − 1 do

Original: Solve for an approximation v ( t ) of the solution v ( t ) ∗ of the problem in (10)
Revised: Solve for an approximation v ( t ) of the solution v ( t ) ∗ of the problem in (10)

Original: [Equation]
Revised: [Equation]

Original:  ( t ) [Equation] w
Revised: α ( t ) i ∇

Original: [Equation]
Revised: [Equation]

Original:  t [Equation] i
Revised: i ( h ( w ( t ) ; i

Original: [Equation]
Revised: [Equation]

Revised: ) (cid:107)

Original: Update w ( t +1) =
Revised: Update w ( t +1) =

Original: w ( t ) − η ( t ) v ( t )
Revised: w ( t ) − η ( t ) v ( t )

Original: 4. T ECHNICAL A SSUMPTIONS
Revised: 4. T ECHNICAL A SSUMPTIONS

Original: Assumption 1.
Revised: Assumption 1.

Original: The loss function φ i is convex and L φ -smooth for i ∈ [ n ] .
Revised: The loss function φ i is convex and L φ -smooth for i ∈ [ n ] .

Original: Moreover, we assume that it is lower bounded, i.e. inf z ∈ R c
Revised: Moreover, we assume that it is lower bounded, i.e. inf z ∈ R c

Original: φ i ( z ) >
Revised: φ i ( z ) >

Original: −∞ for i ∈ [ n ] .
Revised: −∞ for i ∈ [ n ] .

Original: We have shown the convexity and smoothness of squared loss and softmax cross-entropy loss in
Revised: We have shown the convexity and smoothness of squared loss and softmax cross-entropy loss in

Original: Section 3.
Revised: Section 3.

Original: The bounded property of φ i is required in any algorithm for the well-deﬁnedness of (1).
Revised: The bounded property of φ i is required in any algorithm for the well-deﬁnedness of (1).

Original: Now, in order to use the Taylor series approximation, we need the following assumption on the neural network architecture h :
Revised: Now, in order to use the Taylor series approximation, we need the following assumption on the neural network architecture h :

Original: Assumption 2.
Revised: Assumption 2.

Original: We assume that h ( · ; i ) is twice continuously differentiable for all i ∈ [ n ] (i.e. the second-order partial derivatives of all scalars h j ( · ; i ) are continuous for all j ∈ [ c ] and i ∈ [ n ] ), and that their Hessian matrices are bounded, that is, there exists a G > 0 such that for all w ∈ R d , i ∈
Revised: We assume that h ( · ; i ) is twice continuously differentiable for all i ∈ [ n ] (i.e. the second-order partial derivatives of all scalars h j ( · ; i ) are continuous for all j ∈ [ c ] and i ∈ [ n ] ), and that their Hessian matrices are bounded, that is, there exists a G > 0 such that for all w ∈ R d , i ∈

Original: [ n ] and j ∈ [ c ] ,
Revised: [ n ] and j ∈ [ c ] ,

Original: [Equation]
Revised: [Equation]

Original:  where J w denotes the Jacobian 1 .
Revised: 11) where J w denotes the Jacobian 1 .

Original: Remark 1 (Relation to second-order methods) .
Revised: Remark 1 (Relation to second-order methods) .

Original: Although our analysis requires an assumption on the Hessian matrices of h ( w ; i ) , our algorithms do not use any second order information or try to approximate this information.
Revised: Although our analysis requires an assumption on the Hessian matrices of h ( w ; i ) , our algorithms do not use any second order information or try to approximate this information.

Original: Our theoretical analysis focused on the approximation of the classiﬁer and the gradient information, therefore is not related to the second order type algorithms.
Revised: Our theoretical analysis focused on the approximation of the classiﬁer and the gradient information, therefore is not related to the second order type algorithms.

Original: It is currently unclear how to apply second order methods into our problem, however, this is an interesting research question to expand the scope of this work.
Revised: It is currently unclear how to apply second order methods into our problem, however, this is an interesting research question to expand the scope of this work.

Original: Assumption 2 allows us to apply a Taylor approximation of each function h j ( · ; i ) with which we prove the following Lemma that bounds the error in equation (9):
Revised: Assumption 2 allows us to apply a Taylor approximation of each function h j ( · ; i ) with which we prove the following Lemma that bounds the error in equation (9):

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose that Assumption 2 holds for the classiﬁer h .
Revised: Suppose that Assumption 2 holds for the classiﬁer h .

Original: Then for all i ∈ [ n ] and 0 ≤ t < T ,
Revised: Then for all i ∈ [ n ] and 0 ≤ t < T ,

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: is deﬁned as the Jacobian matrix of h ( w ; i ) at w ( t ) and entries (cid:15) ( t ) i,j , j ∈ [ c ] , of vector (cid:15) ( t ) i satisfy
Revised: is deﬁned as the Jacobian matrix of h ( w ; i ) at w ( t ) and entries (cid:15) ( t ) i,j , j ∈ [ c ] , of vector (cid:15) ( t ) i satisfy

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In order to approximate (7) combined with (9), that is, to make sure the right hand sides of (7) and (9) are close to one another, we consider the optimization problem (10):
Revised: In order to approximate (7) combined with (9), that is, to make sure the right hand sides of (7) and (9) are close to one another, we consider the optimization problem (10):

Original: [Equation]
Revised: [Equation]

Original: The optimal value of problem (10) is equal to 0 if there exists a vector v ( t ) ∗ satisfying η ( t ) H ( t ) i v ( t ) ∗ =
Revised: The optimal value of problem (10) is equal to 0 if there exists a vector v ( t ) ∗ satisfying η ( t ) H ( t ) i v ( t ) ∗ =

Original: α ( t ) i ∇ z φ i ( h ( w ( t ) ; i )) for every i ∈ [ n ] .
Revised: α ( t ) i ∇ z φ i ( h ( w ( t ) ; i )) for every i ∈ [ n ] .

Original: Since the solution v ( t ) ∗ is in R d and ∇ z φ i ( h ( w ( t ) ; i )) is in R c , this condition is equivalent to a linear system with n · c constraints and d variables.
Revised: Since the solution v ( t ) ∗ is in R d and ∇ z φ i ( h ( w ( t ) ; i )) is in R c , this condition is equivalent to a linear system with n · c constraints and d variables.

Original: In the overparameterized setting where dimension d is sufﬁciently large ( d (cid:29) n · c ) and there are no identical data, there exists almost surely a vector v ( t ) ∗ that interpolates all the training set, see the Appendix for details.
Revised: In the overparameterized setting where dimension d is sufﬁciently large ( d (cid:29) n · c ) and there are no identical data, there exists almost surely a vector v ( t ) ∗ that interpolates all the training set, see the Appendix for details.

Original: Let us note that an approximation of v ( t ) ∗ serves as the search direction for Framework 1.
Revised: Let us note that an approximation of v ( t ) ∗ serves as the search direction for Framework 1.

Original: For this reason, the solution v ( t ) ∗ of problem (10) plays a similar role as a gradient in the search direction of (stochastic) gradient descent method.
Revised: For this reason, the solution v ( t ) ∗ of problem (10) plays a similar role as a gradient in the search direction of (stochastic) gradient descent method.

Original: It is standard to assume a bounded gradient in the machine learning literature (Nemirovski et al., 2009; Shalev-Shwartz et al., 2007; Reddi et al., 2016).
Revised: It is standard to assume a bounded gradient in the machine learning literature (Nemirovski et al., 2009; Shalev-Shwartz et al., 2007; Reddi et al., 2016).

Original: Motivated by these facts, we assume the following Assumption 3, which implies the existence of a near-optimal bounded solution of (10):
Revised: Motivated by these facts, we assume the following Assumption 3, which implies the existence of a near-optimal bounded solution of (10):

Original: Assumption 3.
Revised: Assumption 3.

Original: We consider an over-parameterized setting where dimension d is sufﬁciently large enough to interpolate all the data and the tolerance ε .
Revised: We consider an over-parameterized setting where dimension d is sufﬁciently large enough to interpolate all the data and the tolerance ε .

Original: We assume that there exists a bound V >such that for ε > 0 and 0 ≤ t < T as in Framework 1, there exists a vector ˆ v ( t ) ∗ ε with (cid:107) ˆ v ( t ) ∗ ε (cid:107) 2 ≤ V so that
Revised: We assume that there exists a bound V >such that for ε > 0 and 0 ≤ t < T as in Framework 1, there exists a vector ˆ v ( t ) ∗ ε with (cid:107) ˆ v ( t ) [Equation] (cid:107) 2 ≤ V so that

Original: [Equation]
Revised: [Equation]

Original: Our Assumption 3 requires a nice dependency on the tolerance ε for the gradient matrices H ( t ) i and ∇ z φ i ( h ( w ( t ) ; i )) .
Revised: Our Assumption 3 requires a nice dependency on the tolerance ε for the gradient matrices H ( t ) i and ∇ z φ i ( h ( w ( t ) ; i )) .

Original: We note that at the starting point t = 0 , these matrices may depend on ε due to the initialization process and the dependence of d on ε .
Revised: We note that at the starting point t = 0 , these matrices may depend on ε due to the initialization process and the dependence of d on ε .

Original: This setting is similar to previous works, e.g. Allen-Zhu et al.
Revised: This setting is similar to previous works, e.g. Allen-Zhu et al.

Original:  N EW A LGORITHMS AND C ONVERGENCE R ESULTS 5. A PPROXIMATING THE SOLUTION USING REGULARIZER
Revised:  N EW A LGORITHMS AND C ONVERGENCE R ESULTS 5. A PPROXIMATING THE SOLUTION USING REGULARIZER

Original: Since problem (10) is convex and quadratic
Revised: Since problem (10) is convex and quadratic

Original: [Equation]
Revised: [Equation]

Original: we consider the following regularized problem:
Revised: we consider the following regularized problem:

Original: [Equation]
Revised: [Equation]

Original: for some small ε > 0 and t ≥ 0 .
Revised: for some small ε > 0 and t ≥ 0 .

Original: It is widely known that problem (15) is strongly convex, and has a unique minimizer v ( t ) ∗ reg .
Revised: It is widely known that problem (15) is strongly convex, and has a unique minimizer v ( t ) ∗ reg .

Original: The global minimizer satisﬁes ∇ v Ψ( v ( t ) ∗ reg )
Revised: The global minimizer satisﬁes ∇ v Ψ( v ( t ) ∗ reg )

Original: = 0 .
Revised: = 0 .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: If ε 2 is small enough, then v ( t ) ∗ reg is a close approximation of the solution v ( t ) ∗ for problem (10).
Revised: If ε 2 is small enough, then v ( t ) ∗ reg is a close approximation of the solution v ( t ) ∗ for problem (10).

Original: Our ﬁrst algorithm updates Framework 1 based on this approximation.
Revised: Our ﬁrst algorithm updates Framework 1 based on this approximation.

Original: Algorithm1 Solve for the exact solution of the regularized problem
Revised: Algorithm

Original: Initialization: Choose an initial point w (0) ∈ R d , tolerance ε > 0 ; for t = 0 , 1 , · · · , T − 1 do
Revised: Initialization: Choose an initial point w (0) ∈ R d , tolerance ε > 0 ; for t = 0 , 1 , · · · , T − 1 do

Original: Update the search direction v ( t ) as the solution v ( t ) ∗ reg of problem in (15): v ( t )
Revised: Update the search direction v ( t ) as the solution v ( t ) ∗ reg of problem in (15): v ( t )

Original: = v( t )
Revised: = v

Original: [Equation]
Revised: [Equation]

Original: reg
Revised: (cid:32)

Original: [Equation]
Revised: [Equation]

Original: (cid:32)
Revised: (cid:33)

Original: [Equation]
Revised: [Equation]

Original: n n (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ( t)
Revised: ( t

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: ( t)
Revised: ( t

Original: [Equation]
Revised: [Equation]

Original: (cid:62)
Revised: (cid:62)

Original: [Equation]
Revised: [Equation]

Original: ( t )

Original: [Equation]
Revised: [Equation]

Original: ( t )

Original: [Equation]
Revised: [Equation]

Original: (cid:33) − 1 (cid:32)
Revised: ( t

Original: [Equation]
Revised: [Equation]

Original: n n (cid:88) i= α ( t ) i η ( t ) H ( t ) i (cid:62) ∇ z φ i ( h ( w ( t )
Revised: i )

Original: [Equation]
Revised: [Equation]

Original: i ))

Original: (cid:33)
Revised: (cid:33)

Original: Update w ( t +1) =
Revised: Update w ( t +1) =

Original: w ( t ) − η ( t ) v ( t ) end for
Revised: w ( t ) − η ( t ) v ( t ) end for

Original: The following Lemma shows the relation between the regularized solution v ( t ) ∗ reg and the optimal solution of the original convex problem ˆ v ( t ) ∗ ε .
Revised: The following Lemma shows the relation between the regularized solution v ( t ) ∗ reg and the optimal solution of the original convex problem ˆ v ( t ) ∗ ε .

Original: Lemma 2.
Revised: Lemma 2.

Original: For given ε > 0 , suppose that Assumption 3 holds for bound V > 0 .
Revised: For given ε > 0 , suppose that Assumption 3 holds for bound V > 0 .

Original: Then, for iteration 0 ≤ t < T , the optimal solution v ( t ) ∗ reg of problem (15) satisﬁes (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V and
Revised: Then, for iteration 0 ≤ t < T , the optimal solution v ( t ) ∗ reg of problem (15) satisﬁes (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V and

Original: [Equation]
Revised: [Equation]

Original: Based on Lemma 2, we guarantee the global convergence of Algorithm 1 and prove our ﬁrst theorem.
Revised: Based on Lemma 2, we guarantee the global convergence of Algorithm 1 and prove our ﬁrst theorem.

Original: Since it is currently expensive to solve for the exact solution of problem (15), our algorithm serves as a theoretical method to obtain the global convergence for the ﬁnite-sum minimization.
Revised: Since it is currently expensive to solve for the exact solution of problem (15), our algorithm serves as a theoretical method to obtain the global convergence for the ﬁnite-sum minimization.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let w ( t ) be generated by Algorithm 1 where we use the closed form solution for the search direction.
Revised: Let w ( t ) be generated by Algorithm 1 where we use the closed form solution for the search direction.

Original: We execute Algorithm 1 for T =
Revised: We execute Algorithm 1 for T =

Original: βε outer loops for some constant β > 0 .
Revised: βε outer loops for some constant β > 0 .

Original: We assume Assumption 1 holds.
Revised: We assume Assumption 1 holds.

Original: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .
Revised: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .

Original: We set the step size equal to η ( t ) =
Revised: We set the step size equal to η ( t ) =

Original: D √ ε for some D > 0 and choose a learning rate α ( t ) i
Revised: D √ ε for some D > 0 and choose a learning rate α ( t ) i

Original: = (1 + ε ) α
Revised: = (1 + ε ) α

Original: ( t − 1)
Revised: ( t − 1)

Original: i =
Revised: i =

Original: (1 + ε ) t α (0) i .
Revised: (1 + ε ) t α (0) i .

Original: Based on β , we deﬁne α (0) i
Revised: Based on β , we deﬁne α (0) i

Original: =
Revised: =

Original: α e β L φ with α ∈ (0 , 13 ) .
Revised: α e β L φ with α ∈ (0 , 13 ) .

Original: Let F ∗ be the global minimizer of F , and h ∗ i =
Revised: Let F ∗ be the global minimizer of F , and h ∗ i =

Original: arg
Revised: arg

Original: min z ∈ R c φ
Revised: min z ∈ R c φ

Original: i ( z ) , i ∈ [ n ] .
Revised: i ( z ) , i ∈ [ n ] .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: We note that β is a constant for the purpose of choosing the number of iterations T .
Revised: We note that β is a constant for the purpose of choosing the number of iterations T .

Original: The analysis can be simpliﬁed by choosing β = 1 with T = 1 ε .
Revised: The analysis can be simpliﬁed by choosing β = 1 with T = 1 ε .

Original: Notice that the common convergence criteria for ﬁnding a stationary point for non-convex problems is 1 T (cid:80)
Revised: Notice that the common convergence criteria for ﬁnding a stationary point for non-convex problems is 1 T (cid:80)

Original: Tt =1 ||∇ F ( w t )
Revised: Tt =1 ||∇ F ( w t )

Original: ||
Revised: ||

Original: 2 ≤ O ( ε ) .
Revised: 2 ≤ O ( ε ) .

Original: This criteria has been widely used in the existing literature for non-convex optimization problems.
Revised: This criteria has been widely used in the existing literature for non-convex optimization problems.

Original: Our convergence criteria 1 T (cid:80) Tt =

Revised: Our convergence criteria 1 T (cid:80) Tt =

Original: [ F ( w t ) − F ∗ ] ≤ O ( ε ) is slightly different, in order to ﬁnd a global solution for non-convex problems.
Revised: [ F ( w t ) − F ∗ ] ≤ O ( ε ) is slightly different, in order to ﬁnd a global solution for non-convex problems.

Original: Our proof for Theorem 1 is novel and insightful.
Revised: Our proof for Theorem 1 is novel and insightful.

Original: It is originally motivated by the Gradient Descent update (7) and the convexity of the loss functions φ i .
Revised: It is originally motivated by the Gradient Descent update (7) and the convexity of the loss functions φ i .

Original: For this reason it may not be a surprise that
Revised: For this reason it may not be a surprise that

Original: Algorithm 1 can ﬁnd an ε -global solution after O (cid:0) 1 ε (cid:1) iterations.
Revised: Algorithm 1 can ﬁnd an ε -global solution after O (cid:0) 1 ε (cid:1) iterations.

Original: However, computing the exact solution in every iteration might be extremely challenging, especially when the number of samples n is large.
Revised: However, computing the exact solution in every iteration might be extremely challenging, especially when the number of samples n is large.

Original: Therefore, we present a different approach to this problem in the following section.
Revised: Therefore, we present a different approach to this problem in the following section.

Original: 5. A PPROXIMATION USING G RADIENT D ESCENT
Revised: 5. A PPROXIMATION USING G RADIENT D ESCENT

Original: In this section, we use Gradient Descent (GD) algorithm to solve the strongly convex problem (15).
Revised: In this section, we use Gradient Descent (GD) algorithm to solve the strongly convex problem (15).

Original: It is well-known that if ψ ( x ) − µ 2 (cid:107) x (cid:107) 2 is convex for ∀ x ∈ R c , then ψ ( x ) is µ -strongly convex (see e.g. Nesterov (2004)).
Revised: It is well-known that if ψ ( x ) − µ 2 (cid:107) x (cid:107) 2 is convex for ∀ x ∈ R c , then ψ ( x ) is µ -strongly convex (see e.g. Nesterov (2004)).

Original: Hence Ψ( · ) is ε 2 -strongly convex.
Revised: Hence Ψ( · ) is ε 2 -strongly convex.

Original: For each iteration t , we use GD to ﬁnd a search direction v ( t ) which is sufﬁciently close to the optimal solution v ( t ) ∗ reg in that
Revised: For each iteration t , we use GD to ﬁnd a search direction v ( t ) which is sufﬁciently close to the optimal solution v ( t ) ∗ reg in that

Original: [Equation]
Revised: [Equation]

Original: Our Algorithm 2 is described as follows.
Revised: Our Algorithm 2 is described as follows.

Original: Algorithm 2 Solve the regularized problem using Gradient Descent
Revised: Algorithm 2 Solve the regularized problem using Gradient Descent

Original: Initialization: Choose an initial point w (0) ∈ R d , tolerance ε > 0 ; for t = 0 , 1 , · · · , T − 1 do
Revised: Initialization: Choose an initial point w (0) ∈ R d , tolerance ε > 0 ; for t = 0 , 1 , · · · , T − 1 do

Original: Use Gradient Descent algorithm to solve Problem (15) and ﬁnd a solution v ( t ) that satisﬁes (cid:107) v ( t
Revised: Use Gradient Descent algorithm to solve Problem (15) and ﬁnd a solution v ( t ) that satisﬁes (cid:107) v ( t

Original: [Equation]
Revised: [Equation]

Original: Update w (
Revised: Update w (

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original:  = w (
Revised: ) = w (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since Algorithm 2 can only approximate a solution within some ε -preciseness, we need a supplemental assumption for the analysis of our next Theorem 2:
Revised: Since Algorithm 2 can only approximate a solution within some ε -preciseness, we need a supplemental assumption for the analysis of our next Theorem 2:

Original: Assumption 4.
Revised: Assumption 4.

Original: Let H ( t ) i be the Jacobian matrix deﬁned in Lemma 1.
Revised: Let H ( t ) i be the Jacobian matrix deﬁned in Lemma 1.

Original: We assume that there exists some constant H > 0 such that, for i ∈ [ n ] , ε > 0 , and 0 ≤ t < T as in Algorithm 2,
Revised: We assume that there exists some constant H > 0 such that, for i ∈ [ n ] , ε > 0 , and 0 ≤ t < T as in Algorithm 2,

Original: [Equation]
Revised: [Equation]

Original: Assumption 4 requires a mild condition on the bounded Jacobian of h ( w ; i ) , and the upper bound may depend on ε .
Revised: Assumption 4 requires a mild condition on the bounded Jacobian of h ( w ; i ) , and the upper bound may depend on ε .

Original: This ﬂexibility allows us to accommodate a good dependence of ε for the theoretical analysis.
Revised: This ﬂexibility allows us to accommodate a good dependence of ε for the theoretical analysis.

Original: We are now ready to present our convergence theorem for Algorithm 2.
Revised: We are now ready to present our convergence theorem for Algorithm 2.

Original: Theorem 2.
Revised: Theorem 2.

Original: Let w ( t ) be generated by Algorithm 2 where v ( t ) satisﬁes (19).
Revised: Let w ( t ) be generated by Algorithm 2 where v ( t ) satisﬁes (19).

Original: We execute Algorithm 2 for T =
Revised: We execute Algorithm 2 for T =

Original: βε outer loops for some constant β > 0 .
Revised: βε outer loops for some constant β > 0 .

Original: We assume Assumption 1 holds.
Revised: We assume Assumption 1 holds.

Original: Suppose that Assumption 2 holds for G > 0 , Assumption 3 holds for V > 0 and Assumption 4 holds for
Revised: Suppose that Assumption 2 holds for G > 0 , Assumption 3 holds for V > 0 and Assumption 4 holds for

Original: H > 0 .
Revised: H > 0 .

Original: We set the step size equal to η ( t ) =
Revised: We set the step size equal to η ( t ) =

Original: D √ ε for some D > 0 and choose a learning rate α ( t ) i
Revised: D √ ε for some D > 0 and choose a learning rate α ( t ) i

Original: = (1 + ε ) α
Revised: = (1 + ε ) α

Original: ( t − 1)
Revised: ( t − 1)

Original: i =
Revised: i =

Original: (1 + ε ) t α (0) i .
Revised: (1 + ε ) t α (0) i .

Original: Based on β , we deﬁne α (0) i
Revised: Based on β , we deﬁne α (0) i

Original: =
Revised: =

Original: α e β L φ with α ∈ (0 , 14 ) .
Revised: α e β L φ with α ∈ (0 , 14 ) .

Original: Let F ∗ be the global minimizer of F , and h ∗ i =
Revised: Let F ∗ be the global minimizer of F , and h ∗ i =

Original: arg
Revised: arg

Original: min z ∈ R c φ
Revised: min z ∈ R c φ

Original: i ( z ) , i ∈ [ n ] .
Revised: i ( z ) , i ∈ [ n ] .

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: h

Original: [Equation]

Original: [Equation]

Original: + e

Original: [Equation]

Original: L

Original: [Equation] (4 ε + 3) [Equation] [Equation]
Revised: ∗ i (cid:107) 2 · ε + e β L φ (4 ε + 3) 2 α (1 − 4 α ) (cid:2) D 2 H 2 + c

Original: [Equation] [Equation] H [Equation]{+(2 + ( V + ε 2 + 2) GD 
+}
Revised: {+(2 + ( V + ε 2 + 2) GD 
+}

Original: ε

Original: [Equation]

Original: [Equation]

Original: [Equation] + [Equation] · ε.
Revised: + 2 + V (cid:3) · ε.

Original: Theorem 2 implies Corollary 2 which provides the computational complexity for Algorithm 2.
Revised: Theorem 2 implies Corollary 2 which provides the computational complexity for Algorithm 2.

Original: Note that for (Stochastic) Gradient Descent, we derive the complexity in terms of component gradient calculations for the ﬁnite-sum problem (1).
Revised: Note that for (Stochastic) Gradient Descent, we derive the complexity in terms of component gradient calculations for the ﬁnite-sum problem (1).

Original: As an alternative, for Algorithm 2 we compare the number of component gradients in problem (15).
Revised: As an alternative, for Algorithm 2 we compare the number of component gradients in problem (15).

Original: Such individual gradient has the following form:
Revised: Such individual gradient has the following form:

Original: [Equation]
Revised: [Equation]

Original: In machine learning applications, the gradient of f ( · ; i ) is calculated using automatic differentiation (i.e. backpropagation).
Revised: In machine learning applications, the gradient of f ( · ; i ) is calculated using automatic differentiation (i.e. backpropagation).

Original: Since f ( · ; i ) is the composition of the network structure h ( · ; i ) and loss function φ i ( · ) , this process also computes the Jacobian matrix H ( t ) i and the gradient ∇ z φ
Revised: Since f ( · ; i ) is the composition of the network structure h ( · ; i ) and loss function φ i ( · ) , this process also computes the Jacobian matrix H ( t ) i and the gradient ∇ z φ

Original: i ( h ( w ( t ) ; i )) at a speciﬁc weight w ( t ) .
Revised: i ( h ( w ( t ) ; i )) at a speciﬁc weight w ( t ) .

Original: Since matrix-vector multiplication computation is not expensive, the cost for computing the component gradient of problem (15) is similar to problem (1).
Revised: Since matrix-vector multiplication computation is not expensive, the cost for computing the component gradient of problem (15) is similar to problem (1).

Original: Corollary 2.
Revised: Corollary 2.

Original: Suppose that the conditions in Theorem 2 hold with η ( t ) =
Revised: Suppose that the conditions in Theorem 2 hold with η ( t ) =

Original: D √ ˆ ε √ N for some D > 0 and 0 < ˆ ε ≤ N (that is, we set ε
Revised: D √ ˆ ε √ N for some D > 0 and 0 < ˆ ε ≤ N (that is, we set ε

Original: = ˆ ε/N ), where
Revised: = ˆ ε/N ), where

Original: [Equation]
Revised: [Equation]

Original: Then, the total complexity to guarantee min 0 ≤ t ≤ T − 1 [ F ( w ( t ) )
Revised: Then, the total complexity to guarantee min 0 ≤ t ≤ T − 1 [ F ( w ( t ) )

Original: − F ∗ ] ≤ 1 T
Revised: − F ∗ ] ≤ 1 T

Original: [Equation]
Revised: [Equation]

Original: [ F ( w
Revised: [ F ( w

Original: [Equation]
Revised: [Equation]

Original: ) ) − F
Revised: ) ) − F

Original: [Equation]
Revised: [Equation]

Original: ] ≤ ˆ ε is O (cid:16)
Revised: ] ≤ ˆ ε is O (cid:16)

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: Corollary 2 shows that O (1 / ˆ ε ) outer loop iterations are needed in order to reach an ˆ ε -global solution, and it proves that each iteration needs the equivalent of O (cid:0) n ˆ ε 2 log( 1ˆ ε ) (cid:1) gradient computations for computing an approximate solution.
Revised: Corollary 2 shows that O (1 / ˆ ε ) outer loop iterations are needed in order to reach an ˆ ε -global solution, and it proves that each iteration needs the equivalent of O (cid:0) n ˆ ε 2 log( 1ˆ ε ) (cid:1) gradient computations for computing an approximate solution.

Original: In total, Algorithm 2 has total complexity
Revised: In total, Algorithm 2 has total complexity

Original: O (cid:0) n ˆ ε 3 log( 1ˆ ε ) (cid:1) for ﬁnding an ˆ ε -global solution.
Revised: O (cid:0) n ˆ ε 3 log( 1ˆ ε ) (cid:1) for ﬁnding an ˆ ε -global solution.

Original: For a comparison, Stochastic Gradient Descent uses a total of O ( 1 ε 2 ) gradient computations to ﬁnd a stationary point satisfying E
Revised: For a comparison, Stochastic Gradient Descent uses a total of O ( 1 ε 2 ) gradient computations to ﬁnd a stationary point satisfying E

Original: [ (cid:107)∇ F ( ˆ w ) (cid:107) 2 ] ≤ ε for non-convex problems (Ghadimi & Lan, 2013).
Revised: [ (cid:107)∇ F ( ˆ w ) (cid:107) 2 ] ≤ ε for non-convex problems (Ghadimi & Lan, 2013).

Original: Gradient Descent has a better complexity in terms of ε , i.e. O ( nε ) such that (cid:107)∇ F ( ˆ w ) (cid:107) 2 ≤ ε (Nesterov, 2004).
Revised: Gradient Descent has a better complexity in terms of ε , i.e. O ( nε ) such that (cid:107)∇ F ( ˆ w ) (cid:107) 2 ≤ ε (Nesterov, 2004).

Original: However, both methods may not be able to reach a global solution of (1).
Revised: However, both methods may not be able to reach a global solution of (1).

Original: In order to guarantee global convergence for nonconvex settings, one may resort to use Polyak-Lojasiewicz (PL) inequality (Karimi et al., 2016; Gower et al., 2021).
Revised: In order to guarantee global convergence for nonconvex settings, one may resort to use Polyak-Lojasiewicz (PL) inequality (Karimi et al., 2016; Gower et al., 2021).

Original: This assumption is widely known to be strong, which implies that every stationary point is also a global minimizer.
Revised: This assumption is widely known to be strong, which implies that every stationary point is also a global minimizer.

Original:  F URTHER D ISCUSSION AND C ONCLUSIONS
Revised:  F URTHER D ISCUSSION AND C ONCLUSIONS

Original: This paper presents an alternative composite formulation for solving the ﬁnite-sum optimization problem.
Revised: This paper presents an alternative composite formulation for solving the ﬁnite-sum optimization problem.

Original: Our formulation allows a new way of exploiting the structure of machine learning problems and the convexity of squared loss and softmax cross entropy loss, and leads to a novel algorithmic framework that guarantees global convergence (when the outer loss functions are convex and
Revised: Our formulation allows a new way of exploiting the structure of machine learning problems and the convexity of squared loss and softmax cross entropy loss, and leads to a novel algorithmic framework that guarantees global convergence (when the outer loss functions are convex and

Original: Lipschitz-smooth).
Revised: Lipschitz-smooth).

Original: Our analysis is general and can be applied to various different learning architectures, in particular, our analysis and assumptions match practical neural networks; in recent years, there has been a great interest in the structure of deep learning architectures for over-parameterized settings (Arora et al., 2018; Allen-Zhu et al., 2019; Nguyen & Mondelli, 2020).
Revised: Our analysis is general and can be applied to various different learning architectures, in particular, our analysis and assumptions match practical neural networks; in recent years, there has been a great interest in the structure of deep learning architectures for over-parameterized settings (Arora et al., 2018; Allen-Zhu et al., 2019; Nguyen & Mondelli, 2020).

Original: Algorithm 2 demonstrates a gradient method to solve the regularized problem, however, other methods can be applied to our framework (e.g. conjugate gradient descent).
Revised: Algorithm 2 demonstrates a gradient method to solve the regularized problem, however, other methods can be applied to our framework (e.g. conjugate gradient descent).

Original: Our theoretical foundation motivates further study, implementation, and optimization of the new algorithmic framework and further investigation of its non-standard bounded style assumptions.
Revised: Our theoretical foundation motivates further study, implementation, and optimization of the new algorithmic framework and further investigation of its non-standard bounded style assumptions.

Original: Possible research directions include more practical algorithm designs based on our Framework 1, and different related methods to solve the regularized problem and approximate the solution.
Revised: Possible research directions include more practical algorithm designs based on our Framework 1, and different related methods to solve the regularized problem and approximate the solution.

Original: This potentially leads to a new class of efﬁcient algorithms for machine learning problems.
Revised: This potentially leads to a new class of efﬁcient algorithms for machine learning problems.

Original: This paper presents a new perspective to the research community.
Revised: This paper presents a new perspective to the research community.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: This paper does not contain ethics concerns.
Revised: This paper does not contain ethics concerns.

Original: A PPENDIX A T ABLE OF NOTATIONS
Revised: A PPENDIX A T ABLE OF NOTATIONS

Original: Notation
Revised: Notation

Original: Meaning
Revised: Meaning

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: Global minimization function of F in (1)
Revised: Global minimization function of F in (1)

Original: [Equation]
Revised: [Equation]

Original: Solution of the convex problem in (10) min v ∈
Revised: Solution of the convex problem in (10) min v ∈

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: i ))
Revised: i ))

Original: [Equation]
Revised: [Equation]

Original: v ( t )
Revised: v ( t )

Original: An approximation of v ( t ) ∗ which is used as the search direction in Framework
Revised: An approximation of v ( t ) ∗ which is used as the search direction in Framework

Original: [Equation]
Revised: [Equation]

Original: A vector that satisﬁes
Revised: A vector that satisﬁes

Original: [Equation]
Revised: [Equation]

Original: for some ε > 0 and (cid:107) ˆ v ( t ) ∗ ε (cid:107) 2 ≤ V , for some V > 0 .
Revised: for some ε > 0 and (cid:107) ˆ v ( t ) ∗ ε (cid:107) 2 ≤ V , for some V > 0 .

Original: v ( t ) ∗ reg
Revised: v ( t ) ∗ reg

Original: Solution of the strongly convex problem in (15) min
Revised: Solution of the strongly convex problem in (15) min

Original: [Equation]
Revised: [Equation]

Original: B U SEFUL R ESULTS
Revised: B U SEFUL R ESULTS

Original: The following lemmas provide key tools for our results.
Revised: The following lemmas provide key tools for our results.

Original: Lemma 3 (Squared loss) .
Revised: Lemma 3 (Squared loss) .

Original: Let b ∈ R c and deﬁne φ ( z ) =
Revised: Let b ∈ R c and deﬁne φ ( z ) =

Original: 12 (cid:107) z − b (cid:107) 2 for z ∈ R c .
Revised: 12 (cid:107) z − b (cid:107) 2 for z ∈ R c .

Original: Then φ is convex and L φ -smooth with L φ = 1 .
Revised: Then φ is convex and L φ -smooth with L φ = 1 .

Original: Lemma 4 (Softmax cross-entropy loss) .
Revised: Lemma 4 (Softmax cross-entropy loss) .

Original: Let index a ∈ [ c ] and deﬁne φ ( z ) = log (cid:34)
Revised: Let index a ∈ [ c ] and deﬁne φ ( z ) = log (cid:34)

Original: [Equation]
Revised: [Equation]

Original: for z = ( z 1 , . . .
Revised: for z = ( z 1 , . . .

Original: , z c ) (cid:62) ∈ R c , where w k = e k − e a with e i representing the i -th unit vector (containing 1 at the i -th position and 0 elsewhere).
Revised: , z c ) (cid:62) ∈ R c , where w k = e k − e a with e i representing the i -th unit vector (containing 1 at the i -th position and 0 elsewhere).

Original: Then φ is convex and L φ -smooth with L φ = 1 .
Revised: Then φ is convex and L φ -smooth with L φ = 1 .

Original: The following lemma is a standard result in (Nesterov, 2004).
Revised: The following lemma is a standard result in (Nesterov, 2004).

Original: Lemma 5 ((Nesterov, 2004)) .
Revised: Lemma 5 ((Nesterov, 2004)) .

Original: If φ is L φ -smooth and convex, then for ∀ z ∈ R c ,
Revised: If φ is L φ -smooth and convex, then for ∀ z ∈ R c ,

Original: [Equation]
Revised: [Equation]

Original: where z ∗ = arg min z φ ( z ) .
Revised: where z ∗ = arg min z φ ( z ) .

Original: The following useful derivations could be used later in our theoretical analysis.
Revised: The following useful derivations could be used later in our theoretical analysis.

Original: Since φ i is convex, by Deﬁnition 2 we have
Revised: Since φ i is convex, by Deﬁnition 2 we have

Original: [Equation]
Revised: [Equation]

Original: If φ i is convex and L φ -smooth, then by Lemma
Revised: If φ i is convex and L φ -smooth, then by Lemma

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where h ∗ i = arg
Revised: where h ∗ i = arg

Original: min z ∈ R c φ
Revised: min z ∈ R c φ

Original: i ( z ) .
Revised: i ( z ) .

Original: We compute gradients of f ( w ; i ) in term of φ i ( h ( w ; i )) .
Revised: We compute gradients of f ( w ; i ) in term of φ i ( h ( w ; i )) .

Original: [Equation]
Revised: [Equation]

Original: where for j ∈ [ c ] ,
Revised: where for j ∈ [ c ] ,

Original: [Equation]
Revised: [Equation]

Original: • Gradient of squared loss:
Revised: • Gradient of squared loss:

Original: [Equation]
Revised: [Equation]

Original: C A DDITIONAL D ISCUSSION C. A BOUT A SSUMPTION
Revised: C A DDITIONAL D ISCUSSION C. A BOUT A SSUMPTION

Original: We make a formal assumption for the case h ( · ; i ) is closely approximated by k ( · ; i ) .
Revised: We make a formal assumption for the case h ( · ; i ) is closely approximated by k ( · ; i ) .

Original: Assumption 5.
Revised: Assumption 5.

Original: We assume that for all i ∈ [ n ] there exists some approximations k ( w ; i ) :
Revised: We assume that for all i ∈ [ n ] there exists some approximations k ( w ; i ) :

Original: R d → R c such that
Revised: R d → R c such that

Original: [Equation]
Revised: [Equation]

Original: where k ( · ; i ) are twice continuously differentiable (i
Revised: where k ( · ; i ) are twice continuously differentiable (i

Original: [Equation]
Revised: [Equation]

Original: e. the second-order partial derivatives of all scalars k j ( · ; i ) are continuous for all i ∈
Revised: e. the second-order partial derivatives of all scalars k j ( · ; i ) are continuous for all i ∈

Original: [ n ] )
Revised: [ n ] )

Original: [Equation]
Revised: [Equation]

Original: and that their Hessian matrices are bounded:
Revised: and that their Hessian matrices are bounded:

Original: [Equation]
Revised: [Equation]

Original: Assumption 5 allows us to prove the following Lemma that bound the error in equation (9):
Revised: Assumption 5 allows us to prove the following Lemma that bound the error in equation (9):

Original: Lemma 6.
Revised: Lemma 6.

Original: Suppose that Assumption 5 holds for the classiﬁer h .
Revised: Suppose that Assumption 5 holds for the classiﬁer h .

Original: Then for all i ∈ [ n ] and 0 ≤ t < T , we have:
Revised: Then for all i ∈ [ n ] and 0 ≤ t < T , we have:

Original: [Equation]
Revised: [Equation]

Original: where H ( t ) i is deﬁned to be the Jacobian matrix of the approximation k ( w ; i ) at w ( t ) :
Revised: where H ( t ) i is deﬁned to be the Jacobian matrix of the approximation k ( w ; i ) at w ( t ) :

Original: [Equation]
Revised: [Equation]

Original: Additionally we have
Revised: Additionally we have

Original: [Equation]
Revised: [Equation]

Original: Note that these result recover the case when h ( · ; i ) is itself smooth.
Revised: Note that these result recover the case when h ( · ; i ) is itself smooth.

Original: Hence we analyze our algorithms using the result of Lemma 6, which generalizes the result from Lemma 1.
Revised: Hence we analyze our algorithms using the result of Lemma 6, which generalizes the result from Lemma 1.

Original: C. A BOUT A SSUMPTION
Revised: C. A BOUT A SSUMPTION

Original: In this section, we justify the existence of the search direction in Assumption 3 (almost surely).
Revised: In this section, we justify the existence of the search direction in Assumption 3 (almost surely).

Original: We argue that there exists a vector ˆ v ( t ) ∗ ε satisfying
Revised: We argue that there exists a vector ˆ v ( t ) ∗ ε satisfying

Original: [Equation]
Revised: [Equation]

Original: It is sufﬁcient to ﬁnd a vector v satisfying that
Revised: It is sufﬁcient to ﬁnd a vector v satisfying that

Original: [Equation]
Revised: [Equation]

Original: Since the solution v is in R d and ∇ z φ i ( h ( w ( t ) ; i )) is in R c , this condition is equivalent to a linear system with n · c constraints and d variables.
Revised: Since the solution v is in R d and ∇ z φ i ( h ( w ( t ) ; i )) is in R c , this condition is equivalent to a linear system with n · c constraints and d variables.

Original: Let A and b be the following stacked matrix and vector:
Revised: Let A and b be the following stacked matrix and vector:

Original: [Equation]
Revised: [Equation]

Original: then the problem reduce to ﬁnding the solution of the equation Av = b .
Revised: then the problem reduce to ﬁnding the solution of the equation Av = b .

Original: In the over-parameterized setting where dimension d is sufﬁciently large ( d (cid:29) n · c ), then rank A = n · c almost surely and there exists almost surely a vector v that interpolates all the training set.
Revised: In the over-parameterized setting where dimension d is sufﬁciently large ( d (cid:29) n · c ), then rank A = n · c almost surely and there exists almost surely a vector v that interpolates all the training set.

Original: To demonstrate this fact easier, we consider a simple neural network where the classiﬁer h ( w ; i ) is formulated as
Revised: To demonstrate this fact easier, we consider a simple neural network where the classiﬁer h ( w ; i ) is formulated as

Original: [Equation]
Revised: [Equation]

Original: where c = 1 , W (1) ∈
Revised: where c = 1 , W (1) ∈

Original: R m × l and W (2) ∈ R l × 1 , w = vec ( { W (1) , W (2) } )
Revised: R m × l and W (2) ∈ R l × 1 , w = vec ( { W (1) , W (2) } )

Original: ∈ R
Revised: ∈ R

Original: d is the vectorized weight where d = l ( m + 1) and σ is sigmoid activation function.
Revised: d is the vectorized weight where d = l ( m + 1) and σ is sigmoid activation function.

Original: H ( t ) i is deﬁned to be the Jacobian matrix of h ( w ; i ) at w ( t ) :
Revised: H ( t ) i is deﬁned to be the Jacobian matrix of h ( w ; i ) at w ( t ) :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: w ; i ) ∂w
Revised: w ; i ) ∂w

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) w = w ( t ) ∈ R
Revised: (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) w = w ( t ) ∈ R

Original: [Equation]
Revised: [Equation]

Original: d , then
Revised: d , then

Original: [Equation]
Revised: [Equation]

Original: We want to show that A has full rank, almost surely.
Revised: We want to show that A has full rank, almost surely.

Original: We consider the over-parameterized setting where the last layer has at least n neuron (i.e. l = n and the simple version when c = 1 .
Revised: We consider the over-parameterized setting where the last layer has at least n neuron (i.e. l = n and the simple version when c = 1 .

Original: We argue that rank of matrix A is greater than or equal to rank of the submatrix B created by the weights of the last layer W (2) ∈ R n :
Revised: We argue that rank of matrix A is greater than or equal to rank of the submatrix B created by the weights of the last layer W (2) ∈ R n :

Original: [Equation]
Revised: [Equation]

Original: Note that h ( · , i ) is a linear function of the last weight layers (in this simple case W (2) ∈ R n and σ ( W (1) (cid:62)
Revised: Note that h ( · , i ) is a linear function of the last weight layers (in this simple case W (2) ∈ R n and σ ( W (1) (cid:62)

Original: x
Revised: x

Original: ( i ) )
Revised: ( i ) )

Original: ∈ R n ), we can compute the partial derivatives as follows:
Revised: ∈ R n ), we can compute the partial derivatives as follows:

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Assuming that there are no identical data, and
Revised: Assuming that there are no identical data, and

Original: σ is the sigmoid activation, the set of weights W (1) that make matrix B degenerate has measure zero.
Revised: σ is the sigmoid activation, the set of weights W (1) that make matrix B degenerate has measure zero.

Original: Hence B has full rank almost surely, and we have the same conclusion for A .
Revised: Hence B has full rank almost surely, and we have the same conclusion for A .

Original: Therefore we are able to prove the almost surely existence of a solution v of the linear equation Av = b for simple two layers network.
Revised: Therefore we are able to prove the almost surely existence of a solution v of the linear equation Av = b for simple two layers network.

Original: Using the same argument, this result can be generalized for larger neural networks where the dimension d is sufﬁciently large ( d (cid:29) nc ).
Revised: Using the same argument, this result can be generalized for larger neural networks where the dimension d is sufﬁciently large ( d (cid:29) nc ).

Original: C. I NITIALIZATION EXAMPLE
Revised: C. I NITIALIZATION EXAMPLE

Original: Our Assumption 3 requires a nice dependency on the tolerance ε for the gradient matrices H (0) i and ∇ z φ i ( h ( w (0) ; i )) .
Revised: Our Assumption 3 requires a nice dependency on the tolerance ε for the gradient matrices H (0) i and ∇ z φ i ( h ( w (0) ; i )) .

Original: We note that at the starting point t = 0 , these matrices may depend on ε due to the initialization process and the dependence of d on ε .
Revised: We note that at the starting point t = 0 , these matrices may depend on ε due to the initialization process and the dependence of d on ε .

Original: In order to accommodate the choice of learning rate η (0) =
Revised: In order to accommodate the choice of learning rate η (0) =

Original: D √ ε in our theorems, in this section we describe a network initialization that satisﬁes (cid:107) H (0)
Revised: D √ ε in our theorems, in this section we describe a network initialization that satisﬁes (cid:107) H (0)

Original: i (cid:107) =
Revised: i (cid:107) =

Original: Θ (cid:16) 1 √ ε (cid:17) where the gradient norm (cid:107)∇ z φ i ( h ( w (0) ; i ))
Revised: Θ (cid:16) 1 √ ε (cid:17) where the gradient norm (cid:107)∇ z φ i ( h ( w (0) ; i ))

Original: (cid:107) is at most constant order with respect to ε .
Revised: (cid:107) is at most constant order with respect to ε .

Original: To simplify the problem, we only consider small-dimension data and networks without activation.
Revised: To simplify the problem, we only consider small-dimension data and networks without activation.

Original: About the target vector: We choose φ i to be the softmax cross-entropy loss.
Revised: About the target vector: We choose φ i to be the softmax cross-entropy loss.

Original: By Lemma 7 (see below), we have that the gradient norm is upper bounded by a constant c , where c is the output dimension of the problem and is not dependent on ε .
Revised: By Lemma 7 (see below), we have that the gradient norm is upper bounded by a constant c , where c is the output dimension of the problem and is not dependent on ε .

Original: Note that when we stack all gradients for n data points, then the size of new vector is still not dependent on ε .
Revised: Note that when we stack all gradients for n data points, then the size of new vector is still not dependent on ε .

Original: About the network architecture:
Revised: About the network architecture:

Original: For simplicity, we consider the following classiﬁcation problem where
Revised: For simplicity, we consider the following classiﬁcation problem where

Original: • The input data is in R 2 .
Revised: • The input data is in R 2 .

Original: There are only two data points { x (1) , x (2) } .
Revised: There are only two data points { x (1) , x (2) } .

Original: Input data is bounded and non-degenerate (we will clarify this property later).
Revised: Input data is bounded and non-degenerate (we will clarify this property later).

Original: • The output data is (categorical) in R 2 : { y (1) =
Revised: • The output data is (categorical) in R 2 : { y (1) =

Original: (1 , 0) , y (2) =
Revised: (1 , 0) , y (2) =

Original: We want to have an over-parameterized setting where the dimension of weight vector is at least nc = 4 .
Revised: We want to have an over-parameterized setting where the dimension of weight vector is at least nc = 4 .

Original: We consider a simple network with two layers, no biases and no activation functions.
Revised: We consider a simple network with two layers, no biases and no activation functions.

Original: Let the number of neurons in the hidden layer be m .
Revised: Let the number of neurons in the hidden layer be m .

Original: The ﬂow of this network is (in) R 2 → R m → R 2 (out).
Revised: The ﬂow of this network is (in) R 2 → R m → R 2 (out).

Original: First, we consider the case where m = 1 .
Revised: First, we consider the case where m = 1 .

Original: • The ﬁrst layer has 2 parameters ( w 1 , w 2 ) and only 1 neuron that outputs z ( i ) =
Revised: • The ﬁrst layer has 2 parameters ( w 1 , w 2 ) and only 1 neuron that outputs z ( i ) =

Original: w 1 x ( i ) 1 + w 2 x ( i ) 2 (the subscript is for the coordinate of input data x ( i ) ).
Revised: w 1 x ( i ) 1 + w 2 x ( i ) 2 (the subscript is for the coordinate of input data x ( i ) ).

Original: • The second layer has 2 parameters ( w 3 , w 4 ) .
Revised: • The second layer has 2 parameters ( w 3 , w 4 ) .

Original: The ﬁnal output is
Revised: The ﬁnal output is

Original: [Equation]
Revised: [Equation]

Original: with w =
Revised: with w =

Original: [ w 1 , w 2 , w 3 , w 4 ] (cid:62) ∈ R 4 .
Revised: [ w 1 , w 2 , w 3 , w 4 ] (cid:62) ∈ R 4 .

Original: This network satisﬁes that the Hessian matrices of h ( w ; i ) are bounded.
Revised: This network satisﬁes that the Hessian matrices of h ( w ; i ) are bounded.

Original: Let Q and b be the following stacked matrix and vector:
Revised: Let Q and b be the following stacked matrix and vector:

Original: [Equation]
Revised: [Equation]

Original: Then we have the following:
Revised: Then we have the following:

Original: [Equation]
Revised: [Equation]

Original: 2) w 1 x (2)1 + w 2 x (2)  w 4 x (2)w 4 x (2) w 1 x (2) 1 + w 2 x (2)   .
Revised: 2) w 1 x (2)1 + w 2 x (2)  w 4 x (2)w 4 x (2) w 1 x (2) 1 + w 2 x (2)   .

Original: The determinant of this matrix is a polynomial of the weight w and the input data.
Revised: The determinant of this matrix is a polynomial of the weight w and the input data.

Original: Under some mild non-degenerate condition of the input data, we can choose some base point w (cid:48) that made this matrix invertible (note that if this condition is not satisﬁed, we can rescale/add a very small noise to the data - which is the common procedure in machine learning).
Revised: Under some mild non-degenerate condition of the input data, we can choose some base point w (cid:48) that made this matrix invertible (note that if this condition is not satisﬁed, we can rescale/add a very small noise to the data - which is the common procedure in machine learning).

Original: Hence the system Qu =
Revised: Hence the system Qu =

Original: b always has a solution.
Revised: b always has a solution.

Original: Now we consider the following two initializations: 1.
Revised: Now we consider the following two initializations: 1.

Original: We choose to initialize the starting point at w (0)
Revised: We choose to initialize the starting point at w (0)

Original: = 1 √ ε w (cid:48) and note that Q ( w ) is a linear function of w and Q ( w (cid:48) ) is independent of ε .
Revised: = 1 √ ε w (cid:48) and note that Q ( w ) is a linear function of w and Q ( w (cid:48) ) is independent of ε .

Original: Then the norm of matrix Q ( w (0) ) has the same scale with 1 √ ε .
Revised: Then the norm of matrix Q ( w (0) ) has the same scale with 1 √ ε .

Original: Instead of choosing m = 1 , we consider an over-parameterized network where m = 1 ε (recall that m is the number of neurons in the hidden layer).
Revised: Instead of choosing m = 1 , we consider an over-parameterized network where m = 1 ε (recall that m is the number of neurons in the hidden layer).

Original: The hidden layer in this case is:
Revised: The hidden layer in this case is:

Original: [Equation]
Revised: [Equation]

Original: The output layer is:
Revised: The output layer is:

Original: [Equation]
Revised: [Equation]

Original: with w =
Revised: with w =

Original: [ w (1)
Revised: [ w (1)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: w (2)1 , 1 , w (2)1 , 2 , . . .
Revised: w (2)1 , 1 , w (2)1 , 2 , . . .

Original: , w (2) m, 1 , w (2) m, 2 ] (cid:62) ∈
Revised: , w (2) m, 1 , w (2) m, 2 ] (cid:62) ∈

Original: R 4 m .
Revised: R 4 m .

Original: Hence,
Revised: Hence,

Original: Q ( w ) =
Revised: Q ( w ) =

Original:   w (2)1 , 1 x (1) . . .
Revised:   w (2)1 , 1 x (1) . . .

Original: w (2) m, 1 x (1) w (2)1 , 1 x (1) . . .
Revised: w (2) m, 1 x (1) w (2)1 , 1 x (1) . . .

Original: w (2) m, 1 x (1) z
Revised: w (2) m, 1 x (1) z

Original: z (1) m w (2)1 , 2 x (1) . . .
Revised: z (1) m w (2)1 , 2 x (1) . . .

Original: w (2) m, 2 x (1) w (2)1 , 2 x (1) . . .
Revised: w (2) m, 2 x (1) w (2)1 , 2 x (1) . . .

Original: w (2) m, 2 x (1)  z (1) . .
Revised: w (2) m, 2 x (1)  z (1) . .

Original: .z
Revised: .z

Original: (1) m w (2)1 , 1 x (2) . . .
Revised: (1) m w (2)1 , 1 x (2) . . .

Original: w (2) m, 1 x (2) w (2)1 , 1 x (2) . . .
Revised: w (2) m, 1 x (2) w (2)1 , 1 x (2) . . .

Original: w (2) m, 1 x (2) z
Revised: w (2) m, 1 x (2) z

Original: z (2) m w (2)1 , 2 x (2) . . .
Revised: z (2) m w (2)1 , 2 x (2) . . .

Original: w (2) m, 2 x (2) w (2)1 , 2 x (2) . . .
Revised: w (2) m, 2 x (2) w (2)1 , 2 x (2) . . .

Original: w (2) m, 2 x (2)  z (2) . .
Revised: w (2) m, 2 x (2)  z (2) . .

Original: .z
Revised: .z

Original: (2) m 
Revised: (2) m 

Original:  .
Revised:  .

Original: Hence, the number of (possibly) non-zero elements in each row is 3 m = 3 ε .
Revised: Hence, the number of (possibly) non-zero elements in each row is 3 m = 3 ε .

Original: For matrix A of rank r , we have (cid:107) A (cid:107) 2 ≤ (cid:107) A (cid:107) F ≤ √ r (cid:107) A (cid:107) 2 .
Revised: For matrix A of rank r , we have (cid:107) A (cid:107) 2 ≤ (cid:107) A (cid:107) F ≤ √ r (cid:107) A (cid:107) 2 .

Original: Since the rank of Q ( w ) is at most( nc = 4 , independent of ε ), we only need to ﬁnd the Frobenius norm of Q ( w ) .
Revised: Since the rank of Q ( w ) is at most( nc = 4 , independent of ε ), we only need to ﬁnd the Frobenius norm of Q ( w ) .

Original: We have (cid:107) Q ( w ) (cid:107) F = (cid:118)(cid:117)(cid:117)(cid:116) 4 (cid:88) i = 4 m (cid:88)
Revised: We have (cid:107) Q ( w ) (cid:107) F = (cid:118)(cid:117)(cid:117)(cid:116) 4 (cid:88) i = 4 m (cid:88)

Original: j = | q ij | 2 .
Revised: j = | q ij | 2 .

Original: Let q min and q max be the element with smallest/largest magnitude of Q ( w ) .
Revised: Let q min and q max be the element with smallest/largest magnitude of Q ( w ) .

Original: Suppose that x ( i ) (cid:54) =
Revised: Suppose that x ( i ) (cid:54) =

Original: (0 , 0) and choose w (cid:54) = 

Revised: (0 , 0) and choose w (cid:54) =

Original: such that z (cid:54) = 0 , q min > 0 and independent of ε .
Revised: such that z (cid:54) = 0 , q min > 0 and independent of ε .

Original: Hence, √ 8 √ ε | q min | ≤ (cid:107) Q ( w ) (cid:107) F ≤ √ 12 √ ε | q max | .
Revised: Hence, √ 8 √ ε | q min | ≤ (cid:107) Q ( w ) (cid:107) F ≤ √ 12 √ ε | q max | .

Original: Hence, (cid:107) Q ( w ) (cid:107) =
Revised: Hence, (cid:107) Q ( w ) (cid:107) =

Original: Θ (cid:16) 1 √ ε (cid:17) .
Revised: Θ (cid:16) 1 √ ε (cid:17) .

Original: Therefore this simple network initialization supports the dependence on ε for our Assumption 3.
Revised: Therefore this simple network initialization supports the dependence on ε for our Assumption 3.

Original: We note that a similar setting is found in (Allen-Zhu et al., 2019), where the authors initialize the weights using a random Gaussian distribution with a variance depending on the dimension of the problem.
Revised: We note that a similar setting is found in (Allen-Zhu et al., 2019), where the authors initialize the weights using a random Gaussian distribution with a variance depending on the dimension of the problem.

Original: In non-convex setting, they prove the convergence of SGD using the assumption that the number of neurons m depends inversely on the tolerance (cid:15) .
Revised: In non-convex setting, they prove the convergence of SGD using the assumption that the number of neurons m depends inversely on the tolerance ε .

Original: Lemma 7.
Revised: Lemma 7.

Original: For softmax cross-entropy loss, and x = h ( w ; i ) ∈ R c , for ∀ w ∈ R d and i ∈ [ n ] , we have
Revised: For softmax cross-entropy loss, and x = h ( w ; i ) ∈ R c , for ∀ w ∈ R d and i ∈ [ n ] , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Proof.
Revised: Proof.

Original: By (24), we have for i = 1 , . . .
Revised: By (24), we have for i = 1 , . . .

Original: , n ,
Revised: , n ,

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: w ;
Revised: w ;

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Hence, for i = 1 , . . . , n ,
Revised: Hence, for i = 1 , . . . , n ,

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof.

Original: D P ROOFS OF L EMMAS AND C OROLLARYP ROOF OF L EMMA
Revised: D P ROOFS OF L EMMAS AND C OROLLARYP ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: Since h ( · ; i ) are twice continuously differentiable for all i ∈ [ n ] , we have the following
Revised: Since h ( · ; i ) are twice continuously differentiable for all i ∈ [ n ] , we have the following

Original: Taylor approximation for each component outputs h j ( · ; i ) where j ∈ [ c ] and i ∈ [ n ] :
Revised: Taylor approximation for each component outputs h j ( · ; i ) where j ∈ [ c ] and i ∈ [ n ] :

Original: [Equation]
Revised: [Equation]

Original: (32) where M i,j ( ˜ w ( t ) ) is the Hessian matrices of h j ( · ; i ) at ˜ w ( t ) and ˜ w ( t ) =
Revised: (32) where M i,j ( ˜ w ( t ) ) is the Hessian matrices of h j ( · ; i ) at ˜ w ( t ) and ˜ w ( t ) =

Original: αw ( t ) + (1 − α ) w ( t +1) for some α ∈ [0 , 1] .
Revised: αw ( t ) + (1 − α ) w ( t +1) for some α ∈ [0 , 1] .

Original: This leads to our desired statement:
Revised: This leads to our desired statement:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Hence we get the ﬁnal bound:
Revised: Hence we get the ﬁnal bound:

Original: [Equation]
Revised: [Equation]

Original: P ROOF OF L EMMA
Revised: P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: From Assumption 3, we know that there exists ˆ v ( t ) ∗ ε so that
Revised: From Assumption 3, we know that there exists ˆ v ( t ) ∗ ε so that

Original: [Equation]
Revised: [Equation]

Original: and (cid:107) ˆ v ( t ) ∗ ε (cid:107) 2 ≤ V , for some V > 0 .
Revised: and (cid:107) ˆ v ( t ) ∗ ε (cid:107) 2 ≤ V , for some V > 0 .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Since v ( t ) ∗ reg is the optimal solution of the problem in (15) for 0 ≤ t < T
Revised: Since v ( t ) ∗ reg is the optimal solution of the problem in (15) for 0 ≤ t < T

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have (17) and (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V for 0 ≤ t < T .
Revised: Therefore, we have (17) and (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V for 0 ≤ t < T .

Original: P ROOF OF L EMMA
Revised: P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: We want to show that for any α ∈ [0 , 1]
Revised: We want to show that for any α ∈ [0 , 1]

Original: [Equation]
Revised: [Equation]

Original: in order to have the convexity of φ with respect to z (see (Nesterov, 2004)).
Revised: in order to have the convexity of φ with respect to z (see (Nesterov, 2004)).

Original: For any α ∈ [0 , 1] , we have for ∀ z 1 , z 2 ∈ R c ,
Revised: For any α ∈ [0 , 1] , we have for ∀ z 1 , z 2 ∈ R c ,

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality follows according to Cauchy-Schwarz inequality (cid:104) a, b
Revised: where the ﬁrst inequality follows according to Cauchy-Schwarz inequality (cid:104) a, b

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: a
Revised: a

Original: [Equation]
Revised: [Equation]

Original: b (cid:107) .
Revised: b (cid:107) .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Therefore, (33) implies the convexity of φ with respect to z .
Revised: Therefore, (33) implies the convexity of φ with respect to z .

Original: We want to show that ∃ L φ > 0 such that
Revised: We want to show that ∃ L φ > 0 such that

Original: [Equation]
Revised: [Equation]

Original: Notice that ∇ φ ( z ) =
Revised: Notice that ∇ φ ( z ) =

Original: z − b , then clearly ∀
Revised: z − b , then clearly ∀

Original: z 1 , z 2 ∈ R c ,
Revised: z 1 , z 2 ∈ R c ,

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: Therefore, (34) implies the L φ -smoothness of φ with respect to z with L φ = 1 .
Revised: Therefore, (34) implies the L φ -smoothness of φ with respect to z with L φ = 1 .

Original: P ROOF OF L EMMA
Revised: P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: For ∀ z 1 , z 2 ∈ R c and 1 ≤ k ≤ c , denote u k, 1 = exp( w (cid:62) k
Revised: For ∀ z 1 , z 2 ∈ R c and 1 ≤ k ≤ c , denote u k, 1 = exp( w (cid:62) k

Original: z 1 ) and u k, 2 = exp( w (cid:62) k
Revised: z 1 ) and u k, 2 = exp( w (cid:62) k

Original: z 2 ) and using Holder inequality
Revised: z 2 ) and using Holder inequality

Original: [Equation]
Revised: [Equation]

Original: , where 1 p + 1 q = 1 ,
Revised: , where 1 p + 1 q = 1 ,

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality since log ( x ) is an increasing function for ∀ x > 0 and exp( v ) > 0 for ∀ v ∈ R .
Revised: where the ﬁrst inequality since log ( x ) is an increasing function for ∀ x > 0 and exp( v ) > 0 for ∀ v ∈ R .

Original: Therefore, (33) implies the convexity of φ with respect to z .
Revised: Therefore, (33) implies the convexity of φ with respect to z .

Original: Note that (cid:107)∇ 2 φ ( z ) (cid:107) ≤ L φ if and only if φ ( z ) is L φ
Revised: Note that (cid:107)∇ 2 φ ( z ) (cid:107) ≤ L φ if and only if φ ( z ) is L φ

Original: -smooth
Revised: -smooth

Original: (see (Nesterov, 2004)).
Revised: (see (Nesterov, 2004)).

Original: First, we compute gradient of φ ( z ) :
Revised: First, we compute gradient of φ ( z ) :

Original: • For i (cid:54)
Revised: • For i (cid:54)

Original: =
Revised: =

Original: a :
Revised: a :

Original: [Equation]
Revised: [Equation]

Original: ( z )
Revised: ( z )

Original: [Equation]
Revised: [Equation]

Original: • For i = a :
Revised: • For i = a :

Original: [Equation]
Revised: [Equation]

Original: We then calculate
Revised: We then calculate

Original: [Equation]
Revised: [Equation]

Original: For i =
Revised: For i =

Original: j
Revised: j

Original: ∂
Revised: ∂

Original: [Equation]
Revised: [Equation]

Original: • For i (cid:54) =
Revised: • For i (cid:54) =

Original: j : ∂ 2 φ ( z ) ∂z j ∂z
Revised: j : ∂ 2 φ ( z ) ∂z j ∂z

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: Denote that y i =
Revised: Denote that y i =

Original: exp( z i − z a ) ≥ 0 , i ∈
Revised: exp( z i − z a ) ≥ 0 , i ∈

Original: [ c ]
Revised: [ c ]

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: • For i =
Revised: • For i =

Original: j : (cid:12) (cid:12) (cid:12) (cid:12) ∂
Revised: j : (cid:12) (cid:12) (cid:12) (cid:12) ∂

Original: [Equation]
Revised: [Equation]

Original: ( z ) ∂z j ∂z
Revised: ( z ) ∂z j ∂z

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: (cid:12)(cid:12)(cid:12) =
Revised: (cid:12)(cid:12)(cid:12) =

Original: (cid:12)
Revised: (cid:12)

Original: [Equation]
Revised: [Equation]

Original: y i ( (cid:80) c k =
Revised: y i ( (cid:80) c k =

Original: [Equation]
Revised: [Equation]

Original: • For i j :
Revised: • For i j :

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Therefore,
Revised: Therefore,

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds since
Revised: where the last inequality holds since

Original: [Equation]
Revised: [Equation]

Original: which follows by the application of Holder inequality (35) with p = 2 , q = 2 , a k = y 3 / 2 k , and
Revised: which follows by the application of Holder inequality (35) with p = 2 , q = 2 , a k = y 3 / 2 k , and

Original: b k = y 1 / 2 k
Revised: b k = y 1 / 2 k

Original: (Note that y k ≥ 0 , k ∈ [ c ] ).
Revised: (Note that y k ≥ 0 , k ∈ [ c ] ).

Original: Hence, (cid:107)∇ 2 φ ( z ) (cid:107) ≤ L φ with L φ = 1 which is equivalent to
Revised: Hence, (cid:107)∇ 2 φ ( z ) (cid:107) ≤ L φ with L φ = 1 which is equivalent to

Original: L φ -smoothness of φ .
Revised: L φ -smoothness of φ .

Original: P ROOF OF L EMMA
Revised: P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: Since k ( · ; i ) are twice continuously differentiable for all i ∈ [ n ] , we have the following
Revised: Since k ( · ; i ) are twice continuously differentiable for all i ∈ [ n ] , we have the following

Original: Taylor approximation for each component outputs k j ( · ; i ) where j ∈ [ c ] and i ∈ [ n ] :
Revised: Taylor approximation for each component outputs k j ( · ; i ) where j ∈ [ c ] and i ∈ [ n ] :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where M i,j ( ˜ w ( t ) ) is the Hessian matrices of k j ( · ; i ) at ˜ w ( t ) and ˜ w ( t ) =
Revised: where M i,j ( ˜ w ( t ) ) is the Hessian matrices of k j ( · ; i ) at ˜ w ( t ) and ˜ w ( t ) =

Original: αw ( t ) + (1 − α ) w ( t +1) for some α ∈ [0 , 1] .
Revised: αw ( t ) + (1 − α ) w ( t +1) for some α ∈ [0 , 1] .

Original: Shifting this back to the original function h j ( · ; i ) we have:
Revised: Shifting this back to the original function h j ( · ; i ) we have:

Original: [Equation]
Revised: [Equation]

Original: which leads to our desired statement:
Revised: which leads to our desired statement:

Original: [Equation]
Revised: [Equation]

Original: i )
Revised: i )

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: η
Revised: η

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: i )
Revised: i )

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: i )
Revised: i )

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: t )
Revised: t )

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Hence we get the ﬁnal bound:
Revised: Hence we get the ﬁnal bound:

Original: [Equation]
Revised: [Equation]

Original: P ROOF OF C OROLLARY
Revised: P ROOF OF C OROLLARY

Original: Proof.
Revised: Proof.

Original: The proof of this corollary follows directly by the applications of Lemmas 3 and 4.
Revised: The proof of this corollary follows directly by the applications of Lemmas 3 and 4.

Original: E T ECHNICAL P ROOFS FOR T HEOREM
Revised: E T ECHNICAL P ROOFS FOR T HEOREM

Original: Lemma 8.
Revised: Lemma 8.

Original: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 , and v ( t ) = v ( t ) ∗ reg .
Revised: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 , and v ( t ) = v ( t ) ∗ reg .

Original: Consider η ( t )
Revised: Consider η ( t )

Original: =
Revised: =

Original: D √ ε for some D > 0 and ε > 0 .
Revised: D √ ε for some D > 0 and ε > 0 .

Original: For i ∈ [ n ] and 0 ≤ t < T , we have
Revised: For i ∈ [ n ] and 0 ≤ t < T , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From (14), for i ∈ [ n ] , j ∈ [ c ] , and for 0 ≤ t < T , by Lemma 1 and Lemma 6 we have
Revised: From (14), for i ∈ [ n ] , j ∈ [ c ] , and for 0 ≤ t < T , by Lemma 1 and Lemma 6 we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows by the fact (cid:107) v ( t ) (cid:107)
Revised: where the last inequality follows by the fact (cid:107) v ( t ) (cid:107)

Original: 2 = (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V of Lemma 2 and η ( t ) =
Revised: 2 = (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V of Lemma 2 and η ( t ) =

Original: D √ ε .
Revised: D √ ε .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Lemma 9.
Revised: Lemma 9.

Original: Let w ( t ) be generated by Algorithm 1 where we use the closed form solution for the search direction.
Revised: Let w ( t ) be generated by Algorithm 1 where we use the closed form solution for the search direction.

Original: We execute Algorithm 1 for T =
Revised: We execute Algorithm 1 for T =

Original: βε outer loops for some constant β > 0 .
Revised: βε outer loops for some constant β > 0 .

Original: We assume Assumption 1 holds.
Revised: We assume Assumption 1 holds.

Original: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .
Revised: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .

Original: We set the step size equal to η ( t ) =
Revised: We set the step size equal to η ( t ) =

Original: D √ ε for some D > 0 and choose a learning rate α ( t ) i ≤ αL φ , for some α ∈ (0 , 13 ) .
Revised: D √ ε for some D > 0 and choose a learning rate α ( t ) i ≤ αL φ , for some α ∈ (0 , 13 ) .

Original: For i ∈ [ n ] and 0 ≤ t < T , we have
Revised: For i ∈ [ n ] and 0 ≤ t < T , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that we have the optimal solution v ( t ) ∗ reg for the optimization problem (15) for 0 ≤ t <
Revised: Note that we have the optimal solution v ( t ) ∗ reg for the optimization problem (15) for 0 ≤ t <

Original: T .
Revised: T .

Original: From (12), we have, for i ∈ [ n ] , h
Revised: From (12), we have, for i ∈ [ n ] , h

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where we expand the square term.
Revised: where we expand the square term.

Original: Now applying Young’s inequalities: 2 |
Revised: Now applying Young’s inequalities: 2 |

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u

Original: [Equation]
Revised: [Equation]

Original: for ε > 0 and 2 |
Revised: for ε > 0 and 2 |

Original: [Equation]
Revised: [Equation]

Original: u, v
Revised: u, v

Original: [Equation]
Revised: [Equation]

Original: | ≤
Revised: | ≤

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u

Original: [Equation]
Revised: [Equation]

Original: 2 +
Revised: 2 +

Original: [Equation]
Revised: [Equation]

Original: v (cid:107) 2 we have:
Revised: v (cid:107) 2 we have:

Original: [Equation]
Revised: [Equation]

Original: − 2 α ( t
Revised: − 2 α ( t

Original: [Equation]
Revised: [Equation]

Original: i [ φ i
Revised: i [ φ i

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: w ( t
Revised: w ( t

Original: [Equation]
Revised: [Equation]

Original: ; i
Revised: ; i

Original: [Equation]
Revised: [Equation]

Original: φ i
Revised: φ i

Original: [Equation]
Revised: [Equation]

Original: h ∗ i )]
Revised: h ∗ i )]

Original: Note that from (23) we get that (cid:107)∇ z φ
Revised: Note that from (23) we get that (cid:107)∇ z φ

Original: i ( h ( w ( t ) ; i ))
Revised: i ( h ( w ( t ) ; i ))

Original: (cid:107) 2 ≤ 2 L φ [ φ i ( h ( w ( t ) ; i ))
Revised: (cid:107) 2 ≤ 2 L φ [ φ i ( h ( w ( t ) ; i ))

Original: − φ
Revised: − φ

Original: i
Revised: i

Original: ( h ∗ i )] .
Revised: ( h ∗ i )] .

Original: Applying this and using the fact that α ( t ) i ≤ αL φ , for some α ∈ (0 , 13 ) , we are able to derive:
Revised: Applying this and using the fact that α ( t ) i ≤ αL φ , for some α ∈ (0 , 13 ) , we are able to derive:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the last inequality follows by Lemma 8.
Revised: where the last inequality follows by Lemma 8.

Original: Lemma 10.
Revised: Lemma 10.

Original: Let w ( t ) be generated by Algorithm 1 where we use the closed form solution for the search direction.
Revised: Let w ( t ) be generated by Algorithm 1 where we use the closed form solution for the search direction.

Original: We execute Algorithm 1 for T =
Revised: We execute Algorithm 1 for T =

Original: βε outer loops for some constant β > 0 .
Revised: βε outer loops for some constant β > 0 .

Original: We assume Assumption 1 holds.
Revised: We assume Assumption 1 holds.

Original: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .
Revised: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .

Original: We set the step size equal to η ( t ) =
Revised: We set the step size equal to η ( t ) =

Original: D √ ε for some D > 0 and choose a learning rate α ( t ) i
Revised: D √ ε for some D > 0 and choose a learning rate α ( t ) i

Original: = (1 + ε ) α
Revised: = (1 + ε ) α

Original: ( t − 1)
Revised: ( t − 1)

Original: i =
Revised: i =

Original: (1 + ε ) t α (0) i .
Revised: (1 + ε ) t α (0) i .

Original: Based on β , we deﬁne α (0) i
Revised: Based on β , we deﬁne α (0) i

Original: = α e β L φ with α ∈ (0 , 13 ) .
Revised: = α e β L φ with α ∈ (0 , 13 ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Rearranging the terms in Lemma 9, we have φ i ( h ( w ( t ) ; i ))
Revised: Rearranging the terms in Lemma 9, we have φ i ( h ( w ( t ) ; i ))

Original: − φ
Revised: − φ

Original: i
Revised: i

Original: ( h ∗ i ) ≤
Revised: ( h ∗ i ) ≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The last inequality follows because the learning rate satisﬁes α (0)
Revised: The last inequality follows because the learning rate satisﬁes α (0)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: β L φ ≤ αL φ and for t = 1 , . .
Revised: β L φ ≤ αL φ and for t = 1 , . .

Original: , T =
Revised: , T =

Original: βε for some β >
Revised: βε for some β >

Original: [Equation]
Revised: [Equation]

Original: since (1 + x ) 1 /x ≤ e , x > 0 .
Revised: since (1 + x ) 1 /x ≤ e , x > 0 .

Original: Moreover, we have 1 α ( t )
Revised: Moreover, we have 1 α ( t )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Taking the average sum from t = 0 , . .
Revised: Taking the average sum from t = 0 , . .

Original: , T − 1 , we have
Revised: , T − 1 , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Taking the average sum from i =
Revised: Taking the average sum from i =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: n , we have
Revised: n , we have

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Therefore, applying (41) to (40)
Revised: Therefore, applying (41) to (40)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: which is our desired result
Revised: which is our desired result

Original: [Equation]
Revised: [Equation]

Original: P ROOF OF T HEOREM
Revised: P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i =
Revised: (cid:88) i =

Original: φ i ( h ∗ i ) .
Revised: φ i ( h ∗ i ) .

Original: Hence F ∗ − 1 n (cid:80) ni
Revised: Hence F ∗ − 1 n (cid:80) ni

Original: =

Revised: =

Original: φ i ( h ∗ i ) ≥ 0 .
Revised: φ i ( h ∗ i ) ≥ 0 .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: ≤

Original: T
Revised: T

Original: T (cid:88) t =  n n (cid:88) i =
Revised: T (cid:88) t =  n n (cid:88) i =

Original: [ f ( w ( t ) ; i ) − φ
Revised: [ f ( w ( t ) ; i ) − φ

Original: i ( h ∗ i )] (39) ≤ e β L φ (1 + ε ) 2(1 − 3 α ) αβ · n n (cid:88)
Revised: i ( h ∗ i )] (39) ≤ e β L φ (1 + ε ) 2(1 − 3 α ) αβ · n n (cid:88)

Original: i =
Revised: i =

Original: (cid:107) h
Revised: (cid:107) h

Original: ( w (0)
Revised: ( w (0)

Original: ; i ) − h ∗
Revised: ; i ) − h ∗

Original: i (cid:107) 2 · ε + e β L φ (3 ε + 2) 8 α (1 − 3 α ) (cid:2) c (4 + ( V + 2) GD 2 ) 2 + 8 + 4 V (cid:3) · ε.
Revised: i (cid:107) 2 · ε + e β L φ (3 ε + 2) 8 α (1 − 3 α ) (cid:2) c (4 + ( V + 2) GD 2 ) 2 + 8 + 4 V (cid:3) · ε.

Original: F T ECHNICAL P ROOFS FOR T HEOREM
Revised: F T ECHNICAL P ROOFS FOR T HEOREM

Original: Lemma 11.
Revised: Lemma 11.

Original: For 0 ≤ t < T , suppose that Assumption 3 holds for V ≥ 0 and v ( t ) satisﬁes (19).
Revised: For 0 ≤ t < T , suppose that Assumption 3 holds for V ≥ 0 and v ( t ) satisﬁes (19).

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From (cid:107) v ( t ) − v ( t ) ∗ reg (cid:107) ≤ ε .
Revised: From (cid:107) v ( t ) − v ( t ) ∗ reg (cid:107) ≤ ε .

Original: Using (cid:107) a (cid:107) 2 ≤ 2 (cid:107) a − b (cid:107) 2 + 2 (cid:107) b (cid:107) 2 , we have
Revised: Using (cid:107) a (cid:107) 2 ≤ 2 (cid:107) a − b (cid:107) 2 + 2 (cid:107) b (cid:107) 2 , we have

Original: [Equation]
Revised: [Equation]

Original: + 2 V. where the last inequality follows since (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V for some V > 0 in Lemma 2.
Revised: + 2 V. where the last inequality follows since (cid:107) v ( t ) ∗ reg (cid:107) 2 ≤ 2 + V for some V > 0 in Lemma 2.

Original: Lemma 12.
Revised: Lemma 12.

Original: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .
Revised: Suppose that Assumption 2 holds for G > 0 and Assumption 3 holds for V > 0 .

Original: Consider η ( t )
Revised: Consider η ( t )

Original: =
Revised: =

Original: D √ ε for some D > 0 and ε > 0 .
Revised: D √ ε for some D > 0 and ε > 0 .

Original: For i ∈ [ n ] and 0 ≤ t < T , we have
Revised: For i ∈ [ n ] and 0 ≤ t < T , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: From (14), for i ∈ [ n ] , j ∈ [ c ] , and for 0 ≤ t < T , by Lemma 1 and Lemma 6 we have
Revised: From (14), for i ∈ [ n ] , j ∈ [ c ] , and for 0 ≤ t < T , by Lemma 1 and Lemma 6 we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality follows by the application of Lemma 11 and η ( t )
Revised: where the last inequality follows by the application of Lemma 11 and η ( t )

Original: =
Revised: =

Original: D √ ε .
Revised: D √ ε .

Original: Hence,
Revised: Hence,

Original: [Equation]
Revised: [Equation]

Original: Lemma 13.
Revised: Lemma 13.

Original: Let w ( t ) be generated by Algorithm 2 where v ( t ) satisﬁes (19).
Revised: Let w ( t ) be generated by Algorithm 2 where v ( t ) satisﬁes (19).

Original: We execute Algorithm 2 for T =
Revised: We execute Algorithm 2 for T =

Original: βε outer loops for some constant β > 0 .
Revised: βε outer loops for some constant β > 0 .

Original: We assume Assumption 1 holds.
Revised: We assume Assumption 1 holds.

Original: Suppose that
Revised: Suppose that

Original: Assumption 2 holds for G > 0 , Assumption 3 holds for V > 0 and Assumption 4 holds for H > 0 .
Revised: Assumption 2 holds for G > 0 , Assumption 3 holds for V > 0 and Assumption 4 holds for H > 0 .

Original: We set the step size equal to η ( t ) =
Revised: We set the step size equal to η ( t ) =

Original: D √ ε for some D > 0 and choose a learning rate α ( t ) i ≤ αL φ , for some α ∈ (0 , 14 ) .
Revised: D √ ε for some D > 0 and choose a learning rate α ( t ) i ≤ αL φ , for some α ∈ (0 , 14 ) .

Original: For i ∈ [ n ] and 0 ≤ t < T , we have (cid:107)
Revised: For i ∈ [ n ] and 0 ≤ t < T , we have (cid:107)

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: i ))
Revised: i ))

Original: − φ
Revised: − φ

Original: i
Revised: i

Original: ( h
Revised: ( h

Original: [Equation]
Revised: [Equation]

Original: i )]
Revised: i )]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Note that v ( t ) is obtained from the optimization problem (15) for 0 ≤ t < T .
Revised: Note that v ( t ) is obtained from the optimization problem (15) for 0 ≤ t < T .

Original: From (9), we have, for i ∈ [ n ] ,
Revised: From (9), we have, for i ∈ [ n ] ,

Original: [Equation]
Revised: [Equation]

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: +
Revised: +

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: + 2 ·
Revised: + 2 ·

Original: [Table]
Revised: [Table]

Original: η
Revised: η

Original: [Table]
Revised: [Table]

Original: H
Revised: H

Original: [Table]
Revised: [Table]

Original: v
Revised: v

Original: [Table]
Revised: [Table]

Original: − v
Revised: − v

Original: [Table] [Table]
Revised: [Table] [Table]

Original: v
Revised: v

Original: [Table]
Revised: [Table]

Original: −
Revised: −

Original: [Table]
Revised: [Table]

Original: ∇
Revised: ∇

Original: [Table]
Revised: [Table]

Original: φ
Revised: φ

Original: [Table] [Table]
Revised: [Table] [Table]

Original: − 2 ·
Revised: − 2 ·

Original: [Table]
Revised: [Table]

Original: α
Revised: α

Original: [Table]
Revised: [Table]

Original: t
Revised: t

Original: [Table]
Revised: [Table]

Original: ∇
Revised: ∇

Original: [Table]
Revised: [Table]

Original: φ
Revised: φ

Original: [Table]
Revised: [Table]

Original: w
Revised: w

Original: [Table] [Table]
Revised: [Table] [Table]

Original: +
Revised: +

Original: [Table]
Revised: [Table]

Original: ∇
Revised: ∇

Original: [Table]
Revised: [Table]

Original: φ
Revised: φ

Original: [Table]
Revised: [Table]

Original: w
Revised: w

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: where we expand the square term
Revised: where we expand the square term

Original: [Equation]
Revised: [Equation]

Original: Now applying Young’s inequalities: 2 |(cid:104) u, v (cid:105)| ≤
Revised: Now applying Young’s inequalities: 2 |(cid:104) u, v (cid:105)| ≤

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u

Original: [Equation]
Revised: [Equation]

Original: 2 ε/ 3 +( ε/ 3) (cid:107) v (cid:107)for ε > 0 and 2 |(cid:104) u, v (cid:105)| ≤ (cid:107) u (cid:107) 

Revised: 2 ε/ 3 +( ε/ 3) (cid:107) v (cid:107)for ε > 0 and 2 |(cid:104) u, v (cid:105)| ≤ (cid:107) u (cid:107)

Original: + (cid:107) v (cid:107) 2 we have:
Revised: + (cid:107) v (cid:107) 2 we have:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that from (23) we get that (cid:107)∇ z φ
Revised: Note that from (23) we get that (cid:107)∇ z φ

Original: i ( h ( w ( t ) ; i ))
Revised: i ( h ( w ( t ) ; i ))

Original: (cid:107) 2 ≤ 2 L φ [ φ i ( h ( w ( t ) ; i ))
Revised: (cid:107) 2 ≤ 2 L φ [ φ i ( h ( w ( t ) ; i ))

Original: − φ
Revised: − φ

Original: i
Revised: i

Original: ( h ∗ i )] .
Revised: ( h ∗ i )] .

Original: Applying this and using the fact that α ( t ) i ≤ αL φ , for some α ∈ (0 , 14 ) , we are able to derive:
Revised: Applying this and using the fact that α ( t ) i ≤ αL φ , for some α ∈ (0 , 14 ) , we are able to derive:

Original: [Equation]
Revised: [Equation]

Original: where ( a ) follows by using matrix vector inequality (cid:107) Hv (cid:107) ≤ (cid:107) H (cid:107)(cid:107) v (cid:107) , where H ∈ R c × d and v ∈ R d and Assumption 4 in (20) and η ( t )
Revised: where ( a ) follows by using matrix vector inequality (cid:107) Hv (cid:107) ≤ (cid:107) H (cid:107)(cid:107) v (cid:107) , where H ∈ R c × d and v ∈ R d and Assumption 4 in (20) and η ( t )

Original: =
Revised: =

Original: D √ ε for some D > 0 and ε > 0 ; ( b ) follows by the fact that (cid:107) v ( t ) − v ( t ) ∗ reg (cid:107) 2 ≤ ε 2 in (19) and Lemma 12.
Revised: D √ ε for some D > 0 and ε > 0 ; ( b ) follows by the fact that (cid:107) v ( t ) − v ( t ) ∗ reg (cid:107) 2 ≤ ε 2 in (19) and Lemma 12.

Original: Lemma 14.
Revised: Lemma 14.

Original: Let w ( t ) be generated by Algorithm 2 where v ( t ) satisﬁes (19).
Revised: Let w ( t ) be generated by Algorithm 2 where v ( t ) satisﬁes (19).

Original: We execute Algorithm 2 for T =
Revised: We execute Algorithm 2 for T =

Original: βε outer loops for some constant β > 0 .
Revised: βε outer loops for some constant β > 0 .

Original: We assume Assumption 1 holds.
Revised: We assume Assumption 1 holds.

Original: Suppose that Assumption 2 holds for G > 0 , Assumption 3 holds for V > 0 and Assumption 4 holds for
Revised: Suppose that Assumption 2 holds for G > 0 , Assumption 3 holds for V > 0 and Assumption 4 holds for

Original: H > 0 .We set the step size equal to η ( t ) =
Revised: H > 0 .We set the step size equal to η ( t ) =

Original: D √ ε for some D > 0 and choose a learning rate α ( t ) i
Revised: D √ ε for some D > 0 and choose a learning rate α ( t ) i

Original: = (1 + ε ) α
Revised: = (1 + ε ) α

Original: ( t − 1)
Revised: ( t − 1)

Original: i = (
Revised: i = (

Original: [Equation]
Revised: [Equation]

Original: α (0) i .
Revised: α (0) i .

Original: Based on β , we deﬁne α (0) i =
Revised: Based on β , we deﬁne α (0) i =

Original: [Equation]
Revised: [Equation]

Original: e β L φ with α ∈ (0 , 14 ) .
Revised: e β L φ with α ∈ (0 , 14 ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Rearranging the terms in Lemma 13, we have φ i (
Revised: Rearranging the terms in Lemma 13, we have φ i (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( t
Revised: ( t

Original: [Equation]
Revised: [Equation]

Original: i )) − φ
Revised: i )) − φ

Original: [Equation]
Revised: [Equation]

Original: ( h
Revised: ( h

Original: [Equation]
Revised: [Equation]

Original: i ) ≤
Revised: i ) ≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: The last inequality follows because the learning rate satisﬁes α (0) i =
Revised: The last inequality follows because the learning rate satisﬁes α (0) i =

Original: [Equation]
Revised: [Equation]

Original: β L φ ≤ αL φ and for t = 1 , . .
Revised: β L φ ≤ αL φ and for t = 1 , . .

Original: , T =
Revised: , T =

Original: βε for some β >
Revised: βε for some β >

Original: [Equation]
Revised: [Equation]

Original: since (1 + x ) 1 /x ≤ e , x > 0 .
Revised: since (1 + x ) 1 /x ≤ e , x > 0 .

Original: Moreover, we have 1 α ( t )
Revised: Moreover, we have 1 α ( t )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: Taking the average sum from t = 0 , . .
Revised: Taking the average sum from t = 0 , . .

Original: , T − 1 , we have
Revised: , T − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Taking the average sum from i = 1 , . .
Revised: Taking the average sum from i = 1 , . .

Original: . , n , we have
Revised: . , n , we have

Original: T T
Revised: T T

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: t =  n n
Revised: t =  n n

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: [ φ i ( h (
Revised: [ φ i ( h (

Original: [Equation]
Revised: [Equation]

Original: ( t )
Revised: ( t )

Original: [Equation]
Revised: [Equation]

Original: i ))
Revised: i ))

Original: [Equation]
Revised: [Equation]

Original: i ( h
Revised: i ( h

Original: [Equation]
Revised: [Equation]

Original: i )]
Revised: i )]

Original: [Equation]
Revised: [Equation]

Original: ε )
Revised: ε )

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: i )
Revised: i )

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: n
Revised: n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Therefore, applying (48) to (47)
Revised: Therefore, applying (48) to (47)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: P ROOF OF T HEOREM
Revised: P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: From (42) we have F ∗ − 1 n (cid:80) ni
Revised: From (42) we have F ∗ − 1 n (cid:80) ni

Original: =

Revised: =

Original: φ i ( h ∗ i ) ≥ 0 .
Revised: φ i ( h ∗ i ) ≥ 0 .

Original: This leads to
Revised: This leads to

Original: T
Revised: T

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: [Equation]
Revised: [Equation]

Original: (cid:88) t = (cid:32)n n (cid:88)
Revised: (cid:88) t = (cid:32)n n (cid:88)

Original: [Equation]
Revised: [Equation]

Original: P ROOF OF C OROLLARY
Revised: P ROOF OF C OROLLARY

Original: Proof.
Revised: Proof.

Original: For each iteration 0 ≤ t < T , we need to ﬁnd v ( t ) satisfying the following criteria:
Revised: For each iteration 0 ≤ t < T , we need to ﬁnd v ( t ) satisfying the following criteria:

Original: [Equation]
Revised: [Equation]

Original: for some ε > 0 .
Revised: for some ε > 0 .

Original: Using Gradient Descent we need O ( n Lµ log( 1 ε 2 ))
Revised: Using Gradient Descent we need O ( n Lµ log( 1 ε 2 ))

Original: =
Revised: =

Original: O (2 n Lµ log( 1 ε )) number of gradient evaluations (Nesterov, 2004), where L and µ = ε 2 are the smooth and strongly convex constants, respectively, of Ψ .
Revised: O (2 n Lµ log( 1 ε )) number of gradient evaluations (Nesterov, 2004), where L and µ = ε 2 are the smooth and strongly convex constants, respectively, of Ψ .

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: Then, for any v ∈ R c
Revised: Then, for any v ∈ R c

Original: [Equation]
Revised: [Equation]

Original: t ) ; i ))] , i ∈ [ n ] .
Revised: t ) ; i ))] , i ∈ [ n ] .

Original: Consider η ( t )
Revised: Consider η ( t )

Original: =
Revised: =

Original: D √ ε for some D > 0 and ε > 0 , we have for i ∈ [ n ] and 0 ≤ t < T
Revised: D √ ε for some D > 0 and ε > 0 , we have for i ∈ [ n ] and 0 ≤ t < T

Original: [Equation]
Revised: [Equation]

Original: Hence, (cid:107)∇
Revised: Hence, (cid:107)∇

Original: [Equation]
Revised: [Equation]

Original: ψ i (
Revised: ψ i (

Original: [Equation]
Revised: [Equation]

Original: (cid:107) + ε 2 for any v ∈ R c which implies that L = D 2 H 2 + ε 2 (Nesterov (2004)) and Lµ =
Revised: (cid:107) + ε 2 for any v ∈ R c which implies that L = D 2 H 2 + ε 2 (Nesterov (2004)) and Lµ =

Original: D 2 H 2 + ε 2 ε.
Revised: D 2 H 2 + ε 2 ε.

Original: Therefore, the complexity to ﬁnd v ( t ) for each iteration t is
Revised: Therefore, the complexity to ﬁnd v ( t ) for each iteration t is

Original: O (2 n D 2 H 2 + ε 2 εlog( 1 ε )) .
Revised: O (2 n D 2 H 2 + ε 2 εlog( 1 ε )) .

Original: Let us choose 0 < ε ≤ 1 .
Revised: Let us choose 0 < ε ≤ 1 .

Original: From (49), we have
Revised: From (49), we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: + V
Revised: + V

Original: [Equation]
Revised: [Equation]

Original: Let ˆ ε =
Revised: Let ˆ ε =

Original: Nε with 0 < ˆ ε ≤ N .
Revised: Nε with 0 < ˆ ε ≤ N .

Original: Then, we need T =
Revised: Then, we need T =

Original: Nβ ˆ ε for some β > 0 to guarantee min
Revised: Nβ ˆ ε for some β > 0 to guarantee min

Original: [Equation]
Revised: [Equation]

Original: [ F ( w ( t ) )
Revised: [ F ( w ( t ) )

Original: [Equation]
Revised: [Equation]

Original: F ∗ ] ≤ 1 T (cid:80) T − 1 t =

Revised: F ∗ ] ≤ 1 T (cid:80) T − 1 t =

Original: [ F ( w ( t ) ) − F ∗ ] ≤ ˆ ε .
Revised: [ F ( w ( t ) ) − F ∗ ] ≤ ˆ ε .

Original: Hence, the total complexity is
Revised: Hence, the total complexity is

Original: O
Revised: O

Original: [Equation]
Revised: [Equation]