Original: F AST CONVERGENCE OF STOCHASTIC SUBGRADIENT METHOD UNDER INTERPOLATION
Revised: F AST CONVERGENCE OF STOCHASTIC SUBGRADIENT METHOD UNDER INTERPOLATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: This paper studies the behaviour of the stochastic subgradient descent (SSGD) method applied to over-parameterized nonsmooth optimization problems that satisfy an interpolation condition.
Revised: This paper studies the behaviour of the stochastic subgradient descent (SSGD) method applied to over-parameterized nonsmooth optimization problems that satisfy an interpolation condition.

Original: By leveraging the composite structure of the empirical risk minimization problems, we prove that SSGD converges, respectively, with rates O
Revised: By leveraging the composite structure of the empirical risk minimization problems, we prove that SSGD converges, respectively, with rates O

Original: (1 /(cid:15) ) and O (log(1 /(cid:15) )) for convex and strongly-convex objectives when interpolation holds.
Revised: (1 /(cid:15) ) and O (log(1 /(cid:15) )) for convex and strongly-convex objectives when interpolation holds.

Original: These rates coincide with established rates for the stochastic gradient descent (SGD) method applied to smooth problems that also satisfy an interpolation condition.
Revised: These rates coincide with established rates for the stochastic gradient descent (SGD) method applied to smooth problems that also satisfy an interpolation condition.

Original: Our analysis provides a partial explanation for the empirical observation that sometimes SGD and SSGD behave similarly for training smooth and nonsmooth machine learning models.
Revised: Our analysis provides a partial explanation for the empirical observation that sometimes SGD and SSGD behave similarly for training smooth and nonsmooth machine learning models.

Original: We also prove that the rate O (1 /(cid:15) ) is optimal for the subgradient method in the convex and interpolation setting.
Revised: We also prove that the rate O (1 /(cid:15) ) is optimal for the subgradient method in the convex and interpolation setting.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Gradient descent (GD) and subgradient descent (subGD) methods are simple and effective ﬁrst-order optimization algorithms for training machine learning models.
Revised: Gradient descent (GD) and subgradient descent (subGD) methods are simple and effective ﬁrst-order optimization algorithms for training machine learning models.

Original: The convergence-rate analyses for these methods depend crucially on the smoothness of the objective function.
Revised: The convergence-rate analyses for these methods depend crucially on the smoothness of the objective function.

Original: It is well understood that there is a fundamental gap between the convergence rates of gradient-like methods for smooth and nonsmooth problems (Shor, 1984; Nemirovski & Yudin, 1983; Nesterov, 2005; Bubeck, 2015;
Revised: It is well understood that there is a fundamental gap between the convergence rates of gradient-like methods for smooth and nonsmooth problems (Shor, 1984; Nemirovski & Yudin, 1983; Nesterov, 2005; Bubeck, 2015;

Original: Beck, 2017).
Revised: Beck, 2017).

Original: Table 1 summarizes the rates for the main problem classes.
Revised: Table 1 summarizes the rates for the main problem classes.

Original: However, in the practice of training machine learning models, the nonsmootheness from the model is not causing much trouble (Glorot et al., 2011; Goodfellow et al., 2016).
Revised: However, in the practice of training machine learning models, the nonsmootheness from the model is not causing much trouble (Glorot et al., 2011; Goodfellow et al., 2016).

Original: Neural networks with nonsmooth activation function such as ReLU activation can usually be trained as fast as the one with smooth activation function such as softplus, see Figure 1.1 as a motivating experiment from us to compare the convergence of gradient and subgradient based methods for smooth and nonsmooth neural networks on the MNIST dataset 1 to classify digits 0 and 1.
Revised: Neural networks with nonsmooth activation function such as ReLU activation can usually be trained as fast as the one with smooth activation function such as softplus, see Figure 1.1 as a motivating experiment from us to compare the convergence of gradient and subgradient based methods for smooth and nonsmooth neural networks on the MNIST dataset 1 to classify digits 0 and 1.

Original: We can see that the gradient and subgradient based methods, either batch or stochastic version, has similar convergence behaviour for smooth and nonsmooth neural networks.
Revised: We can see that the gradient and subgradient based methods, either batch or stochastic version, has similar convergence behaviour for smooth and nonsmooth neural networks.

Original: Therefore there is a discrepancy between theory and practice.
Revised: Therefore there is a discrepancy between theory and practice.

Original: The success of overparameterized models—such as deep and wide neural networks—has instigated a trend in the analysis of stochastic variants of gradient descent in the interpolation setting, in which the model achieves zero training loss, and therefore ﬁts all of the training data (Schmidt & Le Roux, 2013; Bassily et al., 2018;
Revised: The success of overparameterized models—such as deep and wide neural networks—has instigated a trend in the analysis of stochastic variants of gradient descent in the interpolation setting, in which the model achieves zero training loss, and therefore ﬁts all of the training data (Schmidt & Le Roux, 2013; Bassily et al., 2018;

Original: Ma et al., 2018b; Jain et al., 2018; Vaswani et al., 2019a;b; Wu et al., 2019; Liu & Belkin, 2020).
Revised: Ma et al., 2018b; Jain et al., 2018; Vaswani et al., 2019a;b; Wu et al., 2019; Liu & Belkin, 2020).

Original: This recent series of papers offer insight into the fast convergence of
Revised: This recent series of papers offer insight into the fast convergence of

Original: SGD and new approaches for algorithm design.
Revised: SGD and new approaches for algorithm design.

Original: This line of analysis, however, focus exclusively on smooth objective function, and cannot explain the effectiveness of SSGD for training nonsmooth neural networks.
Revised: This line of analysis, however, focus exclusively on smooth objective function, and cannot explain the effectiveness of SSGD for training nonsmooth neural networks.

Original: We present a formal analysis showing that SSGD for nonsmooth objectives could converge as fast as smooth objectives in the interpolation setting.
Revised: We present a formal analysis showing that SSGD for nonsmooth objectives could converge as fast as smooth objectives in the interpolation setting.

Original: Our contributions include:
Revised: Our contributions include:

Original: • A description of semi-smoothness properties of a function useful for the iteration complexity analysis of convex objectives in the interpolation context.
Revised: • A description of semi-smoothness properties of a function useful for the iteration complexity analysis of convex objectives in the interpolation context.

Original: Under mild conditions, semi-
Revised: Under mild conditions, semi-

Original: [Table]
Revised: [Table]

Original: smoothness under interpolation allows us to prove that SSGD has iteration complexity
Revised: smoothness under interpolation allows us to prove that SSGD has iteration complexity

Original: O (1 /(cid:15) ) for convex objectives, and O (log(1 /(cid:15) ))
Revised: O (1 /(cid:15) ) for convex objectives, and O (log(1 /(cid:15) ))

Original: for strongly-convex objectives.
Revised: for strongly-convex objectives.

Original: These rates improved the classic bounds O (1 /(cid:15) 2 ) and ˜ O
Revised: These rates improved the classic bounds O (1 /(cid:15) 2 ) and ˜ O

Original: (1 /(cid:15) ) for convex and strongly-convex objectives and match the convergence rates of SGD for convex and smooth objective under interpolation.
Revised: (1 /(cid:15) ) for convex and strongly-convex objectives and match the convergence rates of SGD for convex and smooth objective under interpolation.

Original: • Proof that the iteration bound O (1 /(cid:15) ) is optimal in the convex and
Revised: • Proof that the iteration bound O (1 /(cid:15) ) is optimal in the convex and

Original: interpolation
Revised: interpolation

Original: setting.
Revised: setting.

Original: In
Revised: In

Original: contrast to the
Revised: contrast to the

Original: case
Revised: case

Original: with a smooth objective function, subgradient-based methods cannot be further accelerated for nonsmooth model—even
Revised: with a smooth objective function, subgradient-based methods cannot be further accelerated for nonsmooth model—even

Original: with the
Revised: with the

Original: interpolation
Revised: interpolation

Original: assumption.
Revised: assumption.

Original:  R ELATED WORK 2. F IRST - ORDER METHODS FOR NONSMOOTH OPTIMIZATION
Revised:  R ELATED WORK 2. F IRST - ORDER METHODS FOR NONSMOOTH OPTIMIZATION

Original: The subgradient method and its convergence analysis for general convex and nonsmooth problems was ﬁrst described by Shor in the late 1960s and 70s; see Shor (1984).
Revised: The subgradient method and its convergence analysis for general convex and nonsmooth problems was ﬁrst described by Shor in the late 1960s and 70s; see Shor (1984).

Original: Nemirovski & Yudin (1983) subsequently established that the iteration complexity O (1 /(cid:15) 2 ) described by Shor (1984) is optimal for methods that can only access a subgradient oracle.
Revised: Nemirovski & Yudin (1983) subsequently established that the iteration complexity O (1 /(cid:15) 2 ) described by Shor (1984) is optimal for methods that can only access a subgradient oracle.

Original: Subsequent works for minimizing general convex and nonsmooth problems include stochastic subgradient methods (Polyak, 1987; Shamir &
Revised: Subsequent works for minimizing general convex and nonsmooth problems include stochastic subgradient methods (Polyak, 1987; Shamir &

Original: Zhang, 2013), dual averaging (Nesterov, 2009), and acceleration via smoothing (Nesterov, 2005;
Revised: Zhang, 2013), dual averaging (Nesterov, 2009), and acceleration via smoothing (Nesterov, 2005;

Original: Beck & Teboulle, 2012).
Revised: Beck & Teboulle, 2012).

Original: More recently, Zhang et al.
Revised: More recently, Zhang et al.

Original: (2020) and Shamir (2020), among others, described the tractability and complexity of getting an approximate stationary point for general nonsmooth and nonconvex problems.
Revised: (2020) and Shamir (2020), among others, described the tractability and complexity of getting an approximate stationary point for general nonsmooth and nonconvex problems.

Original: A related line of work involves the analysis of optimization algorithms for partially nonsmooth objectives that take the form f ( x ) + g ( x ) , where the function f is smooth, and the function g , which usually represents a regularizer, is convex and nonsmooth.
Revised: A related line of work involves the analysis of optimization algorithms for partially nonsmooth objectives that take the form f ( x ) + g ( x ) , where the function f is smooth, and the function g , which usually represents a regularizer, is convex and nonsmooth.

Original: Many models in feature selection and compressed sensing fall into this category of problems, which are usually solved by variations of the proximal-gradient method (Nesterov, 2007; Beck & Teboulle, 2009; Xiao, 2010; Parikh & Boyd, 2014; Defazio et al., 2014; Allen-Zhu, 2017).
Revised: Many models in feature selection and compressed sensing fall into this category of problems, which are usually solved by variations of the proximal-gradient method (Nesterov, 2007; Beck & Teboulle, 2009; Xiao, 2010; Parikh & Boyd, 2014; Defazio et al., 2014; Allen-Zhu, 2017).

Original: These approaches typically use special properties of the regularization function g , which generally do not apply in the context we consider in this paper.
Revised: These approaches typically use special properties of the regularization function g , which generally do not apply in the context we consider in this paper.

Original: Another related line of works focus on compositional optimization (Drusvyatskiy & Paquette, 2019;
Revised: Another related line of works focus on compositional optimization (Drusvyatskiy & Paquette, 2019;

Original: Davis & Drusvyatskiy, 2018; Duchi & Ruan, 2018).
Revised: Davis & Drusvyatskiy, 2018; Duchi & Ruan, 2018).

Original: These works consider the objective as the compositions of convex functions and smooth maps, which is different from the objective formulation in this work, see section 3 for details.
Revised: These works consider the objective as the compositions of convex functions and smooth maps, which is different from the objective formulation in this work, see section 3 for details.

Original: 2. I NTERPOLATION HELPS OPTIMIZATION
Revised: 2. I NTERPOLATION HELPS OPTIMIZATION

Original: The interpolation condition implies that the residual between the prediction of the model and data vanishes.
Revised: The interpolation condition implies that the residual between the prediction of the model and data vanishes.

Original: In the context of nonlinear least-squares, for example, it is known that interpolation allows the Gauss-Newton method to converge at a local superlinear rate, although in general it can only achieve a local linear rate (Bertsekas, 1997).
Revised: In the context of nonlinear least-squares, for example, it is known that interpolation allows the Gauss-Newton method to converge at a local superlinear rate, although in general it can only achieve a local linear rate (Bertsekas, 1997).

Original: Conditional gradient method can also enjoy a linear convergence rate for least square objective when interpolation holds (Beck & Teboulle, 2004).
Revised: Conditional gradient method can also enjoy a linear convergence rate for least square objective when interpolation holds (Beck & Teboulle, 2004).

Original: Recent interest in over-parameterized models for machine learning problems makes interpolation (or near interpolation) a relevant assumption for the analysis and development of new algorithms.
Revised: Recent interest in over-parameterized models for machine learning problems makes interpolation (or near interpolation) a relevant assumption for the analysis and development of new algorithms.

Original: Under interpolation, stochastic gradient descent enjoys the same convergence rate as its deterministic counterpart, gradient descent (Schmidt & Le Roux, 2013;
Revised: Under interpolation, stochastic gradient descent enjoys the same convergence rate as its deterministic counterpart, gradient descent (Schmidt & Le Roux, 2013;

Original: Ma et al., 2018a; Bassily et al., 2018).
Revised: Ma et al., 2018a; Bassily et al., 2018).

Original: Some techniques that originally developed for gradient descent, such as line-search and Nesterov acceleration, are proven to also hold for SGD when interpolation is satisﬁed (Vaswani et al., 2019b;a;
Revised: Some techniques that originally developed for gradient descent, such as line-search and Nesterov acceleration, are proven to also hold for SGD when interpolation is satisﬁed (Vaswani et al., 2019b;a;

Original: Liu & Belkin, 2020; Jain et al., 2018)
Revised: Liu & Belkin, 2020; Jain et al., 2018)

Original: Related work in this vein includes improved analysis of stochastic second-order optimization algorithms (Meng et al., 2020) and adaptive stochastic gradient descent (Loizou et al., 2020; Vaswani et al., 2020) under interpolation.
Revised: Related work in this vein includes improved analysis of stochastic second-order optimization algorithms (Meng et al., 2020) and adaptive stochastic gradient descent (Loizou et al., 2020; Vaswani et al., 2020) under interpolation.

Original: Another line of work (Qian et al., 2019; Khaled & Richt ´ arik, 2020) in this direction aims to explore assumptions other than the classic bounded noise assumption for the optimization theory of training over-parameterized model, such as the expected growth condition (Schmidt & Le Roux, 2013), expected smoothness (Qian et al., 2019;
Revised: Another line of work (Qian et al., 2019; Khaled & Richt ´ arik, 2020) in this direction aims to explore assumptions other than the classic bounded noise assumption for the optimization theory of training over-parameterized model, such as the expected growth condition (Schmidt & Le Roux, 2013), expected smoothness (Qian et al., 2019;

Original: Gower et al., 2018), etc.
Revised: Gower et al., 2018), etc.

Original: All of the work just mentioned involves differentiable and smooth objective functions, and cannot explain the fast convergence of SSGD for nonsmooth objectives that is often observed in practice.
Revised: All of the work just mentioned involves differentiable and smooth objective functions, and cannot explain the fast convergence of SSGD for nonsmooth objectives that is often observed in practice.

Original:  P ROBLEM SETTING
Revised:  P ROBLEM SETTING

Original: We consider the unconstrained empirical risk-minimization problem minimize
Revised: We consider the unconstrained empirical risk-minimization problem minimize

Original: [Equation]
Revised: [Equation]

Original: where each
Revised: where each

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and n is the number of data points.
Revised: and n is the number of data points.

Original: Throughout the paper, we use w ∗ to denote any solution of eq.
Revised: Throughout the paper, we use w ∗ to denote any solution of eq.

Original: (3.1), and thus f ∗ :
Revised: (3.1), and thus f ∗ :

Original: = f ( w ∗ ) is the optimal objective value.
Revised: = f ( w ∗ ) is the optimal objective value.

Original: We assume that the 1-dimensional loss function (cid:96) : R → R ≥ 0 is nonnegative, inf (cid:96) = 0 , convex, and is 1-smooth, i.e.,
Revised: We assume that the 1-dimensional loss function (cid:96) : R → R ≥ 0 is nonnegative, inf (cid:96) = 0 , convex, and is 1-smooth, i.e.,

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, we also assume (cid:96) (0) = 0 .
Revised: Without loss of generality, we also assume (cid:96) (0) = 0 .

Original: Common examples for the loss function include the 2-norm, negative entropy, and the 2-norm hinge loss function.
Revised: Common examples for the loss function include the 2-norm, negative entropy, and the 2-norm hinge loss function.

Original: The n functions h i are Lipschitz continuous with respect to the ﬁxed parameter L :
Revised: The n functions h i are Lipschitz continuous with respect to the ﬁxed parameter L :

Original: [Equation]
Revised: [Equation]

Original: We make no assumption on their smoothness properties.
Revised: We make no assumption on their smoothness properties.

Original: Many key machine learning tasks can be formulated as (3.1), including training deep neural networks with nonsmooth activations, such as the
Revised: Many key machine learning tasks can be formulated as (3.1), including training deep neural networks with nonsmooth activations, such as the

Original: ReLU function.
Revised: ReLU function.

Original: Here and throughout, the function (cid:107) · (cid:107) is the 2-norm of a vector, unless otherwise speciﬁed.
Revised: Here and throughout, the function (cid:107) · (cid:107) is the 2-norm of a vector, unless otherwise speciﬁed.

Original: Our analysis relies on the Clark generalized gradient (Clarke, 1990) of the nonconvex and nonsmooth function h , deﬁned as the convex hull of all valid limiting gradients:
Revised: Our analysis relies on the Clark generalized gradient (Clarke, 1990) of the nonconvex and nonsmooth function h , deﬁned as the convex hull of all valid limiting gradients:

Original: [Equation]
Revised: [Equation]

Original: This deﬁnition additionally requires h to be almost everywhere differentiable by Rademacher’s theorem.
Revised: This deﬁnition additionally requires h to be almost everywhere differentiable by Rademacher’s theorem.

Original: We refer readers to Clarke (1990), and more recently, to Zhang et al.
Revised: We refer readers to Clarke (1990), and more recently, to Zhang et al.

Original: (2020), who use this generalized gradient in a related analysis.
Revised: (2020), who use this generalized gradient in a related analysis.

Original: These properties of the generalized gradient are needed for our analysis:
Revised: These properties of the generalized gradient are needed for our analysis:

Original: [Equation]
Revised: [Equation]

Original: The second property follows from the L -Lipschitz continuity of each function
Revised: The second property follows from the L -Lipschitz continuity of each function

Original: h
Revised: h

Original: i .
Revised: i .

Original: We deﬁne the norm of the generalized gradient at a vector w as
Revised: We deﬁne the norm of the generalized gradient at a vector w as

Original: [Equation]
Revised: [Equation]

Original: Algorithm 1 describes the stochastic subgradient descent (SSGD) method.
Revised: Algorithm 1 describes the stochastic subgradient descent (SSGD) method.

Original: Interpolation means that our model has the ability to ﬁt all training samples perfectly and therefore we can obtain zero training loss at the solution e.g., f ( w ∗ )
Revised: Interpolation means that our model has the ability to ﬁt all training samples perfectly and therefore we can obtain zero training loss at the solution e.g., f ( w ∗ )

Original: = 0 .
Revised: = 0 .

Original: The interpolation assumption holds for overparameterized neural networks and is gaining increasing interest in recent years Jacot et al.
Revised: The interpolation assumption holds for overparameterized neural networks and is gaining increasing interest in recent years Jacot et al.

Original:  M AIN RESULTS 4. B OUNDS AND L IPSCHITZ PROPERTIES OF THE GENERALIZED GRADIENT
Revised:  M AIN RESULTS 4. B OUNDS AND L IPSCHITZ PROPERTIES OF THE GENERALIZED GRADIENT

Original: Our analysis hinges on establishing that the objective function f is differentiable at all of its minimizers, i.e., at all vectors w such that f ( w ) = 0 .
Revised: Our analysis hinges on establishing that the objective function f is differentiable at all of its minimizers, i.e., at all vectors w such that f ( w ) = 0 .

Original: This is implied by the following proposition, which holds even without the interpolation condition.
Revised: This is implied by the following proposition, which holds even without the interpolation condition.

Original: Proposition 4.1 (Generalized gradient bound) .
Revised: Proposition 4.1 (Generalized gradient bound) .

Original: For all d -vectors w ,
Revised: For all d -vectors w ,

Original: [Equation]
Revised: [Equation]

Original: Consequently,
Revised: Consequently,

Original: [Equation]
Revised: [Equation]

Original: Equation (4.2) implies that if the objective value f ( w ) = 0 , then the generalized gradient contains only the origin: ∂f ( w ) = { 0 } .
Revised: Equation (4.2) implies that if the objective value f ( w ) = 0 , then the generalized gradient contains only the origin: ∂f ( w ) = { 0 } .

Original: It thus follows from Clarke (1981, Property 10) that f is differentiable at any point with zero objective value.
Revised: It thus follows from Clarke (1981, Property 10) that f is differentiable at any point with zero objective value.

Original: This means that if there is a solution with a zero value, then it must be a ﬁxed point of the subgradient method.
Revised: This means that if there is a solution with a zero value, then it must be a ﬁxed point of the subgradient method.

Original: This property stands in contrast to a solution where the function is nonsmooth and makes it possible for the subgradient method to converge to solution with constant learning rate.
Revised: This property stands in contrast to a solution where the function is nonsmooth and makes it possible for the subgradient method to converge to solution with constant learning rate.

Original: Proof of Proposition 4.1.
Revised: Proof of Proposition 4.1.

Original: Fix an arbitrary d -vector w .
Revised: Fix an arbitrary d -vector w .

Original: The subdifferential ∂f i ( w ) = (cid:96) (cid:48) ( h i ( w )) · ∂h i ( w ) .
Revised: The subdifferential ∂f i ( w ) = (cid:96) (cid:48) ( h i ( w )) · ∂h i ( w ) .

Original: Because each function h
Revised: Because each function h

Original: i is L -Lipschitz continuous, we can then deduce that
Revised: i is L -Lipschitz continuous, we can then deduce that

Original: [Equation]
Revised: [Equation]

Original: Step (i) follows from the assumption that (cid:96) (0) =
Revised: Step (i) follows from the assumption that (cid:96) (0) =

Original: min λ ∈ R (cid:96) ( λ ) = 0 , which implies (cid:96) (cid:48) (0) = 0 .
Revised: min λ ∈ R (cid:96) ( λ ) = 0 , which implies (cid:96) (cid:48) (0) = 0 .

Original: Step (ii) follows from the fact that any convex L -smooth function u : R n → R satisﬁes the bound
Revised: Step (ii) follows from the fact that any convex L -smooth function u : R n → R satisﬁes the bound

Original: [Equation]
Revised: [Equation]

Original: see Nesterov (2014, Theorem 2.1.5).
Revised: see Nesterov (2014, Theorem 2.1.5).

Original: Make the identiﬁcations u
Revised: Make the identiﬁcations u

Original: [Equation]
Revised: [Equation]

Original: (cid:96), y
Revised: (cid:96), y

Original: [Equation]
Revised: [Equation]

Original: =to immediately obtain eq.
Revised: =to immediately obtain eq.

Original: (4.3) and thus the proof for eq.
Revised: (4.3) and thus the proof for eq.

Original: Equation (4.2) can be obtained directly from Jensen’s inequality:
Revised: Equation (4.2) can be obtained directly from Jensen’s inequality:

Original: [Equation]
Revised: [Equation]

Original: The composite structure of the functions f i allows us develop a semi-Lipschitz bound on their generalized gradients.
Revised: The composite structure of the functions f i allows us develop a semi-Lipschitz bound on their generalized gradients.

Original: Moreover, we show that it is possible to provide a global convex majorant for each function
Revised: Moreover, we show that it is possible to provide a global convex majorant for each function

Original: f i , which holds without assuming convexity.
Revised: f i , which holds without assuming convexity.

Original: Interestingly, eq.
Revised: Interestingly, eq.

Original: (4.5) overlaps with the “semi-smoothness” property of over-parameterized neural networks derived by Allen-Zhu et al.
Revised: (4.5) overlaps with the “semi-smoothness” property of over-parameterized neural networks derived by Allen-Zhu et al.

Original: (2019, Theorem 4).
Revised: (2019, Theorem 4).

Original: In contrast to that result, however, we do not use any special properties of over-parameterized neural networks.
Revised: In contrast to that result, however, we do not use any special properties of over-parameterized neural networks.

Original: Proposition 4.2 may thus be of more general interest.
Revised: Proposition 4.2 may thus be of more general interest.

Original: Proposition 4.2 (Semi-smoothness) .
Revised: Proposition 4.2 (Semi-smoothness) .

Original: For all vectors w 1 and w 2 , and each i ∈ [ n ] ,
Revised: For all vectors w 1 and w 2 , and each i ∈ [ n ] ,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: See Appendix A for the proof of this result.
Revised: See Appendix A for the proof of this result.

Original: 4. C ONVERGENCE RATE OF STOCHASTIC SUBGRADIENT DESCENT
Revised: 4. C ONVERGENCE RATE OF STOCHASTIC SUBGRADIENT DESCENT

Original: We now present a global convergence analysis for the SSGD algorithm under the additional assumption that the objective f of eq.
Revised: We now present a global convergence analysis for the SSGD algorithm under the additional assumption that the objective f of eq.

Original: (3.1) is convex.
Revised: (3.1) is convex.

Original: We develop a bound on the expected progress of the objective value that depends on the minimizing value f ∗ , rather than on the Lipschitz bound on the function itself, which is the usual bound in the literature.
Revised: We develop a bound on the expected progress of the objective value that depends on the minimizing value f ∗ , rather than on the Lipschitz bound on the function itself, which is the usual bound in the literature.

Original: Our proof is based on a simple modiﬁcation of the classical proof of subgradient descent method.
Revised: Our proof is based on a simple modiﬁcation of the classical proof of subgradient descent method.

Original: Theorem 4.3 (Global convergence rate of SSGD) .
Revised: Theorem 4.3 (Global convergence rate of SSGD) .

Original: Assume f is convex.
Revised: Assume f is convex.

Original: Then for any positive integer T and any learning rate function α t that satisﬁes (cid:80) Tt =1 (
Revised: Then for any positive integer T and any learning rate function α t that satisﬁes (cid:80) Tt =1 (

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: L 2 α
Revised: L 2 α

Original: [Equation]
Revised: [Equation]

Original: ) > 0 ,
Revised: ) > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let g ( t ) i ∈
Revised: Let g ( t ) i ∈

Original: ∂f i ( w ( t ) ) .
Revised: ∂f i ( w ( t ) ) .

Original: Then each iterate w ( t ) satisﬁes the bound
Revised: Then each iterate w ( t ) satisﬁes the bound

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: f ( w (
Revised: f ( w (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (i) follows from the convexity of f , and (ii) follows from Proposition 4.1.
Revised: where (i) follows from the convexity of f , and (ii) follows from Proposition 4.1.

Original: Take expectations on both sides of the inequality (4.8) and rearrange to obtain
Revised: Take expectations on both sides of the inequality (4.8) and rearrange to obtain

Original: [Equation]
Revised: [Equation]

Original: Summing inequality (4.9) over t ∈ { 1 , 2 , . .
Revised: Summing inequality (4.9) over t ∈ { 1 , 2 , . .

Original: , T } yields
Revised: , T } yields

Original: [Equation]
Revised: [Equation]

Original: Divide both sides by 2 (cid:80)
Revised: Divide both sides by 2 (cid:80)

Original: Tt =1 ( α t − L 2 α 2 t ) > 0 to obtain the required result.
Revised: Tt =1 ( α t − L 2 α 2 t ) > 0 to obtain the required result.

Original: This proof mirrors closely the classical proof of SSGD, which assumes that the subdifferential of each f i is bounded, i.e., (cid:107) ∂f i ( w ) (cid:107) ≤ G for some constant G .
Revised: This proof mirrors closely the classical proof of SSGD, which assumes that the subdifferential of each f i is bounded, i.e., (cid:107) ∂f i ( w ) (cid:107) ≤ G for some constant G .

Original: We use Proposition 4.1 in eq.
Revised: We use Proposition 4.1 in eq.

Original: (4.7) to avoid the bounded subgradient assumption and to express the convergence rate using the minimal value f ∗ .
Revised: (4.7) to avoid the bounded subgradient assumption and to express the convergence rate using the minimal value f ∗ .

Original: This modiﬁcation allows us to leverage the interpolation assumption that f ∗ = 0 .
Revised: This modiﬁcation allows us to leverage the interpolation assumption that f ∗ = 0 .

Original: We use
Revised: We use

Original: Theorem 4.3 to immediately deduce the following convergence rate result.
Revised: Theorem 4.3 to immediately deduce the following convergence rate result.

Original: Corollary 4.4 (Global convergence rate of SSGD with constant learning rate) .
Revised: Corollary 4.4 (Global convergence rate of SSGD with constant learning rate) .

Original: Assume that f is convex and that the learning rate α t = 1 / (2 L 2 ) is constant for all t > 0 .
Revised: Assume that f is convex and that the learning rate α t = 1 / (2 L 2 ) is constant for all t > 0 .

Original: Then for any positive integer T , the SSGD iterates w t satisfy
Revised: Then for any positive integer T , the SSGD iterates w t satisfy

Original: [Equation]
Revised: [Equation]

Original: Furthermore, when f ∗ = 0 (interpolation holds),
Revised: Furthermore, when f ∗ = 0 (interpolation holds),

Original: [Equation]
Revised: [Equation]

Original: Corollary 4.4 indicates that the SSGD method converges at rate O (1 /(cid:15) ) when interpolation holds.
Revised: Corollary 4.4 indicates that the SSGD method converges at rate O (1 /(cid:15) ) when interpolation holds.

Original: This rate matches the convergence rate of SGD for smooth objective functions under interpolation (Schmidt & Le Roux, 2013).
Revised: This rate matches the convergence rate of SGD for smooth objective functions under interpolation (Schmidt & Le Roux, 2013).

Original: When interpolation does not hold, Corollary 4.4 implies that SSGD, on expectation, could obtain objective value lower than 2 f ∗ + (cid:15) in O (1 /(cid:15) ) time, which could be close to f ∗ when f ∗ is close to 0 (interpolation nearly holds).
Revised: When interpolation does not hold, Corollary 4.4 implies that SSGD, on expectation, could obtain objective value lower than 2 f ∗ + (cid:15) in O (1 /(cid:15) ) time, which could be close to f ∗ when f ∗ is close to 0 (interpolation nearly holds).

Original: Next, we derive convergence rate under the stronger assumption that f is strongly convex, which means that
Revised: Next, we derive convergence rate under the stronger assumption that f is strongly convex, which means that

Original: [Equation]
Revised: [Equation]

Original: for some constant µ > 0 .
Revised: for some constant µ > 0 .

Original: Note that recent works indicated that in order to prove linear convergence rate, the strong convexity assumption can be relaxed to other weaker assumptions that could hold even for some nonconvex functions (Karimi et al., 2016; Qian et al., 2019).
Revised: Note that recent works indicated that in order to prove linear convergence rate, the strong convexity assumption can be relaxed to other weaker assumptions that could hold even for some nonconvex functions (Karimi et al., 2016; Qian et al., 2019).

Original: For simplicity, we assume strong convexity in our analysis.
Revised: For simplicity, we assume strong convexity in our analysis.

Original: Theorem 4.5 (Global convergence rate of SSGD under strong convexity) .
Revised: Theorem 4.5 (Global convergence rate of SSGD under strong convexity) .

Original: Assume that f is µ -strongly convex and that the learning rate α t = 1 /L 2 is constant for all t > 0 .
Revised: Assume that f is µ -strongly convex and that the learning rate α t = 1 /L 2 is constant for all t > 0 .

Original: Then for any positive integer T , the SSGD iterates w ( t ) satisfy
Revised: Then for any positive integer T , the SSGD iterates w ( t ) satisfy

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Use the deﬁnition of the SSGD iteration to obtain
Revised: Use the deﬁnition of the SSGD iteration to obtain

Original: [Equation]
Revised: [Equation]

Original: where (i) follows from the same argument of the proof of Theorem 4.3; (ii) follows from the µ -strong convexity of f (see eq.
Revised: where (i) follows from the same argument of the proof of Theorem 4.3; (ii) follows from the µ -strong convexity of f (see eq.

Original: (4.11)); (iii) follows from eq.
Revised: (4.11)); (iii) follows from eq.

Original: (4.1); and (iv) follows from the deﬁnition of the learning rate α t = 1 L 2 .
Revised: (4.1); and (iv) follows from the deﬁnition of the learning rate α t = 1 L 2 .

Original: Taking expectation to both sides of eq.
Revised: Taking expectation to both sides of eq.

Original: (4.13) and recursively apply it to t ∈ { 1 , 2 , . .
Revised: (4.13) and recursively apply it to t ∈ { 1 , 2 , . .

Original: . , T } to deduce
Revised: . , T } to deduce

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (i) follows from the fact that (cid:80) ∞ t =0 (1 − µ 2 L 2 ) t = 2 L 2 /µ .
Revised: where (i) follows from the fact that (cid:80) ∞ t =0 (1 − µ 2 L 2 ) t = 2 L 2 /µ .

Original: Theorem 4.5 indicates that the SSGD can converges to the ball centered at w ∗ with radius 2 (cid:112) f ∗ /µ at a linear rate.
Revised: Theorem 4.5 indicates that the SSGD can converges to the ball centered at w ∗ with radius 2 (cid:112) f ∗ /µ at a linear rate.

Original: If additionally interpolation holds, SSGD converges to the solution linearly.
Revised: If additionally interpolation holds, SSGD converges to the solution linearly.

Original: Again, this rate matches the convergence rate of SGD for smooth and strongly convex objectives in the interpolation setting (Schmidt & Le Roux, 2013; Ma et al., 2018a).
Revised: Again, this rate matches the convergence rate of SGD for smooth and strongly convex objectives in the interpolation setting (Schmidt & Le Roux, 2013; Ma et al., 2018a).

Original: Similar to Corollary 4.4, when interpolation is nearly satisﬁed, SSGD converges linearly to an 2 (cid:112) f ∗ /µ -approximate solution.
Revised: Similar to Corollary 4.4, when interpolation is nearly satisﬁed, SSGD converges linearly to an 2 (cid:112) f ∗ /µ -approximate solution.

Original: Corollary 4.4 and Theorem 4.5 also provide insight into the effect of learning rate schedules on the performance of SSGD.
Revised: Corollary 4.4 and Theorem 4.5 also provide insight into the effect of learning rate schedules on the performance of SSGD.

Original: A learning rate schedule α t that decays as t − 1 / 2 is optimal because it causes the algorithm to exhibit a complexity bound of O (1 /(cid:15) 2 ) , which is the theoretical lower bound (Nemirovski & Yudin, 1983).
Revised: A learning rate schedule α t that decays as t − 1 / 2 is optimal because it causes the algorithm to exhibit a complexity bound of O (1 /(cid:15) 2 ) , which is the theoretical lower bound (Nemirovski & Yudin, 1983).

Original: However, the learning rate schedule O ( t − 1 / 2 ) is slow in practice.
Revised: However, the learning rate schedule O ( t − 1 / 2 ) is slow in practice.

Original: Corollary 4.4 and Theorem 4.5 partially explain the discrepancy between the theory and practice: many machine learning models exhibit interpolation or near interpolation, and an aggressive constant learning-rate schedule works better than the conservative worst-case optimal learning rate α t
Revised: Corollary 4.4 and Theorem 4.5 partially explain the discrepancy between the theory and practice: many machine learning models exhibit interpolation or near interpolation, and an aggressive constant learning-rate schedule works better than the conservative worst-case optimal learning rate α t

Original: = O ( t − 1 / 2 ) .
Revised: = O ( t − 1 / 2 ) .

Original: Other than the learning rate scheduling α t = 1 /L 2 , we can adopt a more recent developed learning rate scheduling Stich (2019) to obtain a reﬁned convergence rate, see more details in Appendix.
Revised: Other than the learning rate scheduling α t = 1 /L 2 , we can adopt a more recent developed learning rate scheduling Stich (2019) to obtain a reﬁned convergence rate, see more details in Appendix.

Original: 4. L OWER BOUNDS
Revised: 4. L OWER BOUNDS

Original: We have proven that, under interpolation, SGD for smooth problems and SSGD for nonsmooth problems exhibit the same convergence rates.
Revised: We have proven that, under interpolation, SGD for smooth problems and SSGD for nonsmooth problems exhibit the same convergence rates.

Original: This causes us to consider the following questions:
Revised: This causes us to consider the following questions:

Original: • Is it possible to induce momentum-type acceleration for SSGD under interpolation?
Revised: • Is it possible to induce momentum-type acceleration for SSGD under interpolation?

Original: Vaswani et al. (2019a) and Liu & Belkin (2020) showed recently that acceleration is possible for SGD under interpolation.
Revised: Vaswani et al. (2019a) and Liu & Belkin (2020) showed recently that acceleration is possible for SGD under interpolation.

Original: It seems plausible that similar techniques could be used for nonsmooth problems.
Revised: It seems plausible that similar techniques could be used for nonsmooth problems.

Original: • Can the composite structure f i
Revised: • Can the composite structure f i

Original: =
Revised: =

Original: (cid:96) ◦
Revised: (cid:96) ◦

Original: h
Revised: h

Original: i , which is central to our analysis, be used to establish an improved convergence rate for SSGD without interpolation?
Revised: i , which is central to our analysis, be used to establish an improved convergence rate for SSGD without interpolation?

Original: In other words, we know the lower bound on the iteration complexity for any subgradient method for nonsmooth functions is Ω(1 /(cid:15) 2 ) .
Revised: In other words, we know the lower bound on the iteration complexity for any subgradient method for nonsmooth functions is Ω(1 /(cid:15) 2 ) .

Original: What, then, is the lower bound for minimizing the structured nonsmooth function (cid:96) ◦ h ?
Revised: What, then, is the lower bound for minimizing the structured nonsmooth function (cid:96) ◦ h ?

Original: Unfortunately, the answer to these questions is “no”, as we show below.
Revised: Unfortunately, the answer to these questions is “no”, as we show below.

Original: First, we derive the lower iteration bound for any algorithm with access only to a subgradient oracle in the interpolation setting.
Revised: First, we derive the lower iteration bound for any algorithm with access only to a subgradient oracle in the interpolation setting.

Original: Theorem 4.6 (Lower bound with interpolation) .
Revised: Theorem 4.6 (Lower bound with interpolation) .

Original: Given t < d and positive constants L and R , and an initial vector w (1) .
Revised: Given t < d and positive constants L and R , and an initial vector w (1) .

Original: Let (cid:96) =
Revised: Let (cid:96) =

Original: Then there exists an L -Lipschitz function h such that f :=
Revised: Then there exists an L -Lipschitz function h such that f :=

Original: l ◦ h is convex, f ∗ =
Revised: l ◦ h is convex, f ∗ =

Original: min w f ( w ) = 0 , (cid:107) w (1) − w ∗ (cid:107) ≤ R such that
Revised: min w f ( w ) = 0 , (cid:107) w (1) − w ∗ (cid:107) ≤ R such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: With out loss of generality, we assume the initial point w (1) = 0 .
Revised: With out loss of generality, we assume the initial point w (1) = 0 .

Original: Let h ( w ) =
Revised: Let h ( w ) =

Original: L (cid:107) w − w ∗ (cid:107) ∞ , and
Revised: L (cid:107) w − w ∗ (cid:107) ∞ , and

Original: [Equation]
Revised: [Equation]

Original: entries
Revised: entries

Original: [Equation]
Revised: [Equation]

Original: entries
Revised: entries

Original: [Equation]
Revised: [Equation]

Original: where (cid:15) is some arbitrary constant greater than 0.
Revised: where (cid:15) is some arbitrary constant greater than 0.

Original: It is easy to check that (cid:96) is 1-smooth, h is
Revised: It is easy to check that (cid:96) is 1-smooth, h is

Original: L-Lipschitz, f = (cid:96) ◦ h is convex, and f ∗ = 0 .
Revised: L-Lipschitz, f = (cid:96) ◦ h is convex, and f ∗ = 0 .

Original: We follow Nemirovski & Yudin (1983) and assume without loss of generality that any algorithm that uses only subgradient information generates iterates w ( s ) ∈ span { e 1 , e 2 , . . .
Revised: We follow Nemirovski & Yudin (1983) and assume without loss of generality that any algorithm that uses only subgradient information generates iterates w ( s ) ∈ span { e 1 , e 2 , . . .

Original: , e t } for all s ≤ t .
Revised: , e t } for all s ≤ t .

Original: Therefore, for all s ≤ t ,
Revised: Therefore, for all s ≤ t ,

Original: [Equation]
Revised: [Equation]

Original: Note that the above holds ∀ (cid:15) > 0 .
Revised: Note that the above holds ∀ (cid:15) > 0 .

Original: Taking (cid:15) → 0 + , we completes the proof.
Revised: Taking (cid:15) → 0 + , we completes the proof.

Original: Theorem 4.6 established Ω(1 /(cid:15) ) iteration complexity for subgradient method under interpolation.
Revised: Theorem 4.6 established Ω(1 /(cid:15) ) iteration complexity for subgradient method under interpolation.

Original: Combining this result with the O (1 /(cid:15) ) iteration complexity in Corollary 4.4, we can conclude that the rate O (1 /(cid:15) ) is optimal in the interpolation setting and no acceleration is possible for subgradient-based methods.
Revised: Combining this result with the O (1 /(cid:15) ) iteration complexity in Corollary 4.4, we can conclude that the rate O (1 /(cid:15) ) is optimal in the interpolation setting and no acceleration is possible for subgradient-based methods.

Original: Then we proceed to the lower bound with assuming interpolation.
Revised: Then we proceed to the lower bound with assuming interpolation.

Original: Theorem 4.7 (Lower bound without interpolation) .
Revised: Theorem 4.7 (Lower bound without interpolation) .

Original: Given t < d, L, R > 0 and an initial point w (1) .
Revised: Given t < d, L, R > 0 and an initial point w (1) .

Original: Let (cid:96) ( · ) = ( · ) 2 , and there exist L -Lipschitz function h satisfying f :=
Revised: Let (cid:96) ( · ) = ( · ) 2 , and there exist L -Lipschitz function h satisfying f :=

Original: l ◦ h is convex, (cid:107) w
Revised: l ◦ h is convex, (cid:107) w

Original: [Equation]
Revised: [Equation]

Original: 1) − w ∗ (cid:107) ≤ R such that
Revised: 1) − w ∗ (cid:107) ≤ R such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Similar to the proof of Theorem 4.6, we set w (1) = 0 .
Revised: Similar to the proof of Theorem 4.6, we set w (1) = 0 .

Original: But we change the construction of h ( w ) to h ( w ) =
Revised: But we change the construction of h ( w ) to h ( w ) =

Original: L ( (cid:107) w − w ∗ (cid:107) ∞ + R ) , and w ∗ is deﬁned in the same way as in the proof of Theorem 4.6:
Revised: L ( (cid:107) w − w ∗ (cid:107) ∞ + R ) , and w ∗ is deﬁned in the same way as in the proof of Theorem 4.6:

Original: [Equation]
Revised: [Equation]

Original: entries (cid:105) , where (cid:15) is some arbitrary constant greater than 0.
Revised: entries (cid:105) , where (cid:15) is some arbitrary constant greater than 0.

Original: Following the same analysis of the proof of Theorem 4.6.
Revised: Following the same analysis of the proof of Theorem 4.6.

Original: w ( s ) ∈ span { e 1 , e 2 , . . .
Revised: w ( s ) ∈ span { e 1 , e 2 , . . .

Original: , e t } ∀ s ≤ t and
Revised: , e t } ∀ s ≤ t and

Original: [Equation]
Revised: [Equation]

Original: By taking (cid:15) → 0 + , we have (cid:107) w (1) − w ∗ (cid:107) ≤ R and completes the proof.
Revised: By taking (cid:15) → 0 + , we have (cid:107) w (1) − w ∗ (cid:107) ≤ R and completes the proof.

Original: Theorem 4.7 provides an Ω(1 /(cid:15) 2 ) iteration complexity for subgradient-based methods for solving the structured function (cid:96) ◦ h .
Revised: Theorem 4.7 provides an Ω(1 /(cid:15) 2 ) iteration complexity for subgradient-based methods for solving the structured function (cid:96) ◦ h .

Original: This matches the lower bound for solving general nonsmooth objective function with subgradient-based method, and implies that the structure (cid:96) ◦ h itself without interpolation cannot give us an improved iteration complexity of subgradient-based methods.
Revised: This matches the lower bound for solving general nonsmooth objective function with subgradient-based method, and implies that the structure (cid:96) ◦ h itself without interpolation cannot give us an improved iteration complexity of subgradient-based methods.

Original:  N UMERICAL EXPERIMENTS
Revised:  N UMERICAL EXPERIMENTS

Original: The smoothness of the loss function (cid:96) is crucial to our analysis.
Revised: The smoothness of the loss function (cid:96) is crucial to our analysis.

Original: We now present some numerical experiments to compare the convergence of SSGD for training ReLU neural networks with smooth and nonsmooth loss functions.
Revised: We now present some numerical experiments to compare the convergence of SSGD for training ReLU neural networks with smooth and nonsmooth loss functions.

Original: We denote { x i } ni =1 as training samples and { y i } ni =1 as training labels.
Revised: We denote { x i } ni =1 as training samples and { y i } ni =1 as training labels.

Original: ˆ y i stands for the prediction of the i -th sample x
Revised: ˆ y i stands for the prediction of the i -th sample x

Original: i from the trained model.
Revised: i from the trained model.

Original: 5. T EACHER - STUDENT SETUP
Revised: 5. T EACHER - STUDENT SETUP

Original: We randomly generate a small one hidden layer neural network with 16 neurons and ReLU activation as the teacher network, the network takes 16 dimensional vectors as inputs and outputs a scalar.
Revised: We randomly generate a small one hidden layer neural network with 16 neurons and ReLU activation as the teacher network, the network takes 16 dimensional vectors as inputs and outputs a scalar.

Original: Then we generate 128 random vectors from the Gaussian distribution as our training data { x i } 128 i =

Revised: Then we generate 128 random vectors from the Gaussian distribution as our training data { x i } 128 i =

Original: ⊂
Revised: ⊂

Original: Rand get their corresponding labels { y i } 128 i =1 as the output of the teacher network.
Revised: Rand get their corresponding labels { y i } 128 i =1 as the output of the teacher network.

Original: In order to ease the training and satisfy the interpolation assumption, we overparameterize the student neural network and set it to be a one hidden layer network with 512 neurons and ReLU activation.
Revised: In order to ease the training and satisfy the interpolation assumption, we overparameterize the student neural network and set it to be a one hidden layer network with 512 neurons and ReLU activation.

Original: We train the
Revised: We train the

Original: [Figure]
Revised: [Figure]

Original: student network with different loss functions: squared loss e.g., 1 n (cid:80) ni
Revised: student network with different loss functions: squared loss e.g., 1 n (cid:80) ni

Original: =

Revised: =

Original: ( y i − ˆ y i ) 2 and absolute loss e.g., 1 n (cid:80) n i =1 |
Revised: ( y i − ˆ y i ) 2 and absolute loss e.g., 1 n (cid:80) n i =1 |

Original: y i − ˆ y i | with different learning rates.
Revised: y i − ˆ y i | with different learning rates.

Original: The training curves are shown in Figure 5.1a.
Revised: The training curves are shown in Figure 5.1a.

Original: We can observe that the training curve with squared loss is smoother than the curve with absolute loss.
Revised: We can observe that the training curve with squared loss is smoother than the curve with absolute loss.

Original: For absolute loss, the performance of SSGD is more sensitive to the change of learning rate and we need to decrease the learning rate to obtain a lower objective value.
Revised: For absolute loss, the performance of SSGD is more sensitive to the change of learning rate and we need to decrease the learning rate to obtain a lower objective value.

Original: These observations validate the importance of the smoothness of loss function under the interpolation setting.
Revised: These observations validate the importance of the smoothness of loss function under the interpolation setting.

Original: 5. C LASSIFY 4’ S AND 9’ S ON MNIST DATASET
Revised: 5. C LASSIFY 4’ S AND 9’ S ON MNIST DATASET

Original: We train the LeNet (Lecun et al., 1998) on the MNIST dataset to classify 4’s and 9’s.
Revised: We train the LeNet (Lecun et al., 1998) on the MNIST dataset to classify 4’s and 9’s.

Original: To convert this task to a binary classiﬁcation problem, we transform the labels { y i } ni =1 to {− 1 , +1 } n .
Revised: To convert this task to a binary classiﬁcation problem, we transform the labels { y i } ni =1 to {− 1 , +1 } n .

Original: Then we run
Revised: Then we run

Original: SSGD to train the model with difference loss functions: logistic loss e.g., 1 n (cid:80) ni =1 log(1+exp( − y
Revised: SSGD to train the model with difference loss functions: logistic loss e.g., 1 n (cid:80) ni =1 log(1+exp( − y

Original: i
Revised: i

Original: ˆ y i ))
Revised: ˆ y i ))

Original: and L1-hinge loss e.g., 1 n (cid:80) ni
Revised: and L1-hinge loss e.g., 1 n (cid:80) ni

Original: =1 max { 0 , 1 − y
Revised: =1 max { 0 , 1 − y

Original: i ˆ y i } and with different learning rates.
Revised: i ˆ y i } and with different learning rates.

Original: The training curves are presented in Figure 5.1b.
Revised: The training curves are presented in Figure 5.1b.

Original: Different from the teacher-student experiment, SSGD in this task perform similarly with smooth and nonsmooth loss functions.
Revised: Different from the teacher-student experiment, SSGD in this task perform similarly with smooth and nonsmooth loss functions.

Original: We conjecture that this is because the objective function with L1-hinge loss almost satisﬁes Proposition 4.1 locally at the solution, namely eq.
Revised: We conjecture that this is because the objective function with L1-hinge loss almost satisﬁes Proposition 4.1 locally at the solution, namely eq.

Original: (4.1) holds for most training samples in a neighbourhood of the solution.
Revised: (4.1) holds for most training samples in a neighbourhood of the solution.

Original: Our observation supports this conjecture.
Revised: Our observation supports this conjecture.

Original: We observe that more that 95% of our ﬁnal predictions ˆ
Revised: We observe that more that 95% of our ﬁnal predictions ˆ

Original: y
Revised: y

Original: i ’s satisfy the condition | ˆ y i | > 2 .
Revised: i ’s satisfy the condition | ˆ y i | > 2 .

Original: Since the L1-hinge loss is locally smooth when | ˆ y i | > 2 , we can thus say that most training samples satisfy eq.
Revised: Since the L1-hinge loss is locally smooth when | ˆ y i | > 2 , we can thus say that most training samples satisfy eq.

Original: (4.1) locally at the solution.
Revised: (4.1) locally at the solution.

Original: While for the teacher-student training problem, its objective is nonsmooth at solution (when zero residual is attained) since the absolute value function (cid:96) ( x ) := | x | is nonsmooth at 0.
Revised: While for the teacher-student training problem, its objective is nonsmooth at solution (when zero residual is attained) since the absolute value function (cid:96) ( x ) := | x | is nonsmooth at 0.

Original: Therefore the objective of the teacher-student training problem does not satisfy Proposition 4.1 locally as solution and running SSGD to solve it could suffer from slow convergence.
Revised: Therefore the objective of the teacher-student training problem does not satisfy Proposition 4.1 locally as solution and running SSGD to solve it could suffer from slow convergence.

Original:  C ONCLUSION AND DISCUSSION
Revised:  C ONCLUSION AND DISCUSSION

Original: An empirical minimization problem based on composite functions has sufﬁcient structure to allow for a tight convergence analysis that explains the effectiveness of stochastic subgradient descent methods on nonsmooth problems with interpolation.
Revised: An empirical minimization problem based on composite functions has sufﬁcient structure to allow for a tight convergence analysis that explains the effectiveness of stochastic subgradient descent methods on nonsmooth problems with interpolation.

Original: Surprisingly, the complexity bounds O (1 /(cid:15) ) and O (log(1 /(cid:15) )) that we prove under interpolation match those of stochastic gradient descent for smooth functions.
Revised: Surprisingly, the complexity bounds O (1 /(cid:15) ) and O (log(1 /(cid:15) )) that we prove under interpolation match those of stochastic gradient descent for smooth functions.

Original: Our convergence analysis is based on the convexity properties of f .
Revised: Our convergence analysis is based on the convexity properties of f .

Original: As we mention in connection with Theorem 4.5, the strong convexity assumption can be relaxed to weaker conditions, but even these exclude important nonconvex models that appear in neural networks.
Revised: As we mention in connection with Theorem 4.5, the strong convexity assumption can be relaxed to weaker conditions, but even these exclude important nonconvex models that appear in neural networks.

Original: It is still an open problem as to whether a linear convergence rate for subgradient methods under a weaker assumptions, such as the restricted secant inequality (RSI).
Revised: It is still an open problem as to whether a linear convergence rate for subgradient methods under a weaker assumptions, such as the restricted secant inequality (RSI).

Original: In Section 4.3 we proved that the rate O (1 /(cid:15) ) is optimal for subgradient-based methods under interpolation.
Revised: In Section 4.3 we proved that the rate O (1 /(cid:15) ) is optimal for subgradient-based methods under interpolation.

Original: However, this lower bound holds only for subgradient-based algorithms.
Revised: However, this lower bound holds only for subgradient-based algorithms.

Original: Smoothing techniques based on Moreau envelopes (Nesterov, 2005) can sometimes lead to acceleration for nonsmooth optimization, which may be further avenue to explore for obtaining an accelerated SSGD method.
Revised: Smoothing techniques based on Moreau envelopes (Nesterov, 2005) can sometimes lead to acceleration for nonsmooth optimization, which may be further avenue to explore for obtaining an accelerated SSGD method.

Original: A PPENDIX A T HE PROOF OF P ROPOSITION 4. 
Revised: A PPENDIX A T HE PROOF OF P ROPOSITION 4.

Original: Proof of Proposition 4.2.
Revised: Proof of Proposition 4.2.

Original: Given w 1 , w 2 ∈ R d , ∀ i ∈ [ n ] , (cid:107)
Revised: Given w 1 , w 2 ∈ R d , ∀ i ∈ [ n ] , (cid:107)

Original: ∂f
Revised: ∂f

Original: i ( w 2 ) − ∂f i ( w 1 ) (cid:107) ≤ (cid:107) (cid:96) (cid:48)
Revised: i ( w 2 ) − ∂f i ( w 1 ) (cid:107) ≤ (cid:107) (cid:96) (cid:48)

Original: ( h i ( w 2 ))
Revised: ( h i ( w 2 ))

Original: ∂h
Revised: ∂h

Original: i ( w 2 ) − (cid:96)
Revised: i ( w 2 ) − (cid:96)

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where ( i ) follows the same argument as the proof of Proposition 4.1:
Revised: where ( i ) follows the same argument as the proof of Proposition 4.1:

Original: [Equation]
Revised: [Equation]

Original: Exchange w 1 and w 2 , we can also get
Revised: Exchange w 1 and w 2 , we can also get

Original: [Equation]
Revised: [Equation]

Original: Combining Equation (A.2) and Equation (A.1), we ﬁnished the proof for Equation (4.4).
Revised: Combining Equation (A.2) and Equation (A.1), we ﬁnished the proof for Equation (4.4).

Original: Given w 1 , w 2 ∈ R d , note that f i ( w ) is almost every differentiable by Rademacher’s Theorem, thus f i ( w 2 )
Revised: Given w 1 , w 2 ∈ R d , note that f i ( w ) is almost every differentiable by Rademacher’s Theorem, thus f i ( w 2 )

Original: =
Revised: =

Original: f i ( w 1 ) + (cid:90)
Revised: f i ( w 1 ) + (cid:90)

Original: [Equation]
Revised: [Equation]

Original: τ
Revised: τ

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:105) dτ (A
Revised: 1 (cid:105) dτ (A

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Plug Equation (A.4) into Equation (A.3), we ﬁnished the proof for Equation (4.5).
Revised: Plug Equation (A.4) into Equation (A.3), we ﬁnished the proof for Equation (4.5).

Original: B O THER CHOICE OF LEARNING RATE SCHEDULING
Revised: B A NOTHER CHOICE OF LEARNING RATE SCHEDULING

Original: The following theorem is based on the result of Stich (2019)’s analysis on reﬁned learning rate scheduling.
Revised: The following theorem is based on the result of Stich (2019)’s analysis on reﬁned learning rate scheduling.

Revised: Note that the reﬁned learning rate scheduling depends not only on the Lipschitz constant

Revised: L , but also rely on the number of iteration T (we need the number of iterations as an input of our algorithm).

Original: Theorem B.1.
Revised: Theorem B.1.

Original: Assume that f is µ -strongly convex for some µ ≥ 0 .
Revised: Assume that f is µ -strongly convex for some µ ≥ 0 .

Original: There exist a constant learning rate scheduling α t := α ≤ 1 / (2 L 2 ) , such that∀ T > 0 the SSGD iterates w ( t ) satisfy 
Revised: ∀ T > 0 there exist a constant learning rate scheduling α t := α ≤ 1 / (2 L 2 ) , such that the SSGD iterates w ( t ) satisfy min

Original: [Equation]
Revised: [Equation]

Original: for µ > 0 .
Revised: for µ > 0 .

Original: And
Revised: And

Original: [Equation]
Revised: [Equation]

Original: for µ = 0 .
Revised: for µ = 0 .

Original: Proof.
Revised: Proof.

Original: We start with eq.
Revised: We start with eq.

Original: (4.12), eq.

Original: [Equation]
Revised: [Equation]

Original: B
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where (i) is from the fact that 2( α t − α 2 t L 2 ) ≥ α t given that 0 < α t ≤ 1 / (2 L 2 ) .
Revised: where (i) is from the fact that 2( α t − α 2 t L 2 ) ≥ α t given that 0 < α t ≤ 1 / (2 L 2 ) .

Original: Note that eq.
Revised: Note that eq.

Original: (B.3) is in the same form as Eq.
Revised: (B.3) is in the same form as Eq.

Original: (8) from Stich (2019)’s work, then we can directly apply (Stich, 2019,
Revised: (8) from Stich (2019)’s work, then we can directly apply (Stich, 2019,

Original: Lemma 2, Lemma 3) to obtain
Revised: Lemma 2, Lemma 3): there exist a learning rate scheduling such that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Then we can obtain eq.
Revised: Then we can obtain eq.

Original: (B.2) by applying (Stich & Praneeth Karimireddy, 2019, Lemma 13) and therefore completes the proof.
Revised: (B.2) by applying (Stich & Praneeth Karimireddy, 2019, Lemma 13) and therefore completes the proof.

Original: Note that our proof is mostly based on the recurrence relation eq.
Revised: Note that our proof is mostly based on the recurrence relation eq.

Original: (B.3) and Stich (2019)’s analysis, we refer interested readers to Stich (2019)
Revised: (B.3) and Stich (2019)’s analysis, we refer interested readers to Stich (2019)

Original: ’s work for more technical details.
Revised: ’s work for more technical details.