Original: T OWARDS U NDERSTANDING L ABEL S MOOTHING
Revised: T OWARDS U NDERSTANDING L ABEL S MOOTHING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Label smoothing regularization (LSR) is a prevalent component for training deep neural networks and can improve the generalization of models effectively.
Revised: Label smoothing regularization (LSR) is a prevalent component for training deep neural networks and can improve the generalization of models effectively.

Original: Although it achieves empirical success, the theoretical understanding about the power of label smoothing, especially about its inﬂuence on optimization, is still limited.
Revised: Although it achieves empirical success, the theoretical understanding about the power of label smoothing, especially about its inﬂuence on optimization, is still limited.

Original: In this work, we, for the ﬁrst time, theoretically analyze the convergence behaviors of stochastic gradient descent with label smoothing in deep learning.
Revised: In this work, we, for the ﬁrst time, theoretically analyze the convergence behaviors of stochastic gradient descent with label smoothing in deep learning.

Original: Our analysis indicates that an appropriate LSR can speed up the convergence by reducing the variance in gradient, which provides a theoretical interpretation on the effectiveness of LSR.
Revised: Our analysis indicates that an appropriate LSR can speed up the convergence by reducing the variance in gradient, which provides a theoretical interpretation on the effectiveness of LSR.

Original: Besides, the analysis implies that LSR may slow down the convergence at the end of optimization.
Revised: Besides, the analysis implies that LSR may slow down the convergence at the end of optimization.

Original: Therefore, a novel algorithm, namely
Revised: Therefore, a novel algorithm, namely

Original: T wo- S tage LA bel smoothing (TSLA), is proposed to further improve the convergence.
Revised: T wo- S tage LA bel smoothing (TSLA), is proposed to further improve the convergence.

Original: With the extensive analysis and experiments on benchmark data sets, the effectiveness of TSLA is veriﬁed both theoretically and empirically.
Revised: With the extensive analysis and experiments on benchmark data sets, the effectiveness of TSLA is veriﬁed both theoretically and empirically.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Due to the massive number of parameters, over-ﬁtting becomes one of the major challenges in training effective deep neural networks (M ¨ uller et al., 2019).
Revised: Due to the massive number of parameters, over-ﬁtting becomes one of the major challenges in training effective deep neural networks (M ¨ uller et al., 2019).

Original: Various strategies such as weight decay, dropout (Hinton et al., 2012b), batch normalization (Ioffe & Szegedy, 2015), data augmentation (Simard et al., 1998), etc. have been proposed to alleviate the problem.
Revised: Various strategies such as weight decay, dropout (Hinton et al., 2012b), batch normalization (Ioffe & Szegedy, 2015), data augmentation (Simard et al., 1998), etc. have been proposed to alleviate the problem.

Original: Unlike regularization on input data or model weights, label smoothing regularization (LSR) (Szegedy et al., 2016) perturbs labels of data for better generalization.
Revised: Unlike regularization on input data or model weights, label smoothing regularization (LSR) (Szegedy et al., 2016) perturbs labels of data for better generalization.

Original: Concretely, for a K -class classiﬁcation problem, the one-hot label in the prevalent cross entropy loss is replaced by a soft label as y LS =
Revised: Concretely, for a K -class classiﬁcation problem, the one-hot label in the prevalent cross entropy loss is replaced by a soft label as y LS =

Original: (1 − θ )
Revised: (1 − θ )

Original: y
Revised: y

Original: + θ (cid:98) y , where y is the one-hot label and (cid:98)
Revised: + θ (cid:98) y , where y is the one-hot label and (cid:98)

Original: y
Revised: y

Original: = 1 K is a uniform distribution for all labels.
Revised: = 1 K is a uniform distribution for all labels.

Original: By optimizing the soft label instead, LSR helps to improve the generalization of deep neural networks in diverse applications including image classiﬁcation (Zoph et al., 2018;
Revised: By optimizing the soft label instead, LSR helps to improve the generalization of deep neural networks in diverse applications including image classiﬁcation (Zoph et al., 2018;

Original: He et al., 2019), speech recognition (Chorowski &
Revised: He et al., 2019), speech recognition (Chorowski &

Original: Jaitly, 2017; Zeyer et al., 2018), and language translation (Vaswani et al., 2017; Nguyen & Salazar, 2019).
Revised: Jaitly, 2017; Zeyer et al., 2018), and language translation (Vaswani et al., 2017; Nguyen & Salazar, 2019).

Original: Besides, it is complementary to other regularization methods and can work with others to make learned neural networks applicable (He et al., 2019).
Revised: Besides, it is complementary to other regularization methods and can work with others to make learned neural networks applicable (He et al., 2019).

Original: Despite the success of LSR, efforts toward theoretical understanding are still limited.
Revised: Despite the success of LSR, efforts toward theoretical understanding are still limited.

Original: M¨uller et al.
Revised: M¨uller et al.

Original: (2019) have empirically shown that LSR can help improve model calibration.
Revised: (2019) have empirically shown that LSR can help improve model calibration.

Original: However, they also found that LSR could impair knowledge distillation when the learned model is applied as the teacher network.
Revised: However, they also found that LSR could impair knowledge distillation when the learned model is applied as the teacher network.

Original: Yuan et al. (2019a) have proved that LSR provides a virtual teacher model for knowledge distillation.
Revised: Yuan et al. (2019a) have proved that LSR provides a virtual teacher model for knowledge distillation.

Original: As a widely used trick, Lukasik et al.
Revised: As a widely used trick, Lukasik et al.

Original: (2020) have shown that LSR works since it can successfully mitigate label noise.
Revised: (2020) have shown that LSR works since it can successfully mitigate label noise.

Original: Most of existing works try to study LSR as regularizing label space while little work investigates the direct impact of LSR on training deep neural networks.
Revised: Most of existing works try to study LSR as regularizing label space while little work investigates the direct impact of LSR on training deep neural networks.

Original: In this work, we analyze the beneﬁts of LSR for the learning process from the view of optimization.
Revised: In this work, we analyze the beneﬁts of LSR for the learning process from the view of optimization.

Original: Concretely, our theory shows that an appropriate LSR can essentially reduce the variance of stochastic gradient in the assigned class labels and thus it can speed up the convergence of stochastic gradient descent, which is a standard optimizer for deep learning.
Revised: Concretely, our theory shows that an appropriate LSR can essentially reduce the variance of stochastic gradient in the assigned class labels and thus it can speed up the convergence of stochastic gradient descent, which is a standard optimizer for deep learning.

Original: Besides, the analysis implies that LSR may incur slow convergence at the end of optimization.
Revised: Besides, the analysis implies that LSR may incur slow convergence at the end of optimization.

Original: Inspired by the result, a novel algorithm is developed to accelerate the convergence by a two-stage training paradigm.
Revised: Inspired by the result, a novel algorithm is developed to accelerate the convergence by a two-stage training paradigm.

Original: The effectiveness of the proposed method is veriﬁed both theoretically and empirically.
Revised: The effectiveness of the proposed method is veriﬁed both theoretically and empirically.

Original: We summarize the main contributions of this paper as follows.
Revised: We summarize the main contributions of this paper as follows.

Original: • To the best of our knowledge, it is the ﬁrst work that establishes improved iteration complexities of stochastic gradient descent (SGD) (Robbins & Monro, 1951) with LSR for
Revised: • To the best of our knowledge, it is the ﬁrst work that establishes improved iteration complexities of stochastic gradient descent (SGD) (Robbins & Monro, 1951) with LSR for

Original: ﬁnding an (cid:15) -approximate stationary point (Deﬁnition 1) in solving a smooth non-convex problem in the presence of an appropriate label smoothing.
Revised: ﬁnding an (cid:15) -approximate stationary point (Deﬁnition 1) in solving a smooth non-convex problem in the presence of an appropriate label smoothing.

Original: The results theoretically explain why an appropriate LSR can help speed up the convergence.
Revised: The results theoretically explain why an appropriate LSR can help speed up the convergence.

Original: (Subsection 4.1) •
Revised: (Subsection 4.1) •

Original: According to the analysis, we propose a simple yet effective method to accelerate the convergence of LSR (Subsection 4.2).
Revised: According to the analysis, we propose a simple yet effective method to accelerate the convergence of LSR (Subsection 4.2).

Original: It adopts a two-stage training paradigm that applies LSR in the ﬁrst stage and then runs on one-hot labels in the second stage.
Revised: It adopts a two-stage training paradigm that applies LSR in the ﬁrst stage and then runs on one-hot labels in the second stage.

Original: • The theoretical analysis demonstrates that TSLA has an improved iteration complexity (Subsection 4.3) and better generalization (Subsection 4.4) compared to the conventional LSR.
Revised: • The theoretical analysis demonstrates that TSLA has an improved iteration complexity (Subsection 4.3) and better generalization (Subsection 4.4) compared to the conventional LSR.

Original: Extensive experiments on benchmark data sets conﬁrm the effectiveness of our proposed method.
Revised: Extensive experiments on benchmark data sets conﬁrm the effectiveness of our proposed method.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we introduce some related work.
Revised: In this section, we introduce some related work.

Original: A closely related idea to LSR is conﬁdence penalty proposed by Pereyra et al.
Revised: A closely related idea to LSR is conﬁdence penalty proposed by Pereyra et al.

Original: (2017), an output regularizer that penalizes conﬁdent output distributions by adding its negative entropy to the negative log-likelihood during the training process.
Revised: (2017), an output regularizer that penalizes conﬁdent output distributions by adding its negative entropy to the negative log-likelihood during the training process.

Original: The authors (Pereyra et al., 2017) presented extensive experimental results in training deep neural networks to demonstrate better generalization comparing to baselines with only focusing on the existing hyper-parameters.
Revised: The authors (Pereyra et al., 2017) presented extensive experimental results in training deep neural networks to demonstrate better generalization comparing to baselines with only focusing on the existing hyper-parameters.

Original: They have shown that LSR is equivalent to conﬁdence penalty with a reversing direction of KL divergence between uniform distributions and the output distributions.
Revised: They have shown that LSR is equivalent to conﬁdence penalty with a reversing direction of KL divergence between uniform distributions and the output distributions.

Original: DisturbLabel introduced by Xie et al.
Revised: DisturbLabel introduced by Xie et al.

Original: (2016) imposes the regularization within the loss layer, where it randomly replaces some of the ground truth labels as incorrect values at each training iteration.
Revised: (2016) imposes the regularization within the loss layer, where it randomly replaces some of the ground truth labels as incorrect values at each training iteration.

Original: Its effect is quite similar to LSR that can help to prevent the neural network training from overﬁtting.
Revised: Its effect is quite similar to LSR that can help to prevent the neural network training from overﬁtting.

Original: The authors have veriﬁed the effectiveness of DisturbLabel via several experiments on training image classiﬁcation tasks.
Revised: The authors have veriﬁed the effectiveness of DisturbLabel via several experiments on training image classiﬁcation tasks.

Original: Recently, many works (Zhang et al., 2018; Bagherinezhad et al., 2018; Goibert & Dohmatob, 2019;
Revised: Recently, many works (Zhang et al., 2018; Bagherinezhad et al., 2018; Goibert & Dohmatob, 2019;

Original: Shen et al., 2019; Li et al., 2020b) explored the idea of LSR technique.
Revised: Shen et al., 2019; Li et al., 2020b) explored the idea of LSR technique.

Original: Ding et al.
Revised: Ding et al.

Original: (2019) extended an adaptive label regularization method, which enables the neural network to use both correctness and incorrectness during training.
Revised: (2019) extended an adaptive label regularization method, which enables the neural network to use both correctness and incorrectness during training.

Original: Pang et al.
Revised: Pang et al.

Original: (2018) used the reverse cross-entropy loss to smooth the classiﬁer’s gradients.
Revised: (2018) used the reverse cross-entropy loss to smooth the classiﬁer’s gradients.

Original: Wang et al.
Revised: Wang et al.

Original: (2020) proposed a graduated label smoothing method that uses the higher smoothing penalty for high-conﬁdence predictions than that for low-conﬁdence predictions.
Revised: (2020) proposed a graduated label smoothing method that uses the higher smoothing penalty for high-conﬁdence predictions than that for low-conﬁdence predictions.

Original: They found that the proposed method can improve both inference calibration and translation performance for neural machine translation models.
Revised: They found that the proposed method can improve both inference calibration and translation performance for neural machine translation models.

Original: By contrast, we will try to understand the power of LSR from an optimization perspective and try to study how and when to use LSR.
Revised: By contrast, we will try to understand the power of LSR from an optimization perspective and try to study how and when to use LSR.

Original:  P RELIMINARIES AND N OTATIONS
Revised:  P RELIMINARIES AND N OTATIONS

Original: We ﬁrst present some notations.
Revised: We ﬁrst present some notations.

Original: Let ∇ w F S ( w ) denote the gradient of a function F S ( w ) .
Revised: Let ∇ w F S ( w ) denote the gradient of a function F S ( w ) .

Original: When the variable to be taken a gradient is obvious, we use ∇ F S ( w ) for simplicity.
Revised: When the variable to be taken a gradient is obvious, we use ∇ F S ( w ) for simplicity.

Original: We use (cid:107) · (cid:107) to denote the
Revised: We use (cid:107) · (cid:107) to denote the

Original: Euclidean norm.
Revised: Euclidean norm.

Original: Let (cid:104)· , ·(cid:105) be the inner product.
Revised: Let (cid:104)· , ·(cid:105) be the inner product.

Original: In classiﬁcation problem, we aim to seek a classiﬁer to map an example x ∈ X onto one of K labels y ∈ Y ⊂ R K , where y =
Revised: In classiﬁcation problem, we aim to seek a classiﬁer to map an example x ∈ X onto one of K labels y ∈ Y ⊂ R K , where y =

Original: ( y 1 , y 2 , . . .
Revised: ( y 1 , y 2 , . . .

Original: , y K ) is a one-hot label, meaning that y i is “1” for the correct class and “0” for the rest.
Revised: , y K ) is a one-hot label, meaning that y i is “1” for the correct class and “0” for the rest.

Original: Suppose the example-label pairs are drawn from a distribution P , i.e., ( x , y ) ∼ P =
Revised: Suppose the example-label pairs are drawn from a distribution P , i.e., ( x , y ) ∼ P =

Original: ( P x , P y ) .
Revised: ( P x , P y ) .

Original: Let S = { ( x 1 , y 1 ) , . . .
Revised: Let S = { ( x 1 , y 1 ) , . . .

Original: , ( x n , y n ) } denotes a set of n examples drawn from P .
Revised: , ( x n , y n ) } denotes a set of n examples drawn from P .

Original: We denote by E ( x , y )
Revised: We denote by E ( x , y )

Original: [ · ] the expectation that takes over a random variable ( x , y ) .
Revised: [ · ] the expectation that takes over a random variable ( x , y ) .

Original: When the randomness is obvious, we write E[ · ] for simplicity.
Revised: When the randomness is obvious, we write E[ · ] for simplicity.

Original: Our goal is to learn a prediction function f ( w ; x ) : W × X → R K that is as close as possible to y , where w ∈ W is the parameter and W is a closed convex set.
Revised: Our goal is to learn a prediction function f ( w ; x ) : W × X → R K that is as close as possible to y , where w ∈ W is the parameter and W is a closed convex set.

Original: To this end, we want to minimize the following expected loss under P :
Revised: To this end, we want to minimize the following expected loss under P :

Original: [Equation]
Revised: [Equation]

Original: where (cid:96) :
Revised: where (cid:96) :

Original: Y × R K → R + is a cross-entropy loss function given by
Revised: Y × R K → R + is a cross-entropy loss function given by

Original: [Equation]
Revised: [Equation]

Original: The objective function F S ( w ) is not convex since f ( w ; x ) is non-convex in terms of w .
Revised: The objective function F S ( w ) is not convex since f ( w ; x ) is non-convex in terms of w .

Original: To solve the problem (1), one can simply use some iterative methods such as stochastic gradient descent (SGD).
Revised: To solve the problem (1), one can simply use some iterative methods such as stochastic gradient descent (SGD).

Original: Speciﬁcally, at each training iteration t , SGD updates solutions iteratively by w t
Revised: Speciﬁcally, at each training iteration t , SGD updates solutions iteratively by w t

Original: +1 =
Revised: +1 =

Original: w t − η ∇ w (cid:96) ( y t , f ( w t ; x t )) , where η > 0 is a learning rate.
Revised: w t − η ∇ w (cid:96) ( y t , f ( w t ; x t )) , where η > 0 is a learning rate.

Original: Next, we present some notations and assumptions that will be used in the convergence analysis.
Revised: Next, we present some notations and assumptions that will be used in the convergence analysis.

Original: Throughout this paper, we also make the following assumptions for solving the problem (1).
Revised: Throughout this paper, we also make the following assumptions for solving the problem (1).

Original: Assumption 1.
Revised: Assumption 1.

Original: Assume the following conditions hold:
Revised: Assume the following conditions hold:

Original: (i)
Revised: (i)

Original: The stochastic gradient of F S ( w ) is unbiased, i.e., E ( x , y )
Revised: The stochastic gradient of F S ( w ) is unbiased, i.e., E ( x , y )

Original: [ ∇ (cid:96) ( y , f ( w ; x ))]
Revised: [ ∇ (cid:96) ( y , f ( w ; x ))]

Original: = ∇ F S ( w ) , and the variance of stochastic gradient is bounded,
Revised: = ∇ F S ( w ) , and the variance of stochastic gradient is bounded,

Original: i
Revised: i

Original: e., there exists a constant σ 2 > 0 , such that E ( x , y ) (cid:104) (cid:107)∇ (cid:96) ( y , f ( w ; x )) − ∇ F S ( w ) (cid:107) 2 (cid:105) =
Revised: e., there exists a constant σ 2 > 0 , such that E ( x , y ) (cid:104) (cid:107)∇ (cid:96) ( y , f ( w ; x )) − ∇ F S ( w ) (cid:107) 2 (cid:105) =

Original: σ 2 .
Revised: σ 2 .

Original: (ii) F S ( w ) is smooth with an L -Lipchitz continuous gradient, i.e., it is differentiable and there exists a constant L > 

Revised: (ii) F S ( w ) is smooth with an L -Lipchitz continuous gradient, i.e., it is differentiable and there exists a constant L >

Original: such that (cid:107)∇ F S ( w ) − ∇ F S ( u ) (cid:107) ≤ L (cid:107) w − u (cid:107) , ∀ w , u ∈ W .
Revised: such that (cid:107)∇ F S ( w ) − ∇ F S ( u ) (cid:107) ≤ L (cid:107) w − u (cid:107) , ∀ w , u ∈ W .

Original: Remark.
Revised: Remark.

Original: Assumption 1 (i) and (ii) are commonly used assumptions in the literature of non-convex optimization (Ghadimi & Lan, 2013; Yan et al., 2018; Yuan et al., 2019b;
Revised: Assumption 1 (i) and (ii) are commonly used assumptions in the literature of non-convex optimization (Ghadimi & Lan, 2013; Yan et al., 2018; Yuan et al., 2019b;

Original: Wang et al., 2019; Li et al., 2020a).
Revised: Wang et al., 2019; Li et al., 2020a).

Original: Assumption 1 (ii) says the objective function is L -smooth, and it has an equivalent expression (Nesterov, 2004) which is F S ( w ) − F S ( u
Revised: Assumption 1 (ii) says the objective function is L -smooth, and it has an equivalent expression (Nesterov, 2004) which is F S ( w ) − F S ( u

Original: [Equation]
Revised: [Equation]

Original: W .
Revised: W .

Original: For a classiﬁcation problem, the smoothed label y LS is given by
Revised: For a classiﬁcation problem, the smoothed label y LS is given by

Original: [Equation]
Revised: [Equation]

Original: where θ ∈ (0 , 1) is the smoothing strength, y is the one-hot label, (cid:98)
Revised: where θ ∈ (0 , 1) is the smoothing strength, y is the one-hot label, (cid:98)

Original: y is an introduced label.
Revised: y is an introduced label.

Original: For example, one can simply use (cid:98) y
Revised: For example, one can simply use (cid:98) y

Original: = 1 K (Szegedy et al., 2016) for K -class problems.
Revised: = 1 K (Szegedy et al., 2016) for K -class problems.

Original: Similar to label y , we suppose the label (cid:98) y is drawn from a distribution P (cid:98) y .
Revised: Similar to label y , we suppose the label (cid:98) y is drawn from a distribution P (cid:98) y .

Original: We introduce the variance of stochastic gradient using label (cid:98) y as follows.
Revised: We introduce the variance of stochastic gradient using label (cid:98) y as follows.

Original: [Equation]
Revised: [Equation]

Original: where δ > 0 is a constant and σ 2 is deﬁned in Assumption 1 (i).
Revised: where δ > 0 is a constant and σ 2 is deﬁned in Assumption 1 (i).

Original: We make several remarks for (4).
Revised: We make several remarks for (4).

Original: Remark.
Revised: Remark.

Original: (a) We do not require that the stochastic gradient ∇ (cid:96) ( (cid:98) y , f ( w ; x )) is unbiased, i.e., it could be E[ ∇ (cid:96) ( (cid:98) y , f ( w ; x ))]
Revised: (a) We do not require that the stochastic gradient ∇ (cid:96) ( (cid:98) y , f ( w ; x )) is unbiased, i.e., it could be E[ ∇ (cid:96) ( (cid:98) y , f ( w ; x ))]

Original: (cid:54) =
Revised: (cid:54) =

Original: ∇ F S ( w ) .
Revised: ∇ F S ( w ) .

Original: (b) The variance (cid:98) σ 2 is deﬁned based on the label (cid:98) y rather than the smoothed label y LS .
Revised: (b) The variance (cid:98) σ 2 is deﬁned based on the label (cid:98) y rather than the smoothed label y LS .

Original: (c) We do not assume the variance (cid:98) σ 2 is bounded since δ could be an arbitrary value, however, we will discuss the different cases of δ in our analysis.
Revised: (c) We do not assume the variance (cid:98) σ 2 is bounded since δ could be an arbitrary value, however, we will discuss the different cases of δ in our analysis.

Original: If δ ≥ 1 , then (cid:98) σ 2 ≥ σ 2 ; while if 0 < δ < 1 , then (cid:98) σ 2 < σ 2 .
Revised: If δ ≥ 1 , then (cid:98) σ 2 ≥ σ 2 ; while if 0 < δ < 1 , then (cid:98) σ 2 < σ 2 .

Original: It is worth mentioning that δ could be small when an appropriate label is used in the label smoothing.
Revised: It is worth mentioning that δ could be small when an appropriate label is used in the label smoothing.

Original: For example, one can smooth labels by using a teacher model (Hinton et al., 2014) or the model’s own distribution (Reed et al., 2014).
Revised: For example, one can smooth labels by using a teacher model (Hinton et al., 2014) or the model’s own distribution (Reed et al., 2014).

Original: In the ﬁrst paper of label smoothing (Szegedy et al., 2016) and the following related studies (M ¨ uller et al., 2019; Yuan et al., 2019a), researchers consider a uniform distribution over all K classes of labels as the label (cid:98) y , i.e., set (cid:98) y
Revised: In the ﬁrst paper of label smoothing (Szegedy et al., 2016) and the following related studies (M ¨ uller et al., 2019; Yuan et al., 2019a), researchers consider a uniform distribution over all K classes of labels as the label (cid:98) y , i.e., set (cid:98) y

Original: = 1 K .
Revised: = 1 K .

Original: Due to the space limitation, we include more discussions and the evaluations of δ in real-world applications in Appendix A.
Revised: Due to the space limitation, we include more discussions and the evaluations of δ in real-world applications in Appendix A.

Original: For example, we show that δ < 1 for
Revised: For example, we show that δ < 1 for

Original: CIFAR-100 in practice.
Revised: CIFAR-100 in practice.

Original: We now introduce an important property regarding F S ( w ) , i.e. Polyak-Łojasiewicz (PL) condition (Polyak, 1963).
Revised: We now introduce an important property regarding F S ( w ) , i.e. Polyak-Łojasiewicz (PL) condition (Polyak, 1963).

Original: More speciﬁcally, the following assumption holds.
Revised: More speciﬁcally, the following assumption holds.

Original: Assumption 2.
Revised: Assumption 2.

Original: There exists a constant µ > 0 such that 2 µ ( F S ( w ) − F S ( w ∗S )) ≤ (cid:107)∇ F S ( w ) (cid:107) 2 , ∀ w ∈ W , where w ∗S ∈ min w ∈W F S ( w ) is a optimal solution.
Revised: There exists a constant µ > 0 such that 2 µ ( F S ( w ) − F S ( w ∗S )) ≤ (cid:107)∇ F S ( w ) (cid:107) 2 , ∀ w ∈ W , where w ∗S ∈ min w ∈W F S ( w ) is a optimal solution.

Original: Remark.
Revised: Remark.

Original: This property has been theoretically and empirically observed in training deep neural networks (Allen-Zhu et al., 2019; Yuan et al., 2019b).
Revised: This property has been theoretically and empirically observed in training deep neural networks (Allen-Zhu et al., 2019; Yuan et al., 2019b).

Original: This condition is widely used to establish convergence in the literature of non-convex optimization, please see (Yuan et al., 2019b;
Revised: This condition is widely used to establish convergence in the literature of non-convex optimization, please see (Yuan et al., 2019b;

Original: Wang et al., 2019; Karimi et al., 2016; Li & Li, 2018; Charles & Papailiopoulos, 2018; Li et al., 2020a) and references therein.
Revised: Wang et al., 2019; Karimi et al., 2016; Li & Li, 2018; Charles & Papailiopoulos, 2018; Li et al., 2020a) and references therein.

Original: To measure the convergence of non-convex and smooth optimization problems as in (Nesterov, 1998; Ghadimi & Lan, 2013; Yan et al., 2018), we need the following deﬁnition of the ﬁrst-order stationary point.
Revised: To measure the convergence of non-convex and smooth optimization problems as in (Nesterov, 1998; Ghadimi & Lan, 2013; Yan et al., 2018), we need the following deﬁnition of the ﬁrst-order stationary point.

Original: Deﬁnition 1 (First-order stationary point) .
Revised: Deﬁnition 1 (First-order stationary point) .

Original: For the problem of min w ∈W F S ( w ) , a point w ∈ W is called a ﬁrst-order stationary point if (cid:107)∇ F S ( w ) (cid:107) = 0 .
Revised: For the problem of min w ∈W F S ( w ) , a point w ∈ W is called a ﬁrst-order stationary point if (cid:107)∇ F S ( w ) (cid:107) = 0 .

Original: Moreover, if (cid:107)∇ F S ( w ) (cid:107) ≤ (cid:15) , then the point w is said to be an (cid:15) -stationary point, where (cid:15) ∈ (0 , 1) is a small positive value.
Revised: Moreover, if (cid:107)∇ F S ( w ) (cid:107) ≤ (cid:15) , then the point w is said to be an (cid:15) -stationary point, where (cid:15) ∈ (0 , 1) is a small positive value.

Original:  TSLA: A T WO -S TAGE L ABEL S MOOTHING A LGORITHM
Revised:  TSLA: A T WO -S TAGE L ABEL S MOOTHING A LGORITHM

Original: In this section, we present our main method with its convergence analysis.
Revised: In this section, we present our main method with its convergence analysis.

Original: As a warm-up, we ﬁrst show the convergence analysis of SGD with LSR to understand LS from theoretical side.
Revised: As a warm-up, we ﬁrst show the convergence analysis of SGD with LSR to understand LS from theoretical side.

Original: Then we will introduce the proposed TSLA and study its convergence results both in training error (optimization) and testing error (generalization).
Revised: Then we will introduce the proposed TSLA and study its convergence results both in training error (optimization) and testing error (generalization).

Original: 4. C ONVERGENCE A NALYSIS OF SGD WITH LSR
Revised: 4. C ONVERGENCE A NALYSIS OF SGD WITH LSR

Original: To understand LSR from the optimization perspective, we consider SGD with LSR in Algorithmfor the sake of simplicity.
Revised: To understand LSR from the optimization perspective, we consider SGD with LSR in Algorithmfor the sake of simplicity.

Original: The only difference between Algorithm 1 and standard SGD is the use of the output label for constructing a stochastic gradient.
Revised: The only difference between Algorithm 1 and standard SGD is the use of the output label for constructing a stochastic gradient.

Original: The following theorem shows that Algorithm 1 converges to an approximate stationary point in expectation under some conditions.
Revised: The following theorem shows that Algorithm 1 converges to an approximate stationary point in expectation under some conditions.

Original: We include its proof in Appendix C.
Revised: We include its proof in Appendix C.

Original: Theorem 3.
Revised: Theorem 3.

Original: Under Assumption 1, run Algorithm 1 with η = 1 L and θ = 11+ δ , then
Revised: Under Assumption 1, run Algorithm 1 with η = 1 L and θ = 11+ δ , then

Original: E
Revised: E

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: δσ 2 , where R is uniformly sampled from { 0 , 1 , . .
Revised: δσ 2 , where R is uniformly sampled from { 0 , 1 , . .

Original: , T − 1 } .
Revised: , T − 1 } .

Original: Furthermore, given a target accuracy level (cid:15) , we have the following two results.
Revised: Furthermore, given a target accuracy level (cid:15) , we have the following two results.

Original: (1) when δ ≤ (cid:15) 2 4 σ 2 , if we set T = 4 F S
Revised: (1) when δ ≤ (cid:15) 2 4 σ 2 , if we set T = 4 F S

Original: ( w 0 ) η(cid:15), then Algorithm 1 converges to an (cid:15) -stationary point in expectation, i.e., E R
Revised: ( w 0 ) η(cid:15), then Algorithm 1 converges to an (cid:15) -stationary point in expectation, i.e., E R

Original: [ (cid:107)∇ F S ( w R ) (cid:107) 2 ] ≤ (cid:15) 2 .
Revised: [ (cid:107)∇ F S ( w R ) (cid:107) 2 ] ≤ (cid:15) 2 .

Original: The total sample complexity is T
Revised: The total sample complexity is T

Original: =
Revised: =

Original: O (cid:0) 1 (cid:15) 2 (cid:1) .
Revised: O (cid:0) 1 (cid:15) 2 (cid:1) .

Original: (2) when δ > (cid:15) 2 4 σ 2 , if we set T = F S ( w 0 ) ηδσ 2 , then Algorithm 1 does not converge to an (cid:15) -stationary point, but we have E R [ (cid:107)∇ F S ( w R ) (cid:107) 2 ] ≤ 4 δσ 2 ≤ O ( δ ) .
Revised: (2) when δ > (cid:15) 2 4 σ 2 , if we set T = F S ( w 0 ) ηδσ 2 , then Algorithm 1 does not converge to an (cid:15) -stationary point, but we have E R [ (cid:107)∇ F S ( w R ) (cid:107) 2 ] ≤ 4 δσ 2 ≤ O ( δ ) .

Original: Remark.
Revised: Remark.

Original: We observe that the variance term is 2 δσ 2 , instead of ηLσ 2 for standard analysis of
Revised: We observe that the variance term is 2 δσ 2 , instead of ηLσ 2 for standard analysis of

Original: SGD without LSR (i.e., θ = 0 , please see the detailed analysis of Theorem 6 in Appendix D).
Revised: SGD without LSR (i.e., θ = 0 , please see the detailed analysis of Theorem 6 in Appendix D).

Original: For the convergence analysis, the difference between SGD with LSR and SGD without LSR is that ∇ (cid:96) ( (cid:98) y , f ( w ; x )) is not an unbiased estimator of ∇ F S ( w ) when using LSR.
Revised: For the convergence analysis, the difference between SGD with LSR and SGD without LSR is that ∇ (cid:96) ( (cid:98) y , f ( w ; x )) is not an unbiased estimator of ∇ F S ( w ) when using LSR.

Original: The convergence behavior of Algorithm 1 heavily depends on the parameter δ .
Revised: The convergence behavior of Algorithm 1 heavily depends on the parameter δ .

Original: When δ is small enough, say δ ≤
Revised: When δ is small enough, say δ ≤

Original: O ( (cid:15) 2 ) with a small positive value (cid:15) ∈ (0 , 1) , then Algorithm 1 converges to an (cid:15) -stationary point with the total sample complexity of O (cid:0) 1 (cid:15) 2 (cid:1) .
Revised: O ( (cid:15) 2 ) with a small positive value (cid:15) ∈ (0 , 1) , then Algorithm 1 converges to an (cid:15) -stationary point with the total sample complexity of O (cid:0) 1 (cid:15) 2 (cid:1) .

Original: Recall that the total sample complexity of standard
Revised: Recall that the total sample complexity of standard

Original: SGD without LSR for ﬁnding an (cid:15) -stationary point is O (cid:0) 1 (cid:15) 4 (cid:1) ((Ghadimi & Lan, 2016; Ghadimi et al., 2016), please also see the detailed analysis of Theorem 6 in Appendix D).
Revised: SGD without LSR for ﬁnding an (cid:15) -stationary point is O (cid:0) 1 (cid:15) 4 (cid:1) ((Ghadimi & Lan, 2016; Ghadimi et al., 2016), please also see the detailed analysis of Theorem 6 in Appendix D).

Original: The convergence result shows that if we could ﬁnd a label (cid:98) y that has a reasonably small amount of δ , we will be able to reduce sample complexity for training a machine learning model from O (cid:0) 1 (cid:15) 4 (cid:1) to O (cid:0) 1 (cid:15) 2 (cid:1) .
Revised: The convergence result shows that if we could ﬁnd a label (cid:98) y that has a reasonably small amount of δ , we will be able to reduce sample complexity for training a machine learning model from O (cid:0) 1 (cid:15) 4 (cid:1) to O (cid:0) 1 (cid:15) 2 (cid:1) .

Original: Thus, the reduction in variance will happen when an appropriate label smoothing with δ ∈ (0 , 1) is introduced.
Revised: Thus, the reduction in variance will happen when an appropriate label smoothing with δ ∈ (0 , 1) is introduced.

Original: We will ﬁnd in the empirical evaluations that different label (cid:98) y lead to different performances and an appropriate selection of label (cid:98) y has a better performance (see the performances of LSR and LSR-pre in Table 2).
Revised: We will ﬁnd in the empirical evaluations that different label (cid:98) y lead to different performances and an appropriate selection of label (cid:98) y has a better performance (see the performances of LSR and LSR-pre in Table 2).

Original: On the other hand, when the parameter δ is large such that δ > Ω( (cid:15) 2 ) , that is to say, if an inappropriate label smoothing is used, then Algorithm 1 does not converge to an (cid:15) -stationary point, but it converges to a worse level of O ( δ ) , indicating that it may slow down the convergence at the end of optimization.
Revised: On the other hand, when the parameter δ is large such that δ > Ω( (cid:15) 2 ) , that is to say, if an inappropriate label smoothing is used, then Algorithm 1 does not converge to an (cid:15) -stationary point, but it converges to a worse level of O ( δ ) , indicating that it may slow down the convergence at the end of optimization.

Original: 4. T HE TSLA A LGORITHM
Revised: 4. T HE TSLA A LGORITHM

Original: Previous subsection have shown that LSR could not ﬁnd an (cid:15) − stationary point under some situations.
Revised: Previous subsection have shown that LSR could not ﬁnd an (cid:15) − stationary point under some situations.

Original: These motivate us to investigate a strategy that combines the algorithm with and without LSR during the training progress.
Revised: These motivate us to investigate a strategy that combines the algorithm with and without LSR during the training progress.

Original: Let think in this way, one possible scenario is that training one-hot
Revised: Let think in this way, one possible scenario is that training one-hot

Original: Algorithm
Revised: Algorithm

Original: 1: Initialize : w 0 ∈ W , T 1 , θ ∈ (0 , 1) , η 1 , η 2 >//
Revised: 1: Initialize : w 0 ∈ W , T 1 , θ ∈ (0 , 1) , η 1 , η 2 >//

Original: First stage: SGD with LSR 2: for t = 0 , 1 , . . .
Revised: First stage: SGD with LSR 2: for t = 0 , 1 , . . .

Original: , T 1 − 1 do 3: set y LS t = (1 − θ )
Revised: , T 1 − 1 do 3: set y LS t = (1 − θ )

Original: y t + θ (cid:98)
Revised: y t + θ (cid:98)

Original: y t 4: update w t +1 =
Revised: y t 4: update w t +1 =

Original: w t − η 1 ∇ (cid:96) ( y LS t , f ( w t ; x t )) 5: end for //
Revised: w t − η 1 ∇ (cid:96) ( y LS t , f ( w t ; x t )) 5: end for //

Original: Second stage: SGD without LSR 6: for t =
Revised: Second stage: SGD without LSR 6: for t =

Original: T 1 , 1 , . .
Revised: T 1 , 1 , . .

Original: , T 1 + T 2 − 1 do 7: update w t
Revised: , T 1 + T 2 − 1 do 7: update w t

Original: +1 =
Revised: +1 =

Original: w t − η 2 ∇ (cid:96) ( y t , f ( w t ; x t )) .
Revised: w t − η 2 ∇ (cid:96) ( y t , f ( w t ; x t )) .

Original: 8: end for 9: Output: w R , where R is uniformly sampled from { T 1 , . .
Revised: 8: end for 9: Output: w R , where R is uniformly sampled from { T 1 , . .

Original: , T 1 + T 2 − 1 } .
Revised: , T 1 + T 2 − 1 } .

Original: label could be “easier” than training smoothed label 1 .
Revised: label could be “easier” than training smoothed label 1 .

Original: Taking the cross entropy loss in (2) for an example, one need to optimize a single loss function − log (cid:16) exp( f k ( w ; x )) /
Revised: Taking the cross entropy loss in (2) for an example, one need to optimize a single loss function − log (cid:16) exp( f k ( w ; x )) /

Original: (cid:80) Kj =1 exp( f j ( w ; x )) (cid:17) when one-hot label (e.g, y k
Revised: (cid:80) Kj =1 exp( f j ( w ; x )) (cid:17) when one-hot label (e.g, y k

Original: = 

Revised: =

Original: and y i = 

Revised: and y i =

Original: for all i (cid:54)
Revised: for all i (cid:54)

Original: = k ) is used, but need to optimize all K loss functions − (cid:80) Ki
Revised: = k ) is used, but need to optimize all K loss functions − (cid:80) Ki

Original: =

Revised: =

Original: y LS i log (cid:16) exp
Revised: y LS i log (cid:16) exp

Original: ( f i ( w ; x )) /
Revised: ( f i ( w ; x )) /

Original: (cid:80) Kj =1 exp( f j ( w ; x )) (cid:17) when smoothed label (e.g., y LS
Revised: (cid:80) Kj =1 exp( f j ( w ; x )) (cid:17) when smoothed label (e.g., y LS

Original: =
Revised: =

Original: (1 − θ )
Revised: (1 − θ )

Original: y
Revised: y

Original: + θ 1 K so that y LS k = 1 − ( K − 1) θ/K and y LS
Revised: + θ 1 K so that y LS k = 1 − ( K − 1) θ/K and y LS

Original: i = θ/K for all i (cid:54) =
Revised: i = θ/K for all i (cid:54) =

Original: k ) is used.
Revised: k ) is used.

Original: Nevertheless, training deep neural networks is gradually focusing on hard examples with the increase of training epochs.
Revised: Nevertheless, training deep neural networks is gradually focusing on hard examples with the increase of training epochs.

Original: It seems that training with smoothed label in the late epochs makes the learning progress more difﬁcult.
Revised: It seems that training with smoothed label in the late epochs makes the learning progress more difﬁcult.

Original: In addition, after LSR, we focus on optimizing the overall distribution that contains the minor classes, which are probably not important at the end of training progress.
Revised: In addition, after LSR, we focus on optimizing the overall distribution that contains the minor classes, which are probably not important at the end of training progress.

Original: Please see the blue dashed line (trained by SGD with LSR) in Figure 1 2 , and it shows that there is almost no change in loss after 30 epochs.
Revised: Please see the blue dashed line (trained by SGD with LSR) in Figure 1 2 , and it shows that there is almost no change in loss after 30 epochs.

Original: One question is whether LSR helps at the early training epochs but it has less (even negative) effect during the later training epochs?
Revised: One question is whether LSR helps at the early training epochs but it has less (even negative) effect during the later training epochs?

Original: This question encourages us to propose and analyze a simple strategy with LSR dropping that switches a stochastic algorithm with
Revised: This question encourages us to propose and analyze a simple strategy with LSR dropping that switches a stochastic algorithm with

Original: LSR to the algorithm without LSR.
Revised: LSR to the algorithm without LSR.

Original: A natural choice could be just dropping off LSR after certain epochs.
Revised: A natural choice could be just dropping off LSR after certain epochs.

Original: For example, one can run
Revised: For example, one can run

Original: SGD with LSR in the ﬁrst 50 epochs and then run SGD without LSR after that (see red solid line in
Revised: SGD with LSR in the ﬁrst 50 epochs and then run SGD without LSR after that (see red solid line in

Original: Figure 1).
Revised: Figure 1).

Original: From Figure 1, we can see that the drop of LSR after 50 epochs can further decrease the loss function.
Revised: From Figure 1, we can see that the drop of LSR after 50 epochs can further decrease the loss function.

Original: With this in mind in this subsection, we propose a generic framework that consists of two stages, wherein the ﬁrst stage it runs SGD with LSR for T 1 iterations and the second stage it runs SGD without LSR up to T 2 iterations.
Revised: With this in mind in this subsection, we propose a generic framework that consists of two stages, wherein the ﬁrst stage it runs SGD with LSR for T 1 iterations and the second stage it runs SGD without LSR up to T 2 iterations.

Original: This framework is referred to as T wo- S tage LA bel smoothing (TSLA) algorithm, whose updating details are presented in Algorithm 2.
Revised: This framework is referred to as T wo- S tage LA bel smoothing (TSLA) algorithm, whose updating details are presented in Algorithm 2.

Original: Although SGD is considered as the pipeline (Line 4 and Line 8 in Algorithm 2) in the convergence analysis, in practice, SGD can be replaced by any stochastic algorithms such as momentum SGD (Polyak, 1964), Stochastic Nesterov’s
Revised: Although SGD is considered as the pipeline (Line 4 and Line 8 in Algorithm 2) in the convergence analysis, in practice, SGD can be replaced by any stochastic algorithms such as momentum SGD (Polyak, 1964), Stochastic Nesterov’s

Original: Accelerated Gradient (Nesterov, 1983), and adaptive algorithms including A DA G RAD (Duchi et al., 2011), RMSProp (Hinton et al., 2012a), AdaDelta (Zeiler, 2012), Adam (Kingma & Ba, 2015),
Revised: Accelerated Gradient (Nesterov, 1983), and adaptive algorithms including A DA G RAD (Duchi et al., 2011), RMSProp (Hinton et al., 2012a), AdaDelta (Zeiler, 2012), Adam (Kingma & Ba, 2015),

Original: Nadam (Dozat, 2016) and AMSG RAD (Reddi et al., 2018).
Revised: Nadam (Dozat, 2016) and AMSG RAD (Reddi et al., 2018).

Original: In this paper, we will not study the theoretical guarantees and empirical evaluations of other optimizers, which can be considered as future work.
Revised: In this paper, we will not study the theoretical guarantees and empirical evaluations of other optimizers, which can be considered as future work.

Original: Please note that the algorithm can use different learning rates η 1 and η 2 during the two stages.
Revised: Please note that the algorithm can use different learning rates η 1 and η 2 during the two stages.

Original: The last solution of the ﬁrst stage will be used as the initial solution of the second stage.
Revised: The last solution of the ﬁrst stage will be used as the initial solution of the second stage.

Original: If T 1 = 0 , then TSLA reduces to the baseline, i.e., SGD without LSR; while if T 2 = 0 , TSLA becomes to LSR method, i.e., SGD with LSR.
Revised: If T 1 = 0 , then TSLA reduces to the baseline, i.e., SGD without LSR; while if T 2 = 0 , TSLA becomes to LSR method, i.e., SGD with LSR.

Original: 4. O PTIMIZATION R ESULT OF TSLA
Revised: 4. O PTIMIZATION R ESULT OF TSLA

Original: In this subsection, we will give the convergence result of the proposed TSLA algorithm.
Revised: In this subsection, we will give the convergence result of the proposed TSLA algorithm.

Original: For simplicity, we use SGD as the subroutine algorithm A in the analysis.
Revised: For simplicity, we use SGD as the subroutine algorithm A in the analysis.

Original: The convergence result in the following theorem shows the power of LSR from the optimization perspective.
Revised: The convergence result in the following theorem shows the power of LSR from the optimization perspective.

Original: Its proof is presented in Appendix
Revised: Its proof is presented in Appendix

Original: E.
Revised: E.

Original: It is easy to see from the proof that by using the last output of the ﬁrst stage as the initial point of the second stage, TSLA can enjoy the advantage of LSR in the second stage with an improved convergence.
Revised: It is easy to see from the proof that by using the last output of the ﬁrst stage as the initial point of the second stage, TSLA can enjoy the advantage of LSR in the second stage with an improved convergence.

Original: Theorem 4.
Revised: Theorem 4.

Original: Under Assumptions 1, 2, suppose σ 2 δ/µ ≤ F ( w 0 ) , run Algorithm 2, θ = 11+ δ , η 1 = 1 L , T 1 = log (cid:16) 2 µF ( w 0 )(1+ δ ) 2 δσ(cid:17) /
Revised: Under Assumptions 1, 2, suppose σ 2 δ/µ ≤ F ( w 0 ) , run Algorithm 2, θ = 11+ δ , η 1 = 1 L , T 1 = log (cid:16) 2 µF ( w 0 )(1+ δ ) 2 δσ(cid:17) /

Original: ( η 1 µ ) , η 2 = (cid:15) 2 

Revised: ( η 1 µ ) , η 2 = (cid:15) 2

Original: Lσ 2 and T 2 = 8 δσ 2 µη 2 (cid:15) 2 , then E R [ (cid:107)∇ F S ( w R ) (cid:107) 2 ] ≤ (cid:15) 2 , where R is uniformly sampled from { T 1 , . .
Revised: Lσ 2 and T 2 = 8 δσ 2 µη 2 (cid:15) 2 , then E R [ (cid:107)∇ F S ( w R ) (cid:107) 2 ] ≤ (cid:15) 2 , where R is uniformly sampled from { T 1 , . .

Original: , T 1 + T 2 − 1 } .
Revised: , T 1 + T 2 − 1 } .

Original: Remark.
Revised: Remark.

Original: It is obvious that the learning rate η 2 in the second stage is roughly smaller than the learning rate η 1 in the ﬁrst stage, which matches the widely used stage-wise learning rate decay scheme in training neural networks.
Revised: It is obvious that the learning rate η 2 in the second stage is roughly smaller than the learning rate η 1 in the ﬁrst stage, which matches the widely used stage-wise learning rate decay scheme in training neural networks.

Original: To explore the total sample complexity of TSLA, we consider different conditions on δ .
Revised: To explore the total sample complexity of TSLA, we consider different conditions on δ .

Original: For a ﬁxed the target convergence level (cid:15) ∈ (0 , 1) , let us discuss the total sample complexities of ﬁnding (cid:15) -stationary points for SGD with TSLA (TSLA), SGD with LSR (LSR), and SGD without LSR (baseline), where we only consider the orders of the complexities but ignore all constants.
Revised: For a ﬁxed the target convergence level (cid:15) ∈ (0 , 1) , let us discuss the total sample complexities of ﬁnding (cid:15) -stationary points for SGD with TSLA (TSLA), SGD with LSR (LSR), and SGD without LSR (baseline), where we only consider the orders of the complexities but ignore all constants.

Original: When Ω( (cid:15) 2 ) < δ < 1 , LSR does not converge to an (cid:15) -stationary point, while TSLA reduces sample complexity from O (cid:0) 1 (cid:15) 4 (cid:1) to O (cid:0) δ(cid:15) 4 (cid:1) , compared to the baseline.
Revised: When Ω( (cid:15) 2 ) < δ < 1 , LSR does not converge to an (cid:15) -stationary point, while TSLA reduces sample complexity from O (cid:0) 1 (cid:15) 4 (cid:1) to O (cid:0) δ(cid:15) 4 (cid:1) , compared to the baseline.

Original: When δ < O ( (cid:15) 2 ) , the total complexity of TSLA is between log(1 /(cid:15) ) and 1 /(cid:15) 2 , which is always better than
Revised: When δ < O ( (cid:15) 2 ) , the total complexity of TSLA is between log(1 /(cid:15) ) and 1 /(cid:15) 2 , which is always better than

Original: LSR and the baseline.
Revised: LSR and the baseline.

Original: In summary, TSLA achieves the best total sample complexity by enjoying the good property of an appropriate label smoothing (i.e., when 0 < δ < 1 ).
Revised: In summary, TSLA achieves the best total sample complexity by enjoying the good property of an appropriate label smoothing (i.e., when 0 < δ < 1 ).

Original: However, when δ ≥ 1 , baseline has better convergence than TSLA, meaning that the selection of label (cid:98) y is not appropriate.
Revised: However, when δ ≥ 1 , baseline has better convergence than TSLA, meaning that the selection of label (cid:98) y is not appropriate.

Original: Since T 1 contains unknown parameters, it is difﬁculty to know its ground-truth value.
Revised: Since T 1 contains unknown parameters, it is difﬁculty to know its ground-truth value.

Original: However, we can tune different values in practice.
Revised: However, we can tune different values in practice.

Original: 4. G ENERALIZATION R ESULT OF TSLA
Revised: 4. G ENERALIZATION R ESULT OF TSLA

Original: In this subsection, we study the generalization result of TSLA.
Revised: In this subsection, we study the generalization result of TSLA.

Original: First, we give some notations, where most of them are followed by (Hardt et al., 2015; Yuan et al., 2019b).
Revised: First, we give some notations, where most of them are followed by (Hardt et al., 2015; Yuan et al., 2019b).

Original: Let w S =
Revised: Let w S =

Original: A ( S ) be a solution that generated by a random algorithm
Revised: A ( S ) be a solution that generated by a random algorithm

Original: A based on dataset S .
Revised: A based on dataset S .

Original: Recall that problem (1) is called empirical risk minimization in the literature, and the true risk minimization is given by
Revised: Recall that problem (1) is called empirical risk minimization in the literature, and the true risk minimization is given by

Original: [Equation]
Revised: [Equation]

Original: Then the testing error is dedined as
Revised: Then the testing error is dedined as

Original: [Equation]
Revised: [Equation]

Original: We notice that there are several works (Hardt et al., 2015; Yuan et al., 2019b) study the testing error result of SGD for non-convex setting under different conditions such as bounded stochastic gradient (cid:107)∇ w (cid:96) ( y , f ( w ; x ))
Revised: We notice that there are several works (Hardt et al., 2015; Yuan et al., 2019b) study the testing error result of SGD for non-convex setting under different conditions such as bounded stochastic gradient (cid:107)∇ w (cid:96) ( y , f ( w ; x ))

Original: (cid:107) ≤ G and decaying learning rate η t ≤ ct with a constant c > 0 , where t is the optimization iteration.
Revised: (cid:107) ≤ G and decaying learning rate η t ≤ ct with a constant c > 0 , where t is the optimization iteration.

Original: In this paper, we are not interested in establishing fast rate under different conditions, but we want to explore the generalization ability of TSLA with the fewest possible modiﬁcations when building a bridge between theory and practice.
Revised: In this paper, we are not interested in establishing fast rate under different conditions, but we want to explore the generalization ability of TSLA with the fewest possible modiﬁcations when building a bridge between theory and practice.

Original: For example, weight decay is a widely used trick when training deep neural networks.
Revised: For example, weight decay is a widely used trick when training deep neural networks.

Original: With the use of weight decay, the empirical risk minimization in practice becomes min w ∈W { (cid:98) F S ( w ) :
Revised: With the use of weight decay, the empirical risk minimization in practice becomes min w ∈W { (cid:98) F S ( w ) :

Original: = F S ( w ) + λ 2 (cid:107) w (cid:107) 2 } .
Revised: = F S ( w ) + λ 2 (cid:107) w (cid:107) 2 } .

Original: We present the
Revised: We present the

Original: ERB of TSLA in the following Theorem 5, whose proofs can be found in the Appendix F.
Revised: ERB of TSLA in the following Theorem 5, whose proofs can be found in the Appendix F.

Original: Theorem 5.
Revised: Theorem 5.

Original: Under Assumption 1, assume that (cid:96) ( y , f ( w , x )) is L -smooth and B -Lipschitz, suppose (cid:98) F S ( w ) satisﬁes Assumption 2 and min w ∈W (cid:98) F S ( w ) ≤ F S ( w ∗ S )
Revised: Under Assumption 1, assume that (cid:96) ( y , f ( w , x )) is L -smooth and B -Lipschitz, suppose (cid:98) F S ( w ) satisﬁes Assumption 2 and min w ∈W (cid:98) F S ( w ) ≤ F S ( w ∗ S )

Original: + λ 2 (cid:107) w t (cid:107) 2 with λ = 2 L , where w t is the intermediate solution in the second stage of Algorithm 2 by running with θ = 

Revised: + λ 2 (cid:107) w t (cid:107) 2 with λ = 2 L , where w t is the intermediate solution in the second stage of Algorithm 2 by running with θ =

Original: + δ , η 1 < 13 L , T 1 = log (cid:16) 2 µF S ( w 0 )(1+ δ ) 2 δσ(cid:17) / ( η 1 µ ) , η 2 =
Revised: + δ , η 1 < 13 L , T 1 = log (cid:16) 2 µF S ( w 0 )(1+ δ ) 2 δσ(cid:17) / ( η 1 µ ) , η 2 =

Original: O
Revised: O

Original: (1 / √ n ) and T 

Revised: (1 / √ n ) and T

Original: = O ( δn ) then the testing error
Revised: = O ( δn ) then the testing error

Original: E R, A , S [ F ( w R )] − E S
Revised: E R, A , S [ F ( w R )] − E S

Original: [ F S ( w ∗S )] ≤ O (1 / √ n ) , where A is TSLA.
Revised: [ F S ( w ∗S )] ≤ O (1 / √ n ) , where A is TSLA.

Original: Remark.
Revised: Remark.

Original: Theorem 5 shows that in order to have an bound of testing error in the order of O (1 / √ n ) ,
Revised: Theorem 5 shows that in order to have an bound of testing error in the order of O (1 / √ n ) ,

Original: TSLA needs to run T
Revised: TSLA needs to run T

Original: = O ( δn ) iterations.
Revised: = O ( δn ) iterations.

Original: For the standard analysis of SGD with/without LSR (please see the remarks in Appendix F) under the same conditions, SGD without LSR needs to run
Revised: For the standard analysis of SGD with/without LSR (please see the remarks in Appendix F) under the same conditions, SGD without LSR needs to run

Original: T =
Revised: T =

Original: O ( n ) iterations to obtain an O (1 / √ n ) bound of testing error.
Revised: O ( n ) iterations to obtain an O (1 / √ n ) bound of testing error.

Original: Thus, if δ is small enough, the
Revised: Thus, if δ is small enough, the

Original: [Table]
Revised: [Table]

Original: testing error of TSLA is better than that of SGD without LSR, otherwise, SGD without LSR has better testing error than TSLA.
Revised: testing error of TSLA is better than that of SGD without LSR, otherwise, SGD without LSR has better testing error than TSLA.

Original: For SGD with LSR, when δ > Ω(1 / √ n ) , the bound of testing error for LSR can not be bounded by O (1 / √ n ) ; when δ ≤ Ω(1 / √ n ) , the bound for LSR can be bounded by O (1 / √ n ) in T
Revised: For SGD with LSR, when δ > Ω(1 / √ n ) , the bound of testing error for LSR can not be bounded by O (1 / √ n ) ; when δ ≤ Ω(1 / √ n ) , the bound for LSR can be bounded by O (1 / √ n ) in T

Original: = O ( √ n ) iterations.
Revised: = O ( √ n ) iterations.

Original: This shows that TSLA bas better testing error than LSR.
Revised: This shows that TSLA bas better testing error than LSR.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To further evaluate the performance of the proposed TSLA method, we trained deep neural networks on three benchmark data sets, CIFAR-100 (Krizhevsky & Hinton, 2009), Stanford Dogs (Khosla et al., 2011) and CUB-2011 (Wah et al., 2011), for image classiﬁcation tasks.
Revised: To further evaluate the performance of the proposed TSLA method, we trained deep neural networks on three benchmark data sets, CIFAR-100 (Krizhevsky & Hinton, 2009), Stanford Dogs (Khosla et al., 2011) and CUB-2011 (Wah et al., 2011), for image classiﬁcation tasks.

Original: CIFAR-100 3 has 50,000 training images and 10,000 testing images of 32 × 32 resolution with 100 classes.
Revised: CIFAR-100 3 has 50,000 training images and 10,000 testing images of 32 × 32 resolution with 100 classes.

Original: Stanford
Revised: Stanford

Original: Dogs data set 4 contains 20,580 images of 120 breeds of dogs, where 100 images from each breed is used for training.
Revised: Dogs data set 4 contains 20,580 images of 120 breeds of dogs, where 100 images from each breed is used for training.

Original: CUB-2011 5 is a birds image data set with 11,788 images of 200 birds species.
Revised: CUB-2011 5 is a birds image data set with 11,788 images of 200 birds species.

Original: The ResNet-18 model (He et al., 2016) is applied as the backbone in the experiments.
Revised: The ResNet-18 model (He et al., 2016) is applied as the backbone in the experiments.

Original: We compare the proposed TSLA incorporating with SGD (TSLA) with two baselines, SGD with
Revised: We compare the proposed TSLA incorporating with SGD (TSLA) with two baselines, SGD with

Original: LSR (LSR) and SGD without LSR (baseline).
Revised: LSR (LSR) and SGD without LSR (baseline).

Original: The mini-batch size of training instances for all methods is 256 as suggested by (He et al., 2019; 2016).
Revised: The mini-batch size of training instances for all methods is 256 as suggested by (He et al., 2019; 2016).

Original: The momentum parameter is ﬁxed as 0.9.
Revised: The momentum parameter is ﬁxed as 0.9.

Original: We will include more details of experimental settings in Appendix A.
Revised: We will include more details of experimental settings in Appendix A.

Original: 5. S TANFORD D OGS AND CUB- 
Revised: 5. S TANFORD D OGS AND CUB-

Original: We separately train ResNet-18 (He et al., 2016) up to 90 epochs over two data sets Stanford Dogs and CUB-2011.
Revised: We separately train ResNet-18 (He et al., 2016) up to 90 epochs over two data sets Stanford Dogs and CUB-2011.

Original: We use weight decay with the parameter value of 10 − 4 .
Revised: We use weight decay with the parameter value of 10 − 4 .

Original: For all algorithms, the initial learning rates for FC are set to be 0.1, while that for the pre-trained backbones are 0.001 and 0.01 for Standford Dogs and CUB-2011, respectively.
Revised: For all algorithms, the initial learning rates for FC are set to be 0.1, while that for the pre-trained backbones are 0.001 and 0.01 for Standford Dogs and CUB-2011, respectively.

Original: The learning rates are divided by 10 everyepochs.
Revised: The learning rates are divided by 10 everyepochs.

Original: For LSR, we ﬁx the value of smoothing strength θ = 0 .
Revised: For LSR, we ﬁx the value of smoothing strength θ = 0 .

Original: 4 for the best performance, and the label (cid:98) y used for label smoothing is set to be a uniform distribution over all K classes, i.e., (cid:98)
Revised: 4 for the best performance, and the label (cid:98) y used for label smoothing is set to be a uniform distribution over all K classes, i.e., (cid:98)

Original: y
Revised: y

Original: = 1 K .
Revised: = 1 K .

Original: The same values of the smoothing strength θ and the same (cid:98) y are used during the ﬁrst stage of TSLA.
Revised: The same values of the smoothing strength θ and the same (cid:98) y are used during the ﬁrst stage of TSLA.

Original: For TSLA, we drop off the LSR (i.e., let θ = 0 ) after s epochs during the training process, where s ∈ { 20 , 30 , 40 , 50 , 60 , 70 , 80 } .
Revised: For TSLA, we drop off the LSR (i.e., let θ = 0 ) after s epochs during the training process, where s ∈ { 20 , 30 , 40 , 50 , 60 , 70 , 80 } .

Original: We ﬁrst report the highest top-1 and top-5 accuracy on the testing data sets for different methods.
Revised: We ﬁrst report the highest top-1 and top-5 accuracy on the testing data sets for different methods.

Original: All top-1 and top-5 accuracy are averaged over 5 independent random trails with their standard deviations.
Revised: All top-1 and top-5 accuracy are averaged over 5 independent random trails with their standard deviations.

Original: The results of the comparison are summarized in Table 1, where the notation “TSLA( s )” means that the TSLA algorithm drops off LSR after epoch s .
Revised: The results of the comparison are summarized in Table 1, where the notation “TSLA( s )” means that the TSLA algorithm drops off LSR after epoch s .

Original: It can be seen from Table 1 that under an appropriate hyperparameter setting the models trained using TSLA outperform that trained using LSR and baseline, which supports the convergence result in Section 4.2.
Revised: It can be seen from Table 1 that under an appropriate hyperparameter setting the models trained using TSLA outperform that trained using LSR and baseline, which supports the convergence result in Section 4.2.

Original: We notice that the best top-1 accuracy of TSLA are TSLA(40) and TSLA(50) for Stanford Dogs and CUB-2011,
Revised: We notice that the best top-1 accuracy of TSLA are TSLA(40) and TSLA(50) for Stanford Dogs and CUB-2011,

Original: [Figure]
Revised: [Figure]

Original: respectively, meaning that the performance of TSLA( s ) is not monotonic over the dropping epoch s .
Revised: respectively, meaning that the performance of TSLA( s ) is not monotonic over the dropping epoch s .

Original: For CUB-2011, the top-1 accuracy of TSLA(20) is smaller than that of LSR.
Revised: For CUB-2011, the top-1 accuracy of TSLA(20) is smaller than that of LSR.

Original: This result matches the convergence analysis of TSLA showing that it can not drop off LSR too early.
Revised: This result matches the convergence analysis of TSLA showing that it can not drop off LSR too early.

Original: For top-5 accuracy, we found that TSLA(80) is slightly worse than baseline.
Revised: For top-5 accuracy, we found that TSLA(80) is slightly worse than baseline.

Original: This is because of dropping LSR too late so that the update iterations (i.e., T 2 ) in the second stage of TSLA is too small to converge to a good solution.
Revised: This is because of dropping LSR too late so that the update iterations (i.e., T 2 ) in the second stage of TSLA is too small to converge to a good solution.

Original: We also observe that LSR is better than baseline regarding top-1 accuracy but the result is opposite as to top-5 accuracy.
Revised: We also observe that LSR is better than baseline regarding top-1 accuracy but the result is opposite as to top-5 accuracy.

Original: We then plot the averaged top-1 accuracy, averaged top-5 accuracy, and averaged loss among 5 trails of different methods in Figure 2.
Revised: We then plot the averaged top-1 accuracy, averaged top-5 accuracy, and averaged loss among 5 trails of different methods in Figure 2.

Original: We remove the results for TSLA(20) since it dropped off LSR too early as mentioned before.
Revised: We remove the results for TSLA(20) since it dropped off LSR too early as mentioned before.

Original: The ﬁgure shows TSLA improves the top-1 and top-5 testing accuracy immediately once it drops off LSR.
Revised: The ﬁgure shows TSLA improves the top-1 and top-5 testing accuracy immediately once it drops off LSR.

Original: Although TSLA may not converge if it drops off LSR too late, see TSLA(60), TSLA(70), and TSLA(80) from the third column of Figure 2, it still has the best performance compared to LSR and baseline.
Revised: Although TSLA may not converge if it drops off LSR too late, see TSLA(60), TSLA(70), and TSLA(80) from the third column of Figure 2, it still has the best performance compared to LSR and baseline.

Original: TSLA(30), TSLA(40), and TSLA(50) can converge to lower objective levels, comparing to LSR and baseline.
Revised: TSLA(30), TSLA(40), and TSLA(50) can converge to lower objective levels, comparing to LSR and baseline.

Original: 5. CIFAR- 
Revised: 5. CIFAR-

Original: [Figure]
Revised: [Figure]

Original: The total epochs of training ResNet-18 (He et al., 2016) on CIFAR-100 is set to be 200.The weight decay with the parameter value of 5 × 10 − 4 is used.
Revised: The total epochs of training ResNet-18 (He et al., 2016) on CIFAR-100 is set to be 200.The weight decay with the parameter value of 5 × 10 − 4 is used.

Original: We use 0 .
Revised: We use 0 .

Original: 1 as the initial learning rates for all algorithms and divide them by 10 every 60 epochs as suggested in (He et al., 2016; Zagoruyko &
Revised: 1 as the initial learning rates for all algorithms and divide them by 10 every 60 epochs as suggested in (He et al., 2016; Zagoruyko &

Original: Komodakis, 2016).
Revised: Komodakis, 2016).

Original: For LSR and the ﬁrst stage of TSLA, the value of smoothing strength θ is ﬁxed as θ = 0 . 1 , which shows the best performance for LSR.
Revised: For LSR and the ﬁrst stage of TSLA, the value of smoothing strength θ is ﬁxed as θ = 0 . 1 , which shows the best performance for LSR.

Original: We use two different labels (cid:98) y to smooth the one-hot label, the uniform distribution over all labels and the distribution predicted by an ImageNet pre-trained model which is downloaded directly from PyTorch 6 (Paszke et al., 2019).
Revised: We use two different labels (cid:98) y to smooth the one-hot label, the uniform distribution over all labels and the distribution predicted by an ImageNet pre-trained model which is downloaded directly from PyTorch 6 (Paszke et al., 2019).

Original: For TSLA, we try to drop off the LSR after s epochs during the training process, where s ∈ { 120 , 140 , 160 , 180 } .
Revised: For TSLA, we try to drop off the LSR after s epochs during the training process, where s ∈ { 120 , 140 , 160 , 180 } .

Original: [Table]
Revised: [Table]

Original: All top-1 and top-5 accuracy on the testing data set are averaged over 5 independent random trails with their standard deviations.
Revised: All top-1 and top-5 accuracy on the testing data set are averaged over 5 independent random trails with their standard deviations.

Original: We summarize the results in Table 2, where LSR-pre and TSLApre indicate LSR and TSLA use the label (cid:98) y based on the ImageNet pre-trained model
Revised: We summarize the results in Table 2, where LSR-pre and TSLApre indicate LSR and TSLA use the label (cid:98) y based on the ImageNet pre-trained model

Original: [Table]
Revised: [Table]

Original: The results show that LSRpre/TSLA-pre has a better performance than LSR/TSLA.
Revised: The results show that LSRpre/TSLA-pre has a better performance than LSR/TSLA.

Original: The reason might be that the pre-trained model-based prediction is closer to the ground truth than the uniform prediction and it has lower variance (smaller δ ).
Revised: The reason might be that the pre-trained model-based prediction is closer to the ground truth than the uniform prediction and it has lower variance (smaller δ ).

Original: Then, TSLA (LSR) with such pre-trained modelbased prediction converges faster than
Revised: Then, TSLA (LSR) with such pre-trained modelbased prediction converges faster than

Original: TSLA (LSR) with uniform prediction, which veriﬁes our theoretical ﬁndings in
Revised: TSLA (LSR) with uniform prediction, which veriﬁes our theoretical ﬁndings in

Original: Section 4.2 (Section 4.1).
Revised: Section 4.2 (Section 4.1).

Original: This observation also empirically tells us the selection of the prediction function (cid:98) y used for smoothing label is the key to the success of TSLA as well as LSR.
Revised: This observation also empirically tells us the selection of the prediction function (cid:98) y used for smoothing label is the key to the success of TSLA as well as LSR.

Original: Among all methods, the performance of TSLA-pre is the best.
Revised: Among all methods, the performance of TSLA-pre is the best.

Original: For top-1 accuracy,
Revised: For top-1 accuracy,

Original: TSLA-pre(160) outperforms all other algorithms, while for top-5 accuracy, TSLA-pre(180) has the best performance.
Revised: TSLA-pre(160) outperforms all other algorithms, while for top-5 accuracy, TSLA-pre(180) has the best performance.

Original: Finally, we observe from Figure 3 that both TSLA and TSLA-pre converge, while TSLA-pre converges to the lowest objective value.
Revised: Finally, we observe from Figure 3 that both TSLA and TSLA-pre converge, while TSLA-pre converges to the lowest objective value.

Original: Similarly, the top-1 and top-5 accuracies show the improvements of TSLA and TSLA-pre at the point of dropping off LSR.
Revised: Similarly, the top-1 and top-5 accuracies show the improvements of TSLA and TSLA-pre at the point of dropping off LSR.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: We conduct an ablation study for the smoothing parameter θ and the dropping epoch s in TSLA( s ).
Revised: We conduct an ablation study for the smoothing parameter θ and the dropping epoch s in TSLA( s ).

Original: We follow the same settings in Subsection 5.2 but use different values of θ in LSR and TSLA.
Revised: We follow the same settings in Subsection 5.2 but use different values of θ in LSR and TSLA.

Original: Speciﬁcally, θ ∈ { 0 . 2 , 0 . 4 , 0 . 9 } .
Revised: Speciﬁcally, θ ∈ { 0 . 2 , 0 . 4 , 0 . 9 } .

Original: We use the uniform distribution over all labels to smooth the one-hot label.
Revised: We use the uniform distribution over all labels to smooth the one-hot label.

Original: The results are summarized in Table 3, showing that the different values of θ and s can affect the performances of LSR and TSLA.
Revised: The results are summarized in Table 3, showing that the different values of θ and s can affect the performances of LSR and TSLA.

Original: With the increase of θ , the performances become worse.
Revised: With the increase of θ , the performances become worse.

Original: However, with different values of θ ,
Revised: However, with different values of θ ,

Original: TSLA can always outperform LSR when an appropriate dropping epoch s is selected.
Revised: TSLA can always outperform LSR when an appropriate dropping epoch s is selected.

Original: Besides, TSLA(180) has the best performance for each value of θ .
Revised: Besides, TSLA(180) has the best performance for each value of θ .

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we have studied the power of LSR in training deep neural networks by analyzing
Revised: In this paper, we have studied the power of LSR in training deep neural networks by analyzing

Original: SGD with LSR in different non-convex optimization settings.
Revised: SGD with LSR in different non-convex optimization settings.

Original: The convergence results show that an appropriate LSR with reduced label variance can help speed up the convergence.
Revised: The convergence results show that an appropriate LSR with reduced label variance can help speed up the convergence.

Original: We have proposed a simple yet efﬁcient strategy so-called TSLA whose basic idea is to switch the training from smoothed label to one-hot label.
Revised: We have proposed a simple yet efﬁcient strategy so-called TSLA whose basic idea is to switch the training from smoothed label to one-hot label.

Original: Integrating TSLA with SGD, we observe from its improved convergence result that TSLA beneﬁts from LSR in the ﬁrst stage and essentially converges faster in the second stage.
Revised: Integrating TSLA with SGD, we observe from its improved convergence result that TSLA beneﬁts from LSR in the ﬁrst stage and essentially converges faster in the second stage.

Original: Our theoretical result also shows that TSLA has better testing error than that of LSR.
Revised: Our theoretical result also shows that TSLA has better testing error than that of LSR.

Original: Throughout extensive experiments, we have shown that TSLA improves the generalization accuracy of deep models on several benchmark data sets.
Revised: Throughout extensive experiments, we have shown that TSLA improves the generalization accuracy of deep models on several benchmark data sets.

Original: A A DDITIONAL E XPERIMENTS
Revised: A A DDITIONAL E XPERIMENTS

Original: All experiments are implemented using PyTorch 7 with version 1.7.1 on a machine with 8 NVIDIA
Revised: All experiments are implemented using PyTorch 7 with version 1.7.1 on a machine with 8 NVIDIA

Original: P100 GPUs.
Revised: P100 GPUs.

Original: The code is available at https://drive.google.com/drive/folders/1QLvSE6iB2YN7UBpRMKB_ kKQGZzK7M4Qr?usp
Revised: The code is available at https://drive.google.com/drive/folders/1QLvSE6iB2YN7UBpRMKB_ kKQGZzK7M4Qr?usp

Original: =sharing
Revised: =sharing

Revised: A. T HE ESTIMATION OF δ

Original: As mentioned in previsous sections, the value of δ plays a very important role on the theoretical results.
Revised: As mentioned in previsous sections, the value of δ plays a very important role on the theoretical results.

Original: In this section, we provide the estimation of δ for three data sets Stanford Dogs, CUBand CIFAR-100 that used in the experiments.
Revised: In this section, we provide the estimation of δ for three data sets Stanford Dogs, CUBand CIFAR-100 that used in the experiments.

Original: Our results empirically show that δ < 1 for all three data sets.
Revised: Our results empirically show that δ < 1 for all three data sets.

Original: Speciﬁcally, we computed the sample variances (denoted by σ 2 k and (cid:98) σ 2 k , k = 10 , 20 , . . .
Revised: Speciﬁcally, we computed the sample variances (denoted by σ 2 k and (cid:98) σ 2 k , k = 10 , 20 , . . .

Original: ) of two variance σ 2 and (cid:98) σ 2 every 10 epochs, then based on the deﬁnitions in (4), the etimation (denoted by (cid:101) δ ) of δ was taken as
Revised: ) of two variance σ 2 and (cid:98) σ 2 every 10 epochs, then based on the deﬁnitions in (4), the etimation (denoted by (cid:101) δ ) of δ was taken as

Original: [Equation]
Revised: [Equation]

Original: All results of estimated δ are listed in Table 4, empirically showing that (cid:101) δ < 1 .
Revised: All results of estimated δ are listed in Table 4, empirically showing that (cid:101) δ < 1 .

Original: We also plot the estimated values of δ across iterations over three different data sets in Figure 4, showing that (cid:101) δ <for all data sets across iterations.
Revised: We also plot the estimated values of δ across iterations over three different data sets in Figure 4, showing that (cid:101) δ <for all data sets across iterations.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Figure]

Original: [Table]
Revised: More experiments on different models and datasets are included in this subsection. First, we trained models including ResNet-50, VGG (Simonyan & Zisserman, 2014), MobileNet (Howard et al., , Inception (Szegedy et al., 2015) and EfﬁcientNet (Tan & Le, 2019) on three datasets CIFAR100, CUB-2011 and Stanford Dogs.

Original: [Figure]
Revised: We followed the same hyper-parameter setting as in the main paper, except for the smoothing strength θ in LSR.

Original: B T ECHNICAL L EMMA
Revised: We tuned the parameter θ for LSR and used the one with best performance. For TSLA, the same values of smooth strength θ are used during the ﬁrst stage of TSLA. The speciﬁc values of θ for different models and datasets are listed in Table 5. For the simplicity of presentation, we only report the best performance of TSLA among different drop epochs s , where the values of used hyper-parameter s are listed in Table 5.

Revised: [Table]

Revised: All top-1 and top-5 accuracy on the testing datasets are summarized Tables 6, 7 , 8 for CUB-2011,

Revised: Stanford Dogs, and CIFAR-100, respectively.

Revised: Overall, the results show that TSLA has the best performance, comparing with baseline and LSR.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: Besides, we train ResNet-50 over ImageNet with the total epochs of 180.

Revised: The initial learning rate is set to be 0 .

Revised: 2 for all algorithm and divide them by 10 every 60 epochs as suggested in (He et al., 2016;

Revised: Zagoruyko & Komodakis, 2016).

Revised: The mini-batch size of 512 is used.

Revised: The momentum parameter and weight decay are set to be 0 .

Revised: 9 and 10 − 4 respectively.

Revised: We ﬁx the smoothing strength θ = 0 .and

Revised: the drop epoch s

Revised: = 120 for TSLA.

Revised: The top-1 accuracy are presented in Table 9, showing that

Revised: TSLA and LSR are comparable.

Revised: [Table]

Revised: B T ECHNICAL L EMMA

Original: Recall that the optimization problem is
Revised: Recall that the optimization problem is

Original: [Equation]
Revised: [Equation]

Original: where the cross-entropy loss function (cid:96) is given by
Revised: where the cross-entropy loss function (cid:96) is given by

Original: [Equation]
Revised: [Equation]

Revised: (cid:33) .

Revised: [Equation]

Original: If we set 
Revised: If we set p

Original: [Equation]
Revised: [Equation]

Revised: w

Revised: [Equation]

Revised: x ) , . .

Revised: , p K

Revised: [Equation]

Revised: x ))

Revised: [Equation]

Original:  the problem (7) becomes
Revised: (9) the problem (7) becomes

Original: [Equation]
Revised: [Equation]

Original: Then the stochastic gradient with respective to w is
Revised: Then the stochastic gradient with respective to w is

Original: [Equation]
Revised: [Equation]

Original: Lemma 1.
Revised: Lemma 1.

Original: Under Assumption 1 (i), we have
Revised: Under Assumption 1 (i), we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the facts of y LS t = (1 − θ )
Revised: By the facts of y LS t = (1 − θ )

Original: y t + θ (cid:98) y t and the equation in (11), we have 
Revised: y t + θ (cid:98) y t and the equation in (11), we have ∇

Original: [Equation]
Revised: [Equation]

Revised: ∇

Original:  y t , f ( w t ; xt )) .
Revised: [Equation] y [Equation] x

Original: Therefore,
Revised: [Equation] [Equation] ∇

Original: [Equation]
Revised: [Equation]

Original: where (a) uses the convexity of norm, i.e., (cid:107) (1 − θ ) a + θ b (cid:107) 2 ≤ (1 − θ ) (cid:107)
Revised: ( (cid:98) y [Equation] f ( w t ; x t )) .

Original: a
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original:  2 + θ (cid:107) b (cid:107) 2 ; (b) uses assumption 1 (i) and the deﬁnitions in (4), and Assumption 1 (i).
Revised: where (a) uses the convexity of norm, i.e., (cid:107) (1 − θ ) a + θ b (cid:107) 2 ≤ (1 − θ ) (cid:107) a (cid:107) 2 + θ (cid:107) b (cid:107) 2 ; (b) uses assumption 1 (i) and the deﬁnitions in (4), and Assumption 1 (i).

Original: C P ROOF OF T HEOREM
Revised: C P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: By the smoothness of objective function F ( w ) in Assumption 1 (ii) and its remark, we have
Revised: By the smoothness of objective function F ( w ) in Assumption 1 (ii) and its remark

Original: [Equation]
Revised: [Equation]

Original: ≤(cid:104)∇
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: [-η (cid:10) ∇ F S ( w t ) , ∇ (cid:96) ( y LS t , f ( w t ; x t )) (cid:11) + η 2 L 2 (cid:13)(cid:13) ∇ (cid:96) ( y LS t , f ( w t ; x t )) (cid:13)(cid:13)( b ) = − η 
 (cid:107)∇ F S ( w t ) (cid:107) 2 + η 2 (cid:13)(cid:13) ∇ F S ( w t ) − ∇ (cid:96) ( y LS t , f ( w t ; x t )) (cid:13)(cid:13) 2 + η ( ηL − 1)(cid:13)(cid:13) ∇ (cid:96) ( y LS t , f ( w t ; x t ))-]
Revised: [-η (cid:10) ∇ F S ( w t ) , ∇ (cid:96) ( y LS t , f ( w t ; x t )) (cid:11) + η 2 L 2 (cid:13)(cid:13) ∇ (cid:96) ( y LS t , f ( w t ; x t )) (cid:13)(cid:13)( b ) = − η 
 (cid:107)∇ F S ( w t ) (cid:107) 2 + η 2 (cid:13)(cid:13) ∇ F S ( w t ) − ∇ (cid:96) ( y LS t , f ( w t ; x t )) (cid:13)(cid:13) 2 + η ( ηL − 1)(cid:13)(cid:13) ∇ (cid:96) ( y LS t , f ( w t ; x t ))-][Equation]

Original: [-(cid:13)(cid:13)( c ) ≤ − η 
 (cid:107)∇ F S ( w t ) (cid:107) 2 + η 2 (cid:13)(cid:13) ∇ F S ( w t ) − ∇ (cid:96) ( y LS t , f ( w t ; x t )) (cid:13)(cid:13) 2 , (12) where (a) is due to the update of w t +1 ; (b) is due to (cid:104) a , − b (cid:105) = 12 (cid:0) (cid:107) a − b (cid:107) 2 − (cid:107) a (cid:107) 2 − (cid:107) b (cid:107) 2 (cid:1)-] ; (c) is due to η ≤ 1 L .
Revised: [-(cid:13)(cid:13)( c ) ≤ − η 
 (cid:107)∇ F S ( w t ) (cid:107) 2 + η 2 (cid:13)(cid:13) ∇ F S ( w t ) − ∇ (cid:96) ( y LS t , f ( w t ; x t )) (cid:13)(cid:13) 2 , (12) where (a) is due to the update of w t +1 ; (b) is due to (cid:104) a , − b (cid:105) = 12 (cid:0) (cid:107) a − b (cid:107) 2 − (cid:107) a (cid:107) 2 − (cid:107) b (cid:107) 2 (cid:1)-] ; (c) is due to η ≤ 1 L .

Original: Taking the expectation over ( x t , y LS t ) on the both sides of (12), we have
Revised: Taking the expectation over ( x t , y LS t ) on the both sides of (12), we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to Lemma 1.
Revised: where the last inequality is due to Lemma 1.

Original: Then inequality (13) implies
Revised: Then inequality (13) implies

Original: [Equation]
Revised: [Equation]

Original: where (a) is due to θ = 

Revised: where (a) is due to θ =

Original: + δ ; (b) is due to 11+ δ ≤ 1 .
Revised: + δ ; (b) is due to 11+ δ ≤ 1 .

Original: D C ONVERGENCE A NALYSIS OF SGD WITHOUT LSR ( θ = 0 )
Revised: D C ONVERGENCE A NALYSIS OF SGD WITHOUT LSR ( θ = 0 )

Original: Theorem 6.
Revised: Theorem 6.

Original: Under Assumption 1, the solutions w t from Algorithm 1 with θ = 0 satisfy
Revised: Under Assumption 1, the solutions w t from Algorithm 1 with θ = 0 satisfy

Original: [Equation]
Revised: [Equation]

Original: In order to have E R [ (cid:107)∇ F S ( w R ) (cid:107) 2 ] ≤ (cid:15) 2 , it sufﬁces to set η = min (cid:16) 1 L , (cid:15) 2 

Revised: In order to have E R [ (cid:107)∇ F S ( w R ) (cid:107) 2 ] ≤ (cid:15) 2 , it sufﬁces to set η = min (cid:16) 1 L , (cid:15) 2

Original: Lσ 2 (cid:17) and T = 4 F S ( w 0 ) η(cid:15), the total complexity is O (cid:0) 1 (cid:15) 4 (cid:1) .
Revised: Lσ 2 (cid:17) and T = 4 F S ( w 0 ) η(cid:15), the total complexity is O (cid:0) 1 (cid:15) 4 (cid:1) .

Original: Proof.
Revised: Proof.

Original: By the smoothness of objective function F S ( w ) in Assumption 1 (ii) and its remark, we have
Revised: By the smoothness of objective function F S ( w ) in Assumption 1 (ii) and its remark, we have

Original: [Equation]
Revised: [Equation]

Original: where (a) is due to the update of w t +1 .
Revised: where (a) is due to the update of w t +1 .

Original: Taking the expectation over ( x t ; y t ) on the both sides of (14)
Revised: Taking the expectation over ( x t ; y t ) on the both sides of (14), we have

Original: [Equation]
Revised: [Equation]

Original: we have

Original: [Equation]
Revised: [Equation]

Original: (15) where (a) and (b) use Assumption 1 (i); (c) uses the facts that η ≤ 1 L and Assumption 1 (i).
Revised: where (a) and (b) use Assumption 1 (i); (c) uses the facts that η ≤ 1 L and Assumption 1 (i).

Original: The inequality (15) implies
Revised: The inequality (15) implies

Original: [Equation]
Revised: [Equation]

Original: By setting η ≤ {+(cid:15) 2 
+}
Revised: By setting η ≤ {+(cid:15) 2 
+}

Revised: Lσ 2 and T = 4 F S ( w 0 ) η(cid:15), we have 1 T

Original: [Equation]
Revised: [Equation]

Original: 2 Lσ
Revised: −

Original: [Equation]
Revised: [Equation]

Original: and T =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: S ( w 0 ) η(cid:15), we have 1 T (cid:80) T − 1 t =0 E (cid:2) (cid:107)∇ F S ( w t ) (cid:107) 2 (cid:3) ≤ (cid:15) 2 . Thus the total complexity is in the order of O (cid:16) 1 η(cid:15) 2 (cid:17) =
Revised: w t ) (cid:107) 2 (cid:3) ≤ (cid:15) 2 . Thus the total complexity is in the order of O (cid:16) 1 η(cid:15) 2 (cid:17) =

Original: O (cid:0) 1 (cid:15) 4 (cid:1) .
Revised: O (cid:0) 1 (cid:15) 4 (cid:1) .

Original: E P ROOF OF T HEOREM
Revised: E P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Following the similar analysis of inequality (13) from the proof of Theorem 3, we have
Revised: Following the similar analysis of inequality (13) from the proof of Theorem 3, we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Using the condition in Assumption 2 we can simplify the inequality from (16) as
Revised: Using the condition in Assumption 2 we can simplify the inequality from (16) as

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to the deﬁnition of loss function that F S ( w ∗S ) ≥ 0 .
Revised: where the last inequality is due to the deﬁnition of loss function that F S ( w ∗S ) ≥ 0 .

Original: Since η 1 ≤ 1 L < 1 µ , then (1 − η 1 µ ) t +1 < exp( − η 1 µ ( t + 1)) and (cid:80) ti =

Revised: Since η 1 ≤ 1 L < 1 µ , then (1 − η 1 µ ) t +1 < exp( − η 1 µ ( t + 1)) and (cid:80) ti =

Original: (1 − η 1 µ ) i ≤ 1 η 1 µ .
Revised: (1 − η 1 µ ) i ≤ 1 η 1 µ .

Original: As a result, for any T 1 , we have
Revised: As a result, for any T 1 , we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Let θ = 11+ δ and (cid:98) σ 2 :=
Revised: Let θ = 11+ δ and (cid:98) σ 2 :=

Original: (1 − θ ) σ 2 + θδσ 2 = 2 δ 1+ δ σ 2 then 12 µ (cid:0) (1 − θ ) σ 2 + θδσ [Equation] (cid:1) =[Equation]
Revised: (1 − θ ) σ 2 + θδσ 2 = 2 δ 1+ δ σ 2 then 12 µ (cid:0) (1 − θ ) σ 2 + θδσ 2 (cid:1) =

Original:  µ [Equation] + δ ) ≤
Revised: δσ 2 µ (1+ δ ) ≤

Original: F S ( w 0 ) since δ is small enough.
Revised: F S ( w 0 ) since δ is small enough.

Original: By setting
Revised: By setting

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: After T 1 iterations [Equation] we drop off the label smoothing, i
Revised: After T 1 iterations, we drop off the label smoothing, i

Original: [Equation]
Revised: [Equation]

Original: e. θ = 0 , then we know for any t ≥ T 1 , following the inequality (15) from the proof of Theorem [Equation] we have
Revised: e. θ = 0 , then we know for any t ≥ T 1 , following the inequality (15) from the proof of Theorem 6, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore [Equation] we get
Revised: Therefore, we get

Original: [Equation]
Revised: [Equation]

Original: (19) where (a) is due to F S ( w T 1 + T 2 − 1 ) ≥ F S ( w ∗S ) 
Revised: 19) where (a) is due to F S ( w T 1 + T 2 − 1 ) ≥ F S ( w ∗S ) .

Original: [Equation]

Original: By setting η 2 =
Revised: By setting η 2 =

Original: [Equation]
Revised: [Equation]

Original: 2 2 Lσ 2 and T 2 = 8 δσ 2 µη 2 (cid:15) 2 , we have
Revised: 2 2 Lσ 2 and T 2 = 8 δσ 2 µη 2 (cid:15) 2 , we have

Original: [Equation]
Revised: [Equation]

Original: F G ENERALIZATION A NALYSIS
Revised: F G ENERALIZATION A NALYSIS

Original: In this section
Revised: In this section

Original: [Equation]
Revised: [Equation]

Original: let
Revised: let

Original: [Equation]
Revised: [Equation]

Original: Following 
Revised: Following by Yuan et al.

Original: . (2019b), we  following 
Revised: (2019b), we use the following decomposition of testing error.

Original: [Equation]
Revised: [Equation]

Original: Let deﬁne the gradient update rule G (cid:98) (cid:96),η as follows
Revised: Let deﬁne the gradient update rule G (cid:98) (cid:96),η as follows

Original: [Equation]
Revised: [Equation]

Original: Then we have the following lemma, which is similar to Lemma 2.5 and Lemma 4.2 in Hardt et al.
Revised: Then we have the following lemma, which is similar to Lemma 2.5 and Lemma 4.2 in Hardt et al.

Original: Lemma 2.
Revised: Lemma 2.

Original: Assume that (cid:96) ( y , f ( w , x )) is L -smooth and B -Lipschitz.
Revised: Assume that (cid:96) ( y , f ( w , x )) is L -smooth and B -Lipschitz.

Original: Let w t
Revised: Let w t

Original: +

Revised: +

Original: =
Revised: =

Original: G ( w t ) and w (cid:48) t +1 =
Revised: G ( w t ) and w (cid:48) t +1 =

Original: G (cid:48) ( w (cid:48) t ) , then(cid:107)
Revised: G (cid:48) ( w (cid:48) t ) , then

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Recall that (cid:98) (cid:96) ( y i , f ( w ; x i )) =
Revised: Recall that (cid:98) (cid:96) ( y i , f ( w ; x i ) [Equation] =

Revised: [Equation]

Original: (cid:96) ( y i , f ( w ; x i))
Revised: ( y i [Equation] [Equation] x i

Revised: [Equation]

Original:  + λ 2 (cid:107) w (cid:107) 2 .
Revised: ) + λ [Equation] w (cid:107) 2 .

Original: Then (1) When G = G (cid:48) and (cid:96) ( · , f ( w ; · )) is L -smooth, then (cid:107) w t
Revised: Then (1) When G = G (cid:48) and (cid:96) ( · , f ( w ; · )) is L -smooth, then (cid:107) w t

Original: +1 − w (cid:48)
Revised: +1 − w (cid:48)

Original: t +1 (cid:107)

Original:  ≤ (1 − ηλ  (cid:107) w t − w (cid:48) t (cid:107) + η (cid:107)∇ w (cid:96) ( y t , f ( 
Revised: t +1 (cid:107) ≤ (1 − ηλ ) (cid:107) w t − w (cid:48) t (cid:107) + η (cid:107)∇ w (cid:96) ( y t , f ( w t ; x t ))

Original: w

Original: t ; x t )) − ∇ w (cid:96) ( y t , f ( w (cid:48) t ; x t ))
Revised: − ∇ w (cid:96) ( y t , f ( w (cid:48) t ; x t ))

Original: (cid:107) ≤ (1 + ηL − ηλ ) (cid:107) w t − w (cid:48) t (cid:107) 
Revised: (cid:107) ≤ (1 + ηL − ηλ ) (cid:107) w t − w (cid:48) t (cid:107) .

Original: (2) When G (cid:54) =
Revised: (2) When G (cid:54) =

Original: G (cid:48) and (cid:96) ( · , f ( w ; · )) is B -Lipschitz, then (cid:107)
Revised: G (cid:48) and (cid:96) ( · , f ( w ; · )) is B -Lipschitz, then (cid:107)

Original: w t
Revised: w t

Original: +1 − w (cid:48)
Revised: +1 − w (cid:48)

Original: t +1 (cid:107) ≤ (1 − ηλ ) (cid:107) w t − w (cid:48) t (cid:107) + η (cid:107)∇ w (cid:96) ( y t , f ( w t ; x t ))
Revised: t +1 (cid:107) ≤ (1 − ηλ ) (cid:107) w t − w (cid:48) t (cid:107) + η (cid:107)∇ w (cid:96) ( y t , f ( w t ; x t ))

Original: − ∇ w (cid:96) ( y (cid:48) t , f ( w (cid:48) t ; x (cid:48) t ))
Revised: − ∇ w (cid:96) ( y (cid:48) t , f ( w (cid:48) t ; x (cid:48) t ))

Original: (cid:107) ≤ [Equation] ηλ ) (cid:107) 
Revised: (cid:107) ≤ (1 − ηλ ) (cid:107) w t − w (cid:48) t

Original: w t − w (cid:48) t (cid:107) + 2 ηB.
Revised: (cid:107) + 2 ηB.

Original:  ROOF 
Revised: F. P ROOF OF T HEOREM

Original: T HEOREM

Original: Proof.
Revised: Proof.

Original: By Theorem 3.2 of Hardt et al.
Revised: By Theorem 3.2 of Hardt et al.

Original: [Equation] we have
Revised: (2015), we have

Original: [Equation]
Revised: [Equation]

Original: where (a) uses Lemma 2; (b) uses λ = 2 L ; (c) uses ∆ t 0 = 0 ; (d) uses ηL < 1 .
Revised: where (a) uses Lemma 2; (b) uses λ = 2 L ; (c) uses ∆ t 0 = 0 ; (d) uses ηL < 1 .

Original: Then by Lemma 3.11 of Hardt et al.
Revised: Then by Lemma 3.11 of Hardt et al.

Original: (2015) and (cid:98) (cid:96) is L + λ -smooth, we have
Revised: (2015) and (cid:98) (cid:96) is L + λ -smooth, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality holds by selecting t 0 = 1 and λ = 2 L .
Revised: where the last inequality holds by selecting t 0 = 1 and λ = 2 L .

Original: Then by Theorem 2.2 of Hardt et al.
Revised: Then by Theorem 2.2 of Hardt et al.

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: which is the generalization error.
Revised: which is the generalization error.

Original: Next, we will bound the optimization error.
Revised: Next, we will bound the optimization error.

Original: Since (cid:98) F S is ( L + λ ) - smooth and satisﬁes µ -PL condition, then follow the similar analysis of Theorem 4, we have
Revised: Since (cid:98) F S is ( L + λ ) - smooth and satisﬁes µ -PL condition, then follow the similar analysis of Theorem 4, we have

Original: [Equation]
Revised: [Equation]

Original: Then by (22), (26) and (27), we get
Revised: Then by (22), (26) and (27), we get

Original: [Equation]
Revised: [Equation]

Original:  By using the conditions that min w ∈W (cid:98) F S ( w ) ≤ F S ( w ∗ S) + λ 2 (cid:107) w t (cid:107) 2 , deﬁnitions (20) and (21), we get
Revised: [Equation] By using the conditions that min w ∈W (cid:98) F S ( w ) ≤ [Equation] ∗ S

Revised: [Equation]

Revised: + λ 2 (cid:107) w

Original: [Equation]
Revised: [Equation]

Revised: (cid:107) 2 , deﬁnitions (20) and (21), we get

Revised: [Equation]

Original: Therefore, we have the following inequality by (28) and (29):
Revised: [Equation] Therefore [Equation] we have the following inequality by (28) and (29):

Original: [Equation]
Revised: [Equation]

Original: By setting η 2 =
Revised: By setting η 2 =

Original: O (1 / √ n ) and T 

Revised: O (1 / √ n ) and T

Original: = O ( δn ) then the testing error E R, A , S [ F ( w R )] − E S
Revised: = O ( δn ) then the testing error E R, A , S [ F ( w R )] − E S

Original: [ F S ( w ∗S )] ≤
Revised: [ F S ( w ∗S )] ≤

Original: O (1 / √ n ) , where A is TSLA.
Revised: O (1 / √ n ) , where A is TSLA.

Original: Remark 1.
Revised: Remark 1.

Original: (SGD without LSR)
Revised: (SGD without LSR)

Original: Following the similar analysis of Theorem 5 and similar analysis of Theorem 6, we have the testing error for SGD without LSR is
Revised: Following the similar analysis of Theorem 5 and similar analysis of Theorem 6, we have the testing error for SGD without LSR is

Original: [Equation]
Revised: [Equation]

Original: By setting η = O (1 / √ n ) and T = O ( n ) then E R, A , S [ F ( w R )] − E S [ F S ( w ∗S )] ≤ O (1 / √ n ) , where
Revised: By setting η = O (1 / √ n ) and T = O ( n ) then E R, A , S [ F ( w R )] − E S [ F S ( w ∗S )] ≤ O (1 / √ n ) , where

Original: A is SGD without LSR.
Revised: A is SGD without LSR.

Original: Remark 2.
Revised: Remark 2.

Original: (SGD with LSR)
Revised: (SGD with LSR)

Original: Following the similar analysis of Theorem 5 and similar analysis of
Revised: Following the similar analysis of Theorem 5 and similar analysis of

Original: Theorem 3, we have the testing error for SGD with LSR is
Revised: Theorem 3, we have the testing error for SGD with LSR is

Original: [Equation]
Revised: [Equation]

Original: where A is SGD with LSR.
Revised: where A is SGD with LSR.

Original: If δ > Ω(1 / √ n ) , the testing error can not be bounded by O (1 / √ n ) ; if δ ≤ Ω(1 / √ n ) , the testing error can be bounded by O (1 / √ n ) in T
Revised: If δ > Ω(1 / √ n ) , the testing error can not be bounded by O (1 / √ n ) ; if δ ≤ Ω(1 / √ n ) , the testing error can be bounded by O (1 / √ n ) in T

Original: = O ( √ n ) iterations.
Revised: = O ( √ n ) iterations.