Original: T OWARDS U NDERSTANDING L ABEL S MOOTHING
Revised: T OWARDS U NDERSTANDING L ABEL S MOOTHING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Label smoothing regularization (LSR) has a great success in training deep neural networks by stochastic algorithms such as stochastic gradient descent and its variants.
Revised: Label smoothing regularization (LSR) has a great success in training deep neural networks by stochastic algorithms such as stochastic gradient descent and its variants.

Original: However, the theoretical understanding of its power from the view of optimization is still rare.
Revised: However, the theoretical understanding of its power from the view of optimization is still rare.

Original: This study opens the door to a deep understanding of
Revised: This study opens the door to a deep understanding of

Original: LSR by initiating the analysis.
Revised: LSR by initiating the analysis.

Original: In this paper, we analyze the convergence behaviors of stochastic gradient descent with label smoothing regularization for solving non-convex problems and show that an appropriate LSR can help to speed up the convergence by reducing the variance.
Revised: In this paper, we analyze the convergence behaviors of stochastic gradient descent with label smoothing regularization for solving non-convex problems and show that an appropriate LSR can help to speed up the convergence by reducing the variance.

Original: More interestingly, we proposed a simple yet effective strategy, namely T wo- S tage LA bel smoothing algorithm (TSLA), that uses LSR in the early training epochs and drops it off in the later training epochs.
Revised: More interestingly, we proposed a simple yet effective strategy, namely T wo- S tage LA bel smoothing algorithm (TSLA), that uses LSR in the early training epochs and drops it off in the later training epochs.

Original: We observe from the improved convergence result of TSLA that it beneﬁts from LSR in the ﬁrst stage and essentially converges faster in the second stage.
Revised: We observe from the improved convergence result of TSLA that it beneﬁts from LSR in the ﬁrst stage and essentially converges faster in the second stage.

Original: To the best of our knowledge, this is the ﬁrst work for understanding the power of LSR via establishing convergence complexity of stochastic methods with LSR in non-convex optimization.
Revised: To the best of our knowledge, this is the ﬁrst work for understanding the power of LSR via establishing convergence complexity of stochastic methods with LSR in non-convex optimization.

Original: We empirically demonstrate the effectiveness of the proposed method in comparison with baselines on training ResNet models over benchmark data sets.
Revised: We empirically demonstrate the effectiveness of the proposed method in comparison with baselines on training ResNet models over benchmark data sets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In training deep neural networks, one common strategy is to minimize cross-entropy loss with onehot label vectors, which may lead to overﬁtting during the training progress that would lower the generalization accuracy (M ¨ uller et al., 2019).
Revised: In training deep neural networks, one common strategy is to minimize cross-entropy loss with onehot label vectors, which may lead to overﬁtting during the training progress that would lower the generalization accuracy (M ¨ uller et al., 2019).

Original: To overcome the overﬁtting issue, several regularization techniques such as (cid:96) 1 -norm or (cid:96) 2 -norm penalty over the model weights, Dropout which randomly sets the outputs of neurons to zero (Hinton et al., 2012b), batch normalization (Ioffe &
Revised: To overcome the overﬁtting issue, several regularization techniques such as (cid:96) 1 -norm or (cid:96) 2 -norm penalty over the model weights, Dropout which randomly sets the outputs of neurons to zero (Hinton et al., 2012b), batch normalization (Ioffe &

Original: Szegedy, 2015), and data augmentation (Simard et al., 1998), are employed to prevent the deep learning models from becoming over-conﬁdent.
Revised: Szegedy, 2015), and data augmentation (Simard et al., 1998), are employed to prevent the deep learning models from becoming over-conﬁdent.

Original: However, these regularization techniques conduct on the hidden activations or weights of a neural network.
Revised: However, these regularization techniques conduct on the hidden activations or weights of a neural network.

Original: As an output regularizer, label smoothing regularization (LSR) (Szegedy et al., 2016) is proposed to improve the generalization and learning efﬁciency of a neural network by replacing the one-hot vector labels with the smoothed labels that average the hard targets and the uniform distribution of other labels.
Revised: As an output regularizer, label smoothing regularization (LSR) (Szegedy et al., 2016) is proposed to improve the generalization and learning efﬁciency of a neural network by replacing the one-hot vector labels with the smoothed labels that average the hard targets and the uniform distribution of other labels.

Original: Speciﬁcally, for a Kclass classiﬁcation problem, the one-hot label is smoothed by y LS =
Revised: Speciﬁcally, for a Kclass classiﬁcation problem, the one-hot label is smoothed by y LS =

Original: (1 − θ )
Revised: (1 − θ )

Original: y
Revised: y

Original: + θ (cid:98) y , where y is the one-hot label, θ ∈ (0 , 1) is the smoothing strength and (cid:98) y
Revised: + θ (cid:98) y , where y is the one-hot label, θ ∈ (0 , 1) is the smoothing strength and (cid:98) y

Original: = 1 K is a uniform distribution for all labels.
Revised: = 1 K is a uniform distribution for all labels.

Original: Extensive experimental results have shown that LSR has signiﬁcant successes in many deep learning applications including image classiﬁcation (Zoph et al., 2018;
Revised: Extensive experimental results have shown that LSR has signiﬁcant successes in many deep learning applications including image classiﬁcation (Zoph et al., 2018;

Original: He et al., 2019), speech recognition (Chorowski & Jaitly, 2017; Zeyer et al., 2018), and language translation (Vaswani et al., 2017; Nguyen & Salazar, 2019).
Revised: He et al., 2019), speech recognition (Chorowski & Jaitly, 2017; Zeyer et al., 2018), and language translation (Vaswani et al., 2017; Nguyen & Salazar, 2019).

Original: Due to the importance of LSR, researchers try to explore its behavior in training deep neural networks.
Revised: Due to the importance of LSR, researchers try to explore its behavior in training deep neural networks.

Original: M¨uller et al.
Revised: M¨uller et al.

Original: (2019) have empirically shown that the LSR can help improve model calibration, however, they also have found that LSR could impair knowledge distillation, that is, if one trains a teacher model with LSR, then a student model has worse performance.
Revised: (2019) have empirically shown that the LSR can help improve model calibration, however, they also have found that LSR could impair knowledge distillation, that is, if one trains a teacher model with LSR, then a student model has worse performance.

Original: Yuan et al. (2019a) have proved that LSR provides a virtual teacher model for knowledge distillation.
Revised: Yuan et al. (2019a) have proved that LSR provides a virtual teacher model for knowledge distillation.

Original: As a widely used trick,
Revised: As a widely used trick,

Original: Lukasik et al.
Revised: Lukasik et al.

Original: (2020) have shown that LSR works since it can successfully mitigate label noise.
Revised: (2020) have shown that LSR works since it can successfully mitigate label noise.

Original: However, to the best of our knowledge, it is unclear, at least from a theoretical viewpoint, how the introduction of label smoothing will help improve the training of deep learning models, and to what stage, it can help.
Revised: However, to the best of our knowledge, it is unclear, at least from a theoretical viewpoint, how the introduction of label smoothing will help improve the training of deep learning models, and to what stage, it can help.

Original: In this paper, we aim to provide an afﬁrmative answer to this question and try to deeply understand why and how the LSR works from the view of optimization.
Revised: In this paper, we aim to provide an afﬁrmative answer to this question and try to deeply understand why and how the LSR works from the view of optimization.

Original: Our theoretical analysis will show that an appropriate LSR can essentially reduce the variance of stochastic gradient in the assigned class labels and thus it can speed up the convergence.
Revised: Our theoretical analysis will show that an appropriate LSR can essentially reduce the variance of stochastic gradient in the assigned class labels and thus it can speed up the convergence.

Original: Moreover, we will propose a novel strategy of employing LSR that tells when to use LSR.
Revised: Moreover, we will propose a novel strategy of employing LSR that tells when to use LSR.

Original: We summarize the main contributions of this paper as follows.
Revised: We summarize the main contributions of this paper as follows.

Original: • It is the ﬁrst work that establishes improved iteration complexities of stochastic gradient descent (SGD) (Robbins & Monro, 1951) with LSR for ﬁnding an (cid:15) -approximate stationary point (Deﬁnition 1) in solving a smooth non-convex problem in the presence of an appropriate label smoothing.
Revised: • It is the ﬁrst work that establishes improved iteration complexities of stochastic gradient descent (SGD) (Robbins & Monro, 1951) with LSR for ﬁnding an (cid:15) -approximate stationary point (Deﬁnition 1) in solving a smooth non-convex problem in the presence of an appropriate label smoothing.

Original: The results theoretically explain why an appropriate LSR can help speed up the convergence.
Revised: The results theoretically explain why an appropriate LSR can help speed up the convergence.

Original: (Section 4) •
Revised: (Section 4) •

Original: We propose a simple yet effective strategy, namely T wo- S tage LA bel smoothing (TSLA) algorithm, where in the ﬁrst stage it trains models for certain epochs using a stochastic method with LSR while in the second stage it runs the same stochastic method without LSR.
Revised: We propose a simple yet effective strategy, namely T wo- S tage LA bel smoothing (TSLA) algorithm, where in the ﬁrst stage it trains models for certain epochs using a stochastic method with LSR while in the second stage it runs the same stochastic method without LSR.

Original: The proposed TSLA is a generic strategy that can incorporate many stochastic algorithms.
Revised: The proposed TSLA is a generic strategy that can incorporate many stochastic algorithms.

Original: With an appropriate label smoothing, we show that TSLA integrated with SGD has an improved iteration complexity, compared to the SGD with LSR and the SGD without LSR.
Revised: With an appropriate label smoothing, we show that TSLA integrated with SGD has an improved iteration complexity, compared to the SGD with LSR and the SGD without LSR.

Original: (Section 5)
Revised: (Section 5)

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we introduce some related work.
Revised: In this section, we introduce some related work.

Original: A closely related idea to LSR is conﬁdence penalty proposed by Pereyra et al.
Revised: A closely related idea to LSR is conﬁdence penalty proposed by Pereyra et al.

Original: (2017), an output regularizer that penalizes conﬁdent output distributions by adding its negative entropy to the negative log-likelihood during the training process.
Revised: (2017), an output regularizer that penalizes conﬁdent output distributions by adding its negative entropy to the negative log-likelihood during the training process.

Original: The authors (Pereyra et al., 2017) presented extensive experimental results in training deep neural networks to demonstrate better generalization comparing to baselines with only focusing on the existing hyper-parameters.
Revised: The authors (Pereyra et al., 2017) presented extensive experimental results in training deep neural networks to demonstrate better generalization comparing to baselines with only focusing on the existing hyper-parameters.

Original: They have shown that LSR is equivalent to conﬁdence penalty with a reversing direction of KL divergence between uniform distributions and the output distributions.
Revised: They have shown that LSR is equivalent to conﬁdence penalty with a reversing direction of KL divergence between uniform distributions and the output distributions.

Original: DisturbLabel introduced by Xie et al.
Revised: DisturbLabel introduced by Xie et al.

Original: (2016) imposes the regularization within the loss layer, where it randomly replaces some of the ground truth labels as incorrect values at each training iteration.
Revised: (2016) imposes the regularization within the loss layer, where it randomly replaces some of the ground truth labels as incorrect values at each training iteration.

Original: Its effect is quite similar to LSR that can help to prevent the neural network training from overﬁtting.
Revised: Its effect is quite similar to LSR that can help to prevent the neural network training from overﬁtting.

Original: The authors have veriﬁed the effectiveness of DisturbLabel via several experiments on training image classiﬁcation tasks.
Revised: The authors have veriﬁed the effectiveness of DisturbLabel via several experiments on training image classiﬁcation tasks.

Original: Recently, many works (Zhang et al., 2018; Bagherinezhad et al., 2018; Goibert & Dohmatob, 2019;
Revised: Recently, many works (Zhang et al., 2018; Bagherinezhad et al., 2018; Goibert & Dohmatob, 2019;

Original: Shen et al., 2019; Li et al., 2020b) explored the idea of LSR technique.
Revised: Shen et al., 2019; Li et al., 2020b) explored the idea of LSR technique.

Original: Ding et al.
Revised: Ding et al.

Original: (2019) extended an adaptive label regularization method, which enables the neural network to use both correctness and incorrectness during training.
Revised: (2019) extended an adaptive label regularization method, which enables the neural network to use both correctness and incorrectness during training.

Original: Pang et al.
Revised: Pang et al.

Original: (2018) used the reverse cross-entropy loss to smooth the classiﬁer’s gradients.
Revised: (2018) used the reverse cross-entropy loss to smooth the classiﬁer’s gradients.

Original: Wang et al.
Revised: Wang et al.

Original: (2020) proposed a graduated label smoothing method that uses the higher smoothing penalty for high-conﬁdence predictions than that for low-conﬁdence predictions.
Revised: (2020) proposed a graduated label smoothing method that uses the higher smoothing penalty for high-conﬁdence predictions than that for low-conﬁdence predictions.

Original: They found that the proposed method can improve both inference calibration and translation performance for neural machine translation models.
Revised: They found that the proposed method can improve both inference calibration and translation performance for neural machine translation models.

Original: By contrast, in this paper, we will try to understand the power of LSR from an optimization perspective and try to study how and when to use
Revised: By contrast, in this paper, we will try to understand the power of LSR from an optimization perspective and try to study how and when to use

Original: LSR.
Revised: LSR.

Original:  P RELIMINARIES AND N OTATIONS
Revised:  P RELIMINARIES AND N OTATIONS

Original: We ﬁrst present some notations.
Revised: We ﬁrst present some notations.

Original: Let ∇ w F ( w ) denote the gradient of a function F ( w ) .
Revised: Let ∇ w F ( w ) denote the gradient of a function F ( w ) .

Original: When the variable to be taken a gradient is obvious, we use ∇ F ( w ) for simplicity.
Revised: When the variable to be taken a gradient is obvious, we use ∇ F ( w ) for simplicity.

Original: We use (cid:107) · (cid:107) to denote the
Revised: We use (cid:107) · (cid:107) to denote the

Original: Euclidean norm.
Revised: Euclidean norm.

Original: Let (cid:104)· , ·(cid:105) be the inner product.
Revised: Let (cid:104)· , ·(cid:105) be the inner product.

Original: In classiﬁcation problem, we aim to seek a classiﬁer to map an example x ∈ X onto one of K labels y ∈ Y ⊂ R K , where y =
Revised: In classiﬁcation problem, we aim to seek a classiﬁer to map an example x ∈ X onto one of K labels y ∈ Y ⊂ R K , where y =

Original: ( y 1 , y 2 , . . .
Revised: ( y 1 , y 2 , . . .

Original: , y K ) is a one-hot label, meaning that y i is “1” for the correct class and “0” for the rest.
Revised: , y K ) is a one-hot label, meaning that y i is “1” for the correct class and “0” for the rest.

Original: Suppose the example-label pairs are draw from a distribution P , i.e., ( x , y ) ∼ P =
Revised: Suppose the example-label pairs are drawn from a distribution P , i.e., ( x , y ) ∼ P =

Original: ( P x , P y ) .
Revised: ( P x , P y ) .

Original: we denote by E ( x , y )
Revised: we denote by E ( x , y )

Original: [ · ] the expectation that takes over a random variable ( x , y ) .
Revised: [ · ] the expectation that takes over a random variable ( x , y ) .

Original: When the randomness is obvious, we write E[ · ] for simplicity.
Revised: When the randomness is obvious, we write E[ · ] for simplicity.

Original: Our goal is to learn a prediction function f ( w ; x ) : W × X → R K that is as close as possible to y , where w ∈ W is the parameter and W is a closed convex set.
Revised: Our goal is to learn a prediction function f ( w ; x ) : W × X → R K that is as close as possible to y , where w ∈ W is the parameter and W is a closed convex set.

Original: To this end, we want to minimize the following expected loss under P :
Revised: To this end, we want to minimize the following expected loss under P :

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (cid:96) :
Revised: where (cid:96) :

Original: Y × R K → R + is a cross-entropy loss function given by
Revised: Y × R K → R + is a cross-entropy loss function given by

Original: [Equation]
Revised: [Equation]

Original: The objective function F ( w ) is not convex since f ( w ; x ) is non-convex in terms of w .
Revised: The objective function F ( w ) is not convex since f ( w ; x ) is non-convex in terms of w .

Original: To solve the problem (1), one can simply use some iterative methods such as stochastic gradient descent (SGD).
Revised: To solve the problem (1), one can simply use some iterative methods such as stochastic gradient descent (SGD).

Original: Speciﬁcally, at each training iteration t , SGD updates solutions iteratively by
Revised: Speciﬁcally, at each training iteration t , SGD updates solutions iteratively by

Original: [Equation]
Revised: [Equation]

Original: where η > 0 is a learning rate.
Revised: where η > 0 is a learning rate.

Original: Next, we present some notations and assumptions that will be used in the convergence analysis.
Revised: Next, we present some notations and assumptions that will be used in the convergence analysis.

Original: Throughout this paper, we also make the following assumptions for solving the problem (1).
Revised: Throughout this paper, we also make the following assumptions for solving the problem (1).

Original: Assumption 1.
Revised: Assumption 1.

Original: Assume the following conditions hold:
Revised: Assume the following conditions hold:

Original: (i)
Revised: (i)

Original: The stochastic gradient of F ( w ) is unbiased, i.e., E ( x , y )
Revised: The stochastic gradient of F ( w ) is unbiased, i.e., E ( x , y )

Original: [ ∇ (cid:96) ( y , f ( w ; x ))]
Revised: [ ∇ (cid:96) ( y , f ( w ; x ))]

Original: = ∇ F ( w ) , and the variance of stochastic gradient is bounded, i e., there exists a constant σ 2 > 0 , such that E ( x , y ) (cid:104) (cid:107)∇ (cid:96) ( y , f ( w ; x ))
Revised: = ∇ F ( w ) , and the variance of stochastic gradient is bounded, i.e., there exists a constant σ 2 > 0 , such that E ( x , y ) (cid:104) (cid:107)∇ (cid:96) ( y , f ( w ; x ))

Original: − ∇ F ( w ) (cid:107) 2 (cid:105) =
Revised: − ∇ F ( w ) (cid:107) 2 (cid:105) =

Original: σ 2 .
Revised: σ 2 .

Original: (ii) F ( w ) is smooth with an L -Lipchitz continuous gradient,
Revised: (ii) F ( w ) is smooth with an L -Lipchitz continuous gradient,

Original: i
Revised: i

Original: e., it is differentiable and there exists a constant L > 0 such that (cid:107)∇ F ( w ) − ∇ F ( u ) (cid:107) ≤ L (cid:107) w − u (cid:107) , ∀ w , u ∈ W .
Revised: e., it is differentiable and there exists a constant L > 0 such that (cid:107)∇ F ( w ) − ∇ F ( u ) (cid:107) ≤ L (cid:107) w − u (cid:107) , ∀ w , u ∈ W .

Original: Remark.
Revised: Remark.

Original: Assumption 1 (i) and (ii) are commonly used assumptions in the literature of non-convex optimization (Ghadimi & Lan, 2013; Yan et al., 2018; Yuan et al., 2019b;
Revised: Assumption 1 (i) and (ii) are commonly used assumptions in the literature of non-convex optimization (Ghadimi & Lan, 2013; Yan et al., 2018; Yuan et al., 2019b;

Original: Wang et al., 2019; Li et al., 2020a).
Revised: Wang et al., 2019; Li et al., 2020a).

Original: Assumption 1 (ii) says the objective function is L -smooth, and it has an equivalent expression (Nesterov, 2004) which is F ( w ) − F ( u ) ≤ (cid:104)∇ F ( u ) , w − u (cid:105) + L 2 (cid:107) w − u (cid:107) 2 , ∀ w , u ∈ W .
Revised: Assumption 1 (ii) says the objective function is L -smooth, and it has an equivalent expression (Nesterov, 2004) which is F ( w ) − F ( u ) ≤ (cid:104)∇ F ( u ) , w − u (cid:105) + L 2 (cid:107) w − u (cid:107) 2 , ∀ w , u ∈ W .

Original: Let (cid:98) y be a label introduced for smoothing label. Then the smoothed label y LS is given by
Revised: For a classiﬁcation problem, the smoothed label y LS is given by

Original: [Equation]
Revised: [Equation]

Original: where θ ∈ (0 , 1) is the smoothing strength, y is the one-hot label.
Revised: where θ ∈ (0 , 1) is the smoothing strength, y is the one-hot label, (cid:98)

Revised: y is an introduced label.

Revised: For example, one can simply use (cid:98) y

Revised: = 1 K (Szegedy et al., 2016) for K -class problems.

Original: Similar to label y , we suppose the label (cid:98) y is draw from a distribution P (cid:98) y .
Revised: Similar to label y , we suppose the label (cid:98) y is drawn from a distribution P (cid:98) y .

Original: We introduce the variance of stochastic gradient using label (cid:98) y as follows.
Revised: We introduce the variance of stochastic gradient using label (cid:98) y as follows.

Original: [Equation]
Revised: [Equation]

Original: (4) where δ > 0 is a constant and σ 2 is deﬁned in Assumption 1 (i).
Revised: where δ > 0 is a constant and σ 2 is deﬁned in Assumption 1 (i).

Original: We make several remarks for (4).
Revised: We make several remarks for (4).

Original: Remark.
Revised: Remark.

Original: (a) We do not require  the stochastic gradient ∇ (cid:96) ( (cid:98) y , f ( w ; x )) is unbiased, i.e., it could be E[ ∇ (cid:96) ( (cid:98) y , f ( w ; x ))]
Revised: (a) We do not require that the stochastic gradient ∇ (cid:96) ( (cid:98) y , f ( w ; x )) is unbiased, i.e., it could be E[ ∇ (cid:96) ( (cid:98) y , f ( w ; x ))]

Original: (cid:54) =
Revised: (cid:54) =

Original: ∇ F ( w ) .
Revised: ∇ F ( w ) .

Original: (b) The variance (cid:98) σ 2 is deﬁned based on the label (cid:98) y rather than the smoothed label y LS .
Revised: (b) The variance (cid:98) σ 2 is deﬁned based on the label (cid:98) y rather than the smoothed label y LS .

Original: (c) We do not assume the variance (cid:98) σ 2 is bounded since δ could be an arbitrary value, however, we will discuss the different cases of δ in our analysis.
Revised: (c) We do not assume the variance (cid:98) σ 2 is bounded since δ could be an arbitrary value, however, we will discuss the different cases of δ in our analysis.

Original: If δ ≥ 1 , then (cid:98) σ 2 ≥ σ 2 ; while if 0 < δ < 1 , then (cid:98) σ 2 < σ 2 .
Revised: If δ ≥ 1 , then (cid:98) σ 2 ≥ σ 2 ; while if 0 < δ < 1 , then (cid:98) σ 2 < σ 2 .

Original: It is worth mentioning that δ could be small when an appropriate label is used in the label smoothing.
Revised: It is worth mentioning that δ could be small when an appropriate label is used in the label smoothing.

Original: For example, one can smooth labels by using a teacher model (Hinton et al., 2014) or the model’s own distribution (Reed et al., 2014).
Revised: For example, one can smooth labels by using a teacher model (Hinton et al., 2014) or the model’s own distribution (Reed et al., 2014).

Original: In the ﬁrst paper of label smoothing (Szegedy et al., 2016) and the following related studies (M¨uller et al., 2019; Yuan et al., 2019a), researchers consider a uniform distribution over all K classes of labels as the label (cid:98) y , i.e., set (cid:98) y
Revised: In the ﬁrst paper of label smoothing (Szegedy et al., 2016) and the following related studies (M¨uller et al., 2019; Yuan et al., 2019a), researchers consider a uniform distribution over all K classes of labels as the label (cid:98) y , i.e., set (cid:98) y

Original: = 1 K .
Revised: = 1 K .

Original: We now introduce an important assumption regarding F ( w ) , i.e. there is no very bad local optimum on the surface of objective function F ( w ) .
Revised: We now introduce an important property regarding F ( w ) , i.e. Polyak-Łojasiewicz (PL) condition (Polyak, 1963).

Original: More speciﬁcally, the following assumption holds.
Revised: More speciﬁcally, the following assumption holds.

Original: Assumption 2.
Revised: Assumption 2.

Original: There exists a constant µ > 0 such that 2 µ ( F ( w ) − F ∗ ) ≤ (cid:107)∇ F ( w ) (cid:107) 2 , ∀ w ∈ W , where F ∗ = min w ∈W F ( w ) is the optimal value.
Revised: There exists a constant µ > 0 such that 2 µ ( F ( w ) − F ∗ ) ≤ (cid:107)∇ F ( w ) (cid:107) 2 , ∀ w ∈ W , where F ∗ = min w ∈W F ( w ) is the optimal value.

Original: Remark.
Revised: Remark.

Original: This propertyis known as Polyak-Łojasiewicz (PL) condition (Polyak, 1963), and it has been theoretically and empirically observed in training deep neural networks (Allen-Zhu et al., 2019; Yuan et al., 2019b).
Revised: This property has been theoretically and empirically observed in training deep neural networks (Allen-Zhu et al., 2019; Yuan et al., 2019b).

Original: This condition is widely used to establish convergence in the literature of nonconvex optimization, please see (Yuan et al., 2019b;
Revised: This condition is widely used to establish convergence in the literature of non-convex optimization, please see (Yuan et al., 2019b;

Original: Wang et al., 2019; Karimi et al., 2016; Li & Li, 2018; Charles & Papailiopoulos, 2018; Li et al., 2020a) and references therein.
Revised: Wang et al., 2019; Karimi et al., 2016; Li & Li, 2018; Charles & Papailiopoulos, 2018; Li et al., 2020a) and references therein.

Original: To measure the convergence of non-convex and smooth optimization problems as in (Nesterov, 1998; Ghadimi & Lan, 2013; Yan et al., 2018), we need the following deﬁnition of the ﬁrst-order stationary point.
Revised: To measure the convergence of non-convex and smooth optimization problems as in (Nesterov, 1998; Ghadimi & Lan, 2013; Yan et al., 2018), we need the following deﬁnition of the ﬁrst-order stationary point.

Original: Deﬁnition 1 (First-order stationary point) .
Revised: Deﬁnition 1 (First-order stationary point) .

Original: For the problem of min w ∈W F ( w ) , a point w ∈ W is called a ﬁrst-order stationary point if (cid:107)∇ f ( w ) (cid:107) = 0 .
Revised: For the problem of min w ∈W F ( w ) , a point w ∈ W is called a ﬁrst-order stationary point if (cid:107)∇ F ( w ) (cid:107) = 0 .

Original: Moreover, if (cid:107)∇ f ( w ) (cid:107) ≤ (cid:15) , then the point w is said to be an (cid:15) -stationary point, where (cid:15) ∈ (0 , 1) is a small positive value.
Revised: Moreover, if (cid:107)∇ F ( w ) (cid:107) ≤ (cid:15) , then the point w is said to be an (cid:15) -stationary point, where (cid:15) ∈ (0 , 1) is a small positive value.

Original:  C ONVERGENCE A NALYSIS OF SGD WITH LSR
Revised:  C ONVERGENCE A NALYSIS OF SGD WITH LSR

Original: To understand LSR from the optimization perspective, we consider SGD with LSR in Algorithmfor the sake of simplicity.
Revised: To understand LSR from the optimization perspective, we consider SGD with LSR in Algorithmfor the sake of simplicity.

Original: The only difference between Algorithm 1 and standard SGD is the use of the output label for constructing a stochastic gradient.
Revised: The only difference between Algorithm 1 and standard SGD is the use of the output label for constructing a stochastic gradient.

Original: The following theorem shows that Algorithm 1 converges to an approximate stationary point in expectation under some conditions.
Revised: The following theorem shows that Algorithm 1 converges to an approximate stationary point in expectation under some conditions.

Original: We include its proof in Appendix B.
Revised: We include its proof in Appendix B.

Original: Theorem 3.
Revised: Theorem 3.

Original: Under Assumption 1, run Algorithm 1 with η = 1 L and θ = 11+ δ , then
Revised: Under Assumption 1, run Algorithm 1 with η = 1 L and θ = 11+ δ , then

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  + 2 δσ 2 , where R is uniformly sampled from { 0 , 1 , . .
Revised: ηT + 2 δσ 2 , where R is uniformly sampled from { 0 , 1 , . .

Original: , T − 1 } .
Revised: , T − 1 } .

Original: Furthermore,  we have the following two results.
Revised: Furthermore, given a target accuracy level (cid:15) , we have the following two results.

Original: (1) when δ ≤ (cid:15) 2 4 σ 2 , if we set T = 4 F ( w 0 ) η(cid:15) 2 , then Algorithm 1 converges to an (cid:15) -stationary point in expectation, i.e., E R
Revised: (1) when δ ≤ (cid:15) 2 4 σ 2 , if we set T = 4 F ( w 0 ) η(cid:15) 2 , then Algorithm 1 converges to an (cid:15) -stationary point in expectation, i.e., E R

Original: [ (cid:107)∇ F ( w R ) (cid:107) 2 ] ≤ (cid:15) 2 .
Revised: [ (cid:107)∇ F ( w R ) (cid:107) 2 ] ≤ (cid:15) 2 .

Original: The total sample complexity is T
Revised: The total sample complexity is T

Original: =
Revised: =

Original: O (cid:0) 1 (cid:15) 2 (cid:1) .
Revised: O (cid:0) 1 (cid:15) 2 (cid:1) .

Original: (2) when δ > (cid:15) 2 4 σ 2 , if we set T = F ( w 0 ) ηδσ 2 , then Algorithm 1 does not converge to an (cid:15) -stationary point, but we have E R
Revised: (2) when δ > (cid:15) 2 4 σ 2 , if we set T = F ( w 0 ) ηδσ 2 , then Algorithm 1 does not converge to an (cid:15) -stationary point, but we have E R

Original: [ (cid:107)∇ F ( w R ) (cid:107) 2 ] ≤ 4 δσ 2 ≤ O ( δ ) .
Revised: [ (cid:107)∇ F ( w R ) (cid:107) 2 ] ≤ 4 δσ 2 ≤ O ( δ ) .

Original: Remark.
Revised: Remark.

Original: We observe that the variance term is 2 δσ 2 , instead of ηLσ 2 for standard analysis of
Revised: We observe that the variance term is 2 δσ 2 , instead of ηLσ 2 for standard analysis of

Original: SGD without LSR (i.e., θ = 0 , please see the detailed analysis of Theorem 5 in Appendix C).
Revised: SGD without LSR (i.e., θ = 0 , please see the detailed analysis of Theorem 5 in Appendix C).

Original: For the convergence analysis, the different between SGD with LSR and SGD without LSR is that ∇ (cid:96) ( (cid:98) y , f ( w ; x )) is not an unbiased estimator of ∇ F ( w ) when using LSR.
Revised: For the convergence analysis, the difference between SGD with LSR and SGD without LSR is that ∇ (cid:96) ( (cid:98) y , f ( w ; x )) is not an unbiased estimator of ∇ F ( w ) when using LSR.

Original: The convergence behavior of Algorithm 1 heavily depends on the parameter δ .
Revised: The convergence behavior of Algorithm 1 heavily depends on the parameter δ .

Original: When δ is small enough, say δ ≤ O ( (cid:15) 2 ) with a small positive value (cid:15) ∈ (0 , 1) , then Algorithm 1 converges to an (cid:15) -stationary point with the total sample complexity of O (cid:0) 1 (cid:15) 2 (cid:1) .
Revised: When δ is small enough, say δ ≤ O ( (cid:15) 2 ) with a small positive value (cid:15) ∈ (0 , 1) , then Algorithm 1 converges to an (cid:15) -stationary point with the total sample complexity of O (cid:0) 1 (cid:15) 2 (cid:1) .

Original: Recall that the total sample complexity of standard SGD without LSR for ﬁnding an (cid:15) -stationary
Revised: Recall that the total sample complexity of standard SGD without LSR for ﬁnding an (cid:15) -stationary

Original: point is O (cid:0) 1 (cid:15) 4 (cid:1) ((Ghadimi & Lan, 2016; Ghadimi et al., 2016), please also see the detailed analysis of Theorem 5 in Appendix C).
Revised: point is O (cid:0) 1 (cid:15) 4 (cid:1) ((Ghadimi & Lan, 2016; Ghadimi et al., 2016), please also see the detailed analysis of Theorem 5 in Appendix C).

Original: The convergence result shows that if we could ﬁnd a label (cid:98) y that has a reasonably small amount of δ , we will be able to reduce sample complexity for training a  learning model from O (cid:0) 1 (cid:15) 4 (cid:1) to O (cid:0) 1 (cid:15) 2 (cid:1) .
Revised: The convergence result shows that if we could ﬁnd a label (cid:98) y that has a reasonably small amount of δ , we will be able to reduce sample complexity for training a machine learning model from O (cid:0) 1 (cid:15) 4 (cid:1) to O (cid:0) 1 (cid:15) 2 (cid:1) .

Original: Thus, the reduction in variance will happen when an appropriate label smoothing with δ ∈ (0 , 1) is introduced.
Revised: Thus, the reduction in variance will happen when an appropriate label smoothing with δ ∈ (0 , 1) is introduced.

Original: We will ﬁnd in the empirical evaluations that different label (cid:98) y lead to different performances and an appropriate selection of label (cid:98) y has a better performance (see the performances of LSR and LSR-pre in Table 3).
Revised: We will ﬁnd in the empirical evaluations that different label (cid:98) y lead to different performances and an appropriate selection of label (cid:98) y has a better performance (see the performances of LSR and LSR-pre in Table 3).

Original: On the other hand, when the parameter δ is large such that δ > Ω( (cid:15) 2 ) , that is to say, if an inappropriate label smoothing is used, then Algorithm 1 does not converge to an (cid:15) -stationary point, but it converges to a worse level of O ( δ ) .
Revised: On the other hand, when the parameter δ is large such that δ > Ω( (cid:15) 2 ) , that is to say, if an inappropriate label smoothing is used, then Algorithm 1 does not converge to an (cid:15) -stationary point, but it converges to a worse level of O ( δ ) .

Original:  TSLA: A G ENERIC T WO -S TAGE L ABEL S MOOTHING A LGORITHM
Revised:  TSLA: A G ENERIC T WO -S TAGE L ABEL S MOOTHING A LGORITHM

Original: Despite superior outcomes in training deep neural networks, some real applications have shown the adverse effect of LSR.
Revised: Despite superior outcomes in training deep neural networks, some real applications have shown the adverse effect of LSR.

Original: M¨uller et al.
Revised: M¨uller et al.

Original: (2019) have empirically observed that LSR impairs distillation, that is, after training teacher models with LSR, student models perform worse.
Revised: (2019) have empirically observed that LSR impairs distillation, that is, after training teacher models with LSR, student models perform worse.

Original: The authors believed that LSR reduces mutual information between  input example and output logit.
Revised: The authors believed that LSR reduces mutual information between the input example and output logit.

Original: Kornblith et al.
Revised: Kornblith et al.

Original: (2019) have found that LSR impairs the accuracy of transfer learning when training deep neural network models on ImageNet data set.
Revised: (2019) have found that LSR impairs the accuracy of transfer learning when training deep neural network models on ImageNet data set.

Original: Seo et al.
Revised: Seo et al.

Original: (2020) trained deep neural network models for few-shot learning on miniImageNet and found a signiﬁcant performance drop  with LSR.
Revised: (2020) trained deep neural network models for few-shot learning on mini-ImageNet and found a signiﬁcant performance drop for with LSR.

Original: This motivates us to investigate a strategy that combines the algorithm with and without LSR during the training progress.
Revised: This motivates us to investigate a strategy that combines the algorithm with and without LSR during the training progress.

Original: Let think in this way, one possible scenario is that training one-hot label is “easier” than training smoothed label.
Revised: Let think in this way, one possible scenario is that training one-hot label is “easier” than training smoothed label.

Original: Taking the cross entropy loss in (2) for an example, one need to optimize a single loss function − log (cid:16) exp( f k ( w ; x )) /
Revised: Taking the cross entropy loss in (2) for an example, one need to optimize a single loss function − log (cid:16) exp( f k ( w ; x )) /

Original: (cid:80) Kj =1 exp( f j ( w ; x )) (cid:17) when one-hot label (e.g, y k
Revised: (cid:80) Kj =1 exp( f j ( w ; x )) (cid:17) when one-hot label (e.g, y k

Original: = 

Revised: =

Original: and y i = 

Revised: and y i =

Original: for all i (cid:54)
Revised: for all i (cid:54)

Original: = k ) is used, but need to optimize all K loss functions − (cid:80) Ki
Revised: = k ) is used, but need to optimize all K loss functions − (cid:80) Ki

Original: =

Revised: =

Original: y LS i log (cid:16) exp
Revised: y LS i log (cid:16) exp

Original: ( f i ( w ; x )) /
Revised: ( f i ( w ; x )) /

Original: (cid:80) Kj =1 exp( f j ( w ; x )) (cid:17) when smoothed label (e.g., y LS
Revised: (cid:80) Kj =1 exp( f j ( w ; x )) (cid:17) when smoothed label (e.g., y LS

Original: =
Revised: =

Original: (1 − θ )
Revised: (1 − θ )

Original: y
Revised: y

Original: + θ 1 K so that y LS k = 1 − ( K − 1) θ/K and y LS
Revised: + θ 1 K so that y LS k = 1 − ( K − 1) θ/K and y LS

Original: i = θ/K for all i (cid:54) =
Revised: i = θ/K for all i (cid:54) =

Original: k ) is used.
Revised: k ) is used.

Original: Nevertheless, training deep neural networks is gradually focusing on hard examples with the increase of training epochs.
Revised: Nevertheless, training deep neural networks is gradually focusing on hard examples with the increase of training epochs.

Original: It seems that training  smoothed label in the late epochs makes the learning progress more difﬁcult.
Revised: It seems that training with smoothed label in the late epochs makes the learning progress more difﬁcult.

Original: In addition, after LSR, we focus on optimizing the overall distribution that contains the minor classes, which are probably not important at the end of training progress.
Revised: In addition, after LSR, we focus on optimizing the overall distribution that contains the minor classes, which are probably not important at the end of training progress.

Original: One question is whether LSR helps at the early training epochs but it has less (even negative) effect during the later training epochs?
Revised: One question is whether LSR helps at the early training epochs but it has less (even negative) effect during the later training epochs?

Original: This question encourages us to propose and analyze a simple strategy with LSR dropping that switches a stochastic algorithm with LSR to the algorithm without LSR.
Revised: This question encourages us to propose and analyze a simple strategy with LSR dropping that switches a stochastic algorithm with LSR to the algorithm without LSR.

Original: In this subsection, we propose a generic framework that consists of two stages, wherein the ﬁrst stage it runs a stochastic algorithm A (e.g., SGD) with LSR in T 1 iterations and the second stage it runs the same algorithm without LSR up to T 2 iterations.
Revised: In this subsection, we propose a generic framework that consists of two stages, wherein the ﬁrst stage it runs a stochastic algorithm A (e.g., SGD) with LSR for T 1 iterations and the second stage it runs the same algorithm without LSR up to T 2 iterations.

Original: This framework is referred to as T woS tage LA bel smoothing (TSLA) algorithm, whose updating details are presented in Algorithm 2.
Revised: This framework is referred to as T woS tage LA bel smoothing (TSLA) algorithm, whose updating details are presented in Algorithm 2.

Original: The notation A -step ( · ; · , η ) is one update step of a stochastic algorithm A with learning rate η .
Revised: The notation A -step ( · ; · , η ) is one update step of a stochastic algorithm A with learning rate η .

Original: For example, if we select SGD as algorithm A , then
Revised: For example, if we select SGD as algorithm A , then

Original: [Equation]
Revised: [Equation]

Original: The proposed TSLA is a generic strategy where the subroutine algorithm A  can be replaced by any stochastic algorithms such as momentum SGD (Polyak, 1964), Stochastic Nesterov’s Accelerated Gradient (Nesterov, 1983), and adaptive algorithms including A DA G RAD (Duchi et al., 2011), RMSProp (Hinton et al., 2012a), AdaDelta (Zeiler, 2012), Adam (Kingma & Ba, 2015), Nadam (Dozat, 2016) and AMSG RAD (Reddi et al., 2018).
Revised: Although SGD is considered as the subroutine algorithm A in the convergence analysis, in practice, algorithm A can be replaced by any stochastic algorithms such as momentum SGD (Polyak, 1964), Stochastic Nesterov’s Accelerated Gradient (Nesterov, 1983), and adaptive algorithms including A DA G RAD (Duchi et al., 2011), RMSProp (Hinton et al., 2012a), AdaDelta (Zeiler, 2012), Adam (Kingma & Ba, 2015), Nadam (Dozat, 2016) and AMSG RAD (Reddi et al., 2018).

Revised: In this paper, we will not study the theoretical guarantees and empirical evaluations of other optimizers, which can be considered as future work.

Original: Please note that the algorithm can use different learning rates η 1 and η 2 during the two stages.
Revised: Please note that the algorithm can use different learning rates η 1 and η 2 during the two stages.

Original: The last solution of the ﬁrst stage will be used as the initial solution of the second stage.
Revised: The last solution of the ﬁrst stage will be used as the initial solution of the second stage.

Original: If T 1 = 0 , then TSLA reduces to the baseline, i.e., a standard stochastic algorithm A without LSR; while if T 2 = 0 , TSLA becomes to LSR method, i.e., a standard stochastic algorithm A with LSR.
Revised: If T 1 = 0 , then TSLA reduces to the baseline, i.e., a standard stochastic algorithm A without LSR; while if T 2 = 0 , TSLA becomes to LSR method, i.e., a standard stochastic algorithm A with LSR.

Original: 5. C ONVERGENCE R ESULT OF TSLA
Revised: 5. C ONVERGENCE R ESULT OF TSLA

Original: In this subsection, we will give the convergence result of the proposed TSLA algorithm.
Revised: In this subsection, we will give the convergence result of the proposed TSLA algorithm.

Original: For simplicity, we use SGD as the subroutine algorithm A in the analysis.
Revised: For simplicity, we use SGD as the subroutine algorithm A in the analysis.

Original: The convergence result in the following theorem shows the power of LSR from the optimization perspective.
Revised: The convergence result in the following theorem shows the power of LSR from the optimization perspective.

Original: Its proof is presented in Appendix D.
Revised: Its proof is presented in Appendix D.

Original: It is easy to see from the proof that by using the last output of the ﬁrst stage as the initial point of the second stage, TSLA can enjoy the advantage of LSR in the second stage with an improved convergence.
Revised: It is easy to see from the proof that by using the last output of the ﬁrst stage as the initial point of the second stage, TSLA can enjoy the advantage of LSR in the second stage with an improved convergence.

Original: [Table]
Revised: [Table]

Original: Theorem 4.
Revised: Theorem 4.

Original: Under Assumptions 1, 2, suppose σ 2 δ/µ ≤ F ( w 0 ) , run Algorithm 2 with A =
Revised: Under Assumptions 1, 2, suppose σ 2 δ/µ ≤ F ( w 0 ) , run Algorithm 2 with A =

Original: SGD, θ = 11+ δ , η 1 = 1 L , T 1 = log (cid:16) 
Revised: SGD, θ = 11+ δ , η 1 = 1 L , T 1 = log (cid:16) 2 µF ( w 0 )(

Original: [Table]
Revised: [Table]

Original: µF
Revised: + δ

Original: [Table]

Original: [Table]
Revised: [Table]

Original: δσ
Revised: δσ

Revised: [Table]

Revised: [Table]

Revised: =

Original: [Table]
Revised: [Table]

Original: E R
Revised: E R

Original: [ (cid:107)∇ F ( w R ) (cid:107) 2 ] ≤ (cid:15) 2 , where R is uniformly sampled from { T 1 , . .
Revised: [ (cid:107)∇ F ( w R ) (cid:107) 2 ] ≤ (cid:15) 2 , where R is uniformly sampled from { T 1 , . .

Original: , T 1 + T 2 − 1 } .
Revised: , T 1 + T 2 − 1 } .

Original: Remark.
Revised: Remark.

Original: It is obvious that the learning rate η 2 in the second stage is roughly smaller than the learning rate η 1 in the ﬁrst stage, which matches the widely used stage-wise learning rate decay scheme in training neural networks.
Revised: It is obvious that the learning rate η 2 in the second stage is roughly smaller than the learning rate η 1 in the ﬁrst stage, which matches the widely used stage-wise learning rate decay scheme in training neural networks.

Original: To explore the total sample complexity of TSLA, we consider different conditions on δ .
Revised: To explore the total sample complexity of TSLA, we consider different conditions on δ .

Original: We summarize the total sample complexities of ﬁnding (cid:15) -stationary points for SGD with TSLA (TSLA), SGD with LSR (LSR), and SGD without LSR (baseline) in Table 1, where (cid:15) ∈ (0 , 1) is the target convergence level, and we only present the orders of the complexities but ignore all constants.
Revised: We summarize the total sample complexities of ﬁnding (cid:15) -stationary points for SGD with TSLA (TSLA), SGD with LSR (LSR), and SGD without LSR (baseline) in Table 1, where (cid:15) ∈ (0 , 1) is the target convergence level, and we only present the orders of the complexities but ignore all constants.

Original: When Ω( (cid:15) 2 ) < δ < 1 , LSR dose not converge to an (cid:15) -stationary point (denoted by ∞ ), while TSLA reduces sample complexity from O (cid:0) 1 (cid:15) 4 (cid:1) to O (cid:0) δ(cid:15) 4 (cid:1) , compared to the baseline.
Revised: When Ω( (cid:15) 2 ) < δ < 1 , LSR does not converge to an (cid:15) -stationary point (denoted by ∞ ), while TSLA reduces sample complexity from O (cid:0) 1 (cid:15) 4 (cid:1) to O (cid:0) δ(cid:15) 4 (cid:1) , compared to the baseline.

Original: When δ < O ( (cid:15) 2 ) , the total complexity of TSLA is between log(1 /(cid:15) ) and 1 /(cid:15) 2 , which is always better than LSR and the baseline.
Revised: When δ < O ( (cid:15) 2 ) , the total complexity of TSLA is between log(1 /(cid:15) ) and 1 /(cid:15) 2 , which is always better than LSR and the baseline.

Original: In summary, TSLA achieves the best total sample complexity by enjoying the good property of an appropriate label smoothing (i.e., when 0 < δ < 1 ).
Revised: In summary, TSLA achieves the best total sample complexity by enjoying the good property of an appropriate label smoothing (i.e., when 0 < δ < 1 ).

Original: However, when δ ≥ 1 , baseline has better convergence than TSLA, meaning that the selection of label (cid:98) y is not appropriate.
Revised: However, when δ ≥ 1 , baseline has better convergence than TSLA, meaning that the selection of label (cid:98) y is not appropriate.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: To further evaluate the performance of the proposed TSLA method, we trained deep neural networks on three benchmark data sets, CIFAR-100 (Krizhevsky & Hinton, 2009), Stanford Dogs (Khosla et al., 2011) and CUB-2011 (Wah et al., 2011), for image classiﬁcation tasks.
Revised: To further evaluate the performance of the proposed TSLA method, we trained deep neural networks on three benchmark data sets, CIFAR-100 (Krizhevsky & Hinton, 2009), Stanford Dogs (Khosla et al., 2011) and CUB-2011 (Wah et al., 2011), for image classiﬁcation tasks.

Original: CIFAR-100 1 has 50,000 training images and 10,000 testing images of 32 × 32 resolution with 100 classes.
Revised: CIFAR-100 1 has 50,000 training images and 10,000 testing images of 32 × 32 resolution with 100 classes.

Original: Stanford
Revised: Stanford

Original: Dogs data set 2 contains 20,580 images of 120 breeds of dogs, where 100 images from each breed is used for training.
Revised: Dogs data set 2 contains 20,580 images of 120 breeds of dogs, where 100 images from each breed is used for training.

Original: CUB-2011 3 is a birds image data set with 11,788 images of 200 birds species.
Revised: CUB-2011 3 is a birds image data set with 11,788 images of 200 birds species.

Original: The ResNet-18 model (He et al., 2016) is applied as the backbone in the experiments.
Revised: The ResNet-18 model (He et al., 2016) is applied as the backbone in the experiments.

Original: We compare the proposed TSLA incorporating with SGD (TSLA) with two baselines, SGD with LSR (LSR) and
Revised: We compare the proposed TSLA incorporating with SGD (TSLA) with two baselines, SGD with LSR (LSR) and

Original: SGD without LSR (baseline).
Revised: SGD without LSR (baseline).

Original: The mini-batch size of training instances for all methods is 256 as suggested by He et al.
Revised: The mini-batch size of training instances for all methods is 256 as suggested by He et al.

Original: and He et al.
Revised: and He et al.

Original: The momentum parameter is ﬁxed as 0.9.
Revised: The momentum parameter is ﬁxed as 0.9.

Original: 6. S TANFORD D OGS AND CUB- 
Revised: 6. S TANFORD D OGS AND CUB-

Original: We separately train ResNet-18 (He et al., 2016) up to 90 epochs over two data sets Stanford Dogs and CUB-2011.
Revised: We separately train ResNet-18 (He et al., 2016) up to 90 epochs over two data sets Stanford Dogs and CUB-2011.

Original: We use weight decay with the parameter value of 10 − 4 .
Revised: We use weight decay with the parameter value of 10 − 4 .

Original: For all algorithms, the initial learning rates for FC are set to be 0.1, while that for the pre-trained backbones are 0.001 and 0.01 for Standford Dogs and CUB-2011, respectively.
Revised: For all algorithms, the initial learning rates for FC are set to be 0.1, while that for the pre-trained backbones are 0.001 and 0.01 for Standford Dogs and CUB-2011, respectively.

Original: The learning rates are divided by 10 everyepochs.
Revised: The learning rates are divided by 10 everyepochs.

Original: For LSR, we ﬁx the value of smoothing strength θ = 0 .
Revised: For LSR, we ﬁx the value of smoothing strength θ = 0 .

Original: 4 for the best performance, and the label (cid:98) y used for label smoothing is set to be a uniform distribution over all K classes, i.e., (cid:98)
Revised: 4 for the best performance, and the label (cid:98) y used for label smoothing is set to be a uniform distribution over all K classes, i.e., (cid:98)

Original: y
Revised: y

Original: = 1 K .
Revised: = 1 K .

Original: The same values of the smoothing strength θ and the same (cid:98) y are used during the ﬁrst stage of TSLA.
Revised: The same values of the smoothing strength θ and the same (cid:98) y are used during the ﬁrst stage of TSLA.

Original: For TSLA, we drop off the LSR (i.e., let θ = 0 ) after s epochs during the training process, where s ∈ { 20 , 30 , 40 , 50 , 60 , 70 , 80 } .
Revised: For TSLA, we drop off the LSR (i.e., let θ = 0 ) after s epochs during the training process, where s ∈ { 20 , 30 , 40 , 50 , 60 , 70 , 80 } .

Original: We ﬁrst report the highest top-1 and top-5 accuracy on the testing data sets for different methods.
Revised: We ﬁrst report the highest top-1 and top-5 accuracy on the testing data sets for different methods.

Original: All top-1 and top-5 accuracy are averaged over 5 independent random trails with their standard deviations.
Revised: All top-1 and top-5 accuracy are averaged over 5 independent random trails with their standard deviations.

Original: The results of the comparison are summarized in Table 2, where the notation “TSLA( s )” means that the TSLA algorithm drops off LSR after epoch s .
Revised: The results of the comparison are summarized in Table 2, where the notation “TSLA( s )” means that the TSLA algorithm drops off LSR after epoch s .

Original: It can be seen from Table 2 that under an appropriate hyperparameter setting the models trained using TSLA
Revised: It can be seen from Table 2 that under an appropriate hyperparameter setting the models trained using TSLA

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: outperform that trained using LSR and baseline, which supports the convergence result in Section 5.
Revised: outperform that trained using LSR and baseline, which supports the convergence result in Section 5.

Original: We notice that the best top-1 accuracy of TSLA are TSLA(40) and TSLA(50) for Stanford Dogs and CUB-2011, respectively, meaning that the performance of TSLA( s ) is not monotonic over the dropping epoch s .
Revised: We notice that the best top-1 accuracy of TSLA are TSLA(40) and TSLA(50) for Stanford Dogs and CUB-2011, respectively, meaning that the performance of TSLA( s ) is not monotonic over the dropping epoch s .

Original: For CUB-2011, the top-1 accuracy of TSLA(20) is smaller than that of LSR.
Revised: For CUB-2011, the top-1 accuracy of TSLA(20) is smaller than that of LSR.

Original: This result matches the convergence analysis of TSLA showing that it can not drop off LSR too early.
Revised: This result matches the convergence analysis of TSLA showing that it can not drop off LSR too early.

Original: For top-5 accuracy, we found that TSLA(80) is slightly worse than baseline.
Revised: For top-5 accuracy, we found that TSLA(80) is slightly worse than baseline.

Original: This is because of dropping LSR too late so that the update iterations (i.e., T 2 ) in the second stage of TSLA is too small to converge to a good solution.
Revised: This is because of dropping LSR too late so that the update iterations (i.e., T 2 ) in the second stage of TSLA is too small to converge to a good solution.

Original: We also observe that LSR is better than baseline regarding top1 accuracy but the result is opposite as to top-5 accuracy.
Revised: We also observe that LSR is better than baseline regarding top1 accuracy but the result is opposite as to top-5 accuracy.

Original: We then plot the averaged top-1 accuracy, averaged top-5 accuracy, and averaged loss among 5 trails of different methods in Figure 1.
Revised: We then plot the averaged top-1 accuracy, averaged top-5 accuracy, and averaged loss among 5 trails of different methods in Figure 1.

Original: We remove the results for TSLA(20) since it dropped off LSR too early as mentioned before.
Revised: We remove the results for TSLA(20) since it dropped off LSR too early as mentioned before.

Original: The ﬁgure shows TSLA improves the top-1 and top-5 testing accuracy immediately once it drops off LSR.
Revised: The ﬁgure shows TSLA improves the top-1 and top-5 testing accuracy immediately once it drops off LSR.

Original: Although TSLA may not converges if it drops off LSR too late, see TSLA(60), TSLA(70), and
Revised: Although TSLA may not converge if it drops off LSR too late, see TSLA(60), TSLA(70), and

Original: TSLA(80) from the third column of Figure 1, it still has the best performance compared to LSR and baseline.
Revised: TSLA(80) from the third column of Figure 1, it still has the best performance compared to LSR and baseline.

Original: TSLA(30), TSLA(40), and TSLA(50) can converge to lower objective levels, comparing to LSR and baseline.
Revised: TSLA(30), TSLA(40), and TSLA(50) can converge to lower objective levels, comparing to LSR and baseline.

Original: 6. CIFAR- 
Revised: 6. CIFAR-

Original: The total epochs of training ResNet-18 (He et al., 2016) on CIFRA-100 is set to be 200.The weight decay with the parameter value of 5 × 10 − 4 is used.
Revised: The total epochs of training ResNet-18 (He et al., 2016) on CIFAR-100 is set to be 200.The weight decay with the parameter value of 5 × 10 − 4 is used.

Original: We use 0 .
Revised: We use 0 .

Original: 1 as the initial learning rates for all algorithms and divide them by 10 every 60 epochs  suggested in (He et al., 2016; Zagoruyko &
Revised: 1 as the initial learning rates for all algorithms and divide them by 10 every 60 epochs as suggested in (He et al., 2016; Zagoruyko &

Original: Komodakis, 2016).
Revised: Komodakis, 2016).

Original: For LSR and the ﬁrst stage of TSLA, the value of smoothing strength θ is ﬁxed 
Revised: For LSR and the ﬁrst stage of TSLA, the value of smoothing strength θ is ﬁxed as θ = 0 . 1 , which shows the best performance for LSR.

Revised: We use two different labels (cid:98) y to smooth the

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: as θ = 0 . 1 , which shows the best performance for LSR.

Original: We use two different labels (cid:98) y to smooth the one-hot label, the uniform distribution over all labels and the distribution predicted by an ImageNet pre-trained model which  downloaded directly from PyTorch  (Paszke et al., 2019).
Revised: one-hot label, the uniform distribution over all labels and the distribution predicted by an ImageNet pre-trained model which is downloaded directly from PyTorch 4 (Paszke et al., 2019).

Original: For TSLA, we try to drop off the LSR after s epochs during the training process, where s ∈ { 120 , 140 , 160 , 180 } .
Revised: For TSLA, we try to drop off the LSR after s epochs during the training process, where s ∈ { 120 , 140 , 160 , 180 } .

Original: All top-1 and top-5 accuracy on the testing data set are averaged over 5 independent random trails with their standard deviations.
Revised: All top-1 and top-5 accuracy on the testing data set are averaged over 5 independent random trails with their standard deviations.

Original: We summarize the results in Table 3, where LSR-pre and TSLA-pre indicate LSR and TSLA use the label (cid:98) y based on the ImageNet pre-trained model.
Revised: We summarize the results in Table 3, where LSR-pre and TSLA-pre indicate LSR and TSLA use the label (cid:98) y based on the ImageNet pre-trained model.

Original: The results show that LSR-pre/TSLA-pre has a better performance than LSR/TSLA.
Revised: The results show that LSR-pre/TSLA-pre has a better performance than LSR/TSLA.

Original: The reason might be that the pre-trained model-based prediction is closer to the ground truth than the uniform prediction and it has lower variance (smaller δ ).
Revised: The reason might be that the pre-trained model-based prediction is closer to the ground truth than the uniform prediction and it has lower variance (smaller δ ).

Original: Then, TSLA (LSR) with such pre-trained model-based prediction converges faster than TSLA (LSR) with uniform prediction, which veriﬁes our theoretical ﬁndings in Sections 5 (Section 4).
Revised: Then, TSLA (LSR) with such pre-trained model-based prediction converges faster than TSLA (LSR) with uniform prediction, which veriﬁes our theoretical ﬁndings in Section 5 (Section 4).

Original: This observation also empirically tells us the selection of the prediction function (cid:98) y used for smoothing label is the key to the success of TSLA as well as LSR.
Revised: This observation also empirically tells us the selection of the prediction function (cid:98) y used for smoothing label is the key to the success of TSLA as well as LSR.

Original: Among all methods, the performance of TSLA-pre is the best.
Revised: Among all methods, the performance of TSLA-pre is the best.

Original: For top-1 accuracy, TSLA-pre(160) outperforms all other algorithms, while for top-5 accuracy, TSLA-pre(180) has the best performance.
Revised: For top-1 accuracy, TSLA-pre(160) outperforms all other algorithms, while for top-5 accuracy, TSLA-pre(180) has the best performance.

Original: Finally, we observe from Figure 2 that both TSLA and TSLA-pre converge, while TSLA-pre converges to the lowest objective value.
Revised: Finally, we observe from Figure 2 that both TSLA and TSLA-pre converge, while TSLA-pre converges to the lowest objective value.

Original: Similarly, the top-1 and op-5 accuracies show the improvements of TSLA and TSLA-pre at the point of dropping off LSR.
Revised: Similarly, the top-1 and top-5 accuracies show the improvements of TSLA and TSLA-pre at the point of dropping off LSR.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we have studied the power of LSR in training deep neural networks by analyzing
Revised: In this paper, we have studied the power of LSR in training deep neural networks by analyzing

Original: SGD with LSR in different non-convex optimization settings.
Revised: SGD with LSR in different non-convex optimization settings.

Original: The convergence results show that an appropriate LSR with reduced label variance can help speed up the convergence.
Revised: The convergence results show that an appropriate LSR with reduced label variance can help speed up the convergence.

Original: We have proposed a simple and efﬁcient strategy so-called TSLA that can incorporate many stochastic algorithms.
Revised: We have proposed a simple and efﬁcient strategy so-called TSLA that can incorporate many stochastic algorithms.

Original: The basic idea of TSLA is to switch the training from smoothed label to one-hot label.
Revised: The basic idea of TSLA is to switch the training from smoothed label to one-hot label.

Original: Integrating TSLA with SGD, we observe from its improved convergence result that TSLA beneﬁts by LSR in the ﬁrst stage and essentially converges faster in the second stage.
Revised: Integrating TSLA with SGD, we observe from its improved convergence result that TSLA beneﬁts from LSR in the ﬁrst stage and essentially converges faster in the second stage.

Original: Throughout extensive experiments, we have shown that TSLA improves the generalization accuracy of deep models on benchmark data sets.
Revised: Throughout extensive experiments, we have shown that TSLA improves the generalization accuracy of deep models on benchmark data sets.

Original: A T ECHNICAL L EMMA
Revised: A T ECHNICAL L EMMA

Original: Recall that the optimization problem is
Revised: Recall that the optimization problem is

Original: [Equation]
Revised: [Equation]

Original: where the cross-entropy loss function (cid:96) is given by
Revised: where the cross-entropy loss function (cid:96) is given by

Original: [Equation]
Revised: [Equation]

Original: If we set
Revised: If we set

Original: [Equation]
Revised: [Equation]

Original: the problem (7) becomes
Revised: the problem (7) becomes

Original: [Equation]
Revised: [Equation]

Original: Then the stochastic gradient with respective to w is
Revised: Then the stochastic gradient with respective to w is

Original: [Equation]
Revised: [Equation]

Original: Lemma 1.
Revised: Lemma 1.

Original: Under Assumption 1 (i)
Revised: Under Assumption 1 (i)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the facts of y LS t = (1 − θ )
Revised: By the facts of y LS t = (1 − θ )

Original: y t + θ (cid:98) y t and the equation in (11)
Revised: y t + θ (cid:98) y t and the equation in (11)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: ; x t )) =
Revised: ; x t )) =

Original: [Equation]
Revised: [Equation]

Original: 1 − θ ) ∇ (cid:96) ( y t , f ( w t ; x t ))
Revised: 1 − θ ) ∇ (cid:96) ( y t , f ( w t ; x t ))

Original: [Equation]
Revised: [Equation]

Original: (cid:96) ( (cid:98) y t , f ( w t ; x t )) .
Revised: (cid:96) ( (cid:98) y t , f ( w t ; x t )) .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: where (a) uses the convexity of norm, i.e., (cid:107) (1 − θ ) a + θ b (cid:107) 2 ≤ (1 − θ ) (cid:107) a (cid:107) 2 + θ (cid:107) b (cid:107) 2 ; (b) uses assumption 1 (i) and the deﬁnitions in (4), and Assumption 1 (i).
Revised: where (a) uses the convexity of norm, i.e., (cid:107) (1 − θ ) a + θ b (cid:107) 2 ≤ (1 − θ ) (cid:107) a (cid:107) 2 + θ (cid:107) b (cid:107) 2 ; (b) uses assumption 1 (i) and the deﬁnitions in (4), and Assumption 1 (i).

Original: B P ROOF OF T HEOREM
Revised: B P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: By the smoothness of objective function F ( w ) in Assumption 1 (ii) and its remark, we have
Revised: By the smoothness of objective function F ( w ) in Assumption 1 (ii) and its remark, we have

Original: F ( w t +1 ) − F ( w t ) ≤(cid:104)∇
Revised: F ( w t +1 ) − F ( w t ) ≤(cid:104)∇

Original: F ( w t ) , w t +
Revised: F ( w t ) , w t +

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where (a) is due to the update of w t +1 ; (b) is due to (cid:104) a , − b (cid:105)
Revised: where (a) is due to the update of w t +1 ; (b) is due to (cid:104) a , − b (cid:105)

Original: = 12 (cid:0) (cid:107) a − b (cid:107) 2 − (cid:107) a (cid:107) 2 − (cid:107)
Revised: = 12 (cid:0) (cid:107) a − b (cid:107) 2 − (cid:107) a (cid:107) 2 − (cid:107)

Original: b (cid:107) 2 (cid:1) ; (c) is due to η = 1 L .
Revised: b (cid:107) 2 (cid:1) ; (c) is due to η = 1 L .

Original: Taking the expectation over ( x t , y LS t ) on the both sides of (12), we have
Revised: Taking the expectation over ( x t , y LS t ) on the both sides of (12), we have

Original: [Equation]
Revised: [Equation]

Original: (13) where the last inequality is due to Lemma 1.
Revised: (13) where the last inequality is due to Lemma 1.

Original: Then inequality (13) implies
Revised: Then inequality (13) implies

Original: [Equation]
Revised: [Equation]

Original: where (a) is due to θ = 

Revised: where (a) is due to θ =

Original: + δ ; (b) is due to 11+ δ ≤ 1 .
Revised: + δ ; (b) is due to 11+ δ ≤ 1 .

Original: C C ONVERGENCE A NALYSIS OF SGD WITHOUT LSR ( θ = 0 )
Revised: C C ONVERGENCE A NALYSIS OF SGD WITHOUT LSR ( θ = 0 )

Original: Theorem 5.
Revised: Theorem 5.

Original: Under Assumption 1, the solutions w t from Algorithm 1 with θ = 0 satisfy
Revised: Under Assumption 1, the solutions w t from Algorithm 1 with θ = 0 satisfy

Original: [Equation]
Revised: [Equation]

Original: In order to have E R [ (cid:107)∇ F ( w R ) (cid:107) 2 ] ≤ (cid:15) 2 , it sufﬁces to set η = min (cid:16) 1 L , (cid:15) 2 

Revised: In order to have E R [ (cid:107)∇ F ( w R ) (cid:107) 2 ] ≤ (cid:15) 2 , it sufﬁces to set η = min (cid:16) 1 L , (cid:15) 2

Original: Lσ 2 (cid:17) and T = 4 F ( w 0 )
Revised: Lσ 2 (cid:17) and T = 4 F ( w 0 )

Original: η(cid:15) 2 , the total complexity is O (cid:0) 1 (cid:15) 4 (cid:1) .
Revised: η(cid:15) 2 , the total complexity is O (cid:0) 1 (cid:15) 4 (cid:1) .

Original: Proof.
Revised: Proof.

Original: By the smoothness of objective function F ( w ) in Assumption 1 (ii) and its remark, we have
Revised: By the smoothness of objective function F ( w ) in Assumption 1 (ii) and its remark, we have

Original: [Equation]
Revised: [Equation]

Original: (14) where (a) is due to the update of w t +1 .
Revised: (14) where (a) is due to the update of w t +1 .

Original: Taking the expectation over ( x t ; y t ) on the both sides of (14), we have
Revised: Taking the expectation over ( x t ; y t ) on the both sides of (14), we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( w
Revised: ( w

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:3)
Revised: (cid:3)

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: (cid:104) (cid:107)∇ (cid:96) ( y
Revised: (cid:104) (cid:107)∇ (cid:96) ( y

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( w t
Revised: ( w t

Original: [Equation]
Revised: [Equation]

Original: x t ))
Revised: x t ))

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: ( w t ) (cid:107)
Revised: ( w t ) (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: + η
Revised: + η

Original: [Equation]
Revised: [Equation]

Original: E (cid:104) (cid:107)∇ F ( w t ) (cid:107) 2 (cid:105) (
Revised: E (cid:104) (cid:107)∇ F ( w t ) (cid:107) 2 (cid:105) (

Original: [Equation]
Revised: [Equation]

Original: ≤ −
Revised: ≤ −

Original: [Equation]
Revised: [Equation]

Original: (cid:2) (cid:107)∇
Revised: (cid:2) (cid:107)∇

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:3)
Revised: (cid:3)

Original: [Equation]
Revised: [Equation]

Original: 15) where (a) and (b) use Assumption 1 (i); (c) uses the facts that η ≤ 1 L and Assumption 1 (i).
Revised: 15) where (a) and (b) use Assumption 1 (i); (c) uses the facts that η ≤ 1 L and Assumption 1 (i).

Original: The inequality (15) implies
Revised: The inequality (15) implies

Original: [Equation]
Revised: [Equation]

Original: By setting η ≤ (cid:15) 2 2 Lσ 2 and T = 4 F ( w 0 ) η(cid:15) 2 , we have 1 T (cid:80) T − 1 t =

Revised: By setting η ≤ (cid:15) 2 2 Lσ 2 and T = 4 F ( w 0 ) η(cid:15) 2 , we have 1 T (cid:80) T − 1 t =

Original: E (cid:2) (cid:107)∇ F ( w t ) (cid:107) 2 (cid:3) ≤ (cid:15) 2 .
Revised: E (cid:2) (cid:107)∇ F ( w t ) (cid:107) 2 (cid:3) ≤ (cid:15) 2 .

Original: Thus the total complexity is in the order of O (cid:16) 1 η(cid:15) 2 (cid:17) =
Revised: Thus the total complexity is in the order of O (cid:16) 1 η(cid:15) 2 (cid:17) =

Original: O (cid:0) 1 (cid:15) 4 (cid:1) .
Revised: O (cid:0) 1 (cid:15) 4 (cid:1) .

Original: D P ROOF OF T HEOREM
Revised: D P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Following the similar analysis of inequality (13) from the proof of Theorem 3, we have
Revised: Following the similar analysis of inequality (13) from the proof of Theorem 3, we have

Original: [Equation]
Revised: [Equation]

Original: Using the condition in Assumption 2 we can simplify the inequality from (16) as
Revised: Using the condition in Assumption 2 we can simplify the inequality from (16) as

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to the deﬁnition of loss function that F ∗ ≥ 0 .
Revised: where the last inequality is due to the deﬁnition of loss function that F ∗ ≥ 0 .

Original: Since η 1 ≤ 1 L < 1 µ , then (1 − η 1 µ ) t +1 < exp( − η 1 µ ( t + 1)) and (cid:80) ti =

Revised: Since η 1 ≤ 1 L < 1 µ , then (1 − η 1 µ ) t +1 < exp( − η 1 µ ( t + 1)) and (cid:80) ti =

Original: (1 − η 1 µ ) i ≤ 1 η 1 µ .
Revised: (1 − η 1 µ ) i ≤ 1 η 1 µ .

Original: As a result, for any T 1 , we have
Revised: As a result, for any T 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Let θ = 11+ δ and (cid:98) σ 2 :=
Revised: Let θ = 11+ δ and (cid:98) σ 2 :=

Original: [Equation]
Revised: [Equation]

Original: 1 − θ ) σ 2 +
Revised: 1 − θ ) σ 2 +

Original: [Equation]
Revised: [Equation]

Original: 2 = 2 δ 1+ δ σ 2 then 12 µ (cid:0)
Revised: 2 = 2 δ 1+ δ σ 2 then 12 µ (cid:0)

Original: [Equation]
Revised: [Equation]

Original: 1 −
Revised: 1 −

Original: [Equation]
Revised: [Equation]

Original: ) σ 2 + θδσ 2 (cid:1) ≤ F ( w 0 ) since δ is small enough and η 1 L ≤ 1 .
Revised: ) σ 2 + θδσ 2 (cid:1) ≤ F ( w 0 ) since δ is small enough and η 1 L ≤ 1 .

Original: By setting
Revised: By setting

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: After T 1 iterations, we drop off the label smoothing, i.e. θ = 0 , then we know for any t ≥ T 1 , following the inequality (15) from the proof of Theorem 5, we have
Revised: After T 1 iterations, we drop off the label smoothing, i.e. θ = 0 , then we know for any t ≥ T 1 , following the inequality (15) from the proof of Theorem 5, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, we get
Revised: Therefore, we get

Original: [Equation]
Revised: [Equation]

Original: (19) where (a) is due to F ( w T 1 + T 2 − 1 ) ≥ F ∗ .
Revised: (19) where (a) is due to F ( w T 1 + T 2 − 1 ) ≥ F ∗ .

Original: By setting η 2 =
Revised: By setting η 2 =

Original: [Equation]
Revised: [Equation]

Original: 2 2 Lσ 2 and T 2 = 8 δσ 2 µη 2 (cid:15) 2 , we have 1 T 2 (cid:80) T 1 + T 2 − 1 t =
Revised: 2 2 Lσ 2 and T 2 = 8 δσ 2 µη 2 (cid:15) 2 , we have 1 T 2 (cid:80) T 1 + T 2 − 1 t =

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Revised: E A DDITIONAL E XPERIMENTS

Revised: In this section

Revised: [Table]

Revised: we study an ablation study for the smoothing parameter θ .

Revised: We follow the same settings in Subsection 6.2 but use different values of θ in LSR and TSLA.

Revised: We summarize the results in Table 4.

Revised: The results show that the different values of θ can affect the performances of LSR and

Revised: TSLA.

Revised: Besides, TSLA(180) has the best performance for each value of θ

Revised: [Table]

Revised: [Table]