From: ICLR.cc/2021/Conference/Program_Chairs
title: Final Decision
decision: Reject
comment: The paper considers ways to understand label smoothing methods, which are widely used in many applications.  There is some theory on the performance of SGD with and without the methods of the paper, but there is s significant gap in terms of how the theory offers insight into label smoothing.  There are some empirical results, but they are insufficient and there is not much description of the experimental setup.  There was a diversity of reviews.  But, after a discussion among reviewers, it was felt that, overall, another iteration on improving the coherence and presentation of the paper will make it much better for the community.

From: ICLR.cc/2021/Conference/Paper1950/Authors
title: Thanks
comment: We thank you greatly for your time and your reevaluation. Once the experiments are done, we will include the complete results in the final version.

From: ICLR.cc/2021/Conference/Paper1950/AnonReviewer4
title: Response
comment: Thank you for your comments. I have read the updated version of the paper and modified my review accordingly.

* on {\hat y} vs {y^{LS}}, F*, on relation with LSR - i was wrong, thank you for clarifying

Still, I highly encourage authors to report the results they obtained with additional optimizers and architectures in (at least) the appendix section.

From: ICLR.cc/2021/Conference/Paper1950/Authors
title: General Author Response
comment: We would like to thank all reviewers for their precious time and their constructive comments and suggestions. We updated the manuscript according to reviewers' suggestions. All changes in the revision are marked in red for convenience.

From: ICLR.cc/2021/Conference/Paper1950/Authors
title: Response to AnonReviewer4
comment: We would like to thanks the reviewer for carefully reviewing our papers. We fixed all typos in the revision.

Q1. More complex optimizers (e.g. adam) and architectures (e.g. Transformer)

A:  First, for the considered image classification tasks, we used momentum SGD in the experiments since it has the best performance. Second, our preliminary result on NLP where uses transformer architecture and Adam optimizer shows that TSLA has the best performance among LSR and baseline (i.e., without LSR). Third, we modified the description that TSLA can work with any stochastic gradient optimizer. Our initial goal of this claim is to point out that TSLA is a generic algorithm, and in practice, it may use different optimizer for different tasks. 

Q2. TSLA disables label smoothing during the final stages of training.

A: According to the theoretical analysis of LSR, it may not converge. However, the designed TSLA can enjoy the benefit of label smoothing from the first stage and it also can guarantee the theoretical convergence from the second stage.

Q3. About the ImageNet pre-trained model.

A: It is downlowded from https://pytorch.org/docs/stable/torchvision/models.html, as shown after reference [Paszke et. al. 2019]. To use this model from the 1000 ImageNet classes to CIFAR-100, we only need to replace the FC layer and to use fine-tuning technique, which are standard in literature. 

Q4. Momentum parameter is fixed as 0.9.

A: Yes, the momentum parameter is fixed as 0.9 in stage 1 and stage 2 of TSLA.

Q5. about the value of $\theta$.

A: Yes, we tune $\theta\in$ {$0.1, 0.2, 0.4, 0.9$} for LSR and report the best performance. For TSLA, we use the same $\theta$ as LSR. In the revision, we provide a ablation study for $\theta$ in Appendix~E.

Q6. Math

Definition 1: Yes, this is a typo and it should be $F$.

Assumption 2: We agree that some problems may have many global optima, but the optimal value $F_*$ must be unique (otherwise it is not an optimal value by definition). This assumption needs to hold for a unique $F_*$.  

Equation 4: No, it is $\mathbf{\widehat y}$.

Finally, we thank AnonReviewer4 for pointing out many typos and we modified them in the revision.

From: ICLR.cc/2021/Conference/Paper1950/Authors
title: Response to AnonReviewer3
comment: Q1. About the contribution and novelty in this paper.

A: We believe that it is due to critical misunderstanding of the purpose of this paper and our contributions.  1) First, one purpose is to understand the power of LS from the view of optimization with the fewest possible modifications from real tasks. To ensure that the theory is applicable, we focus on non-convex optimization with cross-entropy loss (2). It is worth mentioning that in practice the empirical experiments of label smoothing problems are conducted with cross-entropy loss. To the best of our knowledge, this is the first work that establishes iteration complexities of stochastic gradient descent with LSR for finding an $\epsilon$-approximate stationary point in solving a smooth non-convex problem. The result theoretically explains why an appropriate LSR can help speed up the convergence. 2) Second, the proposed TSLA and its analysis are {\bf new and non-trivial}. We theoretically and empirically shown that TSLA is better than LSR.  In summary, the novelty of this paper lies at that it explains the power of LSR from the view of optimization and it proposes a new and non-trivial TSLA that is better than LSR both in theory and in practice. 

Q2. the role of $\delta$.

A: Based on (4), the definition of $\delta$ is $\delta := \frac{\hat\sigma^2}{\sigma^2}$. One natural information of $\delta$ we have is $\delta>0$, meaning that it can be either smaller than 1 or greater than 1. While we believe that the dimension dependency usually comes from the variance terms $\hat\sigma^2$ and  $\sigma^2$, one can not simply claim $\delta$ is dimension-dependence or not. In fact, one situation is that both $\hat\sigma^2$ and  $\sigma^2$ can have a $O(d)$, and then it will be cancelled in $\delta$, leading to $\delta$ is independent of $d$. 

Q3. comparisons in Table 1 and Theorem 3 for different possibilities of $\delta$.

A:  While we agree that $\delta$ is independent of $\epsilon$, we just list the different possibilities of $\delta$ when the target accuracy $\epsilon$ is given in Table 1 as well as in Theorem 3. In practice, the value of $\epsilon$ is always given in advance due to the settings of parameters (e.g., total iterations, learning rate and mini-batch size) could be dependent of $\epsilon$. For example, when using label smoothing trick in image classification tasks, the total iteration number $T$ which is related to $\epsilon$ (e.g. $T=O(\epsilon^{-4})$) is given before conducting an optimizer such as SGD. In fact, we do the comparisons in Table 1 and Theorem 3 only for ease of reading. If AnonReviewer3 still feel that the comparisons will lead to misleading, we can remove them in the final version and they will not affect the main results of this paper.

Q4. TSLA vs. standard SGD

A:  First, while we agree that the convergence guarantee is due to the standard SGD in the second stage of TSLA, one contribution of our analysis is that it can enjoy the parameter $\delta$ in the final complexity, which can not be done by simply running standard SGD. Second, the comparison of TSLA and standard SGD by AnonReviewer3 is not fair. When it comes to dimension-dependency, standard SGD can also have a $O(d)$ in the complexity since the dimension-dependency can be from the variance terms $\sigma^2$. Third, we do not claim TSLA is always better than vanilla SGD. When the selection of label $\mathbf{\widehat y}$ is not appropriate, $\delta$ can be large so that TSLA is worse than SGD. In this case, however, TSLA still converges faster than LSR. TSLA still has its value in practice because it is a better choice for people who want to use label smoothing in practice to overcome over-fitting issue.

Q5. New algorithm/analysis.

Thank you for your valuable comments. It could be considered as a future work.

From: ICLR.cc/2021/Conference/Paper1950/Authors
title: Response to AnonReviewer2
comment: Q1. This paper focuses on the original ordinary learning.

A: Since LS was firstly proposed in the original ordinary learning (Szegedy et al., 2016), as an initial study of TSLA, we do not consider other settings in this paper.  The theoretical analysis of TSLA and LSR for other settings can be considered as a future work. However, we have a {\bf preliminary} empirical result of knowledge distillation which is implemented on CIFAR-100 dataset. We observe that TSLA does outperform LSR but it is slightly worse than baseline. We would like to mention that this result matches our theoretical findings since the possible reason is that the selection of $\mathbf{\widehat y}$ is not appropriate and thus the $\delta$ is too large.  We will include the complete experimental results in the final version.

Q2. experiments to support these claims given in page 5. 

A:  In fact, our experimental results did so. In Figure 1, TSLA outperforms LSR. Please note that TSLA optimizes the one-hot loss while LSR optimizes all K losses at the second stage.  

Q3. For Stanford Dogs and CUB-2011, 90 epochs looks like it is quite small. 

A: These two datasets are pre-trained, so 90 epochs is sufficient for baseline. For fair comparison, we fixed the total epochs for all algorithms even TSLA(60), TSLA(70) and TSLA(80) do not converge.    


Q4. In the experiments, it would be better to have a validation dataset to choose the best hyper-parameter.

A: While we agree that it is OK in practice, we give ablation study here by choosing different $T_1$.

From: ICLR.cc/2021/Conference/Paper1950/Authors
title: Response to AnonReviewer1
comment: Q1. (i). evidence or analysis of the TSLA method for training teacher models.

A: We have a {\bf preliminary} empirical result which is implemented on CIFAR-100 dataset. We observe that TSLA does outperform LSR but it is slightly worse than baseline. Please note that this result also matches our theoretical findings and the possible reason is that the selection of $\mathbf{\widehat y}$ is not appropriate and the $\delta$ is large. We expect to include the complete experimental results about teacher models in the final version.

Q1. (ii) the performance under different $\theta$ in Theorem 3. 

A:  We've conducted experiments on CIFAR-100 dataset, and the results show that the $\theta$ can influence performance of LSR. See the performance as follows. In the revision, we provide a ablation study for $\theta$ in Appendix E.

$\theta$:  0.2,   0.4,   0.9

LSR: 77.75, 77.72, 76.40

Q1. (iii). experiments on the other optimizers such as Adam or Adagrad. 

A:  We used momentum SGD in the experiments since it has the best performance for the settings of the considered problems. However, our preliminary result on NLP which uses transformer and Adam shows that TSLA has the best performance among LSR and baseline (i.e., without LSR).  

Q2. Even though the convergence speed is theoretically analyzed, it still does not explain why LSR can improve over SGD, namely the better convergence point. In other words, these bounds do not explain the gap among baseline, LSR, and TSLA in Figure 1.

A: When we say the convergence complexity of Algorithm $\mathcal A$ is better than that of Algorithm $\mathcal B$, it means that to obtain a given target accuracy level, Algorithm $\mathcal A$ needs less iterations than Algorithm $\mathcal B$. In other words, given a fixed number of iterations, Algorithm $\mathcal A$ will obtain a better convergence point than Algorithm $\mathcal B$. So for the sake of fair comparison in the experiments, we fixed the total number of epochs for all algorithms, and see which one converges to the best point. This is what many papers have done, and we are following general guidelines in literature.

Q3. experiment setup such as dropout, batch normalization. 

A: As we mentioned in the first paragraph of Section 6 on Page 6, we use standard ResNet model (He et al., 2016) so we do not use dropout. The structure of batch normalization can be found in the original paper of ResNet (He et al., 2016).

From: ICLR.cc/2021/Conference/Paper1950/AnonReviewer1
title: An Interesting Paper Understanding Label Smoothing Techniques
review: [strong points]
1. This paper is smooth and well-motivated. Label smoothing is a well-known trick to improve the multi-class neural network. This is the first work to theoretically understand the effect of label smoothing by analyzing the convergence speed.

2. Based on some reasonable assumption, this paper proves that SGD with LSR can have faster convergence speed than SGD without LSR when delta is small, and it will converge to a worse level of O(delta) when \delta is large.

3. By observing that LSR may have adverse effects during the later training epochs, a simple two-stage algorithm is proposed to take advantage of (i) fast convergence speed of SGD with LSR; (ii) easy training and better convergence guarantee of SGD without LSR.

[negative points]
1. Experiment is not enough. 
  (i). The authors mention several times that LSR may not work well for training teacher models. However, there is no evidence or analysis of whether the TSLA method can help solve this problem. 
  (ii) In theorem  3, \theta is chosen to be 1/(1+delta). However, it is not clear whether different \theta will influence performance. 
  (iii). The experiments are tested based on SGD. It is better to show some results on the other optimizers frequently used in deep learning, such as Adam or Adagrad. Meanwhile, it will be better to show that the bound in SGD can be generalized to the other stochastic optimizers.

2. Even though the convergence speed is theoretically analyzed, it still does not explain why LSR can improve over SGD, namely the better convergence point. In other words, these bounds do not explain the gap among baseline, LSR, and TSLA in Figure 1. 

3. Lack of experiment setup. Little information is provided about the training techniques such as dropout, batch normalization. These techniques may have an influence on LSR.
rating: 4: Ok but not good enough - rejection
confidence: 3: The reviewer is fairly confident that the evaluation is correct

From: ICLR.cc/2021/Conference/Paper1950/AnonReviewer2
title: Gives theoretical understanding of label smoothing from the optimization point of view and gives a practical extension of label smoothing
review: Summarizing the paper:
Label smoothing is a popular regularization method that is used for deep learning models with classification problems, but has not been studied from a theoretical point of view extensively. This paper studies label smoothing from a optimization view and shows the convergence behavior when SGD is used, showing the improvement over the one-hot label learning under certain conditions. The paper then proposes a new method called Two-Stage Label smoothing algorithm (TSLA), which simply turns off the smoother after a certain number of epochs. According to the analysis, TSLA has improved convergence under certain conditions over the original label smoothing baseline. The experiments show how the proposed TSLA performs better than the label smoothing baseline for 3 image datasets.

Strengths of the paper:
The paper's motivation is clear: to study a well known regularizer, label smoothing, from an optimization point of view. It compares the convergence properties of SGD without label smoothing and SGD with label smoothing. It further proposes an original method that first uses label smoothing but later turns it off, which is shown to be empirically better with some convergence properties.

Weaknesses and issues of the paper:
In the beginning of Section 5, it discusses several papers that show the harm of label smoothing, under problem settings such as few-shot learning, knowledge distillation, and transfer learning, as the motivation for thinking of a two step strategy. However, this paper focuses on the original ordinary learning (no transfer learning and not few-shot learning) setup and the connection between these were not so clear.

The intuition behind why label smoothing may be harmful is given in page 5, but it would be nice to have some experiments to support these claims. For example, if optimizing all K loss functions with label smoothing is harder, having more classes should lead to more harm when label smoothing is used.

For Stanford Dogs and CUB-2011, 90 epochs looks like it is quite small. The learning curves for accuracy/loss still seems to be going up/down respectively for (60), (70), and (80).

Other comments:
In the experiments, it would be better to have a validation dataset to choose the best hyper-parameter (the epoch number when LBR is dropped) and report the test performance based on the chosen one. (I'm guessing it will still be better than the baselines since in most hyper-parameter candidates, the proposed method seems to be working better.)

In page 8: op-5 --> top-5

In page 4: different --> difference

================= after response 

Thank you for providing the response. I would like to add some comments to the response.

About Q1-- I agree that it is important to work on the original ordinary learning as the first paper in this direction, but it makes the story of the paper inconsistent. The adverse affect of LSR might only show up in other problem settings (distillation/few-shot/transfer learning) and not in ordinary learning, and if that is the case, it is unclear why the paper focuses on ordinary learning. (This is just a comment on the story/motivation of the paper.)

About Q2-- Although Fig.1 shows the benefits of the one-hot label over label smoothed version in latter stages of training, I believe it does not show how the number of K plays a role in the performance and the difficulty.

About Q3-- I agree that 90 epochs is sufficient for the LSR baseline, and the current figures demonstrate the benefits of the proposed method, but I was more interested in the difference of (s) in TSLA(s). From Fig.1, it still seems like TSLA(60,70,80) are improving at the end of 90 epochs, while TSLA (30,40,50) seem to have converged.

rating: 6: Marginally above acceptance threshold
confidence: 3: The reviewer is fairly confident that the evaluation is correct

From: ICLR.cc/2021/Conference/Paper1950/AnonReviewer3
title: trivial results, strong rejection after reading author response 
review: This paper analyzes the convergence of SGD with a biased gradient, where the bias comes from label smoothing. The paper positions itself as a theorectical work towards understanding the success of such smoothing trick, but I feel the analysis does not add too much to the literature and its main results are somewhat trivial/misleading.

In particular, given that the analysis hinges on a specific loss function, i.e. Eq. (2), which is linear in terms of $y_i$, label smoothing quickly translates to additive terms that can be well-controlled separately from the unbiased gradient. It is thus not surprising that standard analysis of non-convex SGD goes through here with a new additive term appearing in the final gradient upper bound, as shown in the paper. For this reason, I do not see quite technical novelty in this paper.

My another concern is that the role of $\delta$ is unclear. By Eq. (4), this quantity is data-independent and should be treated as only depending on the distribution of $(x, \hat{y})$. Now if we take a close look at the main result, i.e. Theorem 3, it is fairly a weak result saying that SGD converges to a point with constant gradient. This is because the first case in Theorem 3 is essentially ensuring convergence to $\epsilon$-stationary point only when $\epsilon \geq \Omega(\sqrt{\delta}) \geq \Omega(1)$, i.e. a point with constant gradient. Likewise, the second case also boils down to the same guarantee.

As such, the comparison in Table 1 seems problematic since $\epsilon$ should be treated as a variable arbitrarily close to 0. In words, only the first row of Table 1 will make sense. Yet, there are still two issues here.
- I believe that the infinite iteration complexity of LSR is due to the drawback of Theorem 3 as I just pointed out. Namely, running LSR only gives you a point with constant gradient. Although TSLA does converge to $\epsilon$-stationary point, such performance guarantee is *not* due to your design of TSLA, but follows from standard SGD. In fact, simply running SGD from the very beginning already gives such guarantee. Thus, the first row to compare TSLA and LSR is somewhat misleading.
- Now the question boils down to why not simply running SGD. In the first row of the table (and main text), it is argued that TSLA (i.e. LSR + SGD) has improved iteration complexity of $O(\delta / \epsilon^4)$ which is better than the $O(1/\epsilon^4)$ of vanilla SGD. I do not really agree with the conclusion because $\delta$ is independent of $\epsilon$. In fact, it can be a dimension-dependent quantity, and e.g. even blow up to $O(d)$ or so, making the iteration complexity of TSLA worse than SGD by a dimension-dependent factor.

To make the work qualify for a top-tier venue, authors need to either present a new algorithm/analysis with vanishing gradient, i.e. an upper bound of the form $O(\delta \cdot \epsilon)$, or show hardness result, say any algorithm that takes the label smoothing must incur a stationary point with gradient $\geq \Omega(\epsilon + \delta)$.


---------------------Updates after author response---------------------


The authors basically posted their response at the last minute of the window which eliminates the possibility for further discussion. While it is lengthy and point-to-point, I found it failed to clear up any of my concerns.

I am very disappointed that even after they recognized the misleading arguments in "convergence" analysis, in Theorem 3 of the revised version, they are still claiming $\| \nabla F(w_R) \| \leq \epsilon$ when $\delta < O(\epsilon^2)$ as "convergence". Such conclusion will be EXTREMELY MISLEADING if readers missed or did not carefully think about the condition on $\epsilon$. Authors may want to refer to a calculus textbook for the rigorous definition of convergence.

Regarding technical contribution,

- Authors acknowledged that their analysis does *not* guarantee convergence to stationary point, since there is an additive $O(\delta)$ term in the gradient upper bound. This immediately diminishes their theoretical contribution.

- Authors did not justify their technical novelty. In fact, if we carefully look at their analysis in the appendix, it follows from standard SGD with a slight adjustment to the biased gradient induced by label smoothing (which is acknowledged by the authors). This is an additive gradient and thus is easily controlled; it is also the main reason why in their main theorem, the gradient upper bound suffers a non-vanishing $O(\delta)$ term.

- Authors argued that compared to running SGD from scratch, the benefit of LSR+SGD is the introduction of $\delta$. However, this quantity itself is out of control. Note that even they were able to show $\delta=O(1)$, it is not strong enough here since this can easily be obtained by running SGD. The only way that I see will save the paper is to show under distributional assumption of the data, that $\delta = O(1/d)$ for example. However, I did not see how to make it happen, and authors completely ignored such analysis in their response.

Overall, this is a paper playing tricks on its technical parts. It is decorated with bunch of mathmatical analysis most of which is known and standard, and the introduction of new insights is minimal. I will be shocked if it gets accepted in ICLR or equivalent conferences.
rating: 1: Trivial or wrong
confidence: 5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature

From: ICLR.cc/2021/Conference/Paper1950/AnonReviewer4
title: Initial review
review: **Overview**

This paper investigates the effects of Label Smoothing Regularization and proposes two contributions: improved theoretical convergence guarantees for SGD with label-smoothing and a practical two-stage training strategy (TSLA) that uses label smoothing during the first several epochs and disables it thereafter.

The performance gains from TSLA are demonstrated both theoretically and in three image classification experiments (all three use ResNet18 + momentum) against reasonably tuned baselines. The overall readability of the paper is decent. There are numerous minor issues (see below), they are all trivial to fix.


**Score & justification**

While this paper definitely has scientific merit, i cannot recommend acceptance in its current form.
My main issue with the paper is with the claim that TSLA can work with any stochastic gradient optimizer (momentum, adam, etc, see p5 second to last paragraph). While I have no doubts that one can plug in any optimizer, the theoretical guarantees were provided for SGD, and, more importantly, the experimental results only cover SGD+Momentum (not to mention, a single model architecture). More complex optimizers (e.g. adam) and architectures (e.g. Transformer) may introduce problems when switching between training phases. Therefore, I suggest further supporting authors' claim by conducting additional experiments in other popular LSR use cases. 

For example, one such use case is Machine Translation. Most recent models for this task use label smoothing to improve BLEU (e.g. [1, 2]). One can train a transformer-base model with Adam optimizer for IWSLT14 de-en[3] task in around the same time that it takes to train ResNet18 on CIFAR (see fairseq[4]). Of course, any other task would also support the claim, provided it doesn’t use the same model and/or optimizer.

**[resolved]** Another issue I have is a subjective one (so it didn’t affect my score): it feels that the paper contains two independent lines of research with limited synergy. For instance, the theoretical study estimates label smoothing at convergence while TSLA seemingly disregards this by disabling label smoothing during the final stages of training.


**Questions:**

> Page 8: We use two different labels to smooth the one-hot label, the uniform distribution over all labels and the distribution predicted by an ImageNet pre-trained model which downloaded directly from PyTorch

**[resolved]** Please clarify: how exactly do you adapt the predictions from 1000 ImageNet classes to CIFAR-100 classes?
As far as I understand, there are no pre-trained models in PyTorch itself. Did you mean TorchVision or torch Hub? If so, it would be great to state the exact model and library version to facilitate reproducibility. Alternatively, publishing the source code of the proposed solution should also do the trick as long as it clearly describes all the requirements.


> Page 6 (Experiments): The momentum parameter is fixed as 0.9

**[resolved]** Momentum SGD and other popular algorithms maintain a set of statistics accumulated over training batches. Please clarify: do these statistics carry over from stage 1 to stage 2 of TSLA? If not, what happens to them at T1?


> Page 7,8: the value of theta

**[resolved]** As far as i understand, you select theta by maximizing the performance of LSR baseline and then use the same theta for TSLA (if not, please clarify). Please elaborate on how exactly you tuned theta. Is there some intuition why optimal theta for LSR would generalize to TSLA? If not, how does TSLA perform under exaggerated theta?



**Math**

*Definition 1:*
**[resolved]** The formulae use lowercase f, suggesting that the gradient/jacobian of the __model output__  should be zero. Did you mean capital F?

*Assumption 2:*
**[resolved]** Q1: Neural networks typically have many global optima due to inherent invariances in the model. Does this assumption need to hold for every F*?


*Equation 4: {\hat y} vs {y^{LS}}:*
**[i was wrong]** As I understood, \hat y is the common smoothing vector that is used to construct y^LS. Did you mean to use y^LS in the objective function? Note: the same expression with \hat y is also used later on pages 3-4.


**Typos & presentation:**

**Page 2 (last paragraph)** *the example-label pairs are draw*

**Page 3 (after remark 1)** *label y is draw from*

I’d recommend changing “draw” to “drawn”.


**Page 3 (after remark 1)** *Let y be a label introduced for smoothing label*

While the sentence is technically correct, I would suggest paraphrasing.


**Page 3, remark 2** *we do not require the stochastic gradient … is unbiased*

Suggestion: “require **that** ... is unbiased” or "require ... **to be** unbiased"


**Page 3 (assumption 2)** *i.e. there is no very bad local optimum on the surface of objective function*

Again, technically correct, but i’d recommend paraphrasing.


**Page 4 (remark)**  *sample complexity for training a learning model from*

Did you mean “training a model” (w/o learning) or “training a machine learning model”?


**Page 4** *the different between SGD with LSR and SGD without LSR*

Did you mean “the difference”?


**Page 4 (bottom)** *miniImageNet*

This issue is of negligible importance, but i’d recommend using one of mini-ImageNet, Mini-ImageNet or MinImageNet.


**Page 4 (bottom)** *between input example and output logit*

Maybe there’s a missing “the”


**Algorithm 2**

Should T1 be listed as an input (along with theta, eta1, eta2)


**Page 5** *It seems that training smoothed label in the late epochs*

Training **with** smoothed labels?


**Page 5** *it runs a stochastic algorithm A(e.g., SGD) with LSR in T1 iterations*

Runs … **for** T1 iterations?


**Page 6** *LSR dose not converge*

LSR **does** not converge


**Page 7 (bottom)** *CIFRA-100*

Did you mean **CIFAR**-100?


**Page 7** *TSLA may not converges if it drops*

may not **converge** (without s)


**Page 7** *… and divide them by 10 every 60 epochs suggested in ...*

… **as** suggested in …?


**Page 8** *pre-trained model which downloaded*

which **is/was** downloaded?


**Page 8 (conclusion)** *that TSLA benefits by LSR in the first stage*

benefits **from** LSR?


**Page 8** *which verifies our theoretical findings in Sections 5 (Section 4)* 

Section/sections & formatting


**References**

[1] Attention is all you need, Vaswani et al. https://arxiv.org/pdf/1706.03762.pdf
[2] Joint Source-Target Self Attention with Locality Constraints, Iyer et al, https://arxiv.org/pdf/1905.06596v1.pdf 
[3] http://workshop2014.iwslt.org/downloads/proceeding.pdf
[4] see Fairseq library, examples/translation is a tutorial for IWSLT14 github.com/pytorch/fairseq/tree/master/examples/translation 


**EDIT** my original score was 5; i have read the authors response and updated my score to 6.

My score represents the experimental side of the paper, i do not feel confident judging the impact of theoretical contributions.

rating: 6: Marginally above acceptance threshold
confidence: 3: The reviewer is fairly confident that the evaluation is correct