Towards a Better Understanding of Label Smoothing in Neural Machine Translation (2020)

Abstract: In order to combat overfitting and in pursuit of better generalization, label smoothing is widely applied in modern neural machine translation systems. The core idea is to penalize over-confident outputs and regularize the model so that its outputs do not diverge too much from some prior distribution. While training perplexity generally gets worse, label smoothing is found to consistently improve test performance. In this work, we aim to better understand label smoothing in the context of neural machine translation. Theoretically, we derive and explain exactly what label smoothing is optimizing for. Practically, we conduct extensive experiments by varying which tokens to smooth, tuning the probability mass to be deducted from the true targets and considering different prior distributions. We show that label smoothing is theoretically well-motivated, and by carefully choosing hyperparameters, the practical performance of strong neural machine translation systems can be further improved.

## introduction in recent years, neural network (nn) models bring steady and concrete improvements on the task of machine translation (mt). from the introduction of sequence-to-sequence models @xcite , to the invention of the attention mechanism @xcite , end-to-end sequence learning with attention becomes the dominant design choice for neural machine translation (nmt) models. from the study of convolutional sequence to sequence learning @xcite , to the prosperity of self-attention networks @xcite , modern nmt systems, especially transformer-based ones @xcite , often deliver state-of-the-art performances @xcite , even under the condition of large-scale corpora @xcite . in transformer-based models, label smoothing is a widely applied method to improve model performance. @xcite initially introduce the method when making refinements to the inception @xcite model, with the motivation to combat overfitting and improve adaptability. in principle, label smoothing discounts a certain probability mass from the true label and redistributes it uniformly across all the class labels. this lowers the difference between the largest probability output and the others, effectively discouraging the model to generate overly confident predictions. since information entropy @xcite can be thought of as a confidence measure of a probability distribution, @xcite add a negative entropy regularization term to the conventional cross entropy training criterion and compare it with uniform smoothing and unigram smoothing. @xcite deliver further insightful discussions about label smoothing, empirically investigating it in terms of model calibration, knowledge distillation and representation learning. label smoothing itself is an interesting topic that brings insights about the general learnability of a neural model. while existing methods are rather heuristical in their nature, the fact that simply discounting some probability mass from the true label and redistributing it with some prior distribution (see figure 1 for an illustration) works in practice, is worth to be better understood. in this paper, we raise two high-level research questions to outline our work: ## related work the extensive use of nns in mt @xcite @xcite is a result of many pioneering and inspiring works. continuousvalued word vectors lay the foundation of modern natural language processing (nlp) nns, capturing semantic and syntactic relations and providing numerical ways to calculate meaningful distances among words @xcite @xcite @xcite . the investigations of sequence-to-sequence learning @xcite , the studies of attention mechanism @xcite and the explorations into convolutional and self-attention nns @xcite ) mark steady and important steps in the field of nmt. since the introduction of bert @xcite , the transformer model @xcite becomes the de facto architectural choice for many competitive nlp systems. among the numerous ingredients that make transformer networks successful, label smoothing is one that must not be overlooked and shall be the focus of this work. the idea of smoothing is not new in itself. for instance, many smoothing heuristics and functions are investigated in the context of count-based language modeling @xcite @xcite @xcite . interestingly, when training nns, the idea of smoothing comes in a new form and is applied on the empirical one-hot target distributions. proposed to counteract overfitting and pursue better generalization, label smoothing @xcite finds its first applications in nns in the field of computer vision. later, the method is shown to be effective in mt @xcite . furthermore, it is also helpful when applied in other scenarios, e.g. generative adversarial networks (gans) @xcite , automatic speech recognition @xcite , and person re-identification @xcite . since the method centralizes on the idea of avoiding over-confident model outputs on training data, it is reanalyzed in @xcite . the authors include an additional confidence penalty regularization term in the training loss, and compare it to standard label smoothing with uniform or unigram prior. while label smoothing boosts performance significantly compared to using hard target labels, the difference in performance gains when comparing different smoothing methods is relatively small. @xcite bring recent advancements towards better intuitive understandings of label smoothing. they observe a clustering effect of learned features and argue that label smoothing improves model calibration, yet hurting knowledge distillation when the model is used as a teacher for another student network. as a regularization technique in training, label smoothing can be compared against other methods such as dropout @xcite and dis-turblabel @xcite . intuitively, dropout can be viewed as ensembling different model architectures on the same data and disturblabel can be viewed as ensembling the same model architecture on different data, as pointed out in @xcite . interestingly, label smoothing can also be understood as estimating the marginalized label dropout during training @xcite . in this paper, we propose two straightforward extensions to label smoothing, examining token selection and prior distribution. @xcite and @xcite investigate a similar issue to the former. in the context of gans, they select only those positive examples to smooth while we consider the task of mt, discussing how many tokens to smooth and how they should be selected. @xcite and @xcite talk about ideas similar to the latter. in their respective contexts, one experiments with unigram probabilities for label smoothing and the other uses language model (lm) posteriors to softly augment the source and target side of mt training data. ## solving the training problem the standard label smoothing (stn) loss, as used by @xcite , can be expressed as: the confidence penalty (cfd) loss, as used by @xcite , can be expressed as: in both cases, the outer summation is over all of the training tokens n , implicating that all of the target token probabilities are smoothed. the dependencies of q v and p v on n are omitted for simplicity. additionally for equation 1, authors of both papers @xcite point out that the uniform prior can be replaced with alternative distributions over the target vocabulary. one more thing to notice is the negative sign in front of the non-negative term m in equation 2 , which means that p v -m q v is not a probability distribution anymore. one can nonetheless apply tricks to normalize the term inside the parentheses so that it becomes a probability distribution, e.g.: and implement it as an additional layer of activation during training, where v is an alternative running index in the vocabulary. in any case, the integration of equation 2 into the form of equation 1 cannot be done without significantly modifying the original confidence penalty, and we leave it for future work. ## finding a good recipe in this section, we describe our results and insights towards a good recipe to successfully apply label smoothing. we experiment with six iwslt2014 datasets: german (de), spanish (es), italian (it), dutch (nl), romanian (ro), russian (ru) to english (en), and one wmt2014 dataset: english to german. the statistics of these datasets are summarized in ## analyzing the mismatch as discussed in section 4.3, models smoothed with lms or mt model posteriors yield very good development set perplexities but no big improvements in terms of test bleu scores. here, we further investigate this phenomenon in terms of search and scoring. ## conclusion in this work, we investigate label smoothing in neural machine translation. considering important aspects in label smoothing: token selection, probability mass and prior distribution, we introduce a generalized formula and derive theoretical solutions to the training problem. examining the effect of various hyperparameter choices, practically we show that with a good label smoothing recipe, one can obtain consistent improvements over strong baselines. delving into search and scoring, we finally emphasize the mismatch between training and testing, and motivate future research. reassuring that label smoothing brings concrete improvements and considering that it only operates at the output side of the model, our next step is to explore similar smoothing ideas at the input side.