Unifying Input and Output Smoothing in Neural Machine Translation (2020)

Abstract: Soft contextualized data augmentation is a recent method that replaces one-hot representation of words with soft posterior distributions of an external language model, smoothing the input of neural machine translation systems. Label smoothing is another effective method that penalizes over-confident model outputs by discounting some probability mass from the true target word, smoothing the output of neural machine translation systems. Having the benefit of updating all word vectors in each optimization step and better regularizing the models, the two smoothing methods are shown to bring significant improvements in translation performance. In this work, we study how to best combine the methods and stack the improvements. Specifically, we vary the prior distributions to smooth with, the hyperparameters that control the smoothing strength, and the token selection procedures. We conduct extensive experiments on small datasets, evaluate the recipes on larger datasets, and examine the implications when back-translation is further used. Our results confirm cumulative improvements when input and output smoothing are used in combination, giving up to +1.9 BLEU scores on standard machine translation tasks and reveal reasons why these smoothing methods should be preferred.

## introduction nowadays, neural network models are commonly used for the task of machine translation. the transformer @xcite architecture is the default choice for many competitive systems @xcite @xcite . in order to make use of large amount of available data in the target language, among others, back-translation is a frequently used method @xcite @xcite . recently, a method under the name "soft contextualized data augmentation" @xcite ) is introduced and focus on the input side of neural machine translation models. intuitively, the method smoothes both the source input and the target input to the model. while one-hot vectors are traditionally used to feed the word / token information to the network, this method instead uses an external language model (lm) trained on the parallel data without additional monolingual data, and achieve a "smoother" input where a "linear interpolation" operation is done on the word embedding matrices, instead of the usual "table lookup". conceptually, the "input smoothing" method above largely resembles the "label smoothing" method used in the output side. initially introduced in the field of image recognition @xcite , label smoothing is included in the original transformer setup @xcite by default. deducting a certain probability mass from the true target and redistributing uniformly across the vocabulary, label smoothing can also be thought of as penalizing over-confident system outputs. from here on, to highlight our motivation, we will use "input smoothing" and "output smoothing" to refer to "soft contextualized data augmentation" and "label smoothing" respectively. in this paper, our motivation is straight-forward, that we want to carefully look at both input smoothing and output smoothing in combination. specifically, below is a list of important research questions that we wish to answer: consequently, we try to organize the paper in a reader-friendly structure: ## related work in the recent work by @xcite , input smoothing is motivated and introduced. conceptually, the parallel data used to train neural machine translation models is only a sampled subset of the total "true" translation data. under the name of "soft contextualized data augmentation", the authors use an additional lm to give soft distributions across the vocabulary for randomly selected positions. compared to previous data augmentaion methods @xcite @xcite @xcite , the soft smoothing method achieves more significant improvements. the method is quickly adopted and motivates many interesting work, e.g. selecting words to "smooth" with dependency parsing @xcite , automatic repairing noisy synthetic parallel data @xcite , and distilling knowledge from bert @xcite via "text smoothing" @xcite . on the other hand, output smoothing enjoys a slightly longer history. in this era where neural network are extensively used for modeling, @xcite initially introduce the method of "label smoothing" and @xcite include it by default into the transformer setup. discounting a certain probability mass from to one-hot true target distribution, and redistributing uniformly across the vocabulary, label smoothing wishes to solve the problem of over-fitting and boost adaptability. relatedly, @xcite introduces a confidence penalty term, which is also mentioned in @xcite , to encourage more even model outputs. investigating the implications of label smoothing in terms of properties of learned features and in the case of knowledge distillation, @xcite further extends the knowledge of the smoothing method when used at the output side of the network. to build a competitive machine translation system, the large amount of target-side monolingual data can not be neglected. traditionally for statistical machine translation, back-translating target-side monolingual data already sees some use @xcite . for neural machine translation, @xcite formally introduces the method of back-translation, which marks an important improvement in the training pipelines. @xcite extends the method to the data scenario where huge amounts of monolingual data is available. while a separate target-to-source back-translation model is commonly used, how exactly the artificial source-side texts should be generated largely remains a yet-to-resolve research question. the sampling vs. search comparisons are already made in several works @xcite @xcite , while @xcite being a most recent work introducing uncertainty-based confidence estimation as an alternative. the term "smoothing" is also traditionally used in the context of count-based language modeling @xcite @xcite @xcite . in a broader sense, one-hot vectors commonly used at the input-side and output-side of neural networks serve as empirical counts to the model. in this interpretation, both input smoothing and output smoothing are similar in that they both discount certain probability masses from the what is observed in data, to those rare events (rare combinations of input and output words / tokens). in other words, the smoothing methods discussed in this paper should have an overall effect of combating over-fitting and boosting generalization. closely related, the standing investigations into knowledge distillation @xcite @xcite are also connected works and may be further referred to. ## methodology in this section, we define the translation models and training criterions using different combinations of smoothing at the input side and output side. the distinction among these models is necessarily made in order to draw a fair conclusion if the improvements from input and output smoothing can stack. consider parallel training data f j 1 = f 1 ...f j ...f j , @xmath0...e i ...e i , @xmath1, where f and e denote tokens in the source and target vocabularies f and e, j, j, i and i mark token positions in sentences. a typical neural model parametrized by θ can be written as p θ (e i |f j ## experiments for fast iterations of experiments, we first conduct experiments on a smaller dataset, namely iwslt2014 @xmath2. after obtaining meaningful and conclusive results, we repeat the core experiments and extend to two more small datasets, iwslt2014 @xmath3, plus a larger dataset, wmt2014 @xmath4. for the latter, we look at both the base and big model from the original transformer paper and evaluate on newstest2014, newstest2015 and newstest2016. we implement the various smoothing methods in the fairseq toolkit @xcite and conduct experiments with it. for the vocabulary, we use 10,000 and 32,000 merge operations with joint byte pair encoding from @xcite for iwslt and wmt respectively. for iwslt, we largely follow @xcite , lowercase all sentences, remove extremely long sentence pairs, subsample sentence pairs from training data as our development set, and concatenate all available development and test sets. for wmt, we adopt the preprocessing steps in @xcite . for all of our experiments, the three embedding matrices are tied, similar to that in @xcite . for more details about preprocessing, we refer the readers to the fairseq source, where the preprocessing scripts are available and easy to adapt. during training, we monitor the development set perplexity. after convergence, we further average the most recent checkpoints for a small performance boost. note that this checkpoint averaging step is done for all of the models mentioned in this paper, so it is fair to compare the results to draw conclusions about smoothing. during beam search, we consistently use a beam size of five for iwslt and four for wmt for fair comparisons to literature. finally, we report case-insensitive bleu scores on iwslt and case-sensitive bleu scores on wmt @xcite on the test sets. ## analyses so far, it is clear that significant empirical improvements can be obtained with both input smoothing and output smoothing. a natural question is then, why? our interpretation of the improvements lies in how frequent the word vectors are updated during the training process. in @xcite , the authors stress that word vector parameters make up the lion's share in long short-term memory @xcite neural network models, where the percentage even go up to 90% for a model with a large vocabulary. in our case, the percentage is lower for transformer models because byte pair encoding reduces vocabulary size drastically, but the word vector parameters still make up a significant portion of total parameter count. for example, for transformer base with three-way shared vocabulary and embeddings on en-de, the percentage is still 27.5%. when no smoothing is applied, each word vector in the input embedding matrices gets updated precisely the empirical count of the word times for each training epoch, because the one-hot vectors result in simple table lookups. with input smoothing, if the helper model's posterior probability distribution covers the whole vocabulary, this number increases to the empirical count of all tokens in the training corpus. at the output side, the non-target tokens also show up in the denominator in the softmax calculation, complicating things a bit. one can nonetheless calculate the partial derivatives of the local losses without and with output smoothing, l i,no smooth and l i,smooth , on a certain non-target token e ι , e.g. with zero-gram lm output smoothing shown in equation 7 . here, h denotes the context vector for position i and z denotes the denominator which is a sum over all logits. it is clear that for non-target output word embeddings, the magnitude of the gradient increases with an increasing m. in other words, output smoothing tends to update the non-target words more drastically compared to when no smoothing is applied. in summary, both input embeddings and output embeddings, including those embeddings of infrequent tokens, are more strongly updated with the smoothing methods, explaining why these smoothing methods work well in practice. finally, we plot the negative log-probabilities with respect to the frequency of words. in figure 4 , for frequent tokens as well as the infrequent tokens, our model is able to perform better than the transformer baseline. notice that the negative log-probabilities become more noisy as the tokens appear less often in the data, which is expected. ## conclusion in this paper, we combine input smoothing and output smoothing for neural machine translation. we confirm strong cumulative improvements of the two methods, giving up to +1.9 bleu scores in our experiments. contrary to prior work, we find that simply smoothing everywhere with position-independant language models such as a zerogram langauge model already gives significant improvements, setting us free from pre-training complex external helper models. we show that the improvements also sustain in case of back-translation. finally, we explain that the key to the improvements lies in the increased update frequency of word vectors. since the benefits almost come for free, we encourage more usage as well as more theoretical justifications of such smoothing methods, e.g. making symmetric smoothing work.