Investigating Softmax Tempering for Training Neural Machine Translation Models (2021)

Abstract: Neural machine translation (NMT) models are typically trained using a softmax cross-entropy loss where the softmax distribution is compared against the gold labels. In low-resource scenarios and NMT models tend to perform poorly because the model training quickly converges to a point where the softmax distribution computed using logits approaches the gold label distribution. Although label smoothing is a well-known solution to address this issue and we further propose to divide the logits by a temperature coefficient greater than one and forcing the softmax distribution to be smoother during training. This makes it harder for the model to quickly over-fit. In our experiments on 11 language pairs in the low-resource Asian Language Treebank dataset and we observed significant improvements in translation quality. Our analysis focuses on finding the right balance of label smoothing and softmax tempering which indicates that they are orthogonal methods. Finally and a study of softmax entropies and gradients reveal the impact of our method on the internal behavior of our NMT models.

## introduction neural machine translation (nmt) @xcite enables endto-end training of translation models and is known to give state-of-the-art results for a large variety of language pairs. nmt for high-resource language pairs is straightforward: choose an nmt architecture and implementation, and train a model on all existing data by minimizing the softmax cross-entropy loss, i.e., cross-entropy between the softmax distribution and the label distribution typically represented with a one-hot vector. in contrast, for low-resource language pairs, this does not work well due to the inability of neural networks to generalize from small amounts of data. one reason for this is over-fitting @xcite , where the softmax distribution (sparse vector) ends up resembling the label distribution (one-hot vector). there are several solutions that address this issue, of which the two most effective ones are transfer learning and model regularization. transfer learning can sometimes be considered as data regularization and comes in the form of monolingual or cross-lingual (multilingual) fashion @xcite , pseudo-parallel data generation (back-translation) @xcite , or multi-task learning @xcite . on the other hand, model regularization techniques place constraints on the learning of model parameters in order to aid the model to learn robust representations that positively impact model performance. among existing model regularization methods, dropout @xcite is most commonly used and is known to be effective regardless of the size of data. label smoothing @xcite is another effective approach that uses smoothed label vectors as opposed to onehot label vectors. previous work on nmt has shown that label smoothing is very effective in low-resource settings @xcite and we believe that this deserves further study. we thus focus on a technique that does not need additional data and can complement dropout and label smoothing in an extremely low-resource situation. in this paper, we propose to apply softmax tempering @xcite to the training of nmt models. softmax tempering is realized by dividing the pre-softmax logits with a positive real number greater than 1.0. this leads to a smoother softmax probability distribution, which is then used to compute the cross-entropy loss. softmax tempering has been devised and used regularly in knowledge distillation @xcite and model calibration @xcite albeit for different purposes. we regard softmax tempering as a means of deliberately making the softmax distribution noisy during training with the expectation that this will have a positive impact on the final translation quality. it is especially important to note that calibration involves tempering after a model has been trained whereas we perform tempering during training. we primarily evaluate the utility of softmax tempering on extremely low-resource settings involving english and 11 languages in the asian languages treebank (alt) @xcite . our experiments reveal that softmax tempering with a reasonably high temperature improves the translation quality. furthermore, greedy-search performance of models trained with softmax tempering becomes comparable to or better than the beam-search performance of models that are trained without softmax tempering. our analysis focuses on the orthogonality of softmax tempering and label smoothing. we additionally compare these methods with the related softmax entropy maximization method @xcite . finally, we analyze the impact of softmax tempering on the softmax distributions and on the gradient flows during training. ## related work the method presented in this paper is a training technique aimed to improve the quality of nmt models in low-resource scenarios. work on knowledge distillation @xcite for training compact models is highly related to our application of softmax tempering. however, the purpose of softmax tempering for knowledge distillation is to smooth the student and teacher distributions which is known to have a positive impact on the quality of student models. in our case, we use softmax tempering to make softmax distributions noisy during training a model from scratch to avoid over-fitting. in the context of nmt, @xcite conducted experiments with softmax tempering. however, their focus was on model compression and they did not experiment with low-resource settings. softmax tempering is also used in model calibration @xcite , where the temperature coefficient is optimized on the development set in order to penalize overconfident predictions, which is a common practice in low-resource settings. while model calibration is performed after a model is trained, we use softmax tempering during training. we regard softmax tempering as a regularization technique, since it adds noise to nmt model training. thus, it is related to techniques, such as l n regularization @xcite , dropout @xcite , and tuneout @xcite . the most important aspect of our method is that it is only applied at the softmax layer whereas other regularization techniques add noise to several parts of the entire model. label smoothing @xcite , which is known to help low-resource nmt @xcite , is highly related to our idea, where the key difference is that label smoothing affects the label distributions whereas softmax tempering affects the softmax distributions. on a related note, softmax entropy maximization @xcite seeks to mitigate overconfident predictions but is not known to work well for nmt. our method is intended to complement these techniques, i.e., label smoothing and softmax entropy maximization, and not necessarily replace them. existing methods effective for low-resource language pairs include data augmentation via back-translating additional monolingual data @xcite , exploitation of multilingualism @xcite @xcite , and pre-training on monolingual data @xcite @xcite . these require more training time and resources, while ours does not. ## softmax tempering softmax tempering @xcite consists of two tiny changes in the implementation of the training phase of any neural model used for classification. assume that d @xmath0, y i , where v stands for the target vocabulary size, and that let @xmath1. then, the prediction with softmax tempering (p temp i ) and the corresponding cross-entropy loss (l temp i ) are formalized as follows. by referring to equation (1), when t is greater than 1.0, the logits, d i , are down-scaled which leads to a smoother probability distribution before loss is computed. the smoother the distribution becomes, the higher its entropy is and hence the more uncertain the prediction is. because loss is to be minimized, back-propagation will force the model to generate logits to counter the smoothing effect of temperature. during decoding with a model trained in this way, the temperature coefficient is also used which mitigates overconfident predictions foot_0 stemming from tempering during training. the gradients are altered by tempering, and we thus re-scale the loss by the temperature as shown in equation (2). this is inspired by the loss scaling method used in knowledge distillation @xcite , where both the student and teacher's softmax distributions are tempered and the loss is multiplied by the square of the temperature. ## experiments to evaluate the effectiveness of softmax tempering, we conducted experiments on both lowresource and high-resource settings. ## analysis and further exploration softmax tempering directly manipulates the softmax distribution making it noisy (smoother). in this section, we explore the relationship between softmax tempering and its closest related methods that directly affect the softmax distribution, i.e., label smoothing and entropy maximization. we also study the internal working of the softmax-tempered models during training. for these analyses, we focus on extremely low-resource settings, since our results in section 4.5 demonstrate that softmax tempering is less impactful in high-resource settings. we especially take @xmath2, @xmath3, @xmath4, and @xmath5. we focus on the models with optimal hyper-parameters determined via a grid search on greedy-search bleu scores on the development set, and report on greedy-search bleu scores on the test set. 7 for alt tasks, decoding times are very similar when translating into english due to it being a multi-parallel corpus. config. @xmath6, s) bleu (t, s) bleu (t, s) bleu (t, s) default 7.1 (1.0, 0.1) 8.7 (1.0, 0.1) 27.4 (1.0, 0.1) 19.4 (1.0, 0.1) ls 8.4 (1.0, 0.5) 10.6 (1.0, 0.5) 29.1 (1.0, 0.5) 20.7 (1.0, 0.5) temp. 9.1 (5.0, 0.1) 11.0 (5.0, 0.1) 29.7 (4.0, 0.1) 21.3 (4.0, 0.1) temp. & ls 9.4 (5.0, 0.5) 11.8 (5.0, 0.5) 30.1 (5.0, 0.45) 22.1 (4.0, 0.45) ## conclusion in this paper, we explored the utility of softmax tempering for training nmt models. our experiments in low-resource and high-resource settings revealed that softmax tempering leads to an improvement in the greedy-and beam-search decoding quality. as an indirect consequence, in latency sensitive scenarios, we can use greedy search while achieving better translation quality than non-tempered models leading to 1.5 to 3.5 times faster decoding. we also explored the compatibility of softmax tempering with label smoothing and softmax entropy maximization where we showed that the combination of tempering and label smoothing is very important. we also identified settings where each method works best. furthermore, our analysis of the softmax entropies and gradients during training confirms that tempering gives precise softmaxes while enabling the model to learn with strong gradient signals even during late training stages. in the future, we will explore the effectiveness of softmax tempering in other natural language processing tasks.