Prediction Difference Regularization against Perturbation for Neural Machine Translation (2022)

Abstract: Regularization methods applying input perturbation have drawn considerable attention and have been frequently explored for NMT tasks in recent years. Despite their simplicity and effectiveness, we argue that these methods are limited by the under-fitting of training data. In this paper, we utilize prediction difference for ground-truth tokens to analyze the fitting of token-level samples and find that under-fitting is almost as common as over-fitting. We introduce prediction difference regularization (PD-R), a simple and effective method that can reduce over-fitting and under-fitting at the same time. For all token-level samples, PD-R minimizes the prediction difference between the original pass and the input-perturbed pass, making the model less sensitive to small input changes, thus more robust to both perturbations and under-fitted training data. Experiments on three widely used WMT translation tasks show that our approach can significantly improve over existing perturbation regularization methods. On WMT16 En-De task, our model achieves 1.80 SacreBLEU improvement over vanilla transformer.

## introduction neural machine translation models have achieved great success in recent years @xcite @xcite . despite their efficiency and superb performance, nmt models are prone to over-fitting that universal regularization techniques such as dropout @xcite and label smoothing @xcite have been indispensable. however, over-fitting is still a significant problem for nmt, especially for small and medium tasks, which motivates researchers to constantly explore more specialized and sophisticated regularization techniques. particularly, regularization methods applying input perturbation have been frequently explored for nmt models in recent years @xcite @xcite . in these methods, neural models are trained to maximize the likelihood of perturbed samples that perturbed by a certain type of perturbations, with a primary intention to enhance model's robustness to perturbations, since neural models have been discovered fragile to small input noises @xcite @xcite . in the past few years, many types of perturbations have been proposed to machine translation and been shown effective, including word-dropout @xcite , word-replacement @xcite and adversarial perturbation @xcite , etc. in this paper, unlike previous works which are devoted to finding stronger perturbations and more appropriate perturbation schedules, we rethink the existing perturb-and-fit mechanism and prove that indiscriminate fitting of perturbed samples ignores and aggravates under-fitting, which dramatically limits the effectiveness of perturbation regularization. we further propose prediction difference regularization (pd-r), a simple and effective method that can alleviate over-fitting and under-fitting at the same time and significantly enhance the effectiveness of perturbation regularization. specifically, we use the prediction difference for ground-truth labels before and after input perturbation as an indicator of over-fitting and underfitting for token-level samples. quantitative analysis shows that a considerable part of token-level predictions get improved after input perturbation, indicating that the model is less fitted to those original samples compared to the perturbed samples, which has been ignored by previous works. we then divide labels in a batch into relatively underfitted and over-fitted subsets according to real-time prediction difference and train only one subset to fit the perturbed inputs and the other subset to fit the original inputs. experiments show that training only the relatively under-fitted subset to further fit the perturbed inputs dramatically degrade the model performance, while the opposite gets better results than the existing indiscriminate way. this indicates that existing methods are hindered by the excessive fitting of perturbed data. we further propose to use prediction difference as a regularization term, where the prediction difference is the divergence of prediction distribution caused by input perturbation. since the value of prediction difference reflects the severity of overfitting or under-fitting, both of which are cases we want to avoid for training models, regularizing prediction difference has been a natural solution to avoid above fitting problems. by combining crossentropy loss and the prediction difference term, a model can be trained to fit training data with control of over-fitting and under-fitting. we apply pd-r on simplest word dropout regularization and conduct experiments on three widely used wmt translation tasks covering small-scale, medium-scale, and large-scale data sets. our method significantly improves over existing perturbation regularization methods. on wmt16 en-de translation task, our method achieves 1.80 sacrebleu improvement over vanilla transformer model and 1.12 sacrebleu over traditional word dropout regularization. ## background in this section, we introduce basic principles of neural machine translation, representative types of perturbations as well as their training objectives. ## prediction difference fitting analysis it is usually believed that a model's prediction for ground-truth target tokens will be hindered when the input is perturbed, which has been the initial motivation for perturbation regularization. however, this conclusion is not necessarily true in experiment for many reasons: firstly, neural networks are complex and may not behave in an ideally logical way. secondly, perturbations are randomly produced and may have complex properties. for example, word-replacement may induce synonyms or heteronyms, and high-dimension embedding perturbation is hard to interpret. thirdly, the model is uncertain during training due to parameter dropout, which further brings uncertainty to the model's reaction to perturbations. in this section, we analyze the influence of perturbations leveraging token-level prediction difference. here the prediction difference is defined as the change of model's prediction probabilities for ground-truth target tokens @xcite : we apply random perturbations to samples in the test set and divide all target labels into two subsets according to their prediction change: positively influenced subset s p containing labels whose prediction probabilities get bigger after input perturbation and negatively influenced subset s n containing labels whose prediction probabilities get smaller. we compute the quantitative proportion and the average value of ∆p for these two subsets to evaluate the influence of different perturbations. since the prediction difference is also under the influence of parameter dropout during training, we also conduct experiments both with and without parameter dropout difference. the original pass and the perturbed pass are carried out by two different sub-models if their parameter dropout mask is different. in experiment, we use a transformer base model trained on wmt16 en-de data set and conduct our experiments on a test set which is a combination of 5 test sets from wmt16 to wmt20. our analysis covers different kinds of perturbations, including word-dropout, word-replacement, and adversarial perturbation. the word-dropout and word-replacement probabilities are set as 0.05, and all perturbations are applied on both sides of the model. as illustrated in figure 1 , for any certain type of perturbation, the negative impact is principal, especially for adversarial perturbation. however, the positive influence is non-negligible since the proportion of positively influenced tokens could reach 30%-40% for word-dropout and word-replacement. with parameter dropout difference, the positive influence could get further bigger and become more crucial. we attribute prediction difference to relatively over-fitting and under-fitting of token-level samples. since perturbations are very small, perturbed samples can be approximately viewed as good samples. for one target label, if its prediction probability gets smaller after a small input perturbation, it indicates the model is relatively over-fitted to the original sample, while the contrary case is the reflection of relatively under-fitting. with parameter dropout, predictions are carried out by sub-models, and these fitting problems also reflect the relative fitting bias of sub-models, which is also what we want to avoid. we further carry out selective training for worddropout regularization, where one subset is trained to fit the perturbed inputs and the other subset is trained to fit the original inputs. as presented in ## prediction difference regularization (pd-r) since both positive and negative prediction difference is a sign of improper fitting of samples, we therefore propose prediction difference regularization (pd-r), to regularize the model directly with the prediction difference: ## experiments we evaluate pd-r on three public wmt machine translation tasks and compare it with representative related works. ## analysis in this section, we analyze the robustness of our methods and distinguish the contribution of different components via ablation study. ## related work works involving input perturbation apart from the works mentioned above, some works introduce subword uncertainty at the subword segmentation stage, including sampling multiple subword candidates @xcite , applying subword dropout @xcite or producing adversarial subword segmentation @xcite . for character-level tasks, there are also works using character-level perturbation including characterlevel random deletion, insertion, substitution and swap @xcite and adversarial substitution @xcite . the mixup technique for nlp tasks can also be seen as a form of perturbation where samples are perturbed (mixed) with other samples for data augmentation or generation diversity @xcite @xcite . our work can be regarded as one example of perturbation regularization. however, unlike previous perturbation regularization works which are focused on finding better perturbation, our work improves the training mechanism and can be applied to any type of perturbations. influence of perturbation perturbation is commonly considered as a negative factor for neural models by previous works @xcite @xcite , which is generally correct with the fact that perturbation does degrade the training and inference accuracy of a model. @xcite demonstrates that the performance of nmt systems degrades monotonously as input modification increases, which is consistent with our observations. based on the above facts, perturbation regularization is frequently studied to enhance models' robustness to unexpected inputs at the inference stage. from a data selection perspective, khayrallah and @xcite and @xcite demonstrate that noisy or semantically divergent data is harmful to the training of nmt models. in this paper, we find that the interaction between perturbation and model is complicated and positive influence of perturbation is very common, which is further regarded by us as a sign of relatively under-fitting and a variable that needs to be restricted. prediction difference prediction difference is usually considered as a reflection of the relationship between input and output and is often used to analyze model behavior. @xcite utilizes prediction difference to visualize the importance of a specific input image area to model decision. @xcite uses the prediction difference of a target word when a source word is removed to induce word alignment and find it more accurate than attention weights. @xcite finds that adversarial examples can be accurately and efficiently detected via prediction difference. @xcite proposes r-drop and take prediction difference as a regularization term to regularize sub-model divergence. in this work, prediction difference is used as an analytical tool to detect improper fitting problems and also a regularization term to regularize the model's fitting bias to in this paper, we propose to use probability difference for ground-truth tokens before and after input perturbation as an indicator to analyze the influence of different types of perturbations and attribute probability difference to improper fitting of token-level samples. we find that under-fitting is almost as common as over-fitting, which is totally ignored and further aggravated by existing perturbation regularization methods. to regularize both under-fitting and over-fitting, we use prediction difference as a regularization term (pd-r) and apply it on word-dropout regularization. our method achieves significant improvement over existing methods on three wmt translation tasks and is proved more robust to input perturbation.