Token Drop mechanism for Neural Machine Translation (2020)

Abstract: Neural machine translation with millions of parameters is vulnerable to unfamiliar inputs. We propose Token Drop to improve generalization and avoid overfitting for the NMT model. Similar to word dropout, whereas we replace dropped token with a special token instead of setting zero to words. We further introduce two self-supervised objectives: Replaced Token Detection and Dropped Token Prediction. Our method aims to force model generating target translation with less information, in this way the model can learn textual representation better. Experiments on Chinese-English and English-Romanian benchmark demonstrate the effectiveness of our approach and our model achieves significant improvements over a strong Transformer baseline.

## introduction neural machine translation (nmt) achieved enormous success in advancing the quality of translation @xcite @xcite . in spite of the impressive performance, nmt models are still vulnerable to perturbations in the input sentences @xcite , i.e. a tiny perturbation will affect hidden representation and lead to low quality of translation @xcite . moreover, nmt commonly consists of millions of parameters, which making it prone to overfitting especially in low resource scene. a natural way to improve generalization is synthesizing natural noise @xcite or adopting arbitrary noise @xcite . another way is exploring regularization techniques to avoid overfitting @xcite , making model robust to unseen or unfamiliar inputs. however, as discrete data, the text is hard to retain the semantic information after corruption. in this paper, we propose token drop to prevent overfitting and improve generalization. different from standard dropout @xcite that drops neurons in network randomly, we drop tokens of the input sentences. in order to retain semantic information, we replace tokens with a special @xmath0. this allows model learn hidden representation from rest token's context, and predict target translation condition on latent variable. on the one hand, our method allows model meeting exponentially different sentences can be explained as data augmentation; on the other hand, our method corrupts input sentences with natural noise can be seen as regularization term for nmt. we investigate two self-supervised objectives: replaced token detection and dropped token prediction. considering our token drop method regularize parameters by weakening model inputs, making nmt suitable for applying self-supervised objective. during training: (1) use a discriminator to detect whether input tokens are dropped or not; (2) leverage hidden state to predict original tokens of dropped tokens inspired by cloze task @xcite . both of them guide model to generate semantically similar representation, leading to a better generalization capacity. ## token drop training standard dropout prevents overfitting by setting input neurons or hidden neurons to zero with a certain probability p @xcite . whereas we consider the input sequences of machine translation models instead of the network's neurons, which named token drop. given a input tokens sequence of sentence @xmath1, x 2 , ..., x n } and posit a |x| independent drop rate p. the token x i in x will be dropped if m i is 1 . this process as equation 1: the token drop can be interpreted as data augmentation and regularization technique for nmt. seeing that nmt model commonly adopts encoder and decoder architecture, therefore our method drops tokens for both source and target inputs. for the source side, model encoder learns intermediate representation by exponentially different incomplete sentences. for the target side, model decoder generates target translation condition on latent variable, weakening the constraint caused by teacher forcing. both of them receives incomplete information from inputs, simulating the real situation (e.g unknown or unfamiliar data) at test time. ## conclusion in this paper, we have proposed token drop mechanism for neural machine translation task. inspired by self-supervised learning, we introduced replaced token detection and dropped token prediction training objective. we found that nmt model trained with token drop gains larger generalization capacity and reduction in overfitting. even without prior knowledge and additional parameters, our proposed approach reports convincing results on neural machine translation. in future work, we plan to investigate impact of dropping on different words, e.g. word importance and word type.