Self-supervised and Supervised Joint Training for Resource-rich Machine Translation
Authors: Yong Cheng, Wei Wang., Lu Jiang, Wolfgang Macherey
Keywords: resource-rich machine translation, neural machine translation, pre-training, self-supervised learning, joint training
Abstract: Self-supervised pre-training of text representations has been successfully applied to low-resource Neural Machine Translation (NMT). However, it usually fails to achieve notable gains on resource-rich NMT. In this paper, we propose a joint training approach, $F_2$-XEnDec, to combine self-supervised and supervised learning to optimize NMT models. To exploit complementary self-supervised signals for supervised learning,  NMT models are trained on examples that are interbred from monolingual and parallel  sentences through a new process called crossover encoder-decoder. Experiments on two resource-rich translation bench-marks, WMT’14 English-German and WMT’14 English-French, demonstrate that our approach achieves substantial improvements over a vanilla Transformer and obtains a new state of the art of 46 BLEU on English-French. Results also show that our approach is capable of improving model robustness against input perturbations which is known as a key weakness in contemporary NMT systems.