Pretrained Bidirectional Distillation for Machine Translation (2023)

Abstract: Knowledge transfer can boost neural machine translation (NMT), for example, by finetuning a pretrained masked language model (LM). However, it may suffer from the forgetting problem and the structural inconsistency between pretrained LMs and NMT models. Knowledge distillation (KD) may be a potential solution to alleviate these issues, but few studies have investigated language knowledge transfer from pretrained language models to NMT models through KD. In this paper, we propose Pretrained Bidirectional Distillation (PBD) for NMT, which aims to efficiently transfer bidirectional language knowledge from masked language pretraining to NMT models. Its advantages are reflected in efficiency and effectiveness through a globally defined and bidirectional context-aware distillation objective. Bidirectional language knowledge of the entire sequence is transferred to an NMT model concurrently during translation training. Specifically, we propose self-distilled masked language pretraining to obtain the PBD objective. We also design PBD losses to efficiently distill the language knowledge, in the form of token probabilities, to the encoder and decoder of an NMT model using the PBD objective. Extensive experiments reveal that pretrained bidirectional distillation can significantly improve machine translation performance and achieve competitive or even better results than previous pretrain-finetune or unified multilingual translation methods in supervised, unsupervised, and zero-shot scenarios. Empirically, it is concluded that pretrained bidirectional distillation is an effective and efficient method for transferring language knowledge from pretrained language models to NMT models.

## introduction initializing parameters by a pretrained masked language model (lm) @xcite is a knowledge transfer method widely applied to natural language processing tasks. following its success, pretrained neural machine translation (nmt) models have attracted more and more research interest @xcite @xcite . however, the pretrain-finetune paradigm may suffer from potential issues. as is pointed out in @xcite , the finetuned model may forget some critical language generation skills learned from the pretraining phase. the catastrophic forgetting problem @xcite ) commonly exists in transfer learning, leading to overfitting to target domains. @xcite ; @xcite also observe similar forgetting problems in pretrained nmt tasks. besides, in the pretrain-finetune paradigm, model parameters are initialized by a pretrained model; this requires structure consistency (e.g., exact dimensions, layers, attention heads, etc.) between the pretrained lm and the nmt models to some extent. however, a powerful but structurally inconsistent pretrained lm may incorporate more language knowledge. knowledge distillation (kd) @xcite may be a potential solution to alleviate these issues, but few studies investigate language knowledge transfer from pretrained language models to nmt models by kd. previous works use kd for model compression @xcite , or data complexity reduction @xcite , or multilingual translation @xcite . @xcite utilizes confidence-based knowledge distillation to incorporate bidirectional global context into nmt models. in this paper, we propose pretrained bidirectional distillation (pbd) for nmt, which can alleviate the difference caused by pretraining (mask language modeling, perturbed sentences) and mt fine-tuning (full sentences) in the pretrain-finetune paradigm and boost large-scale translation training. in pretrained bidirectional distillation, language knowledge acquired from pretraining is continuously transferred to the nmt model. knowledge transfer runs through the training process to address the forgetting problem. we deal with the pretrained language knowledge by pretrained bidirectional distillation objectives, which are the token probabilities generated by the pretrained lm about potential tokens matching a global context. the pretrained bidirectional distillation objectives are distilled to the encoder and decoder of an nmt model. therefore, there is no need to require structure consistency between pretrained lms and nmt models, and bidirectional distillation enriches the nmt decoder with bidirectional semantic information. to guarantee the effectiveness and efficiency of pretrained bidirectional distillation, we propose self-distilled masked language pretraining, which can generate globally defined and bidirectional context aware token probabilities and use them as the pretrained bidirectional distillation objectives. "globally defined" lets us obtain the full probabilities of each token in a single forward pass, guaranteeing distillation effect and execution efficiency. "bidirectional context aware" distillation objectives incorporate bidirectional language knowledge of the whole sequence, guaranteeing effectiveness. extensive experiments are conducted on widely used benchmark datasets. in a supervised scenario, the proposed method achieves +2.7 and +8.5 absolute average bleu improvement using the unified multilingual translation model and pretrainfinetune paradigm, respectively. and our model obtains 19.28 and 16.55 average bleu in unsupervised and zero-shot scenarios, respectively, outper-algorithm 1 pretrained bidirectional distillation for nmt require: language model lm , nmt model t m , unlabeled lm data d lm , parallel data d t m 1: initialize lm by random 2: for each @xmath0 update lm ← backprop(l, lm ) 5: end for 6: initialize t m by random or pretraining 7: for each (x, y ) ∈ d t m do 8: update t m ← backprop(l, t m ) 12: end for 13: return t m forming previous models. to summarize, our contributions are as follows: ## experiments we primarily study the proposed pretrained bidirectional distillation by conducting experiments on supervised, unsupervised, and zero-shot multilingual machine translation scenarios. ## conclusion in this paper, we proposed the pretrained bidirectional distillation to investigate language knowledge transfer from pretrained language models to nmt models by knowledge distillation. the proposed approach has the advantages of distillation effectiveness and efficiency, and achieves new stateof-the-art performance in supervised, unsupervised, and zero-shot multilingual translation experiments. the model analysis also shows that the proposed self-distilled language model is critical to generating globally defined distillation objectives. in the future, we will do more research on optimizing the self-distilled language model and pretrained bidirectional distillation losses.