Encoder and Decoder, Not One Less for Pre-trained Language Model Sponsored NMT (2023)

Abstract: Well pre-trained contextualized representations from pre-trained language model (PLM) have been shown helpful for enhancing various natural language processing tasks, surely including neural machine translation (NMT). However, existing methods either consider encoder-only enhancement or rely on specific multilingual PLMs, which leads to a much larger model or give up potentially helpful knowledge from target PLMs. In this paper, we propose a new monolingual PLM-sponsored NMT model to let both encoder and decoder enjoy PLM enhancement to alleviate such obvious inconvenience. Especially, incorporating a newly proposed frequency-weighted embedding transformation algorithm, PLM embeddings can be effectively exploited in terms of the representations of the NMT decoder. We evaluate our model on IWSLT14 En-De, De-En, WMT14 En-De, and En-Fr tasks, and the results show that our proposed PLM enhancement gives significant improvement and even helps achieve new state-of-the-art.

## introduction pre-trained language models (plm) such as bert @xcite , roberta @xcite and xlnet @xcite have significantly improved a wide range of natural language processing (nlp) tasks. however, neural machine translation (nmt) tasks often require big models and large scale of training data, which leads to fine-tuned plms easily forgetting supposed-not-tobe-forgotten knowledge. as a result, nmt model cannot yield promising results only by fine-tuning plms according to extensive studies @xcite . to make effective use of plms in nmt, there has been a series of explorations. @xcite proposed ctnmt based on asymptotic distillation to keep reminding the nmt model of bert knowledge and dynamic switching gate to combine the encoded embedding from bert and the encoder of nmt model. @xcite proposed bert-fused model using bert as context-aware embedding for nmt, which exploits the representation from bert and feeds it into every layer of the encoder. ab-net @xcite incorporates adapter layers into each bert layer to conduct non-auto-regressive and auto-regressive decoding of sequence-to-sequence models for improvement. @xcite propose a framework to fuse bert into nmt model using a layer-wise coordination structure with a partitioned multi-task learning method. in this paper, we also focus on incorporating bert into nmt model. these works do achieve improvement on nmt tasks with plms more or less while leaving huge challenges behind. for example, @xcite ; @xcite simply put plms as a plugin for nmt model, which leads to oversized models. to avoid forgetting plms knowledge, @xcite completely abandon fine-tuning by simply freezing plm parameters. besides, @xcite ; @xcite show that using monolingual plms for decoder initialization may hurt nmt performance so that nmt decoder cannot conveniently benefit from plm enhancement. in existing nmt models using plms @xcite @xcite , nmt is enhanced by plms in the same way as other nlp tasks: plms offer representations first and then the model makes a prediction based on such representations. however, the behavior of nmt differs from other nlp tasks in a small but vital detail. most nlp tasks simply perform in a strict pipeline of first-encodingthen-decoding way, while nmt models cannot perform similarly as other tasks in which the nmt decoder has to encode predicted target words and encoder decoder plm (a) @xcite encoder decoder plm (b) @xcite encoder plm decoder (c) @xcite plm decoder coordinator encoder (d) @xcite plm coordinator decoder encoder plm (e) our plm-sponsored nmt model. figure 1 : structures of some existing autoregressive models and our model. all structures shown in figures are used for prediction. note that the model in figure 1(c ) is an autoregressive model @xcite . in light of such an observation on the difference between nmt and other nlp tasks, we propose a novel design to let both encoder and decoder of nmt model enjoy the empowerment of plm for the first time. for the decoder part, we reorganize the decoder by replacing all self-attention sub-layers with plms to encode predicted target sentences and moving all cross-attention sub-layers to a novel coordinator. our decoder consisting of self-attention sub-layers has the same structure as the encoder which can be replaced with plms completely without any structure change and focuses on encoding the representations of predicted words. the novel coordinator views the encoder and decoder as equal and combines representations from the encoder and decoder for the prediction of unknown words. in our model, the encoder and decoder can be replaced by plms completely and the coordinator is randomly initialized, the same as other downstream tasks in which randomly initialized parameters are light and the plms are heavy. our model design facilitates plms of source and target to focus on encoding source and target sentences respectively which are the original roles of plms. we evaluate our model on four nmt tasks. experimental results show that our model can achieve improvement of all tasks significantly using plms. our model achieves 37.5 bleu score on iwslt14 german-english task outperforming other works, 31.0 and 44.5 bleu score on wmt14 english-german and english-french tasks with fewer parameters compared to other works. we also give some ablations to evaluate the effectiveness of different methods in our models, and ablations show that plms indeed achieve the performance of the nmt model from both sides of the encoder and decoder. ## our model in this section, we introduce the architecture of our plm-sponsored nmt model and the training objective. ## ablation in this section, we perform ablation experiments over a number of facets of our model on iwslt14 de-en and wmt14 en-de tasks. to finish our ablation experiments with limited resources, the update steps for de-en and en-de are 100,000 and 400,000 respectively, and the coordinator for en-de has 6 layers only to speed up the training. ## related work various pre-trained language models @xcite @xcite @xcite have improved the performance of various nlp tasks, which makes designing and training plm a popular task. it is common to train a plm for nmt tasks. conneau and lample (2019) proposed two methods to learn cross-lingual language models to provide improvements on cross-lingual classification and nmt tasks. liu et al. (2020) proposed mbart which is a multilingual seq2seq transformer based denoising auto-encoder and can achieve improvement of performance on poor and medium resources nmt tasks. chen et al. (2021) proposed sixt model to directly translate unseen languages using twostage training schedules. xu et al. (2021) proposed bibert which is a bilingual pre-trained language model and can improve the performance on several different nmt tasks. weng et al. (2019) trained a bi-directional self-attention language model to generate representations and proposed two different mechanisms to fuse representations into the encoder and decoder respectively. some works also incorporate representations of plm into nmt model for improvements of performance. imamura and sumita (2019) proposed a two-stage optimization to fine-tune bert for nmt model. yang et al. (2020) proposed a framework of knowledge distillation and dynamic switch to incorporate bert into nmt model. zhu et al. (2020) proposed a bertfused model which uses bert as contextual embedding to extract representations for sentences and incorporate representations into each layer of the encoder and the decoder through attention mechanisms. weng et al. (2022) proposed a framework to deep fuse the representations into nmt by replacing the encoder with bert and using a layer-wise coordination structure. ab-net (guo et al., 2020) is based on a parallel sequence decoding algorithm to improve the performance of non-autoregressive and autoregressive nmt model. 6 conclusion to improve nmt by incorporating pre-trained contextualized representations from plm during training, we propose a new monolingual plmsponsored nmt model to enhance both encoder and decoder by replacing the randomly initialized encoder and decoder with bert. besides, we also propose a frequency-weighted embedding transformation algorithm to effectively exploit plm embeddings in terms of the representations of nmt decoder. all parameters including plms can be trained together with mlm on both encoder and decoder instead of training plm using nmt data first. we evaluate our model on four nmt tasks,and the experiment results show that our model with plms can improve the performance of nmt model. compared to other works, our model can outperform or achieve comparable results using smaller and monolingual bert with fewer parameters, which also shows the effectiveness of our model. we also evaluate our model with different model configurations and study the characteristic of our model. ## limitations although our model can achieve better results compared to other works, there are some limitations of our model: