Multilingual Translation via Grafting Pre-trained Language Models (2021)

Abstract: Can pre-trained BERT for one language and GPT for another be glued together to translate texts? Self-supervised training using only monolingual data has led to the success of pre-trained (masked) language models in many NLP tasks. However, directly connecting BERT as an encoder and GPT as a decoder can be challenging in machine translation, for GPT-like models lack a cross-attention component that is needed in seq2seq decoders. In this paper, we propose Graformer to graft separately pre-trained (masked) language models for machine translation. With monolingual data for pre-training and parallel data for grafting training, we maximally take advantage of the usage of both types of data. Experiments on 60 directions show that our method achieves average improvements of 5.8 BLEU in x2en and 2.9 BLEU in en2x directions comparing with the multilingual Transformer of the same size.

## introduction in recent years, pre-trained (masked) language models have achieved significant progress in all kinds of nlp tasks @xcite . among them, neural machine translation (nmt) is also explored by several attempts @xcite @xcite . the pre-training and fine-tuning style becomes an important alternative to take advantage of monolingual data @xcite @xcite . an intuitive question comes as: can we bridge bert-like pre-trained encoders and gpt-like decoders to form a high-quality translation model? since they only need monolingual data, we can reduce the reliance on the large parallel corpus. moreover, if the combination of models is universal, it can be applied to translation for multiple languages, as is shown in figure 1 . however, though many works successfully gain improvements by loading encoder/decoder parameters from bert-like pre-trained encoders @xcite , they do not achieve satisfactory results with loading decoder parameters from gpt-like pre-trained decoders @xcite . theoretically, the well-trained decoder model like gpt should bring better generation ability to the translation model. we suggest the outcome may be attributed to the architecture mismatch. pre-trained (masked) language models predict the current word solely based on the internal context while the translation decoder has to capture the source context. specifically, the decoder in nmt has a "cross-attention" sub-layer that plays a transduction role @xcite , while pretrained models have none, as is shown in figure 2 . this mismatch between the generation models and conditional generation models makes it a challenge for the usage of pre-trained models as translation decoders. therefore, some previous works manually insert cross-attention sub-layer or adapters @xcite @xcite . however, the extra implantation may influence the ability of the pre-trained model. other works try to avoid cross attention self attention feed-forward network feed-forward network self attention ? pre-trained (masked) language models translation decoder figure 2 : taking the popular architecture transformer @xcite as an example, the translation model has a "cross-attention" sub-layer, while pre-trained (masked) language models have none. in this paper, we mainly focus on exploring the best way to simultaneously take advantage of the pre-trained representation model and generation model (e.g., bert+gpt) without limiting their strengths. the primary target is to link the generation model to the source side and maintain the invariability of the architecture in the meantime. therefore, we propose graformer, with pretrained models grafted by a connective sub-module. the structure of the pre-trained parts remains unchanged, and we train the grafting part to learn to translate. for universality and generalization, we also extend the model to multilingual nmt, achieving mbert+mgpt. generally, the translation process can be divided into three parts: representation, transduction, and generation, respectively achieved by the encoder, cross-attention, and decoder. in multilingual nmt, the transduction can only be trained with multiple parallel data. but the rest two can be pre-trained with multiple monolingual data, which is tens or hundreds of the size of parallel one. to maximize the efficacy of each part, we firstly pre-train a multilingual bert and multilingual gpt. then they are grafted to implement translation. with the architecture consistency, we can reserve the language knowledge of the pre-trained models and obtain a strong translation model flexibly at the same time. experiments on 30 language directions show that our method improves the results of multilingual nmt by 2.9 and 5.8 bleu on average. it also achieves gains of 9.2 to 13.4 bleu scores on zeroshot translation settings. in addition, it verifies that such translation capability can be well transferred to other languages without fine-tuning on the target parallel corpus. ## related work this paper is related to a chain of studies of multilingual translation and pre-trained models. ## approach to maintain the original model structure of pretrained models, we propose graformer, as is in figure 3 . for the encoder side, we stack another k-layers encoder (@xmath0, in this paper) on pretrained mbert to help it adapt to the translation training. for the decoder side, we do similarly, except we append cross-attention layers to extract conditional context from the source. unlike previous works, we maintain the integrality of mbert and mgpt and do not change their architectures. finally, we employ a residual connection @xcite that we combine the hidden state outputted by mgpt and the grafting decoder. the summed context is then fed into the softmax layer. this integration is for utilizing the generation ability of the pre-trained decoder to help to generate a better language model. as mentioned in section 1, we try to take advantage of both multiple parallel data and multiple monolingual data so as to maximize the efficacy of representation, transduction, and generation, respectively. therefore, our training methods can be separated into two stages: 1) pre-train on the multiple monolingual data and obtain independent encoder (representation) and decoder (generation); 2) fine-tune on the multilingual parallel data to graft two models (transduction). 2020 ), we use masked language model (mlm) as the training goal with the masked probability of 15%. specifically, we adopt transformer @xcite encoder with n layers (@xmath1, in this paper). to make cross-lingual token representation more universal, we add no language token as previous works do. the training goal is as follows: ## experiments in this paper, we perform many-to-many style multilingual translation @xcite . the detailed illustrations of the datasets and implementation are as follows. ## conclusion in this paper, we propose graformer, grafting multilingual bert and multilingual gpt for multilingual neural machine translation. by pre-training the representation part (encoder) and generation part (decoder) of the model, we leverage the monolingual data to boost the translation task. and different from other previous fusing methods, we maintain the original architectures. with this approach, we can fully take advantage of the pre-trained models, including their well-trained capacity for representation and generation. experimental results show that our method can significantly improve the performance and outperform similar related works. a series of empirical analyses of perplexity, few-shot translation, and zero-shot translation also shows its universality.