S em F ace: Pre-training Encoder and Decoder with a Semantic Interface for Neural Machine Translation (2021)

Abstract: While pre-training techniques are working very well in natural language processing, how to pre-train a decoder and effectively use it for neural machine translation (NMT) still remains a tricky issue. The main reason is that the cross-attention module between the encoder and decoder cannot be pre-trained, and the combined encoder-decoder model cannot work well in the fine-tuning stage because the inputs of the decoder cross-attention come from unknown encoder outputs. In this paper, we propose a better pre-training method for NMT by defining a semantic interface (SemFace) between the pre-trained encoder and the pre-trained decoder. Specifically, we propose two types of semantic interfaces, including CL-SemFace which regards cross-lingual embeddings as an interface, and VQ-SemFace which employs vector quantized embeddings to constrain the encoder outputs and decoder inputs in the same language-independent space. We conduct massive experiments on six supervised translation pairs and three unsupervised pairs. Experimental results demonstrate that our proposed SemFace can effectively connect the pre-trained encoder and decoder, and achieves significant improvement by 3.7 and 1.5 BLEU points on the two tasks respectively compared with previous pre-training-based NMT models.

## introduction in recent years, pre-trained language models @xcite @xcite @xcite significantly boost the performances of various natural language processing (nlp) tasks, receiving extensive attention in nlp communities. following the idea of unsupervised pre-training methods in the nlp area, several approaches @xcite @xcite have been proposed to improve neural machine translation (nmt) models with pretraining by leveraging the large-scale monolingual corpora. the typical training process usually consists of two stages: pre-training an encoder and a decoder separately with a large monolingual corpus in a self-supervised manner, and then fine-tuning on specific nmt tasks @xcite . the above method essentially pre-trains a bertlike @xcite transformer encoder, and uses it to initialize both the encoder and decoder. although it shows promising results, pre-training decoder benefits little in their results. the potential reason is that the cross-attention between the encoder and decoder is not pre-trained, which is randomly initialized when they are connected for fine-tuning, resulting in a lack of semantic interfaces between the pre-trained encoder and decoder. another line of work attempts to pre-train a sequence-to-sequence model directly, e.g., mass @xcite and bart @xcite . but these methods usually use monolingual denoising auto-encoder as the main training objective, and cannot learn the corss-lingual mapping between source and target languages explicitly. in parallel to the idea of dall•e foot_0 which defines the cross-modality interface of image and text, we propose to pre-train the encoder and decoder with a language-independent semantic interface (semface) for neural machine translation. with the semantic interface, the encoder is pre-trained to extract features to this space, and the decoder is pretrained to generate contents with features provided by it. by defining this interface, we can decouple the encoder-decoder network and pre-train them separately. during the decoder pre-training, the cross-attention module is also pre-trained, thus the pre-trained encoder and decoder can be naturally our contributions are listed as follows: ## related work pre-training has been widely used in nlp tasks to learn better language representations @xcite @xcite @xcite @xcite . typically, these methods first pre-train neural networks on largescale unlabeled corpora, and then fine-tune the models on downstream tasks @xcite . the early pre-training techniques mainly focused on the natural language understanding tasks such as the glue benchmark @xcite , and later it was gradually extended to the natural language generation tasks, e.g., nmt. recently, a prominent line of work has been proposed to improve nmt with pre-training. these techniques can be broadly classified into two categories. the first category usually uses pre-trained models as feature extractors of a source language, or initializes the encoder and decoder with pretrained models separately @xcite @xcite . for example, @xcite proposed a cross-lingual language model with a supervised translation language modeling objective, and used mlm or clm to pre-train the encoder and decoder of nmt. however, the combined encoder-decoder model, where the crossattention is randomly initialized, often does not work well because of the lack of semantic interfaces between the pre-trained encoder and decoder. there is also some work trying to leverage bertlike pre-trained models for mt with an adapter @xcite or an apt framework @xcite . the former defines additional layers in the pre-trained encoder and decoder during finetuning, while the last adopts a fusion mechanism or knowledge distillation to leverage knowledge in bert for mt. different from them, we enable the encoder and decoder to interact with a semantic interface during pre-training, and they can be connected directly for the mt fine-tuning without any other additional layers or training loss. the second category methods pre-train a whole sequence-to-sequence model for nmt. mass @xcite employed the encoder-decoder framework to reconstruct a sentence fragment given the remaining part of the sentence. bart @xcite adopted a similar framework and trained the model as a denoising auto-encoder. mbart @xcite trained bart model on large-scale monolingual corpora in many languages. although the above work can pre-train the cross-attention of decoder, they are learned on monolingual denoising auto-encoding and cannot learn the corss-lingual transformation between source and target languages. there is also some work trying to explicitly introduce cross-lingual information in a code-switch way during the sequence-to-sequence pre-training, such as csp @xcite and mrasp @xcite . however, their methods need a lexicon or phrase translation table, which is inferred from unsupervised cross-lingual embeddings. therefore, they depend on the quality of the dictionary. the most similar work to ours is probably the one of dall•e and clip @xcite . dall•e is a transformer language model that receives both the text and the image as a single stream of data. the core idea is to define the cross-modality interface of image and text, which can generate images from text descriptions. in this paper, to address the above limitations of pretraining methods for nmt, we attempt to define a cross-lingual semantic interface to connect the pre-trained encoder and decoder. ## conclusion we propose semface, a better pre-training method for neural machine translation. the key point is to use a semantic interface to connect the pre-trained encoder and decoder. by defining this interface, we can pre-train the encoder and decoder separately with the same intermediate language-independent space. the cross-attention can also be pre-trained with our method so that we can naturally combine the pre-trained encoder and decoder for fine-tuning. we introduce and compare two semantic interfaces, e.g., cl-semface and vq-semface, which leverage unsupervised cross-lingual embeddings and vector quantized embeddings as the intermediate interfaces respectively. massive experiments on supervised and unsupervised nmt translation tasks show that our proposed semface obtains substantial improvements over the state-of-the-art baseline models. in the future, we will design and test more semantic interface types for extensions.