CSP :Code-Switching Pre-training for Neural Machine Translation (2020)

Abstract: This paper proposes a new pre-training method, called Code-Switching Pre-training (CSP for short) for Neural Machine Translation (NMT). Unlike traditional pre-training method which randomly masks some fragments of the input sentence, the proposed CSP randomly replaces some words in the source sentence with their translation words in the target language. Specifically, we firstly perform lexicon induction with unsupervised word embedding mapping between the source and target languages, and then randomly replace some words in the input sentence with their translation words according to the extracted translation lexicons. CSP adopts the encoder-decoder framework: its encoder takes the code-mixed sentence as input, and its decoder predicts the replaced fragment of the input sentence. In this way, CSP is able to pre-train the NMT model by explicitly making the most of the alignment information extracted from the source and target monolingual corpus. Additionally, we relieve the pretrain-finetune discrepancy caused by the artificial symbols like [mask]. To verify the effectiveness of the proposed method, we conduct extensive experiments on unsupervised and supervised NMT. Experimental results show that CSP achieves significant improvements over baselines without pre-training or with other pre-training methods.

## introduction neural machine translation @xcite @xcite which typically follows the encoder-decoder framework, directly applies a single neural network to transform the source sentence into the target sentence. with tens of millions of trainable parameters in the nmt model, translation tasks are usually datahungry, and many of them are low-resource or even zero-resource in terms of training data. following the idea of unsupervised and self-supervised pre-training methods in the nlp area @xcite @xcite @xcite , some works are proposed to improve the nmt model with pretraining, by making full use of the widely available monolingual corpora @xcite @xcite @xcite @xcite . typically, two different branches of pre-training approaches are proposed for nmt: model-fusion and parameterinitialization. the model-fusion approaches seek to incorporate the sentence representation provided by the pretrained model, such as bert, into the nmt model @xcite @xcite @xcite . these approaches are able to leverage the publicly available pre-trained checkpoints in the website but they need to change the nmt model to fuse the sentence embedding calculated by the pre-trained model. large-scale parameters of the pre-trained model significantly increase the storage cost and inference time, which makes it hard for this branch of approaches to be directly used in production. as opposed to model-fusion approaches, the parameter-initialization approaches aim to directly pre-train the whole or part of the nmt model with tailored objectives, and then initialize the nmt model with pre-trained parameters @xcite . these approaches are more production-ready since they keep the size and structure of the model same as standard nmt systems. while achieving substantial improvements, these pre-training approaches have two main cons. firstly, as pointed out by @xcite , the artificial symbols like [mask] used by these approaches during pre-training are absent from real data at finetuning time, resulting in a pretrain-finetune discrepancy. secondly, while each pre-training step only involves sentences from the same language, these approaches are unable to make use of the cross-lingual alignment information contained in the source and target monolingual corpus. we argue that, as a cross-lingual sequence generation task, nmt requires a tailored pre-training objective which is capable of making use of cross-lingual alignment signals explicitly, e.g., word-pair information extracted from the source and target monolingual corpus, to improve the performance. to address the limitations mentioned above, we propose code-switching pre-training (csp) for nmt. we extract the word-pair alignment information from the source and target monolingual corpus automatically, and then apply the extracted alignment information to enhance the pre-training performance. the detailed training process of csp can be presented in two steps: 1) perform lexicon induction to get translation lexicons by unsupervised word embedding mapping @xcite ; 2) randomly replace some words in the input sentence with their translation words in the extracted translation lexicons and train the nmt model to predict the replaced words. csp adopts the encoder-decoder framework: its encoder takes the code-mixed sentence as input, and its decoder predicts the replaced fragments based on the context calculated by the encoder. by predicting the sentence fragment which is replaced on the encoder side, csp is able to either attend to the remaining words in the source language or to the translation words of the replaced fragment in the target language. therefore, csp trains the nmt model to: 1) learn how to build the sentence representation for the input sentence as the traditional pre-training methods do; 2) learn how to perform cross-lingual translation with extracted word-pair alignment information. in summary, we mainly make the following contributions: there have also been works on applying prespecified translation lexicons to improve the performance of nmt. @xcite and @xcite proposed an altered beam search algorithm, which took target-side pre-specified translations as lexical constraints during beam search. @xcite investigated a data augmentation method, making code-switched training data by replacing source phrases with their target translations according to the pre-specified translation lexicons. recently, motivated by the success of unsupervised cross-lingual embeddings, @xcite al. (2018b), lample et al. (2018a) and yang et al. (2018) applied the pre-trained translation lexicons to initialize the word embeddings of the unsupervised nmt model. @xcite applied translation lexicons to unsupervised domain adaptation in nmt. in this paper, we apply the translation lexicons automatically extracted from the monolingual corpus to improve the pre-training of nmt. ## csp in this section, we firstly describe how to build the shared vocabulary for the nmt model; then we present the way extracting the probabilistic translation lexicons; and we introduce the detailed training process of csp finally. ## experiments and results this section describes the experimental details about csp pre-training and fine-tuning on the supervised and unsupervised nmt tasks. to test the effectiveness and generality of csp, we conduct extensive experiments on english-german, english-french and chinese-to-english translation tasks. ## conclusions and future work this work proposes a simple yet effective pretraining approach, i.e., csp for nmt, which randomly replaces some words in the source sentence with their translation words in the probabilistic translation lexicons extracted from monolingual corpus only. to verify the effectiveness of csp, we investigate two downstream tasks, supervised and unsupervised nmt, on english-german, english-french and chinese-to-english translation tasks. experimental results show that the proposed approach achieves substantial improvements over strong baselines consistently. additionally, we show that csp is able to enhance the ability of nmt on handling code-switching inputs. there are two promising directions for the future work. firstly, we are interested in applying csp to other related nlp areas for code-switching problems. secondly, we plan to investigate the pre-training objectives which are more effective in utilizing the cross-lingual alignment information for nmt.