Universal Conditional Masked Language Pre-training for Neural Machine Translation (2022)

Abstract: Pre-trained sequence-to-sequence models have significantly improved Neural Machine Translation (NMT). Different from prior works where pre-trained models usually adopt an unidirectional decoder, this paper demonstrates that pre-training a sequence-to-sequence model but with a bidirectional decoder can produce notable performance gains for both Autoregressive and Non-autoregressive NMT. Specifically, we propose CeMAT, a conditional masked language model pre-trained on large-scale bilingual and monolingual corpora in many languages. We also introduce two simple but effective methods to enhance the CeMAT, aligned code-switching & masking and dynamic dual-masking. We conduct extensive experiments and show that our CeMAT can achieve significant performance improvement for all scenarios from low- to extremely high-resource languages, i.e., up to +14.4 BLEU on low resource and +7.9 BLEU improvements on average for Autoregressive NMT. For Non-autoregressive NMT, we demonstrate it can also produce consistent performance gains, i.e., up to +5.3 BLEU. To the best of our knowledge, this is the first work to pre-train a unified model for fine-tuning on both NMT tasks. Code, data, and pre-trained models are available at https://github.com/huawei-noah/Pretrained-Language-Model/CeMAT

## introduction pre-trained language models have been widely adopted in nlp tasks @xcite . for example, xlm @xcite demonstrated that crosslingual pre-training is effective in improving neural machine translation (nmt), especially on lowresource languages. these methods all directly pretrain a bidirectional encoder or an unidirectional decoder. the encoder and decoder in nmt models are then independently initialized with them and approach enc. dec. mono. para. mbert @xcite • • xlm @xcite • • • mass @xcite • → • mbart @xcite • → • mrasp @xcite • → • cemat (ours) inspired by that, we present cemat, a multilingual conditional masked language pretraining model for machine translation, which cat sat on the mat [en] we dance on the grass [de] wir tanzen auf dem gras mono. para. original [en] cat sat on the mat [en] [mask] sat on the [mask] [en] we danse [mask] the grass [de] wir [mask] auf dem [mask] mono. para. masked [en] kedi sat on the [mask] encoder self-attention feed forward bidirectional decoder cross-attention self-attention feed forward encoder decoder [en] who are you [de] wer bist du wer bist @xmath0. para. consists of a bidirectional encoder, a bidirectional decoder, and a cross-attention module for bridging them. specifically, the model is jointly trained by mlm on the encoder and conditional mlm (cmlm) on the decoder with large-scale monolingual and bilingual texts in many languages. to better train the representation capability of the model, the masking operations are applied in two steps. first, some source words that have been aligned with target words are randomly selected and then substituted by new words of similar meanings in other languages, and their corresponding target words are masked. we call this method aligned code-switching & masking. then, the remaining words in both source and target languages will be masked by dynamic dual-masking. extensive experiments on downstream at and nat tasks show significant gains over prior works. specifically, under low-resource conditions (< 1m bitext pairs), our system gains up to +14.4 bleu points over baselines. even for extremely highresource settings (> 25m), cemat still achieves significant improvements. in addition, experiments on the wmt16 @xmath1.1 bleu) by the back-translation (bt; @xcite . the main contributions of our work can be summarized as follows: figure 2 : the details of our two-step masking. we first obtain the aligned pair set λ = {("dance","tanzen"),...} (marked with ) from the original inputs by looking up the cross-lingual dictionary (denote as 1.aligned), and then randomly select a subset (marked as "dance" "tanzen" with red color) from it, in the lower left of the figure. for each element in the subset, we select a new word by f m (x i m ), and perform csr to replace the source fragment ("danse" marked as red color) and csm for target ("[mask]" marked as red color) respectively. finally, we do the dm process to mask the contents of the source and target respectively ("[mask]" marked as light-blue color). ## pre-training approach our cemat is jointly trained by mlm and cmlm on the source side and the target side, respectively. the overall framework is illustrated in figure 1 . in this section, we first introduce the multilingual cmlm task (section 2.1). then, we describe the two-step masking, including the aligned codeswitching & masking (section 2.2) and the dynamic dual-masking (section 2.3). finally, we present training objectives of cemat (section 2.4). formally, our training data consists of m language-pairs @xmath2, d 2 , ..., d m }. d k (m, n) is a collection of sentence pairs in language l m and l n , respectively. in the description below, we denote a sentence pair as (x m , y n ) ∈ d k (m, n), where x m is the source text in the language l m , and y n is the corresponding target text in the language l n . for monolingual corpora, we create pseudo bilingual text by copying the sentence, namely, x @xmath3. ## pre-training settings pre-training data we use the english-centric multilingual parallel corpora of pc32 foot_1 , and then collect 21-language monolingual corpora from common crawl foot_2 . in this paper, we use iso language code foot_3 to identify each language. a "[language code]" token will be prepended to the beginning of the source and target sentence as shown in figure 2 . this type of token helps the model to distinguish sentences from different languages. the detailed correspondence and summary of our pre-training corpora can be seen in appendix a. ## autoregressive neural machine translation in this section, we verify cemat provides consistent performance gains in low to extremely high resource scenarios. we also compare our method with other existing pre-training methods and further present analysis for better understanding the contributions of each component. ## non-autoregressive neural machine translation in this section, we will verify the performance of our cemat on the nat, which generates translations in parallel, on widely-used translation tasks. ## conclusion in this paper, we demonstrate that multilingually pre-training a sequence-to-sequence model but with a bidirectional decoder produces significant performance gains for both autoregressive and non-autoregressive neural machine translation. benefiting from conditional masking, the decoder module, especially the cross-attention can learn the word representation and cross-lingual representation ability more easily. we further introduce the aligned code-switching & masking to align the representation space for words with similar semantics but in different languages, then we use a dynamic dual-masking strategy to induce the bidirectional decoder to actively obtain the information from the source side. finally, we verified the effectiveness of these two methods. in the future, we will investigate more effective word alignment method for aligned code-switching & masking.