XLM - D : Decorate Cross-lingual Pre-training Model as Non-Autoregressive Neural Machine Translation (2022)

Abstract: Pre-training language models have achieved thriving success in numerous natural language understanding and autoregressive generation tasks, but non-autoregressive generation in applications such as machine translation has not sufficiently benefited from the pre-training paradigm. In this work, we establish the connection between a pre-trained masked language model (MLM) and non-autoregressive generation on machine translation. From this perspective, we present XLM-D, which seamlessly transforms an off-the-shelf cross-lingual pre-training model into a non-autoregressive translation (NAT) model with a lightweight yet effective decorator. Specifically, the decorator ensures the representation consistency of the pre-trained model and brings only one additional trainable parameter. Extensive experiments on typical translation datasets show that our models obtain state-of-the-art performance while realizing the inference speed-up by 19.9x. One striking result is that on WMT14 En-De, our XLM-D obtains 29.80 BLEU points with multiple iterations, which outperforms the previous mask-predict model by 2.77 points.

## introduction as a dominant pre-training paradigm in natural language processing (nlp), masked language models such as bert and its variants @xcite , were initially proposed and have achieved state-of-the-art performance on various natural language understanding tasks. for generation tasks, previous studies either leverage the pre-trained bert as an external component for representation fusing @xcite or simply initialize the generation models with a pre-trained model @xcite . although straightforward, these methods suffer from either heavy computation costs during inference or only supporting autoregressive generation with sequential dependency. additionally, these approaches result in inconsistency from two perspectives: 1) mismatch of architectures between pre-trained models and the generation model; 2) disagreement of training objectives between pre-training and autoregressive generation tasks. different from previous studies, we reexamine the problem from another point of view. we find that the training objective in pre-trained mlm can align well with the one in non-autoregressive generation, which is an emergent generation paradigm due to its excellent inference speed-up @xcite . intuitively, both training objectives are formalized as a series of independent token predictions in the output sequence. based on this observation, we incorporate a pretrained mlm into the non-autoregressive generation. specifically, in this work, we focus on the classic non-autoregressive machine translation task. albeit studied for ages, nat models have not performed very well, lagging behind the autoregressive translation counterpart. we propose to adapt the cross-lingual pretraining model (xlmr) into nat models with a lightweight, effective and user-configurable decorator. following occam's razor principle and to better leverage the capability of pre-training models, we design the decorator component based on two key criteria: 1) involve additional trainable parameters as few as possible, e.g., parameter-free; 2) keep the model intermediate representation consistent after using the decorator. guided by the two criteria, the decorator consists of a distance-based latent transformation module and a position-wise add and scale module, which contains only one trainable scalar parameter. moreover, the decorator can be flexibly incorporated into a user-specified layer of xlmr to balance the translation performance and inference speed. we use the connectionist temporal classification (ctc) @xcite loss as the training objective. to validate the effectiveness of our approach, we systematically conduct the evaluation on several widely-used translation datasets. extensive experiments demonstrate that our proposed model significantly and consistently improves the translation performance and achieves new state-of-theart results in both single-step and iterative nat models. results show that our single-step xlm-d model achieves 27.46/34.70 bleu points on wmt14 en⇒de and wmt16 en⇒ro translation tasks with 19.9× speed-up. encouragingly, our iterative xlm-d model obtains 29.80/35.65 points on both tasks and outperforms previous wellperformed models (cmlm) @xcite by a large margin, i.e., 2.77/2.57 points. further analyses reveal that our approach enhances the capability in long sentence translation and can be user-configured to balance the trade-off between the translation quality and inference speed. ## related work pre-training models unsupervised representation learning @xcite @xcite has achieved remarkable success in natural language understanding. peters et al. (2018) first introduced a general approach for learning high-quality context-dependent representations, which encode syntactic and semantic information efficiently. by using the objective of masked lan-guage model, devlin et al. (2019) introduced a new bidirectional representation model and obtained state-of-the-art results on various tasks. most recently, conneau et al. (2020) extended pre-training models to one hundred languages by using more than two terabytes of data. different from these studies, we observe that cross-lingual pre-trained masked language models are endowed with the attribute of non-autoregressive generation inherently. hence, we propose to decorate a cross-lingual pretraining model as non-autoregressive translation. non-autoregressive neural machine translation many previous works investigate reducing the decoding latency by empowering the parallel sequence generation capability @xcite . @xcite pioneered to propose a non-autoregressive translation model by modeling fertility as latent variables. although accelerating the inference process significantly, nat models suffer from the serious multi-modality problem, which results in considerably worse performance than their autoregressive counterparts. to alleviate this issue, extensive methods have been investigated in vanilla non-autoregressive training @xcite @xcite . another line of researches @xcite @xcite @xcite employ an iterative refinement process to maintain the translation quality. @xcite used a masked language modeling to predict any subset of target words by conditioning on a partially masked target translation. ## conclusion we propose to decorate cross-lingual pre-training models (xlmr) as non-autoregressive machine translation in a lightweight, effective and flexible way. firstly, we bridge pre-training masked language models with non-autoregressive generation, revealing that a pre-trained masked language model can be directly used to produce sequences in parallel. moreover, we propose to incorporate distancebased latent transformation and position-wise add and scale components into a user-specified layer of xlmr. empirical results on a variety of language pairs demonstrate the effectiveness and universality of our approach. further analyses confirm that the proposed method benefits translations for long sentences and achieves a better balance between translation quality and inference speedup.