Con- NAT : Contrastive Non-autoregressive Neural Machine Translation (2022)

Abstract: Inspired by the success of contrastive learning in natural language processing, we incorporate contrastive learning into the conditional masked language model which is extensively used in non-autoregressive neural machine translation (NAT). Accordingly, we propose a Contrastive Non-autoregressive Neural Machine Translation (Con-NAT) model. Con-NAT optimizes the similarity of several different representations of the same token in the same sentence. We propose two methods to obtain various representations: Contrastive Common Mask and Contrastive Dropout. Positive pairs are various different representations of the same token, while negative pairs are representations of different tokens. In the feature space, the model with contrastive loss pulls positive pairs together and pushes negative pairs away. We conduct extensive experiments on six translation directions with different data sizes. The results demonstrate that Con-NAT showed a consistent and significant improvement in fully and iterative NAT. Con-NAT is state-of-the-art on WMT’16 Ro-En (34.18 BLEU).

## introduction neural machine translation has developed rapidly with the development of deep learning. the traditional neural machine translation models @xcite @xcite are autoregressive (at), which means that they predict target tokens one by one based on source tokens and previously predicted tokens. this dependence leads to the limitation of translation speed, and the time required for translation is directly proportional to the sentence length. recently, non-autoregressive machine translation (nat) becomes a research hotspot. the nonautoregressive generation mode eliminates token dependency in the target sentence and generates all tokens in parallel, considerably improving translation speed. however, the speed increase is accompanied by a decrease in translation quality. many iterative models have been developed to make a trade-off between translation speed and quality. the iterative model improves translation quality by continually and iteratively optimizing the generated target sentence. the iterative model is usually to predict the masked token in the target sentence, such as bert @xcite . the masked tokens are usually chosen at random. a sentence can be masked in a variety of ways. in different masked sequences of the same sentence, the predicted tokens at the same position should be the same. embodied in token representations is the similarity of representations. the representation of the same masked token should be similar because they are from the same token and have the same semantics in a similar context (the same source sentence and the different masked results of the same target sentence). we think about how to make these different representations of the same token more similar. inspired by the successful use of contrastive learning in nlp pre-trained models (e.g., @xcite , we explore combining contrastive learning and the conditional masked language model, treating different representations of the same masked token as positive pairs and representations of different tokens as negative pairs. we pull in positive pairs and push out negative pairs using contrastive learning. as illustrated in figure 1 , we propose two strategies for constructing positive pairs in this paper. contrastive common mask is a method that utilizes representations of the same token in different masked sequences of the same sentence. as shown in figure 1 (a), "fell" is masked both in "he [mask] asleep almost [mask]" and "he [mask] asleep @xcite instantly", which are different randomly masked results of "he fell asleep almost instantly". the other is inspired by @xcite , where we feed the same input to the decoder twice and get two different representations due to the dropout setting, which we call contrastive dropout. the two representations of the same token should be similar, as shown in figure 1(b) . we use the constructed positive and negative pairs to calculate the contrastive loss and jointly optimize it with the cross-entropy loss. we verify the effectiveness of our model in six translation directions of three standard datasets with varying data sizes. experiments show that our model beats cmlm @xcite with 0.80-1.46 bleu margins and glat @xcite with 0.18-0.65 bleu margins at the same translation speed. it also outperforms other cmlm-based models and beats the state-of-the-art nat model on . the main contributions of this work can be concluded as follows: ## preliminaries non-autoregressive machine translation the machine translation task is defined as generating a target sentence @xmath0, . . . , y ty under the condition of a given source sentence @xmath1, . . . , x tx }. most models factorize the conditional probability p θ (y | x) by: although the assumption of conditional independence improves the translation speed, it also impairs the model performance. ## methodology in this section, we present how we incorporate contrastive learning into nat. we begin by introducing the structure of our model con-nat, followed by two positive pair construction methods for contrastive learning, and lastly, the training objective combined with the contrastive loss. figure 2 shows the overall framework. ## related work in order to speed up the translation process, @xcite introduced non-autoregressive translation. we divide nat models into three types according to the training loss. the first is the conditional independent language model, which includes: enhancing the decoder input @xcite @xcite , enhancing the decoder output @xcite , learning or transforming from autoregressive model @xcite @xcite @xcite , latent variable-based model @xcite @xcite . the second is the conditional masked language model, includes: strong baseline model cmlm @xcite , disentangled context transformer @xcite , jointly masked sequence-to-sequence model @xcite , semi-autoregressive training @xcite , increasing the mask ratio gradually @xcite , learning autoregressive model @xcite , progressive multi-granularity training @xcite , using the bi-direction distillation data @xcite , improving the alignment of cross entropy @xcite . the last is the ctc model, which includes ctc @xcite and imputer @xcite which combines the ctc and the masked language model. other excellent approaches include: flow-based generative model @xcite , adding a lite autoregressive module @xcite , training with monolingual data @xcite , incorporating the pre-trained model @xcite , and tricks of the trade @xcite . ## conclusion in this work, we propose con-nat, which is the first effort to combine token-level contrastive learning and the conditional masked language model. con-nat optimizes the similarity of different representations of the same token in the same sentence by contrastive learning. we propose contrastive common mask and contrastive dropout to construct positive pairs, using different random masks and dropout masks, respectively. our model achieves consistent and significant improvement in the six translation tasks and is state-of-the-art on wmt'16 ro-en. the lightweight contrastive module is removed during inference, so it does not affect the translation speed. in the future, we will focus on combining the idea with the ctc and the pre-trained masked language model.