M i SS @ WMT 21: Contrastive Learning-reinforced Domain Adaptation in Neural Machine Translation (2021)

Abstract: In this paper, we describe our MiSS system that participated in the WMT21 news translation task. We mainly participated in the evaluation of the three translation directions of English-Chinese and Japanese-English translation tasks. In the systems submitted, we primarily considered wider networks, deeper networks, relative positional encoding, and dynamic convolutional networks in terms of model structure, while in terms of training, we investigated contrastive learning-reinforced domain adaptation, self-supervised training, and optimization objective switching training methods. According to the final evaluation results, a deeper, wider, and stronger network can improve translation performance in general, yet our data domain adaption method can improve performance even more. In addition, we found that switching to the use of our proposed objective during the finetune phase using relatively small domain-related data can effectively improve the stability of the model’s convergence and achieve better optimal performance.

## introduction news translation @xcite @xcite is one of the most prominent and appealing tasks in machine translation evaluation @xcite . our miss system took part in the wmt21 news translation task, including @xmath0, @xmath1, and @xmath2. we developed translation systems for this year's submission to investigate machine translation techniques from two perspectives: model structure and model training. all of the data used by the submitted systems is constrained. due to a lack of training resources, the english->japanese translation direction is only investigated from the model structure perspective. from the perspective of model structure, we choose the transformer @xcite ) model based on self-attention, which is extensively utilized in neural machine translation systems, as our basis @xcite . on this strong foundation, we opt to simply deepen the model by increasing the number of encoder layers or widen the model by increasing the hidden size of the model to obtain a deeper or wider model. when deepening or widening the model, we found that there is no need for additional sophisticated structure design (e.g., layer drop @xcite / sublayer drop @xcite ) or training strategy when there is adequate training data available. in addition to transformer architecture, @xcite propose a dynamic convolution structure that can perform competitively or better to the self-attention structure. follow the practice in wmt20 @xcite , we also applied the dynamic convolution architecture as another basis. according to our preliminary results on the development set, domain has a significant impact on performance, despite the fact that we are working with the resource-rich en-zh and en-ja language pairs. this year's submissions are mostly concerned with utilizing training approaches to mitigate the impact of domain differences. specifically, we first use data in all hybrid domains to train the initial nmt model, and then, based on sentence embedding model enhanced by contrastive learning, the parallel/monolingual corpus is filtered monolingually or cross-lingually, and the filtered domainrelated parallel corpus is used for further finetuning, and the domain-related monolingual corpus is used for in-domain back-translation enhancement. in addition, we also adopted a self-supervised training method to train the model on the given source text of the test set and its domain-related monolingual text obtained by filtering. in self-supervised training, we combine our data-dependent gaussian prior objective (d2gpo) objective @xcite to alleviate the collapse due to non-golden targets. in the finetune stage with the domainrelated parallel corpus, we adopted the training strategy of switching the optimization objective from the mle to our proposed dual skew divergence (dsd) @xcite . the results demonstrated that switching to the dsd objective resulted in improved convergence. from the evaluation results, we observe substantial improvements over the strong baseline with 4.3 (@xmath3, 4.8 (@xmath4, 3.2 (@xmath5, respectively. the gains can be attributed to larger model capacity and better training strategies. and the results suggest that the cost of domain adaptation to improve performance is less than the cost of increasing model capacity. ## model perspective with the development of deep learning in nlp @xcite @xcite , model ensembling can usually produce better results than single models, and the bigger the difference between the models used for ensembling, within a certain limit, the higher the improvement will be. as a result, we chose four distinct typical architectures as the basis for single nmt models and trained them on the same data. the detailed parameters of each model architecture are shown in deep transformer some related works @xcite @xcite have revealed that deep networks have great advantages in nmt performance compared to shallow networks recently. based on the transformer nmt model architecture, we found that in the presence of sufficient training data, merely increasing the number of stacked layers of the encoder can fulfill the goal of deep transformer without the use of additional initialization, dropout, or layer skipping techniques. wide transformer recent researches @xcite @xcite @xcite have demonstrated that, in addition to deepening the nmt model, widening the model can also effectively improve translation performance, with increasing the feed-forward network (ffn) size in the trans- deep dynamicconv dynamic convolution (dynamicconv) @xcite was proposed as a replacement for transformer architecture and has piqued much interest @xcite due to its good speed advantage and comparable performance. to enhance the performance of single model, we also deepen the dynamicconv model by increasing the number of encoder layers, denoted as deep dynamicconv. the original dy-namicconv model consists of 7 encoder layers and 6 decoder layers. we deepen the dynamicconv model's encoder layers to deep dynamicconv. because the kernel size of each convolution layer in the dynamicconv model differs, we set the kernel sizes of the 16 encoder layers in deep dynamicconv to @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite and leave the other settings unchanged from the original model. relative position encoding because selfattention in the convention transformer model is position-independent, the encoded features must be enhanced with explicit positional information for natural language processing. absolute position encoding is usually employed in the transformer nmt model. @xcite proposed to add relative position encoding (rpe) for improving self-attentional features and shown additional performance gains. we also applied relative position encoding to the wide transformer model and created another strong baseline. we use the identical vocabulary and data to train these four baseline models separately, and then average the best 5 checkpoints in each model's training phase to generate the final model output in the corresponding stage. according to wu et al. ## training perspective contrastive learning-reinforced domain adaptation data domain issues have been found to have a significant impact on machine translation performance @xcite . the official training data is of hybrid domain, despite the fact that the evaluation task is news translation. and, while news translation corpora can be deemed to be in the news domain, there are significant variances in news styles within the same domain. as a result, one of the keys to performance enhancement will be how to utilize the data training model that is closer to the evaluation data domain and style. using languages l 1 and l 2 as an example, the data that may be used comprises the parallel corpus d p l 1 -l 2 , as well as their respective large-scale monolingual corpus d m l 1 and d m l 2 . parallel corpora are typically utilized for direct training of nmt models, whereas monolingual corpora are used for back-translation @xcite and self-supervised training @xcite . the domain filtering method can be utilized in these three training procedures to create corpus whose domain is more similar to the development and test sets. instead of relying on the co-occurrence probabil-ity of the surface tokens in the sentence, we based the domain filtering on the hypothesis that the more similar the sentence representations generated by the transformer encoder are, the more likely they are to be dispersed in the same domain. because the current transformer encoder's representation is based on the bidirectional and full attention of all tokens, the combination and order of tokens have a significant impact on the final representation, the sentence representation is adequate for capturing domain information. as a result, we use the sentence embedding distance to measure the domain similarity. we leveraged a universal paraphrastic sentence encoder @xcite @xcite to embed each given sentence to a dense representation. on a large scale monolingual corpus, we train our own monolingual and multilingual sentence encoder, a transformer that has been pre-trained using masked language modeling @xcite @xcite , with the xlm toolkit @xcite and fine-tuned to maximize cosine similarity between similar sentences. contrastive learning seeks to acquire effective representation by pulling semantically close neighbors and pushing nonneighbors apart @xcite . since this criterion precisely meets the requirements of sentence representation learning, we use contrastive learning to finetune the pre-trained sentence encoder. figure 1 illustrates our contrastive learning-reinforced domain adaptation method. according to the domain adaptation requirements in actual machine translation, the trained sentence encoder needs respond to four scenar-ios: original input monolingual filter, translated input monolingual filter, original input crosslingual filter, translated input cross-lingual filter. because the fourth scenario can be covered by the first, we only employ the first three scenarios in our experiment. for all scenarios, we first follow @xcite 's approach to perform unsupervised training in which the input sentence itself is used as a positive instance due to there will be some differences between the sentence representations of the two pass input with the presence of the model dropout, and other sentences in the in-batch are used as negative instances. the unsupervised contrastive learning-trained monolingual sentence encoder can be used directly as an evaluator of the similarity of sentences in the same language and to mine similar sentences from the sentence bank. however, for the non-gold translated sentences filtering, we apply the baseline nmt models to translate parallel corpus and to back-translated monolingual corpus to generate pseudo-paraphrase corpus. and then triplet loss is used to fine-tune the unsupervised sentence encoder: likewise, we still need cross-language filtering, therefore we use parallel corpus instead of synthetic pseudo-restatement corpus and triplet loss for additional finetuning on the multilingual sentence encoder. as shown in figure 1 , taking the l 2 in-domain source sentences in development set as an example, we first use the initial nmt model to translate these sentences to l 1 translated text. the different trained sentence encoder is then used to encode these sentences and the large-scale monolingual or parallel corpus based on different scenarios respectively. then, using the faiss toolkit 1 , a query procedure is performed to locate related in-domain monolingual or parallel corpora with similarity calculation and ranking. ## results and analysis ## conclusion in this paper, we introduce our miss translation system, which participated in the wmt21 news translation task. we developed a new contrast learning-reinforced domain adaptation strategy in this work, and the experimental findings suggest that this method may significantly increase translation performance. furthermore, we conducted experiments on a range of model architectures. our domain adaption strategy improved these strong baseline models significantly, illustrating the method's generality and indicating that the performance deficiency is not due to a specific model structure.