Does Masked Language Model Pre-training with Artificial Data Improve Low-resource Neural Machine Translation? (2023)

Abstract: Pre-training masked language models (MLMs) with artificial data has been proven beneficial for several natural language processing tasks such as natural language understanding and summarization; however, it has been less explored for neural machine translation (NMT).A previous study revealed the benefit of transfer learning for NMT in a limited setup, which differs from MLM.In this study, we prepared two kinds of artificial data and compared the translation performance of NMT when pre-trained with MLM.In addition to the random sequences, we created artificial data mimicking token frequency information from the real world. Our results showed that pre-training the models with artificial data by MLM improves translation performance in low-resource situations. Additionally, we found that pre-training on artificial data created considering token frequency information facilitates improved performance.

## introduction transfer learning is an effective method for improving the performance of various natural language processing tasks @xcite @xcite . this has been proven for neural machine translation (nmt) in low-resource situations @xcite @xcite . general explanations attribute the performance improvements in various downstream tasks to the transfer of linguistic traits (e.g., frequency, co-occurrence, and structure of words) in pre-training data @xcite @xcite . meanwhile, some studies have focused on identifying the specific traits in pre-training data that improve downstream task performance by employing artificial data for pre-training @xcite @xcite . transformer 45 190 4 21 _ _ _ _ 50 304 85 6 45 190 4 21 50 304 85 6 artificial data masking transformer transfer the parameters how do you explain this progression ? wie erklären sie diesen fortschritt ? source data target data fine-tuning on parallel data mass pre-training with artificial data sequences (see §2.2) brings better translation performance in low-resource situations. their pretraining tasks included 1) autoencoding: translating one token into the same, and 2) substitution: translating one token into another; however, their solutions were uncommon for pre-training nmt models. thus, the improvement of the translation performance when performing pre-training through the masked sequence-to-sequence model @xcite @xcite with artificial data was not addressed. in this work, we use masked language modeling for an encoder-decoder model called masked sequence-to-sequence pre-training (mass; @xcite as the pre-training task and investigate the translation performance of the nmt model pre-trained on artificial data in simulated (@xmath0. additionally, other than random sequences, we create artificial data containing token frequency information from the real world and examine whether injecting this information into pre-training data affects translation performance. we compare the performance when pre-trained on each dataset with the mass objective. furthermore, we perform ablation studies to investigate how each part of the network affects the translation performance when pre-trained on artificial data by transferring or freezing some parameters of the pre-trained model. our findings can be summarized as follows: ## experimental setup simulated low-resource situation we use 30k and 100k paired sentences randomly sampled from wmt14 @xmath1, common crawl, and news commentary; approximately 4.5m sentences in total) to compare how pre-training with artificial data affects translation performance on different sizes. we use newstest2013 of wmt as the validation set and newstest2016 as the test set. we calculate casesensitive bleu using sacrebleu foot_2 , foot_3 @xcite for evaluation. ## analysis we investigate which parts of the nmt model pretrained on artificial data contribute to improved performance and verify whether the effect of each part on translation performance differs from the case where pre-trained on real-world data. specifically, we divide the model parameters into four components: embeddings (emb), encoders (enc), cross-attentions (x-attn), and decoders except for cross-attentions (dec), and perform two ablation studies. firstly, we transfer a part of the components from a pre-trained model and finetune the model ( token frequency information imparts embeddings with beneficial information for translation from on the other hand, when pre-training with the "random" dataset, the performance gains by transferring emb are negligible (rows 3 and 4). both "english shuf" and "zipf" datasets contain token frequency information from the real world; that is, the distribution of token frequency follows zipf's law. this brings about multiple occurrences of the same n-gram to a certain extent, even when the tokens in pre-training data are shuffled @xcite . figure 2 shows the rank-frequency distribution of n-grams (@xmath4, 2, 3, 4) in each pretraining dataset. 9 even though tokens are sampled independently to form a sequence for the "english shuf" and "zipf" datasets, we can observe a power trend in the frequency of n-grams at @xmath5, 3, 4. therefore, we consider that the presence of multiple instances of the same n-gram, which results in the emergence of local contexts within a sentence, and embeddings pre-trained on this dataset obtain beneficial information for translation performance. encoders pre-trained on artificial data obtain enhanced representations to understand the inputs from we attribute this tendency to the mechanism of the mass pre-training depending on data property. when pre-training with the "english" dataset, the dec predicts a masked span of an english text that contains linguistic traits like structure, which makes the dec's prediction easier. therefore, the dec can make predictions without requiring much information from the enc, which makes the enc understand the input sequence moderately. this explains why the best score is achieved when freezing dec and the worst score is achieved when freezing enc compared to other datasets. however, when pre-training with other datasets in which the tokens in a sequence are independent of each other, it is challenging for the dec to predict a masked span autoregressively. this incentivizes the dec to extract beneficial information for predictions from the enc; that is, the dec relies more on enc's information. this encourages the enc to understand the input sequence more, and transferring enc enhanced to capture the input meaning results in higher translation performance. this consideration is consistent with the assertion of @xcite . ## conclusion in this work, we chose mass for the pretraining task and explored the effects on translation performance in low-resource situations when pre-training the nmt model on artificial data. both in simulated (@xmath6, pretraining with artificial data improved the performance, and further improvements could be obtained by injecting token frequency information when the parallel data size was very small. through ablation studies, we found that token frequency information generates contexts within a dataset, and pre-training on such datasets enables embeddings to obtain beneficial information for translation performance. in addition, pre-training on artificial datasets in which tokens are independent of each other enhances the capability of encoders to understand inputs, resulting in improved translation performance.