Synthetic Pre-Training Tasks for Neural Machine Translation (2023)

Abstract: Pre-training models with large crawled corpora can lead to issues such as toxicity and bias, as well as copyright and privacy concerns. A promising way of alleviating such concerns is to conduct pre-training with synthetic tasks and data, since no real-world information is ingested by the model. Our goal in this paper is to understand the factors that contribute to the effectiveness of pre-training models when using synthetic resources, particularly in the context of neural machine translation. We propose several novel approaches to pre-training translation models that involve different levels of lexical and structural knowledge, including: 1) generating obfuscated data from a large parallel corpus 2) concatenating phrase pairs extracted from a small word-aligned corpus, and 3) generating synthetic parallel data without real human language corpora. Our experiments on multiple language pairs reveal that pre-training benefits can be realized even with high levels of obfuscation or purely synthetic parallel data. We hope the findings from our comprehensive empirical analysis will shed light on understanding what matters for NMT pre-training, as well as pave the way for the development of more efficient and less toxic models.

## introduction and motivation neural machine translation (nmt) models depend on large quantities of aligned training data @xcite @xcite . for many language pairs of interest, however, high quality parallel data is either unavailable or exists only in limited quantities. training robust nmt systems with such limited data can be a significant challenge. even for high-resource language pairs, parallel data can be noisy and frequently contains toxic speech or biased language. such problems are particularly acute for comparable corpora crawled automatically from the web @xcite figure 1 : a comparison of the extent to which the synthetic data generation methods described in section 3 encode lexical and/or structural translation knowledge. the vertical axis compares methods with respect to lexical knowledge. the horizontal axis compares structural knowledge. bleu scores correspond to the indonesianto-english translation task described in section 4. our primary motivation is to investigate how knowledge transfer from nmt pre-training can help to avoid or minimize the data issues described above. we study the impact of pre-training and transfer learning on translation tasks by comparing various procedural approaches to synthetic data generation. each approach has varying degrees of inherited or artificially constructed lexical and structural translation knowledge. the degree to which each method encodes lexical and/or structural translation knowledge is plotted in abstract form in figure 1 . we describe each of our synthetic data generation methods in section 3. our first approach ( §3.1) studies the extent to which the transfer benefits of regular pre-training can be realized when using obfuscated or encrypted data. our obfuscated corpus is derived from real parallel data by mapping the original words to a vocabulary of 'nonsense' tokens. experiments on six different language pairs show that obfuscated pretraining is able to capture much of the transferable knowledge: pre-training with an obfuscation ratio as high as 75% is still able to achieve bleu scores close to those obtained by a model pre-trained on the original un-obfuscated parallel data. our second approach ( §3.2) seeks to maximize the benefit that can be derived from a specific limited quantity of fine-tuning data. we do this by pretraining on newly constructed artificial sentence pairs synthesized directly from the fine-tuning corpus. the synthetic sentence pairs are created by concatenating randomly sampled aligned phrase pairs extracted from the fine-tuning corpus. although the sentence-level fluency and grammaticality of sentences constructed using this technique are both quite poor, they do retain word-and phraselevel correspondences and local reordering information that can greatly improve translation quality and robustness compared to models trained using only the original fine-tuning data. our third approach ( §3.3) explores the pretraining impact of important translation phenomena such as alignments and reordering. we pre-train models on procedurally generated synthetic parallel data that does not derive from any real human language corpus. we design three simple synthetic sequence-to-sequence translation tasks and associated data sets. since our data is procedurally generated, problems of toxicity, attribution and copyright can be avoided. we evaluate the effectiveness of pre-training and transfer for our synthetic tasks in the context of low-resource nmt. our results show that -to a surprising degree -the transfer benefits of pre-training can be realized even with purely synthetic tasks and data. our analysis shows that structure, in the form of aligned sub-trees, matters in synthetic pre-training for nmt. we empirically evaluate the impact of each of our proposed synthetic pre-training methods in lowresource mt settings ( §4), followed by a discussion and analysis explaining our insights into what makes for a good pre-trained model ( §5). we also consider the question of model toxicity. we measure the extent of hallucinated toxicity in each syn-thetic data generation method, showing that synthetic methods can result in substantially reduced toxicity compared to models pre-trained on real parallel corpora. the primary contributions of our paper are as follows: (i) we propose several novel synthetic pre-training tasks, that encode varying degrees of structural and lexical knowledge, in order to gain insights into what makes for a good pre-trained nmt model; (ii) we conduct a comprehensive empirical evaluation of knowledge transfer in nmt from synthetic data pre-training, considering metrics of both translation quality and toxicity; and (iii) we demonstrate that synthetic data is a promising stepping stone towards relieving the data burden in low-resource translation and building more accurate and trustworthy nmt systems. ## related work transferring knowledge from pre-trained language models @xcite @xcite ) is a common technique for ensuring robust nlp downstream task performance. early work by @xcite explored transfer learning for nmt from a model pre-trained on a single language pair. more recently, methods that transfer from large-scale multilingual pre-trained models @xcite @xcite have achieved improved translation performance across a wide range of language pairs. aji et al. (2020) conducted a study on pre-training and transfer for low-resource nmt. these works depend on real human language for pre-training and therefore inherit data issues such as toxicity and bias. in contrast, our work studies nmt pre-training and transfer from synthetic data based on 'nonsense' words. only a few methods have addressed the problem of pre-training from synthetic data in nlp. @xcite proposed pre-training for summarization using synthetic article and summary pairs derived from manually curated tasks and a vocabulary of nonsense symbols. @xcite have shown that masked language model pre-training with limited word-order information can be almost as effective as regular pre-training. @xcite show that non-human language data and artificial datasets (e.g. nested sequences of parentheses), can still demonstrate knowledge transfer to downstream nlp tasks. @xcite compare the effect of pre-training on many sim-ple synthetic tasks. our work in this paper represents the first empirical evaluation of synthetic pre-training for neural machine translation. to the best of our knowledge, our proposed synthetic tasks have not been explored in previous work. the quality of a pre-trained model should not be measured purely by performance. we should also consider trustworthiness @xcite @xcite . recent works have noted that translation systems pre-trained on webscale corpora are prone to produce toxic @xcite or biased outputs @xcite @xcite , and/or present privacy issues @xcite , which reduces user trustworthiness. bias mitigation for nmt has been well-investigated while privacy and toxicity issues for translation are still not extensively explored @xcite . @xcite propose federated neural machine translation to protect privacy such as commercial leakage or copyright. @xcite mitigate toxicity by filtering training data that matches pre-defined multilingual toxic word lists. ## synthetic pre-training for nmt pre-training followed by fine-tuning is a common approach to training robust nmt models @xcite . our motivation is to understand the extent to which the transfer benefits of pre-training can be replicated using synthetic tasks and data. in this section, we describe three approaches to the programmatic generation of synthetic data: (i) pre-training with obfuscated parallel data that implicitly preserves certain language properties such as distributional frequencies, (ii) pre-training with synthetic data created by concatenating aligned phrases, and (iii) pre-training with synthetic tasks designed to encourage transfer learning of important translation properties such as long-distance reordering. ## experimental framework we evalute our synthetic pre-training data generation methods for nmt using using both englishcentric and non-english-centric language pairs. ## conclusion our study of synthetic pre-training tasks for nmt showed that pre-training benefits can still be achieved even when using synthetic or obfuscated data. additionally, we have shown that synthetic data has the potential to reduce model toxicity compared to models trained on web-scale crawled corpora. our research provides insights into what types of knowledge transfer make for a good pretrained model. we believe that synthetic data augmentation techniques based on synthetic tasks and procedurally generated data are a promising solution for addressing pre-training data concerns, and can lead to efficient, accurate, and trustworthy nmt. in future work, we plan to further investigate synthetic pre-training by exploring more advanced data generation models and directly optimizing the parameters for specific downstream fine-tuning tasks. increasing the effectiveness of synthetic data at different data scales is also worthy of further exploration.