Integrating Unsupervised Data Generation into Self-Supervised Neural Machine Translation for Low-Resource Languages (2021)

Abstract: For most language combinations and parallel data is either scarce or simply unavailable. To address this and unsupervised machine translation (UMT) exploits large amounts of monolingual data by using synthetic data generation techniques such as back-translation and noising and while self-supervised NMT (SSNMT) identifies parallel sentences in smaller comparable data and trains on them. To this date and the inclusion of UMT data generation techniques in SSNMT has not been investigated. We show that including UMT techniques into SSNMT significantly outperforms SSNMT (up to +4.3 BLEU and af2en) as well as statistical (+50.8 BLEU) and hybrid UMT (+51.5 BLEU) baselines on related and distantly-related and unrelated language pairs.

## introduction neural machine translation (nmt) achieves high quality translations when large amounts of parallel data are available @xcite . unfortunately, for most language combinations, parallel data is non-existent, scarce or low-quality. to overcome this, unsupervised mt (umt) @xcite @xcite focuses on exploiting large amounts of monolingual data, which are used to generate synthetic bitext training data via various techniques such as back-translation or denoising. self-supervised nmt (ssnmt) @xcite learns from smaller amounts of comparable data -i.e. topic-aligned data such as wikipedia articles-by learning to discover and exploit similar sentence pairs. however, both umt and ssnmt approaches often do not scale to low-resource languages, for which neither monolingual nor comparable data are available in sufficient quantity @xcite @xcite . to date, umt data augmentation techniques have not been explored in ssnmt. however, both approaches can benefit from each other, as i) ssnmt has strong internal quality checks on the data it admits for training, which can be of use to filter low-quality synthetic data, and ii) umt data augmentation makes monolingual data available for ssnmt. in this paper we explore and test the effect of combining umt data augmentation with ssnmt on different data sizes, ranging from very low-resource (∼ 66k non-parallel sentences) to high-resource (∼ 20m sentences). we do this using a common high-resource language pair (en-f r), which we downsample while keeping all other parameters identical. we then proceed to evaluate the augmentation techniques on different truly low-resource similar and distant language pairs, i.e. english (en)-{afrikaans (af ), kannada (kn), burmese (my), nepali (ne), swahili (sw), yorùbá (yo)}, chosen based on their differences in typology (analytic, fusional, agglutinative), word order (svo, sov) and writing system (latin, brahmic). we also explore the effect of different initialization techniques for ssnmt in combination with finetuning. ## related work substantial effort has been devoted to muster training data for low-resource nmt, e.g. by identifying parallel sentences in monolingual or noisy corpora in a pre-processing step @xcite @xcite and also by leveraging monolingual data into supervised nmt e.g. by including autoencoding @xcite or language modeling tasks @xcite . low-resource nmt models can benefit from high-resource languages through transfer learning @xcite , e.g. in a zero-shot setting @xcite , by using pre-trained language models @xcite , or finding an optimal path for pivoting through related languages @xcite . back-translation often works well in high-resource settings @xcite @xcite . nmt training and back-translation have been used in an incremental fashion in both unidirectional @xcite and bidirectional systems @xcite . unsupervised nmt @xcite @xcite applies bi-directional back-translation in combination with denoising and multilingual shared encoders to learn mt on very large monolingual data. this can be done multilingually across several languages by using language-specific decoders @xcite , or by using additional parallel data for a related pivot language pair @xcite . further combining unsupervised neural mt with phrase tables from statistical mt leads to top results @xcite @xcite . however, unsupervised systems fail to learn when trained on small amounts of monolingual data @xcite , when there is a domain mismatch between the two datasets @xcite or when the languages in a pair are distant @xcite . unfortunately, all of this is the case for most truly low-resource language pairs. self-supervised nmt @xcite jointly learns to extract data and translate from comparable data and works best on 100s of thousands of documents per language, well beyond what is available in true low-resource settings. ## umt-enhanced ssnmt ssnmt jointly learns mt and extracting similar sentences for training from comparable corpora in a loop on-line. sentence pairs from documents in languages l1 and l2 are fed as input to a bidirectional nmt system {l1, l2} → {l1, l2}, which filters out non-similar sentences after scoring them with a similarity measure calculated from the internal embeddings. ## exploration of corpus sizes (en-f r) to explore which technique works best with varying data sizes, and to compare with the highresource ssnmt setting in @xcite , we train ssnmt on en-f r, with different combinations of techniques (+bt, +wt, +n) over decreasingly small corpus sizes. the base (b) model is a simple ssnmt model with spe. figure 2 (left) shows that translation quality as measured by bleu is very low in the lowresource setting. for experiments with only 4 k comparable articles (similar to the corpus size available for en-yo), bleu is close to zero with base (b) and b+bt models. only when wt is applied to rejected back-translated pairs does training become possible, and is further improved by adding noise, yielding bleus of 3.38 foot_5 (en2f r) and 3.58 (f r2en). the maximum gain in performance obtained by wt is at 31 k comparable articles, where it adds ∼ 9 bleu over the b+bt performance. while the additional supervisory signal provided by wt is useful in the low and medium resource settings, up until ∼ 125 k articles, its benefits are overcome by the noise it introduces in the high-resource scenario, leading to a drop in translation quality. similarly, the utility of adding noise varies with corpus size. only bt constantly adds a slight gain in performance of ∼1-2 over all base models, where training is possible. in the high resource case, the difference between b and b+bt is not significant, with bleu 29.64 (en2f r) and 28.56 (f r2en) for b+bt, which also leads to a small, yet statistically insignificant gain over the en-f r ssnmt model in @xcite , i.e. +0.1 (en2f r) and +0.9 (f r2en) bleu. at the beginning of training, the number of extracted sentence pairs (spe) of the b+bt+wt+n model trained on the most extreme low-resource setting (4 k articles), is low (figure 2 , right), with 4 k sentence pairs extracted in the first epoch. this number drops further to 2 k extracted pairs in the second epoch, but then continually rises up to 13 k extracted pairs in the final epoch. this is not the case for the base (b) model, which starts with a similar amount of extracted parallel data but then continually extracts less as training progresses. the difference between the two models is due to the added bt and wt techniques. at the beginning of training b+bt+wt is not able to generate backtranslations of decent quality, with only few (196) backtranslations accepted for training. rejected backtranslations are passed into wt, which leads to large numbers of wt sentence pairs up to the second epoch (56 k). these make all the difference: through wt, the system is able to gain noisy supervisory signals from the data, which leads to the internal representations to become more informative for spe, thus leading to more and better extractions. then, bt and spe enhance each other, as spe ensures original (clean) parallel sentences to be extracted, which improves translation accuracy, and hence more and better backtranslations (e.g. up to 20 k around epoch 15) are accepted. ## exploration of language distance bt, wt and n data augmentation techniques are especially useful for the low-and mid-resource settings of related language pairs such as english and french (both indo-european). to apply the approach to truly low-resource language pairs, and to verify which language-specific characteristics impact the effectiveness of the different augmentation techniques, we train and test our model on a selected number of languages ( ## comparison to other nmt architectures we compare the best ssnmt model configuration per language pair with the ssnmt baseline system, and with monoses @xcite , an unsupervised machine translation model in its statistical (usmt) and hybrid (usmt+unmt) version ( we also compare our results with a supervised nmt system trained on wp parallel sentences extracted by laser 16 @xcite (en-{af ,my}) in a preprocessing data extraction step with the recommended extraction threshold of 1.04. we use the preextracted and similarity-ranked wikimatrix @xcite corpus, which uses laser to extract parallel sentences, for en-{ne,sw}. laser is not trained on kn and yo, thus these languages are not included in the analysis. for en-af , our model and the supervised model trained on laser extractions perform equally well. in all other cases, our model statistically significantly outperforms the supervised laser model, which is surprising, given the fact that the underlying laser model was trained on parallel data in a highly multilingual setup (93 languages), while our mdae setup does not use any parallel data and was trained on the monolingual data of much fewer language directions (7 languages) only. this again underlines the effectiveness of joining ssnmt with bt, multilingual pre-training and bilingual finetuning. for reference, we also report the top-scoring system (tss) per language direction based on top results reported on the relevant test sets together with the amount of parallel training data available to tss systems. in case of language pairs whose test set is part of ongoing shared tasks (en-{kn,my}), we report the most recent results reported on the shared task web-pages (section 4). the amount of parallel data available for these tss varies greatly across languages, from 37 k (en-af ) to 995 k (often noisy) sentences. in general, tss systems perform much better than any of the ssnmt configurations or unsupervised models. this is natural, as tss systems are mostly supervised @xcite , semi-supervised @xcite or multilingual models with parallel pivot language pairs @xcite , none of which is used in the umt and ssnmt models. for en2af only, our best configuration and the supervised nmt model trained on laser extractions outperform the current tss, with a gain in bleu of +16.9 (b+bt), which may be due to the small amount of parallel data the tss was trained on (37 k parallel sentences). ## discussion and conclusion across all tested low-resource language pairs, joining ssnmt-style online sentence pair extraction with umt-style online back-translation significantly outperforms the ssnmt baseline and unsupervised mt models, indicating that the small amount of available supervisory signals in the data is exploited more efficiently. our models also outperform supervised nmt systems trained on laser extractions, which is remarkable given that our systems are trained on non-parallel data only, while laser has been trained on massive amounts of parallel data. while ssnmt with data augmentation and mdae pre-training is able to learn mt even on a low-resource distant language pair such as en-kn, it can fail when a language does not have any relation to other languages included in the multilingual pre-training, which was the case for my in our setup. this can be overcome by being conscientious of the importance of language distance and including related languages during mdae pre-training and ssnmt training. we make our code and data publicly available. 17