R eusing a P retrained L anguage M odel on L anguages with L imited C orpora for U nsupervised NMT (2020)

Abstract: Using a language model (LM) pretrained on two languages with large monolingual data in order to initialize an unsupervised neural machine translation (UNMT) system yields state-of-the-art results. When limited data is available for one language, however, this method leads to poor translations. We present an effective approach that reuses an LM that is pretrained only on the high-resource language. The monolingual LM is fine-tuned on both languages and is then used to initialize a UNMT model. To reuse the pretrained LM, we have to modify its predefined vocabulary, to account for the new language. We therefore propose a novel vocabulary extension method. Our approach, RE-LM, outperforms a competitive cross-lingual pretraining model (XLM) in English-Macedonian (En-Mk) and English-Albanian (En-Sq), yielding more than +8.3 BLEU points for all four translation directions.

## introduction neural machine translation (nmt) has recently achieved remarkable results @xcite , based on the exploitation of large parallel training corpora. such corpora are only available for a limited number of languages. unmt has attempted to address this limitation by training nmt systems using monolingual data only @xcite . top performance is achieved using a bilingual masked language model @xcite to initialize a unmt encoder-decoder system @xcite . the model is then trained using denoising auto-encoding @xcite and back-translation @xcite . the approach was mainly evaluated by translating between high-resource languages. translating between a high-resource and a lowresource language is a more challenging task. in this setting, the unmt model can be initialized with a pretrained cross-lingual lm. however, training this unmt model has been shown to be ineffective when the two languages are not related @xcite . moreover, in order to use a pretrained cross-lingual lm to initialize a unmt model, the two models must have a shared vocabulary. thus, a bilingual lm needs to be trained from scratch for each language pair, before being transferred to the unmt model (e.g. en-de lm for en-de unmt). motivated by these issues, we focus on the question: how can we accurately and efficiently translate between a high-monolingual-resource (hmr) and a low-monolingual-resource (lmr) language? to address this question, we adapt a monolingual lm, pretrained on an hmr language to an lmr language, in order to initialize a unmt system. we make the following contributions: (1) we propose reused-lm foot_0 (re-lm), an effective transfer learning method for unmt. our method reuses a pretrained lm on an hmr language, by finetuning it on both lmr and hmr languages. the fine-tuned lm is used to initialize a unmt system that translates the lmr to the hmr language (and vice versa). (2) we introduce a novel vocabulary extension method, which allows fine-tuning a pretrained lm to an unseen language. (3) we show that re-lm outperforms a competitive transfer learning method (xlm) for unmt on three language pairs: english-german (en-de) on a synthetic setup, en-mk and en-sq. (4) we show that re-lm is effective in low-resource supervised nmt. (5) we conduct an analysis of fine-tuning schemes for re-lm and find that including adapters @xcite in the training procedure yields almost the same unmt results as re-lm at a lower computational price. we also run experiments to identify the contribution of the vocabulary extension method. 2 related work transfer learning for unmt. the field of unmt has recently experienced tremendous progress. artetxe et al. (2018); lample et al. ( 2018 ) train unmt models with monolingual data only, using denoising auto-encoding @xcite and online back-translation @xcite as training objectives. this approach is successful for languages with high-quality, large, comparable data. when these conditions are not met, though, unmt provides near-zero scores @xcite . unmt is further improved when initialized with a cross-lingual pretrained model, trained on large corpora @xcite . however, many languages have only limited monolingual data available, a setting where unmt is not effective @xcite . @xcite , whose work is close to our work in motivation, train a unmt model for an hmr-lmr language pair. iteratively, every subset (e.g. 10%) of hmr and all lmr data is backtranslated and the pseudo-parallel corpus is added to the training process. just like xlm, this training procedure needs to run from scratch for every new language pair. by contrast, our method fine-tunes a monolingual pretrained lm for unmt, so it is computationally faster and simpler. vocabulary. transferring a pretrained model (source) to a new model (target) requires the use of a shared vocabulary (nguyen and chiang, 2017). @xcite propose a linear alignment of the source and target model embeddings using an unsupervised dictionary. however, when the embeddings of the two models do not have enough overlapping strings, dictionary induction might fail @xcite . @xcite transfer a source nmt model to a target nmt model (e.g. de-en to nl-en). to enable transfer, they overwrite the source vocabulary with the target vocabulary. by contrast, we keep the union of the two vocabularies. we fine-tune a pretrained monolingual lm to an lmr language, to initialize an nmt model. thus, we need the vocabularies of both languages. adapters. residual adapters @xcite are feed-forward networks, added to each of to the original model's layers. during fine-tuning, the model parameters are frozen and only the adapters are fine-tuned. this can prevent catastrophic forgetting @xcite . adapters show promising results in domain adaptation @xcite and cross-lingual classification @xcite . motivated by this, we study the use of adapters during lm fine-tuning in our analysis. ## proposed approach we describe our method for translation between a high-resource (hmr) and a low-resource language (lmr) using monolingual data in this section. ## experimental setup datasets. we experiment with two setups. in the first synthetic setup we use en-de. we sample 8m en sentences from newscrawl. to simulate an lmr language, we gradually sample 0.05m, 0.5m and 1m de sentences. we use the wmt dev/test sets @xcite . the second, real-world setup is en-mk, en-sq. we use 68m en sentences from newscrawl. for mk and sq, we use 2.4m mk and 4m sq, obtained from oscar foot_1 (ortiz suárez et al., 2019) and wikipedia. we randomly select 3k sentences from setimes foot_2 as dev and 3k as test set. we tokenize data with standard moses @xcite scripts. for the low-resource supervised case, we sample 10k, 100k, and 200k parallel sentences from setimes for mk and sq. preprocessing. we train a standard xlm model @xcite as a baseline using 32k bpe merge operations, learned on the concatenation of sentences sampled randomly from the corpora of each language pair with α = 0.5. for re-lm, we learn 32k bpes on the hmr corpus and extract the initial vocabulary (v hmr ). then, we learn 32k bpes on the joint lmr and hmr corpus (bpe joint ). we extend the initial v hmr vocabulary by the amount of lmr vocabulary items that are not already present in v hmr . to identify whether a smaller number of bpe merges would be useful for splitting the lmr language, we conduct experiments varying their number in the analysis. ## conclusions training competitive unsupervised nmt models for hmr-lmr scenarios is important for many real low-resource languages. we proposed re-lm, a novel approach that fine-tunes a high-resource lm on a low-resource language and initializes an nmt model. re-lm outperformed a strong baseline in unmt, while also improving translations on a lowresource supervised setup. in future work, we will apply our method to languages with corpora from diverse domains and also to other languages. each lm using as criterion the validation perplexity on the lmr language, with a patience of 10. the training details of the two pretraining methods are presented here: our approach also requires an lm fine-tuning step. the runtimes, parameters and gpu details are shown in nmt configuration and training details. the parameters and runtimes of the unmt models we used are shown in