Multilingual Bidirectional Unsupervised Translation through Multilingual Finetuning and Back-Translation (2023)

Abstract: We propose a two-stage approach for training a single NMT model to translate unseen languages both to and from English. For the first stage, we initialize an encoder-decoder model to pretrained XLM-R and RoBERTa weights, then perform multilingual fine-tuning on parallel data in 40 languages to English. We find this model can generalize to zero-shot translations on unseen languages. For the second stage, we leverage this generalization ability to generate synthetic parallel data from monolingual datasets, then bidirectionally train with successive rounds of back-translation. Our approach, which we EcXTra (uE/unglish-uc/uentric Crosslingual (uX/u) uTra/unsfer), is conceptually simple, only using a standard cross-entropy objective throughout. It is also data-driven, sequentially leveraging auxiliary parallel data and monolingual data. We evaluate unsupervised NMT results for 7 low-resource languages, and find that each round of back-translation training further refines bidirectional performance. Our final single EcXTra-trained model achieves competitive translation performance in all translation directions, notably establishing a new state-of-the-art for English-to-Kazakh (22.9 10.4 BLEU). Our code is available at [this URL]( https://github.com/manestay/EcXTra ).

## introduction current neural machine translation (nmt) systems owe much of their success to efficient training over large corpora of parallel sentences, and consequently tend to struggle in low-resource scenarios and domains @xcite . this has motivated investigation into the field of zero-resource nmt, in which no parallel sentences are available for the source-target language pair. this is especially valuable for lowresource languages, which by nature have little to no parallel data. there are two mainstream lines of inquiry towards developing models to tackle zero-resource machine translation. unsupervised machine translation learns a model from monolingual data from the source and target languages. some research involves introducing new unsupervised pre-training objectives between monolingual datasets @xcite . others devise training schemes with composite loss functions on various objectives @xcite . in contrast, zero-shot machine translation learns a model by training on other datasets @xcite or other language pairs @xcite , then directly employ this model for translating unseen languages. this work leverages both mainstream approaches in zero-resource translation. we propose a conceptually simple, yet effective, two-stage approach for training a single nmt model to translate unseen languages both to and from english. the first stage model is trained on real parallel data from 40 high-resource languages to english. this results in a strong zero-shot model, which we use to translate unseen languages to english. by applying back-translation to flip the order, we obtain english-to-unseen synthetic parallel data. in the second stage, we continue training the model on successive rounds of offline back-translation, where each round uses the prior round for both for weight initialization and for synthetic parallel data. we term our overall unsupervised translation approach ecxtra (english-centric crosslingual (x) transfer). ecxtra can be thought of as a data-driven approach, which sequentially leverages auxiliary parallel data then monolingual data. each stage's model is initialized to an informed pretrained model, before fine-tuning. we initialize the first stage model's encoder and decoder to xlm-roberta @xcite and roberta @xcite respectively, and we initialize the second stage model's weights to those of the first stage. in doing so, ecxtra importantly avoids the complicated training schemes and custom training objectives of prior work. as our approach is simple to train and extend to new unseen languages, we release all code, data and pretrained models. 1 our contributions are: ## our approach our training procedure closely follows the standard machine translation task. machine translation involves developing models to output text in a target language t , given text in a source language s. in a typical supervised mt setting, it is assumed there is a parallel corpus @xmath0, t i )} n @xmath1. a model is then trained on these examples, to minimize the cross-entropy loss given by given enough parallel data, this training framework allows contemporary nmt models to achieve strong performance @xcite . however, in the unsupervised setting arises the fundamental challenge that we no longer have any parallel data between the source and target languages of interest. conceptually, we divide the two stages of our training procedure into four steps: 1a. zero-shot model transfer by initializing to pretrained multilingual lms. we use an xlm-roberta encoder and a roberta decoder. observe that these are are widely-used techniques in the field of machine translation. our main contribution is in presenting an effective synthesis of the techniques to enable a single model to perform zero-shot and bidirectional translation (while using only a standard loss function). terminology it is worthwhile formalizing our exact terminology, given that prior work in this field uses terms rather inconsistently. 2 our setting is english-centric, as the language pairs include english as either the source or target foot_2 our final model is bidirectional, in that it can translate s to t and also translate t to s. we call the non-english side of a pair a foreign language. therefore, we use the terms foreign-english and many-to-english interchangeably (likewise with english-foreign and english-to-any). languages seen during training on parallel datasets are auxiliary languages. ## datasets used here we succinctly describe the data, providing further details in appendix b. training for the zero-shot stage, we use parallel corpora from higher-resource auxiliary languages to english. we utilize a subset of the many-to-english v1 dataset @xcite . we consider only the 40 largest foreign-english pairs, foot_3 and equally sample 2 million examples from each. 5 the resulting dataset, which we term m2e-40, consists of 80 million sentence pairs from 40 source languages. note that unlike most prior work, we have taken an agnostic view towards multilinguality -we do not choose the training languages with reference to the testing languages. for the unsupervised stage, we use monolingual corpora in the 7 test languages (below) from com-moncrawl and cc-100. testing we evaluate our approach on 7 languages: kazakh (kk), gujarati (gu), sinhala (si), nepali (ne), pashto (ps), icelandic (is), and burmese (my). test sets are taken from wmt21, flores-101 and wat21. the languages were chosen for both their diversity and for comparison to prior unsupervised nmt work. validation to validate the zero-shot stage, we select 15 foreign-english parallel datasets from wmt19 development data; these languages are seen during training. in the unsupervised stage we only have access to monolingual data. for validation purposes, we thus reserve a small number of synthetic sentence pairs (250 per direction * 14 directions). ## experimental setup we move from the overall ecxtra approach, to the specifics of using ecxtra to train an nmt model. ## results we evaluate our models on test sets for 7 lowresource-to-english pairs in both translation directions (14 directions total). we use evaluation metrics with are consistent with prior work. by default, we report detokenized sacrebleu @xcite . 7 for the indic languages (gu, si, ne), we report tokenized bleu with the indic-nlp library @xcite . for burmese (my), we report spm-bleu @xcite to handle the language's optional spacing. ## discussion and analysis enabling english-foreign translation in the second stage seems to come at the cost of some foreign-english performance. this may be an instance of the insufficient modeling capacity problem of multilingual nmt models @xcite . still, r 2 improves over r 1 , while training on entirely synthetic parallel data generated from backtranslations in both directions. this finding underscores the effectiveness of successive rounds of back-translation. the ecxtra-trained model r 0 underperforms sixt+ @xcite for foreign-english translations. because ecxtra is a training approach, we can use sixt+ as a drop-in replacement for r 0 for both weight initialization, and for its backtranslations. we suspect that training such a combined model would achieve even better englishforeign performance, and leave this to future work. the ecxtra-trained model r 2 underpeforms garcia et al. (2021) for english-indic translations. this is likely a function of our m2e-40 dataset having a much lower proportion of hindi that the dataset of @xcite . 11 while we take an agnostic 11 this is not explicitly specified in their paper, but is clear view of multilinguality, our training data is by no means writing script-centric; possibly making our model worse at outputting indic texts. the exceeding en-kk and high en-is scores of r 2 provide some evidence for this. overall, the r 2 achieves competitive unsupervised translation results. our model supports 3 additional language pairs over prior bidirectional unsupervised translation models, and the ecxtra approach makes it simple to extend to even more translation pairs. we underscore the overall appeal of our approach, in that we can use the zero-shot model to bootstrap back-translations for any unseen language, and train a bidirectional translation system from there. ## related work the field of low-resource and zero-resource neural machine translation is an area of continued interest. below, we describe related works those which follow our data constraint: parallel foreign-english data in auxiliary languages, and monolingual data in unseen languages. @xcite propose sixt, a fine-tuning method for foreign-english zero-shot nmt. they initialize both the encoder and decoder to xlm-r. they follow a two-stage fine-tuning approach, first only fine-tuning the decoder layers, then continuing training by unfreezing the encoder layers and decoder embeddings. the model is trained on a parallel corpora in only de-en, and they report zero-shot to-english performance for 10 languages. @xcite propose sixt+, which builds upon the authors' prior work, and is trained on a parallel corpus in 6 source languages. this is concurrent to the first submission of our work. they show their model can address zero-shot tasks from nmt to cross-lingual abstractive summarization. this work has the same goal as our first stage of training. 12 the main differences are in our training data (40 vs 6 source languages, 80m vs 120m pairs), and our simpler zero-shot training stage (no unfreezing, no position disentangled encoder). their method jointly optimizes four tasks: denoising autoencoder, adversarial training, high-resource translation, and low-resource backtranslation -the latter two of which we also use. however, their work trains individual models for each direction, and furthermore for each model explicitly trains on related high-resource language datasets. this approach is thus more expensive and less adaptable to new languages as ours. ## conclusion we have described a two-stage training approach for developing a single bidirectional, unsupervised nmt model, which we term ecxtra. the main contribution of ecxtra is in its effective synthesis of techniques from both zero-shot nmt, multilingual fine-tuning, and from unsupervised nmt, back-translation. while prior work also uses similar underlying techniques, they have much more involved training processes, either to consider the bidirectional and zero-shot direction, or introduce additional loss functions (which make training more involved). furthermore, in this work we have taken an agnostic view towards multilinguality. we trained a single nmt model through ecxtra, and find that each round of back-translation training further refines bidirectional translation performance. this gives rise to the view of ecxtra as successive rounds of informed initialization into further fine-tuning. the final, unsupervised ecxtratrained model achieves competitive performance on 7 foreign-english tasks, in both directions. the straightforward nature of ecxtra allows it to be easily extended to new unseen languages. for computational linguistics: acl-ijcnlp 2021, pages 3450-3466, online. association for computational linguistics. thomas wolf, lysandre debut, victor sanh, julien chaumond, clement delangue, anthony moi, pierric cistac, tim rault, rémi louf, morgan funtowicz, et al. 2020. transformers: state-of-the-art natural language processing. in proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pages 38-45. biao zhang, philip williams, ivan titov, and rico sennrich. 2020. improving massively multilingual neural machine translation and zero-shot translation. arxiv preprint arxiv:2004.11867. shiyue zhang, vishrav chaudhary, naman goyal, james cross, guillaume wenzek, mohit bansal, and francisco guzman. 2022. how robust is neural machine translation to language imbalance in multilingual tokenizer training? in proceedings of the 15th biennial conference of the association for machine translation in the americas (volume 1: research track), pages 97-116, orlando, usa. association for machine translation in the americas.