T he LMU M unich S ystem for the WMT 2020 U nsupervised M achine T ranslation S hared T ask (2020)

Abstract: This paper describes the submission of LMU Munich to the WMT 2020 unsupervised shared task, in two language directions, German↔Upper Sorbian. Our core unsupervised neural machine translation (UNMT) system follows the strategy of Chronopoulou et al. (2020), using a monolingual pretrained language generation model (on German) and fine-tuning it on both German and Upper Sorbian, before initializing a UNMT model, which is trained with online backtranslation. Pseudo-parallel data obtained from an unsupervised statistical machine translation (USMT) system is used to fine-tune the UNMT model. We also apply BPE-Dropout to the low resource (Upper Sorbian) data to obtain a more robust system. We additionally experiment with residual adapters and find them useful in the Upper Sorbian→German direction. We explore sampling during backtranslation and curriculum learning to use SMT translations in a more principled way. Finally, we ensemble our best-performing systems and reach a BLEU score of 32.4 on German→Upper Sorbian and 35.2 on Upper Sorbian→German.

## introduction neural machine translation achieves remarkable results @xcite when large parallel training corpora are available. however, such corpora are only available for a limited number of languages. unmt addresses this issue by using monolingual data only @xcite . the performance of unmt models is further improved using transfer learning from a pretrained cross-lingual model @xcite . however, pretraining also demands large monolingual corpora for both languages. without abundant data, unmt methods are often ineffective @xcite . therefore, effectively trans-lating between a high-resource and a low-resource language, in terms of monolingual data, which is the target of this year's unsupervised shared task, is challenging. we participate in the wmt 2020 unsupervised machine translation shared task. the task includes two directions: @xmath0. our systems are constrained, using only the provided hsb monolingual data and de newscrawl monolingual data released for wmt. we pretrain a monolingual encoder-decoder model on a language generation task with the masked sequence to sequence model (mass) @xcite and fine-tune it on both languages of interest, following @xcite . we then train it on unmt, using online backtranslation. we use our usmt system to backtranslate monolingual data in both languages. this pseudo-parallel corpus serves to fine-tune our unmt model. iterative offline backtranslation is later leveraged, yielding a performance boost. we use bpe-dropout @xcite as a data augmentation technique, sampling instead of greedy decoding in online backtranslation, and curriculum learning to best include the smt pseudo-parallel data. we also use residual adapters @xcite to translate to the low-resource language (hsb). ## model description figure 1 presents all the different components of our system and how they are connected to each other. we train both an unsupervised smt (#1) and nmt (#2) model. the unmt model is based on a pretrained mass model (#0), which is monolingual (de). the model is later fine-tuned on both hsb and de. we additionally explore fine-tuning only on hsb using adapters. these models are used to initialize an nmt model (#2, #4) which is trained with online backtranslation. we additionally experiment with sampling (#3) during backtranslation. the usmt model is used to backtranslate hsb and de data. this synthetic bi-text is used to fine-tune the baseline unmt model (#5). we use the synthetic bi-text also to fine-tune directly the adapteraugmented mass model, while employing online backtranslation and sampling (#8). we experiment with curriculum learning (#6) to estimate the optimal way to feed the model this pseudo-parallel data. we also use our unmt model to generate backtranslations and fine-tune existing models (#7). further usmt-backtranslated data is used in #9. finally, some models are fine-tuned with monolingual data which is oversampled and segmented 3 https://github.com/microsoft/mass with bpe-dropout (#10, #11). the details of these components are outlined in the following. ## results the results of our systems on the test set provided during development are presented in after these initial experiments, we use the usmt model (#1) to backtranslate all hsb monolingual data and 7m de sentences. this pseudo-parallel data is leveraged to fine-tune our unmt models alongside online backtranslation. this approach, denoted as model #5, improves the unmt baseline (#3) by more than 5.5 bleu for @xmath2.5 bleu for @xmath3. the curriculum learning approach (#6) yields a small improvement of 0.6 bleu for @xmath4. unfortunately, the curriculum learning model ran without the use of sampling. we later train the model with sampling (#6*) and obtain slight improvements in both directions. using nmt backtranslations in an offline manner (#7) provides for a large improvement in the @xmath5, obtaining 33.2 bleu. further training our high scoring model #7 on usmt backtranslations, depicted as model #9, degrades performance on @xmath6. this might indicate that usmt backtranslations alone are not very important for high performance, but simply adding any kind of pseudo-parallel data during training. the adapter-augmented model with usmt backtranslations (#8) manages to close the gap to the baseline model. comparing #5 and #8, we can see that the model with adapters is worse by 0.9 bleu on @xmath7, but better by 0.4 on @xmath8. due to time constraints, we train #4 and #8 in parallel and #8 is not fine-tuned from #4. overall, adapters are a promising research direction as they lead to faster mass fine-tuning and comparable performance. we observe considerable improvements using bpe-dropout. as noted before, we oversample the parallel and hsb monolingual data and apply bpe-dropout only on hsb. we use this data to fine-tune some of our already trained models, specifically #5 and #7 which results in models #10 and #11, respectively. this approach improves the @xmath9.5 bleu and up to 1.0 bleu for @xmath10. system #11 proved to be our best single system in both translation directions. we hypothesize that using bpe-dropout while simultaneously oversampling the data provides for a data augmentation effect. in future work, it would be interesting to decouple these two steps and measure their effect separately. ensembling further boosts performance. ensemble #12 is used for @xmath11. we note that while computing ensemble bleu scores during development, we did not fix the issue with german-style quotes. this resulted in ensemble #13 obtaining better scores on @xmath12. we later fix the quotes issue and find out that ensemble #12 is better on both translation directions and is the best system overall. ## conclusion in this paper, we present the lmu munich system for the wmt 2020 unsupervised shared task for translation between german and upper sorbian. our system is a combination of an smt and an nmt model trained in an unsupervised way. the unmt model is trained by fine-tuning a mass model, according to the recently proposed re-lm approach. the experiments show that the mass fine-tuning technique is efficient even if little monolingual data is available for one language and results in a strong unmt model. we also show that using pseudoparallel data from usmt and unmt backtranslations improves performance considerably. furthermore, we show that oversampling the low-resource upper sorbian and applying bpe-dropout, which can effectively be seen as data augmentation, results in further improvements. adapters in mass fine-tuning provided for a balance between performance and computational efficiency. finally, smaller but noticeable gains are obtained from us-ing curriculum learning and sampling during decoding in backtranslation.