Data Selection for Unsupervised Translation of G erman– U pper S orbian (2020)

Abstract: This paper describes the methods behind the systems submitted by the University of Groningen for the WMT 2020 Unsupervised Machine Translation task for German–Upper Sorbian. We investigate the usefulness of data selection in the unsupervised setting. We find that we can perform data selection using a pretrained model and show that the quality of a set of sentences or documents can have a great impact on the performance of the UNMT system trained on it. Furthermore, we show that document-level data selection should be preferred for training the XLM model when possible. Finally, we show that there is a trade-off between quality and quantity of the data used to train UNMT systems.

## introduction unsupervised neural machine translation (unmt) has recently become the dominant paradigm for unsupervised mt, with the advent of cross-lingual language model pretraining as used in the xlm model (conneau and lample, 2019). however, much of the existing research in unmt assumes that the amount of data available for one language is roughly equivalent to the other. the wmt 2020 unsupervised machine translation task is unique in that monolingual data is abundant for one language (german), with hundreds of millions of sentences available, and sparse for the other (upper sorbian), which only has around 750 thousand sentences available. with a wealth of data available on the german side, it is natural to ask: how can we best use this data? viewing this under the lens of data selection, we break this broad question down into 3 concrete sub-questions, tailored for the unsupervised setting. they are as follows: section 2 describes the general setup pertaining to every experiment, including datasets, data processing steps, model architecture, and training details. in section 3, we detail our individual experiments and their corresponding results. finally, in section 4, we make our conclusions and discuss paths for future work. ## setup for upper sorbian, we use the 3 monolingual datasets provided by the sorbian institute, the witaj sprachzentrum, and the web data from cis, lmu. we also use the upper sorbian side of the parallel corpus from train.hsb-de.hsb.gz. for german, we use monolingual data from news crawl and common crawl. for validation and testing, we use the data provided in devtest.tar.gz. all data is tokenized and truecased using the moses toolkit (koehn et al., 2007). for bpe segmentation (sennrich et al., 2016), we apply a joint segmentation for both languages. this is done by first taking a sample of the german data of the same length as the upper sorbian data (around 750 thousand sentences). the bpe codes are learned and applied using fastbpe. @xcite after bpe is applied, we remove duplicate sentences while retaining the order of the corpora. @xcite we used the xlm model (conneau and lample, 2019) using the default parameters, with the excep-tion of allowing for sentences of max length 200 rather than 100. @xcite the language model pretraining step includes only masked language modelling, and training is limited to 24 hours. the nmt step is also limited to 24 hours, with the additional stopping criterion of no improvement on the @xmath0. @xcite 3 experiments for all of our data selection experiments, we start by training an initial model. our initial model is trained on 10 million german sentences and all of the available upper sorbian sentences. the 10 million german sentences include all of the data from years 2007 and 2010, and the remaining sentences are taken from 2014. @xcite our initial model achieves bleu scores of 17.43 and 19.05 for @xmath1. ## conclusion in the unmt setting where one has access to a wealth of resources for one language, we investigated the feasibility of data selection. we attempt both document-level and sentence-level selection, finding that both methods are capable of distinguishing low quality data from high quality data, with quality in this case defined as the efficacy for training an xlm model. we found that while document-level selection chooses poorer sentences on average, the xlm model can leverage the intersentence information to achieve better results than when simply using the highest quality sentences. we also found that there appears to be a point where adding more monolingual data is not beneficial, but rather potentially harmful, indicating a need for data selection. finally, we noted some potential drawbacks to using this form of data selection, particularly that data cannot be used for both initial training of the nmt model and subsequent selection. future work could continue along many avenues, such as the effectiveness of data selection on other language pairs, or even on the upper sorbian side.