Unsupervised Translation of G erman– L ower S orbian: Exploring Training and Novel Transfer Methods on a Low-Resource Language (2021)

Abstract: This paper describes the methods behind the systems submitted by the University of Groningen for the WMT 2021 Unsupervised Machine Translation task for German–Lower Sorbian (DE–DSB): a high-resource language to a low-resource one. Our system uses a transformer encoder-decoder architecture in which we make three changes to the standard training procedure. First, our training focuses on two languages at a time, contrasting with a wealth of research on multilingual systems. Second, we introduce a novel method for initializing the vocabulary of an unseen language, achieving improvements of 3.2 BLEU for DE->DSB and 4.0 BLEU for DSB->DE.Lastly, we experiment with the order in which offline and online back-translation are used to train an unsupervised system, finding that using online back-translation first works better for DE->DSB by 2.76 BLEU. Our submissions ranked first (tied with another team) for DSB->DE and third for DE->DSB.

## introduction unsupervised neural machine translation (unmt) has become increasingly useful in the field of mt, given that monolingual data is easier to gather compared to bilingual (or parallel) data. such is especially the case for low-resource languages, which constitute the majority of languages in the world. the wmt 2021 unsupervised mt task focuses on one such low-resource language: lower sorbian (dsb). the task is to translate between german (de), a high-resource language, and lower sorbian, which is a very low resource language with roughly 150 thousand sentences of monolingual data available for the task at hand. the unsupervised task from prior years of wmt focused on german-czech and german-upper sorbian translation. unique to this year however is the relatively small amount of monolingual data available for dsb, compared to last year in which roughly 750 thousand sentences of upper sorbian were available. this makes it increasingly difficult to rely on the ubiquitous state-of-the-art unmt methods @xcite @xcite , as they typically rely on a large amount of monolingual data available for both languages. to alleviate the difficulty that comes with the lack of monolingual data for dsb, this year's wmt task allows for the use of monolingual and parallel data outside of de-dsb. specifically, all upper sorbian (hsb) data from wmt20 and all parallel data for german (de) from wmt and opus @xcite are made available to use. additionally, as auxiliary languages related to dsb, monolingual data for czech (cs) and polish (pl) is also provided. given the success of language transfer via multilingual models such as mbart @xcite , this fundamentally changes this year's unsupervised task from a bilingual nmt task to a multilingual task. however, pretrained multilingual models like mbart cannot be used as they do not fit the limitations on the training data that one is allowed to use for this shared task. as the problem is unique to date due to the limited available data as well as the limitation on preexisting pretrained models, we aim to establish a standard for training systems under these restrictions. specifically, we ask three research questions (rqs): concerning rq1, there is a wealth of research into multilingual models, however these typically require a large amount of monolingual data for each language in addition to a wealth of computational resources and time. therefore, we develop the hypothesis that under limited constraints (time and computational resources) training only on two languages at a time will result in better performance due to there being fewer training objectives. specific to this task, we propose training on de coupled with cs, then hsb, then dsb itself, following the order of least to most linguistically similar to the low resource language that we target: dsb. as for rq2, due to the scarcity of monolingual data and complete absence of parallel data for dsb, our ability to train the model on this language is limited, thus the model's initialization for the language plays an increasingly important role in its resulting performance. relying on the similarity of the two sorbian languages, we aim to improve this initialization by transferring the model's knowledge of the hsb vocabulary to the dsb vocabulary. finally, with respect to rq3, garcia et al. (2020) established a method for incorporating offline and online back-translation (bt) into their mt system by first using offline bt then online bt, both following a multilingual pretraining. however, the reverse order (i.e. online bt followed by offline bt) has not been tested to the best of our knowledge, and theoretically doing online bt should improve the quality of the synthetic data that would be used for offline bt. therefore, we test garcia et al.'s method as well as the reverse order to establish the best practice for this task specifically. the remaining of the paper is organized as follows. we first outline the data we chose to use and our preprocessing steps in section 2. we then specify our architecture and training methods in section 3. our results are in section 4, followed by our conclusions in section 5. ## data apart from our main language pair of de and dsb, we opted to use data from two languages that are related to the latter, namely cs and hsb. 1 for hsb and dsb, we use all of the data provided for training by wmt. for de and cs, we use data from wmt newscrawl, years 2010-11 and 2018-20. we chose these years as they are the most frequent years occurring in the hsb data, following @xcite . for de, we take the first 1 million sentences each from 2018-20 and 0.5 million from 2010-11, totalling 4 million. for cs, we take 0.5 million and 0.25 million from the respective years, totalling 2 million. in terms of parallel cs-de data, we use multi-paracrawl, europarl v8, wmt news v2019, and news commentary v16, all available from the opus project @xcite . the datasets are shown in for development and testing, we use the de-hsb and de-dsb devel and devel_test datasets provided by wmt. for cs-de, we make use of the wmt news translation task dev set, using newstest2012 for development and newstest2013 for testing. all data is tokenized using the moses toolkit @xcite . we then apply bpe @xcite , for all languages jointly, using fastbpe. 2 the segmentation is applied on the same number of randomly-selected sentences for each language, roughly 145 thousand, matching the number of sentences in the dsb training data. we experiment with the number of joins used (trying {20, 40, 50, 60, 80} thousand), finding 50 thousand to perform the best, according to bleu scores after step 4. ## results ## conclusion the translation of lower sorbian to and from german presents a unique challenge in the field of unsupervised mt, due to the absence of parallel data and the scarcity of monolingual data for training. therefore, the task necessitates an initial pretraining with similar, higher-resource languages. with this assumption, we experimented with various methods of pretraining, positing that training on 2 languages at a time is competitive with training with all languages at once, while allowing for a better initialization of dsb. we also showcase a new method for transferring knowledge to the word embeddings of a transformer, provided a similar language is used in pretraining. we intend to experiment with this method further to gauge its applicability for more distantlyrelated languages. finally, the use of both online and offline back-translation can improve the performance of a model, and if not done in an iterative fashion, the order in which they are performed can greatly affect the results.