The LMU M unich Systems for the WMT 21 Unsupervised and Very Low-Resource Translation Task (2021)

Abstract: We present our submissions to the WMT21 shared task in Unsupervised and Very Low Resource machine translation between German and Upper Sorbian, German and Lower Sorbian, and Russian and Chuvash. Our low-resource systems (German↔Upper Sorbian, Russian↔Chuvash) are pre-trained on high-resource pairs of related languages. We fine-tune those systems using the available authentic parallel data and improve by iterated back-translation. The unsupervised German↔Lower Sorbian system is initialized by the best Upper Sorbian system and improved by iterated back-translation using monolingual data only.

## introduction in this paper, we describe systems for translation between german (de) and upper sorbian (hsb), german (de) and lower sorbian (dsb), and russian (ru) and chuvash (cv) developed at lmu munich for the wmt21 shared task on unsupervised and very low resource machine translation (mt). upper sorbian is a minority language spoken by around 30,000 people in today's german federal state of saxony, lower sorbian has around 7,000 speakers and is spoken in the german federal state of brandenburg. with such a small number of speakers, machine translation and automatic processing of sorbian language is an inherently lowresource problem without any chance that the resources available for sorbian would ever approach the size of resources for languages spoken by millions of people. on the other hand, being western slavic languages related to czech and polish, it is possible to take advantage of relatively rich resources collected for these two languages. unlike our last year's submission for upper sorbian @xcite , we decided not to use synthetic data from unsupervised translation between czech and upper sorbian and only did iterative back-translation. despite having more authentic parallel data than last year, our system reaches approximately the same translation quality. our upper sorbian systems ranked third out of six systems in the official ranking. we leverage the relatedness between the sorbian languages and use the upper sorbian system as a starting point for iterative back-translation using monolingual data only. our lower sorbian systems ranked second (@xmath0. chuvash is a minority language spoken in the volga region in the southwest of russia. although it uses the cyrillic script, it is not related to eastern slavic languages, but it is a turkic language, relatively isolated in the turkic language family. as a language with the highest number of speakers in this shared task, it also has the highest amount of available parallel data. we adopt a similar approach as for german-upper sorbian translation and pretrain our models on the related kazakh language. in addition, we experiment with character-level models in the hope that they will be particularly effective for agglutinative morphology. ## experimental setup most of our experimental setup is shared across all the language pairs. all our models use the transformer architecture @xcite as implemented in fairseq @xcite . all data is segmented using bpe @xcite with 16k merge operations as implemented in youtokentome foot_0 without previous explicit tokenization. the merges are computed using a concatenation of all training data: german, czech, upper and lower sorbian in the first set of experiments, russian, kazakh, and chuvash in the second set of experiments. for the supervised task, we first pre-train models on high-resource related languages: russian-kazakh for chuvash and german-czech for upper sorbian. we first train transformer base models on authentic data. these systems are used to generate back-translation @xcite ) of monolingual data. using tagged back-translation @xcite , we trained transformer big models for german↔czech and russian↔kazakh translation. all back-translation steps use sampling and simple length-based filtering as proposed by @xcite foot_1 . we upsample the authentic parallel data to match the size of the synthetic data. we keep most default hyperparameters from the predefined architectures in fairseq (transformer for the base model, transformer_wmt_en_de_big_t2t for the big model. the batch size is 6k tokens for the base models, 2k tokens for big models on a single gpu, because we always start with high-resource training, we keep the dropout on the standard value of 0.1. we use these models to initialize the weights @xcite of the supervised low-resource models without restarting the optimizer. because the learning rate is already low at that stage of training, we do not need to change the dropout to prevent overfitting. first, we train the supervised models using the authentic parallel data only, then we continue with iterated back-translation. the best upper sorbian-to-german model is used to translate lower sorbian monolingual data into german. in the next steps, we continue with a standard iterative back-translation procedure for unsupervised neural machine translation @xcite . our final submission is an ensemble (with the vote strategy) of the best-scoring systems in the process of iterated back-translation. language-pairspecific descriptions and results are discussed in the following sections. we evaluate our systems using the bleu score @xcite , chrf score @xcite as implemented in sacrebleu @xcite . 3 further, we evaluate the models using bertscore @xcite foot_3 with xlm-roberta large @xcite ) as an underlying model for german and russian and mbert @xcite for chuvash. similar to the official task evaluation, we also report for each system the number of significantly worse systems in each metric at the significance level 0.95 with bootstrap resampling @xcite with 1k samples. for each metric, each system receives one point for each system it significantly outperforms in the metric at the significance level of 0.95. ## conclusions we presented our systems for low-resourced translation between german and upper sorbian, unsu-pervised translation between german and lower sorbian, and translation between chuvash and russian. our systems used standard state-of-the-art techniques for low-resource and unsupervised machine translation but did not exhaust all available methods. better results could be achieved using more monolingual data and by more careful filtering of the synthetic parallel data.