CUNI Systems for the Unsupervised and Very Low Resource Translation Task in WMT 20 (2020)

Abstract: This paper presents a description of CUNI systems submitted to the WMT20 task on unsupervised and very low-resource supervised machine translation between German and Upper Sorbian. We experimented with training on synthetic data and pre-training on a related language pair. In the fully unsupervised scenario, we achieved 25.5 and 23.7 BLEU translating from and into Upper Sorbian, respectively. Our low-resource systems relied on transfer learning from German-Czech parallel data and achieved 57.4 BLEU and 56.1 BLEU, which is an improvement of 10 BLEU points over the baseline trained only on the available small German-Upper Sorbian parallel corpus.

## introduction an extensive area of the machine translation (mt) research focuses on training translation systems without large parallel data resources @xcite @xcite . the wmt20 translation competition presents a separate task on unsupervised and very low-resource supervised mt. the organizers prepared a shared task to explore machine translation on a real-life example of a lowresource language pair of german (de) and @xcite . there are around 60k authentic parallel sentences available for this language pair which is not sufficient to train a high-quality mt system in a standard supervised way, and calls for unsupervised pre-training @xcite , data augmentation by synthetically produced sentences @xcite or transfer learning from different language pairs @xcite . the wmt20 shared task is divided into two tracks. in the unsupervised track, the participants are only allowed to use monolingual german and upper sorbian corpora to train their systems; the low-resource track permits the usage of auxiliary parallel corpora in other languages as well as a small parallel corpus in german-upper sorbian. we participate in both tracks in both translation directions. section 2 describes our participation in the unsupervised track and section 3 describes our systems from the low-resource track. section 4 introduces transfer learning via czech (cs) into our low-resource system. we conclude the paper in section 5. ## unsupervised mt unsupervised machine translation is the task of learning to translate without any parallel data resources at training time. both neural and phrasebased systems were proposed to solve the task @xcite . in this work, we train several neural systems and compare the effects of different training approaches. ## very low-resource supervised mt with transfer learning one of the main approaches to improving performance under low-resource conditions is transferring knowledge from different high-resource language pairs @xcite . this section describes the unmodified strategy for transfer learning as presented by @xcite , using german-czech as the parent language pair. since we do not modify the approach nor tune hyperparameters of the nmt model, we consider our system a transfer learning baseline for low-resource supervised machine translation. ## conclusion we participated in the unsupervised and lowresource supervised translation task of wmt20. in the fully unsupervised scenario, the best scores of 25.5 (@xmath0.7 (@xmath1. we submitted this system under the name cuni-synthetic-ii. in the low-resource supervised scenario, the best scores of 57.4 (@xmath2.1 (@xmath3. we submitted this system under the name cuni-transfer. we showed that transfer learning plays an important role in the low-resource scenario, bringing an improvement of ∼10 bleu points over a vanilla supervised mt model trained on the small parallel data only. additional techniques used by other competing teams yield further improvements of up to 4 bleu over our transfer learning baseline.