U d S - DFKI @ WMT 20: Unsupervised MT and Very Low Resource Supervised MT for G erman- U pper S orbian (2020)

Abstract: This paper describes the UdS-DFKI submission to the shared task for unsupervised machine translation (MT) and very low-resource supervised MT between German (de) and Upper Sorbian (hsb) at the Fifth Conference of Machine Translation (WMT20). We submit systems for both the supervised and unsupervised tracks. Apart from various experimental approaches like bitext mining, model pre-training, and iterative back-translation, we employ a factored machine translation approach on a small BPE vocabulary.

## introduction this paper describes the uds-dfki submission to the unsupervised and very low resource supervised tasks of wmt20 for german to upper sorbian (@xmath0. our submitted systems are constrained for the very low resource supervised and unconstrained for the unsupervised task, in that we use wikipedia dumps as additional data. current machine translation systems that deal with low-resource languages are based on unsupervised neural machine translation, semi-supervised methods and pre-trained models leveraging monolingual data @xcite , and multilingual systems among others. in this work, we explore different systems which include baseline nmt, factored nmt @xcite , iterative backtranslation, self-supervised nmt (ssnmt) @xcite and pretraining with xlm @xcite using transformer-base models @xcite for the training of the systems. this paper begins by presenting the data we used for the tasks and the preprocessing pipeline (section 2). this is followed by an overview of the training setup (section 3) and the methods we applied 2 data unsupervised task for sorbian, we use the given data from the sorbian institute (ins hsb ), from witaj sprachzentrum (witaj hsb ), and web-scraped data (web hsb ). supervised task apart from the provided parallel data, we use high-quality europarl (ep, @xcite ) and medium-quality jw300 @xcite corpora for de-cs-hsb. for parallel text mining with laser @xcite , we use the combination of all the monolingual corpora of german and upper sorbian from the unsupervised section of preprocessing our preprocessing steps include normalization, tokenization, deduplication, and truecasing. we attach feature labels related to the source language (<@xmath1, target language (<@xmath2, and the data quality (<@xmath3, for every individual sentence. the quality of a sentence depends on the corpus it is from and the quality tags @xmath4, <medium>, @xmath5: e.g. ins hsb is high quality sorbian. a typical sentence from the corpus after our preprocessing pipeline has the following format: <@xmath6.2), we proceed to apply joint byte-pair encoding (bpe) @xcite on the corpora to finally get our preprocessed data which we use for training all our nmt models. unless specified otherwise, we use a default of 5k merge operations. ## results tables 7 (submitted systems) and 8 (unfactored baseline systems) show a summary of all bleu scores. ## conclusion and future work this paper describes the uds-dfki submission to the shared task of unsupervised and very low resource supervised machine translation between german and upper sorbian at wmt20. for all our systems, we have used the standard transformerbase architecture. we have extracted parallel data from wikipedia dumps using ssnmt @xcite , followed by iterative back-translation for the unsupervised task. for the supervised track, we have tried to factor morphological information into our data to improve our results further. for the constrained supervised task, we achieve 40.9 bleu for de(fac.)→hsb. we obtain bleu scores of 10.3, 8.9, and 9.0 for the @xmath7, @xmath8, and hsb(fac.)→de translation directions respectively in the unsupervised track. as discussed in section 6, one approach for future work is to combine xlm pre-training along with ssnmt directly to improve system initialization. it would be interesting to explore linguistic and syntactic information from other closelyrelated languages to further enhance the performance of the multilingual models.