NRC Systems for Low Resource G erman- U pper S orbian Machine Translation 2020: Transfer Learning with Lexical Modifications (2020)

Abstract: We describe the National Research Council of Canada (NRC) neural machine translation systems for the German-Upper Sorbian supervised track of the 2020 shared task on Unsupervised MT and Very Low Resource Supervised MT. Our models are ensembles of Transformer models, built using combinations of BPE-dropout, lexical modifications, and backtranslation.

## introduction we describe the national research council of canada (nrc) neural machine translation systems for the shared task on unsupervised mt and very low resource supervised mt. we participated in the supervised track of the low resource task, building upper sorbian-german neural machine translation (nmt) systems in both translation directions. upper sorbian is a minority language spoken in germany. we built baseline systems (standard transformer @xcite with a byte-pair encoding vocabulary (bpe; @xcite ) trained on all available parallel data (60,000 lines), which resulted in unusually high bleu scores for a language pair with such limited data. in order to improve upon this baseline, we used transfer learning with modifications to the training lexicon. we did this in two ways: by experimenting with the application of bpe-dropout @xcite to the transfer learning setting (section 2.3), and by modifying czech data used for training parent systems with word and character replacements in order to make it more "upper sorbian-like" (section 2.4). our final systems were ensembles of systems built using transfer learning and these two approaches to lexicon modification, along with iterative backtranslation. ## data we used all provided parallel german-upper sorbian data and all monolingual upper sorbian data (after filtering), along with german-czech parallel data from open subtitles (lison and tiedemann, 2016), foot_0 dgt @xcite , jw300 @xcite , europarl v10 @xcite , news-commentary v15, and wmt-news foot_1 for building the bpe vocabularies. the monolingual upper sorbian web and witaj datasets foot_2 were filtered to remove lines containing characters that had not been observed in the upper sorbian parallel data or in the czech data; this removed sentences that contained text in other scripts and other languages. the czech-german data was used for training parent models, while monolingual german and upper sorbian were used (along with parallel german-upper sorbian data) for training child models. a table of data sizes and how they were used is shown in appendix a. ## preprocessing we build bpe vocabularies of size 2k, 5k, 10k, 15k, and 20k using subword-nmt foot_3 @xcite . after building the vocabulary, we add a set of 25 generic tags, plus a special backtranslation tag "<@xmath0, which we use in future experiments for indicating when training data has been backtranslated @xcite . we also add all moses and sockeye special tags (ampersand, <@xmath1.) to a glossary file used for applying bpe, which prevents them from being segmented. because there is so much more czech data than upper sorbian data, we duplicate the in-domain parallel hsb-de data and the monolingual hsb data 25 times when training bpe in order to make sure that hsb data is adequately represented (and not overwhelmed by czech data) in training the encoding. after training bpe, we extract (and fix for the remainder of our experiments) a single de vocabulary and a single hsb-cs vocabulary, covering all the relevant data used to train bpe for that language pair. we ran bpe-dropout with a rate of 0.1 over the training data 5 times using the same bpe merge operations, vocabularies and glossaries as before, concatenating these variants to form an extended training set. ## software and systems we used sockeye's @xcite implementation of transformer @xcite with 6 layers, 8 attention heads, network size of 512 units, and feedforward size of 2048 units. we have changed the default gradient clipping type to absolute, used the whole validation set during validation, an initial learning rate of 0.0001, batches of ∼8192 tokens/words, maximum sentence length of 200 tokens, optimizing for bleu. parent systems used checkpoint intervals of 2500 and 4000. child system checkpoint intervals varied from 65 to 4000 to balance frequent checkpointing with efficiency. decoding was performed with beam size 5. our results for this are somewhat mixed, owing in part to the relatively small bleu gains produced by bpe-dropout (as compared to backtranslation). in as a result of these experiments, many of the systems we used in our final ensembles were trained with source-side bpe-dropout, though when it appeared promising we also ensembled with systems without bpe-dropout. ## conclusions in this work, we demonstrated that transfer learning, bpe-dropout, and backtranslation all provide improvements for this low-resource setting. our experiments on lexical modifications, building pseudo-sorbian text for training parent models, performed approximately on-par with standard transfer learning approaches, and could be trivially combined with bpe-dropout. while the lexical modification approach did not outperform the standard transfer learning setup, we found that it still improved ensembles, possibly due to the increase in system diversity.