The LMU M unich System for the WMT 20 Very Low Resource Supervised MT Task (2020)

Abstract: We present our systems for the WMT20 Very Low Resource MT Task for translation between German and Upper Sorbian. For training our systems, we generate synthetic data by both back- and forward-translation. Additionally, we enrich the training data with German-Czech translated from Czech to Upper Sorbian by an unsupervised statistical MT system incorporating orthographically similar word pairs and transliterations of OOV words. Our best translation system between German and Sorbian is based on transfer learning from a Czech-German system and scores 12 to 13 BLEU higher than a baseline system built using the available parallel data only.

## introduction in this paper, we describe systems for translation between german and upper sorbian developed at lmu munich for the wmt20 shared task on very low-resource supervised mt. upper sorbian is a minority language spoken by around 30,000 people in today's german state of saxony. with such a small number of speakers, machine translation and automatic processing of sorbian is an inherently low-resource problem without any chance that the resources available for sorbian would ever approach the size of resources for languages spoken by millions of people. on the other hand, being a western slavic language related to czech and polish, it is possible to take advantage of relatively rich resources collected for these two languages. the german-sorbian systems presented in this paper are neural machine translation (nmt) systems based on the transformer architecture @xcite . we experiment with various data preparation and augmentation techniques: back-translation @xcite , finetuning systems trained for translation between czech and german @xcite , and data augmentation by including german-czech parallel data with the czech side translated to upper sorbian by an unsupervised system that includes an unsupervised transliteration model for guessing how to translate out-of-vocabulary czech words to upper sorbian. our experiments show the importance of data augmentation via stochastic pre-processing and synthetic data generation. the best systems were trained by transfer-learning from a czech-german system. however, compared to data augmentation, transfer learning from czech-german translation only produces a minor improvement. based on the preliminary shared task results, the presented systems scored on the 4th place among 10 competing teams in the shared task. ## related work until recently, phrase-based approaches were believed to be more suitable for low-resource translation. @xcite claimed that a parallel dataset of at least 10 7 tokens is required for nmt to outperform phrase-based mt. this view was also supported by the results of @xcite and @xcite , who showed that phrase-based approaches work well for unsupervised mt, at least in the early stages of the iterative back-translation procedure. recently, @xcite revisited the claims about data needs of supervised nmt and showed that with recent innovations in neural network and careful hyper-parameter tuning, nmt models outperform their phrase-based counterparts with training data as small as 100k tokens (15 times smaller than the data provided for this shared task). standard techniques for low-resource machine translation include data augmentation with rulebased substitutions @xcite , by sam-data # sent. # tok. ## data we used several types of data to train our systems. the organizers provided authentic parallel data and sorbian monolingual data. we also use german and czech news crawl data and czech-german parallel data available in opus @xcite . ## synthetic data from czech-german since upper sorbian is related to czech, we generate additional synthetic parallel german-sorbian data by translating the czech side of the german-czech parallel data. for this, we use an unsupervised statistical mt system which includes mined czech-sorbian transliteration word pairs for better performance. ## experimental setup parallel data system monolingual data authentic de↔hsb ft and bt de↔hsb authentic de↔cs ft and bt de↔cs de↔hsb from de↔cs 1 2 transformer base de hsb 3 transformer base de hsb 4 transformer big de hsb unsupervised cs hsb 5 transformer big de hsb transformer base de cs 6 init. by de cs de hsb cs de hsb figure 1: overview of datasets and systems that were used to generate synthetic data. solid arrows denote training a system, dashed gray arrows denote using the model for data generation. synthetic datasets have dashed boxes. transformers @xcite . we followed known best practices for architecture and optimization choices. in our experiments, we mostly focus on data engineering. ## results the quantitative results in terms of bleu score @xcite and chrf @xcite score are presented in the data augmentation using bpe-dropout (model 2) seems to have a substantial effect on the translation quality, improving the translation by 6-7 bleu points. this is a much larger effect than @xcite reported. however, they also observed a larger positive effect on smaller datasets. unlike @xcite , we 3 https://github.com/mjpost/sacrebleu did not find any benefits of using a small bpebased vocabulary or tuning learning rate. however, the positive effect of the small vocabulary might be partially emulated by the bpe dropout. adding the back-and forward-translated data in the training data improved the translation quality only slightly (model 3). a large positive effect can be achieved by switching to the big architecture (model 4). adding the synthetic data generated from czech-german parallel data improve only the sorbian-to-german translation direction (model 5), presumably because the quality of the synthetic sorbian side of the corpus is too low to be used as a target side. transfer learning from german-czech models further improves the translation quality by approximately 1 bleu point. these are thus the best single models we have developed and our contrastive submission to the shared task. additional improvements were reached by model ensembling. ensembling both the model trained from random initialization and transfer learning models improves the translation by approx. 1 bleu point. ensembling these two model types together further improves the translation quality by around half bleu point. the model generating the translation right-toleft reach translation quality that is comparable to the left-to-right models. however, rescoring of the n-best lists generated by left-to-right ensembles by the right-to-left models improves the translation quality only negligibly. the rescored ensemble was our primary submission to the shared task. ## conclusions we presented nmt systems for translation between german and upper sorbian. due to the domain closeness and relative simplicity of the test data, we were able to achieve bleu scores over 50 using the parallel data only. the crucial component was the use of bpe-dropout for both the source and target side. further translation quality improvements were achieved by generating synthetic training data by back-and forward-translation. additionally, we generated synthetic data by machine-translating the czech side of a parallel german-czech corpus. for that, we built an unsupervised smt system that additionally utilizes an unsupervised transliteration system for the translation of oov tokens. our best single system is based on transfer learning, i.e., initializing the model by a czech-german system, reaching 1-2 higher bleu scores compared to systems based on sorbian and german data only. further minor improvements were achieved by model ensembling and right-to-left rescoring.