The U niversity of H elsinki and Aalto University submissions to the WMT 2020 news and low-resource translation tasks (2020)

Abstract: This paper describes the joint participation of University of Helsinki and Aalto University to two shared tasks of WMT 2020: the news translation between Inuktitut and English and the low-resource translation between German and Upper Sorbian. For both tasks, our efforts concentrate on efficient use of monolingual and related bilingual corpora with scheduled multi-task learning as well as an optimized subword segmentation with sampling. Our submission obtained the highest score for Upper Sorbian -> German and was ranked second for German -> Upper Sorbian according to BLEU scores. For English–Inuktitut, we reached ranks 8 and 10 out of 11 according to BLEU scores.

## introduction our work is motivated by @xcite , who provide a detailed study of different transfer learning and regularization approaches for lowresource machine translation. they focus on an asymmetric-resource scenario in which the target language is underresourced, but related to a higherresource language that can be used in a multilingual setting. for example, in the english-to-estonian task, estonian is assumed to be a low-resource language (lrl) which is complemented by a second higher-resource target language (hrl), finnish. among the wmt 2020 shared tasks, the @xmath0, with czech being a high-resource language closely related to upper sorbian. we adapt the approach proposed by @xcite also to three slightly different scenarios: in the upper @xmath1, the low-resource language is on the source side, but can be complemented with czech in the same way; for the @xmath2, no related high-resource language is available; and for @xmath3, the low-resource language is on the source side and no high-resource language is available. @xcite recommend the combination of the following techniques to reach optimal translation performance in their examined setup: scheduled multi-task learning the learning process is split in two phases. the first phase only sees data from the source and the hrl, whereas lrl data is only added in the second phase. backtranslation the addition of synthetic data has become a staple of neural machine translation. they recommend marking synthetic data and controlling its weight in the task scheduler. ## data both the inuktitut-english and upper sorbian-german tasks can be qualified as low-resource settings, with less than 800k (deduplicated) parallel training instances for the former and 60k for the latter. for both tasks, we follow the constrained setting, which limits the allowed data to those made available on the wmt website. in this section, we present the parallel and monolingual resources that we used for our systems. ## segmentation models nmt models should ideally be able to represent the entire vocabulary of their source and target languages. the simplest solution however, in which word forms are represented as atomic vocabulary items, leads to sparse statistics, issues with out-ofvocabulary words, and heavy computational costs due to large vocabularies. moreover, such wordlevel modeling does not allow the productive recombination of morphemes and is thus unsuitable for morphologically rich languages such as inuktitut or sorbian. in recent years, a consensus has emerged that nmt vocabularies should consist of subwords of variable size. various unsupervised word segmentation algorithms have been proposed, among which byte-pair encoding (bpe) @xcite , sentencepiece @xcite , and several variants of morfessor @xcite @xcite . besides the actual word segmentation algorithm, various parameters influence the quality of the resulting translation system: we carried out some additional experiments with the english-inuktitut task, which differs from their setup in the sense that the languages use different scripts and there is no third language involved. for german-upper sorbian, the setup differs from @xcite with respect to the amount of available training data. we therefore ran additional experiments to measure the impact of both the training data used for the segmentation model and the training data used for the translation model. when training both the segmentation model and the translation model on the provided parallel data (experiments 1 and 2), sentencepiece performs much better than morfessor em+prune. the addition of monolingual training data for the segmentation model (experiments 3 and 4) helps both segmentation algorithms about equally well (+ 1-2 bleu). in contrast, the further addition of czech data for the segmentation model (experiments 5 and 6) benefits morfessor more than sentence-piece on average. 5 finally, augmenting the trans- @xmath5. tasks nh dev newsdev nh dev newsdev 1 en↔iu + bt --24.13 15.72 41.07 32.86 2 en↔iu + bt noise + translit. *25.15 *15.95 41.89 33.47 3 en↔iu + bt noise + taboo 25.28 16.15 *41.67 *33.49 ## translation models all our models are based on the transformer architecture and use, by and large, the same hyperparameters as @xcite . the transformer contains 8 encoder and 8 decoder layers with 16 attention heads each. the hidden layer size is 1024, the filter size 4096. the minibatch varies between 7200 and 9200 tokens, depending on the task, and gradients are accumulated over 4 minibatches. all models were trained for 200 000 steps, which corresponded to 5-7 days training time on a single v100 gpu. the best savepoint was selected on the basis of development set accuracy; this measure turned out to be more stable than development set bleu score. we use the dynamicdata branch of the opennmt-py toolkit @xcite for our experiments. 6 this branch provides the neces-witaj and sorbian institute corpora. we added an equivalent amount of german data from newscommentary and wmt-news. the czech data also stems from newscommentary and wmt-news and is complemented by a subset of czech europarl. the experiments presented in tables 3 and 4 already confirmed the positive impact of subword regularization and backtranslation. row 1 of tables 5 and 6 provide baseline results with these two techniques. backtranslated training instances are marked with a special token. scheduled multi-task learning as row 2 in a training schedule with two phases (row 3 in ## submissions and results for the best-performing configurations, we trained two models each, one ("basic") with the hyperparameters listed above, and an alternative one with relative position distance clipping at 4 (see @xcite . however, this setting did not yield any consistent accuracy gains or losses. for the inuktitut task, we submitted single systems of settings 2 and 3 for both directions. for @xmath6, the alternative model of setting 2 obtained the best scores on the test set (10.1 bleu / 0.301 chrf), whereas for @xmath7, the basic model of setting 3 obtained the best scores on the test set (23.0 bleu / 0.455 chrf). among the 11 primary submissions in both translation directions, our submission obtained rank 8 for @xmath8. it will be instructive to examine the manual evaluation results and the other system descriptions to identify the reasons behind these rather disappointing results. for the sorbian task, we submitted ensembles of the basic and alternative models. setting 3 turned out to be the best choice for @xmath9.9 bleu / second rank), and setting 4 for @xmath10.6 bleu / first rank). for both directions, ensembling has raised the bleu scores by 0.6. our submissions would obtain first rank in both directions if only single systems were considered. ## conclusion in this work, we tested various methods for lowresource machine translation proposed by @xcite on the english-inuktitut and german-upper sorbian tasks in wmt 2020. in particular, we investigated several subword segmentation approaches and the inclusion of monolingual tasks. in terms of subword segmentation, we were not able to reproduce the reported gains for the morfessor em+prune method over sentencepiece. we obtained comparable results with both methods though. we also found that increasing the size of segmentation model training data was useful, and that morfessor em+prune was more sensitive to training data size than sentencepiece. furthermore, we obtained slight improvements from subword sampling, confirming earlier results. during the development phase, we also found curious interactions between the subword vocabulary size and different nmt toolkits. we were able to reproduce the organizer-provided inuktitut baselines with both small and large vocabularies using the sockeye toolkit, but obtained significantly lower scores with opennmt-py and large vocabularies, even after harmonizing the training hyper-parameters between toolkits. with small subword vocabularies, opennmt-py became competitive again. the inclusion of monolingual tasks yielded clear improvements for the inuktitut experiments. the noise model had the most positive effect, whereas the transliteration and taboo sampling tasks showed minor effects. in contrast, the effect of the monolingual tasks on the sorbian experiments was more subtle. the two-phase training schedule introduced by @xcite proved useful in the sorbian experiments.