Boosting Neural Machine Translation from F innish to N orthern S ámi with Rule-Based Backtranslation (2021)

Abstract: We consider a low-resource translation task from Finnish into Northern Sámi. Collecting all available parallel data between the languages, we obtain around 30,000 sentence pairs. However, there exists a significantly larger monolingual Northern Sámi corpus, as well as a rule-based machine translation (RBMT) system between the languages. To make the best use of the monolingual data in a neural machine translation (NMT) system, we use the backtranslation approach to create synthetic parallel data from it using both NMT and RBMT systems. Evaluating the results on an in-domain test set and a small out-of-domain set, we find that the RBMT backtranslation outperforms NMT backtranslation clearly for the out-of-domain test set, but also slightly for the in-domain data, for which the NMT backtranslation model provided clearly better BLEU scores than the RBMT. In addition, combining both backtranslated data sets improves the RBMT approach only for the in-domain test set. This suggests that the RBMT system provides general-domain knowledge that cannot be found from the relative small parallel training data.

## introduction machine translation from and to minority languages is challenging because large parallel corpora are typically hard to obtain. two strategies have proven most successful to eliminate this bottleneck: using rule-based machine translation (rbmt) systems that do not rely on large data, or training data-driven translation systems with automatically created synthetic data, e.g. backtranslation @xcite . in this paper, we com-bine both strategies in the context of neural machine translation (nmt) from finnish to northern sámi. in particular, we investigate the impact of rbmt in data augmentation in comparison to standard nmt-based backtranslation. northern sámi is a uralic minority language spoken in norway, sweden and finland. historically, most of the work on machine translation from and to sámi languages is based on rbmt @xcite @xcite . data-driven approaches such as nmt are generally more competitive, but require large amounts of training data in the form of parallel translated sentences. for minority languages, finding parallel data sets is usually more difficult than collecting monolingual data, which is also the case for northern sámi. a common way of leveraging monolingual data for nmt is the above mentioned backtranslation strategy, a method where monolingual data of the target language is translated automatically to the source language to create additional parallel training data. in this work, we use two reverse translation models to produce the backtranslations: a neural model trained only on the available parallel data and a rule-based approach. the latter is a system developed for the translation from northern sámi to finnish @xcite within the apertium framework @xcite . we also combine both methods to further augment the data. our experiments demonstrate the positive effects of both strategies and the possibility of obtaining complementary information from different backtranslation engines. ## related work using backtranslations from different sources as training data has been shown to be beneficial for improving machine translation quality. in addition to proposing training data augmentation methods that do not require reverse translation systems, burlot and yvon (2018) compare the effects of using statistical machine translation (smt) and nmt based backtranslations for @xmath0. they show that both types of backtranslations improve translation quality, nmt slightly more than smt. poncelas et al. (2019) also produce backtranslations with smt and nmt. they show that the translation quality of a @xmath1. the greatest improvement is observed when both types of backtranslations are used. augmenting training data with rbmt backtranslations has also proven to be useful for boosting translation quality. dowling et al. (2019) use rbmt backtranslations to improve statistical machine translation performance for scottish @xmath2. the authors show that backtranslations can be beneficial even in cases where the translation quality of the mt system used to produce the backtranslations is low. @xcite study the performance of nmt systems trained with augmented training data backtranslated using rbmt, smt and nmt. they experiment with @xmath3. @xcite also analyze the effects of using augmented training data backtranslated with the three different paradigms. they focus on two language pairs: a low-resource language pair, @xmath4, and a high-resource language pair, @xmath5. in addition to showing similar results as @xcite , they show further improvement in translation performance when all types of augmented training data are combined. ## data the uit freecorpus 1 contains a finnish -northern sámi (fin-sme) parallel corpus with 110k sentence pairs and a distinct set of 868k monolingual northern sámi sentences. the uit corpora are collected from multiple sources and cover various domains. both the parallel and the monolingual corpora contain considerable amounts of duplicate lines. in this section, we describe our data cleaning and filtering efforts and the data split. for ad-1 https://giellatekno.uit.no/ ditional evaluation, we collected a small test set consisting of translated yle news articles foot_0 . data filtering and cleaning is carried out with the opusfilter toolbox @xcite . our opusfilter configuration files are available online foot_1 , which helps to replicate the data preprocessing steps. first, we remove duplicate lines from the parallel corpus. this process removes 67.7% of the sentence pairs, leaving us with 35,426 unique sentence pairs. the remaining data set is then cleaned with a set of filters from opusfilter. similar filtering setups have been confirmed to improve translation quality @xcite . in particular, we remove sentence pairs that satisfy one of the following conditions: after filtering, 29,106 clean sentence pairs remain in the parallel data set. from this clean set, 2000 pairs are randomly selected to form a validation set and another 2000 pairs to form a test set, leaving 25,106 pairs for training. note that all subsets are disjoint due to the initial deduplication. the additional test set consists of two news articles describing sámi culture in finland available in both finnish and northern sámi on yle news. it was extracted from the web and manually aligned to create a clean reference set. this test set is, however, small (151 sentence pairs) and may not produce completely reliable evaluation scores, but it should still provide additional insights about the quality of the translation models and their ability to generalize to new domains. the monolingual northern sámi data is processed in a similar way as the parallel data above. duplicate removal discards 35.6% of the total of 867,677 sentences, leaving 559,074 sentences in the data set. for corpus cleaning, we use all filters of those cited above that are applicable to monolingual data, i.e. the sentence length filter, the word length filter, the html element filter, the latin script filter, and the language identification filter. the resulting clean monolingual corpus contains 462,803 sentences. ## method in this section, we compare a baseline fin-sme nmt model trained only with the available parallel data to nmt models trained with additional backtranslated data. the backtranslations are produced by translating the clean monolingual northern sámi data to finnish either with a nmt system trained on the parallel data in the reverse direction (sme-fin), or with the sme-fin rbmt system. this yields three additional synthetic training sets that augment the original parallel training data: one with the nmt backtranslations, one with rbmt translations, and one with both types of backtranslations. each of them is then used to train a separate nmt model that we can compare to the baseline model, which is trained on the original parallel data only. note that we do not use any data sampling or weighting scheme to balance original and augmented training data. all nmt models in our experiments are trained with @xcite version 1.8.33. the backtranslation model is based on a rnn architecture with gru cells @xcite from @xcite . for subword segmentation, we use the sentencepiece tokenizer @xcite with vocabulary size 8000, which has been shown to produce the best results with the data set sizes that we are dealing with @xcite . we train the models until the cross-entropy of the validation set does not improve for 10 consecutive validation steps. for the rbmt backtranslations, we use apertium with the sme-fin model by @xcite . this system implements a shallow transfer-based translation engine consisting of modules for morphological analysis, disambiguation and generation, modules for lexical translation based on context rules, and a module for syntactic transformation operations. ## results the upper part of the significant boost from rbmt backtranslations is quite remarkable considering that apertium does not seem to perform very well on the reverse translation direction on uit data. this result stresses once more that the effect of backtransla-tion is to a larger extent due to improved target language coverage than to the quality of the translations. instead, the additional, less domain-specific knowledge encoded in the rbmt model seems to lead to the additional push even in the uit domain and it certainly carries over to the out-of-domain data represented by the yle news data. the simple combination of both types of backtranslations only provides a modest additional boost on the uit test set. the out-of-domain performance drops substantially compared to using rbmt-based backtranslations alone. adding nmt-based translations seem to hurt the model in this regard. next, we study the effect of filtering the backtranslations before training the augmented nmt models. ## conclusion in this work, we confirm that the addition of backtranslations produced with multiple paradigms, including rbmt, improves the quality of nmt models. additionally, the translation performance can be further improved by removing noisy sentence pairs from the nmt backtranslations. we show that these methods are beneficial in a real-world low-resource setting with the @xmath6. in the future, we plan to extend our work in various ways including more careful data selection and filtering, the use of subword regularization, domain labeling, improved sampling strategies and further data augmentation techniques such as pivot-based translations and transfer learning using multilingual nmt models. furthermore, we would like to optimize hyper-parameters such as vocabulary size, network architectures and training parameters to maximize the translation performance in low-resource scenarios.