Machine Translation for Low-resource F inno- U gric Languages (2023)

Abstract: This paper focuses on neural machine translation (NMT) for low-resource Finno-Ugric languages. Our contributions are three-fold: (1) we extend existing and collect new parallel and monolingual corpora for 20 languages, (2) we expand the 200-language translation benchmark FLORES-200 with manual translations into nine new languages, and (3) we present experiments using the collected data to create NMT systems for the included languages and investigate the impact of back-translation data on the NMT performance for low-resource languages. Experimental results show that carefully selected limited amounts of back-translation directions yield the best results in terms of translation scores, for both high-resource and low-resource output languages.

## introduction neural networks have caused rapid growth in output quality for many natural language processing tasks, including neural machine translation (nmt, @xcite . however, the output quality crucially depends on the availability of large amounts of parallel and monolingual data for the covered languages. recently synthetic data and cross-lingual transfer have not only shown potential for low-resource language nmt but also have been taken to the extreme through open massively multilingual translation models @xcite . in addition to translation models, a massive translation benchmark flores-200 @xcite has been created, consisting of multi-parallel translations of the same sentences into 200 languages. here we focus on nmt for low-resource languages from a family of languages spoken in europe, but not part of the indo-european family: finno-ugric languages. three members of that family (estonian, finnish and hungarian) are commonly included in massively multilingual efforts and can be considered medium-resource languages. at the same time, several lower-resource finno-ugric languages are not included in the existing massively multilingual models (m2m-100, nllb). in terms of the number of speakers, they range from 20 near-native speakers of livonian to several hundred thousand speakers of mordvinic languages. our contributions are three-fold. first, we present a collection of parallel and monolingual corpora that can be used for training nmt systems for 20 low-resource finno-ugric languages. the resources are collected from sources that are already digital (primarily online sources); the languages and the data are described in section 3. secondly, we expand a part of the 200-language translation benchmark flores-200 with manual translations into the low-resource finno-ugric languages. this includes the first 250 sentences of flores-200 and the following languages: komi, udmurt, hill and meadow mari, erzya, livonian, mansi, moksha and livvi karelian. this new benchmark is described in section 4. finally, we use the collected parallel and monolingual data in experiments to create nmt systems for the covered languages. the main question we address is which subsets of translation directions yield the best results for the included low-resource languages. we achieve an average chrf++ score of 26.8 when translating from high-resource to lowresource languages included in our expansion of flores-200. the complete experiments and results are presented in sections 5 and 6. in @xcite and @xcite , the authors investigate the best nmt model setups, with @xcite showing a comparison to phrase-based systems that are not that common these days. @xcite and @xcite indicate that training universal models (sharing parameters between multiple languages) and transfer learning are two aspects that get significant gains for lowresource language pairs in translation quality. more recently, low-resource machine translation has risen to the attention of more and more research groups with multiple comprehensive surveys emerging @xcite , showing that there has already been a lot of work done that can now be systematically aggregated and utilized in further research. ## languages and data the finno-ugric language group has two major branches: finno-permic and ugric. although both branches share common linguistic roots, they are quite distant. the finno-permic branch includes two highresource languages, estonian and finnish, and several low-resource languages, such as komi, komi permyak, udmurt, hill and meadow mari, erzya and moksha, proper and livvi karelian, ludian, võro, veps, livonian, sami languages. the ugric branch comprises three languages: highresource hungarian and two low-resource mansi and khanty. in this work, we develop an nmt system between 20 low-resource finno-ugric (fu) languages shown in figure 1 and seven high-resource languages (english, estonian, finnish, hungarian, latvian, norwegian (bokmål), and russian). the selection of the high-resource languages is not accidental: estonian, finnish, and hungarian belong to the fu language family, while latvian has markedly influenced livonian, norwegian has deeply affected the sami languages and russian has had a profound impact on the permic, mordvinic, mari, karelian, veps, and ob-ugric languages. figure 1: languages from the finno-ugric language family, for which we have created mt systems. green colour represents branches, orange -languages. the finno-permic languages are visualized according to the janhunen classification (janhunen, 2009). ## benchmark dataset in order to create a multilingual benchmark for finno-ugric languages foot_7 , we took the first 250 rows of the flores dataset @xcite and had them translated into nine finno-ugric languages: komi, udmurt, hill and meadow mari, erzya, moksha, livonian, mansi, and livvi karelian by a team of bilingual speakers, both natives and fluent speakers, of estonian or russian and low-resource fu languages. most translators have an academic degree in linguistics or have extensive translation experience. while translating, translators have encountered the following problems: ## experiments one of the goals of our paper was to find out which language pairs are needed to reach a certain level of quality for low-resource nmt models in the finno-ugric setting. more specifically, the question is whether it is necessary for low-resource multilingual systems to back-translate in all directions (which is costly) or subsets of translation directions can suffice? by finding optimal amounts of synthetic data we can optimize the overall system creation process by making it less costly and less time-consuming while being able to increase the number of iterations performed. ## results quantitative analysis to get an overview of the quality of the models and compare different synthetic data settings, we compare chrf++ @xcite results for all of the experiments, calculated using sacrebleu @xcite 13 . as we are evaluating morphologically rich languages, reporting chrf++ as the main automatic metric gives the most truthful results, whereas bleu @xcite is too punishing on this type of languages. in the cause of this situation is the fact that although we limited monolingual data to 100k for each language pair, some smaller language pairs had a lot less than 100k monolingual sentences. taking only 10% of the synthetic data leveled the distribution of high-and low-resource synthetic data and allowed high-resource to low-resource pairs to get more attention during training. the best scenario for translating into lowresource languages seems to be to use synthetic data from low-resource language into related highresource languages (l-rh). this is shown by the subtables 3a and 3c. for translating from lowresource languages to high-resource languages, however, the most efficient is to add synthetic data from each low-resource language to all the high-resource languages involved in the initial fine-tuning (l-h), instead of using the larger all-all model. comparing the baseline to all the other models, we see significant improvements which can be explained by the fact that the parallel data for low-resource languages originated mainly from the bible, but monolingual data originated from different domains, even more for the high-resource languages. one anomaly clear from sub we do not report results for low-resource languages that lack the flores benchmark, because the held-out test set is too biased towards the bible domain and there is no other comprehensive benchmark for the rest of the low-resource languages. in addition to the mentioned experiments, we tried filtering the back-translation data with some of the same filters used to filter the original parallel data. however, the results of the experiments with filtered back-translation data were the same or even a little worse than with the non-filtered back-translation data. thus, we do not report these results and leave the thorough back-translation filtering analysis for future work. ## conclusion we presented a flores-based benchmark dataset for nine low-resource finno-ugric languages: erzya, komi, livvi karelian, livonian, hill and meadow mari, mansi, moksha, and udmurt. in this study, we trained and evaluated multiple models for these languages and generated a large amount of synthetic parallel data through back-translation. the results showed that the models achieved promising performance on the benchmark dataset and demonstrated the potential of these methods for low-resource machine translation. our experiments also showed that it could be useful to choose back-translation settings more strategically, selecting certain language pairs, to achieve better results while using fewer resources for back-translation and training.