Language Model Priors and Data Augmentation Strategies for Low-resource Machine Translation: A Case Study Using F innish to N orthern S ámi (2024)

Abstract: We investigate ways of using monolingual data in both the source and target languages for improving low-resource machine translation. As a case study, we experiment with translation from Finnish to Northern Sámi.Our experiments show that while conventional backtranslation remains a strong contender, using synthetic target-side data when training backtranslation models can be helpful as well.We also show that monolingual data can be used to train a language model which can act as a regularizer without any augmentation of parallel data.

## introduction this paper focuses on improving machine translation (mt) between the northern sámi (iso 639-3: sme) and finnish (fin) languages. with approximately 25,000 speakers, northern sámi has the most speakers of the sámi languages, which are spoken by the sámi people. the sámi are the indigenous peoples of sápmi, which spans modernday norway, sweden, finland, and russia. 1 this paper provides a contribution towards improving northern sámi-finnish machine translation, with the hope that these findings may be generalized to other languages in subsequent work. we advance the state of the art for northern sámi-finnish mt established by @xcite using a combination of techniques. the contributions of this paper are as follows. first, we demonstrate that current llm-based mt models are insufficient for translating this language pair-and likely other pairs including less-resourced/minoritized languages-by evaluating how well the recent madlad-400 model @xcite performs on our datasets. second, we demonstrate that the parallel data-only baseline of @xcite was undertrained, 1 further reading: the sámi language crisis, language policy and the sámi languages: an investigation using the comparative perspective, and indigenous peoples in sápmi. ## related work northern sámi language technology much of the northern sámi language technology research to date has been performed at the giellatekno institute at the arctic university of norway. although a large part of the research focuses on building tools like keyboards and dictionaries @xcite , recent work using their corpora has also focused on mt @xcite . there has also been work on integrating northern sámi with existing multilingual mt models such as m2m-100 @xcite @xcite . backtranslation and forward translation backand forward translation also have a long history in neural machine translation research. backtranslation was introduced in the nmt context by @xcite where the authors observe 2-3 bleu increases on english-german and english-turkish translation tasks. forward translation was also introduced around the same time by @xcite who use it as both a standalone technique and in a multi-task framework. massively multilingual language models recently, @xcite introduced madlad-400, a massively multilingual encoderdecoder language model (mmlm) with support for both finnish and northern sámi. to gauge whether our translation task has already been solved by developments in mmlms, we evaluated the performance of the 3b parameter mt variant of madlad-400, madlad400-3b-mt, using test sets we describe later. based on the results given in ## methodology we expand upon the experiments of @xcite in several ways: using more powerful model architectures for backtranslation (transformer instead of rnn), more elaborate data augmentation including synthetic target-side data, and transformer-based autoregressive language model priors. ## conclusion we have shown several ways to augment extremely small parallel datasets with synthetic data to enable performance improvements. in the absence of publicly-available large monolingual data sets, language models may provide another avenue for regularization, potentially even using off-the-shelf language models. a key question is whether the approaches described in this work will generalize well to other languages. we were not able to answer this question primarily due to computational resource considerations, as we only had enough compute resources to run our experiments in the language pairs explicitly supported by our funding source. other settings, but we do not expect that others will find results completely consistent with ours. our work also deals with an indigenous language, which opens the door for potential misuse of nlp models against marginalized peoples. even when used by those peoples themselves, quality issues with the models (such as mistranslations) can have negative impact due to providing incorrect information. this can be especially problematic with out-of-domain data as our experiments have shown. however, as our models establish a state of the art for the datasets we examine and our limited analysis of a popular multilingual model (madlad-400) suggests its performance is much lower than our models, we believe the risks from our model are less than those of existing ones.