Bootstrapping M oksha- E rzya Neural Machine Translation from Rule-Based Apertium (2023)

Abstract: Neural Machine Translation (NMT) has made significant strides in breaking down language barriers around the globe. For lesser-resourced languages like Moksha and Erzya, however, the development of robust NMT systems remains a challenge due to the scarcity of parallel corpora. This paper presents a novel approach to address this challenge by leveraging the existing rule-based machine translation system Apertium as a tool for synthetic data generation. We fine-tune NLLB-200 for Moksha-Erzya translation and obtain a BLEU of 0.73 on the Apertium generated data. On real world data, we got an improvement of 0.058 BLEU score over Apertium.

## introduction a significant number of the world's languages are currently at risk of becoming endangered to varying degrees @xcite . this endangered status presents particular challenges when it comes to conducting modern nlp research with these languages. the primary issue stems from the fact that many endangered languages lack extensive textual resources that are readily accessible online. moreover, even when some resources are available, there are concerns regarding the quality of the data, which can be influenced by factors like the author's fluency level, spelling accuracy, and basic character encoding inconsistencies, as discussed in hämäläinen 2021. for over two centuries, scholars have been investigating the cohesion and variety within the contemporary mordvin literary languages, namely erzya (myv) and moksha (mdf). the first comprehensive grammatical works on these languages were published in the 1830s, with moksha in 1838 @xcite and erzya in 1838-1839 @xcite . in the subsequent 180 years, researchers have engaged in extensive fieldwork, compiled grammars, created dictionaries, and worked towards popularizing these languages. notably, in 2002, the inaugural monolingual erzya dictionary was published, authored by abramov @xcite , with plans for future expansion. recent years have also witnessed continued academic interest in the mordvin languages @xcite @xcite , highlighting their enduring significance in linguistic research. it is crucial to provide newcomers to language documentation with chances to enhance their comprehension of languages by involving them in projects. a noteworthy period to highlight in this regard is the years spanning from 1988 to 1997 when many of today's researchers were engaged in word processing for the extensive 'dictionary of mordvin dialects', compiled on the basis of language materials whose collection was originated and orchestrated by prof. heikki paasonen at the turn of the twentieth century, and which, when completed, comprised a substantial 2073 pages. the research conducted in this paper is based on data generated using the rule-based machine translation system apertium. while rule-based tradition has influenced the current nlp for endangered uralic languages (cf. @xcite , our aim is to study the degree to which more modern neural models can be incorporated into the existing paradigm. the largest obstacle in using machine learning models is the scarcity of data available in these languages. using apertium to generate training data is our attempt at overcoming this problem. ## related work many contemporary machine translation models heavily rely on the presence of parallel texts. however, finding parallel texts is a challenging endeavor, particularly when dealing with lessresourced languages across various domains. this challenge becomes even more pronounced when attempting to train data-intensive models for these languages. a method proposed by @xcite addresses this issue by utilizing a large, nonparallel but comparable corpus, such as news articles, in conjunction with relatively small parallel corpora from a different domain, like the united nations corpus. by matching sentences from comparable articles that share the same topic, this method attempts to determine if two sentences are translations of each other. although this approach enhances translations within the news domain, it's not always viable, especially for extremely lowresource languages like erzya and moksha, which lack the necessary comparable corpora. to develop machine translation models for languages with limited resources, another approach involves leveraging a resource-rich language closely related to the low-resource one as a parent language. this entails acquiring some of the resourcerich language's characteristics, such as syntax and morphology, and transferring them to the lowresource machine translation model @xcite @xcite . these techniques don't necessarily require parallel texts in the lowresource language but rely on the resource-rich language, which may result in limited coverage of the low-resource language's morphology. researchers have also explored methods for constructing parallel texts through crowdsourcing, wherein online workers are tasked with translating expressions into another language @xcite @xcite . crowdsourcing, however, proves to be a challenging endeavor when dealing with low-resource languages due to the limited number of native speakers. additionally, the absence of a standardized language form or even linguistic variation further complicates the quality control of crowdsourced translation tasks, even if a significant number of workers are involved. a different approach proposed by @xcite involves translating english into morphologically complex languages by creating a model that predicts word inflections in the target language. this model is then used to generate synthetic phrases, potentially with new inflections, which are incorporated into the training data alongside a parallel corpus to train a machine translation model. @xcite introduced a method for creating parallel data for low-resource endangered languages with complex morphology. they demonstrated their approach using finnish as a pilot language, matching the resource limitations to those of erzya. additionally, the authors described a technique for automatically aligning the abstract morphosyntactic structures of two languages to generate a set of parallel templates. however, the system could only translate phrases as opposed to complete sentences. ## apertium in data generation apertium @xcite is rule-based machine translation system that is available for several language pairs. there is a special version of the system for endangered languages hosted by giellalt @xcite that has support for erzya and moksha. the erzya-moksha translation has been developed through a shallow transfer approach @xcite and it utilizes fst transducers developed for these languages @xcite . we use a monolingual erzya corpus of around 220 000 sentences (the erzya-language novel purgaz @xcite ). we feed this corpus to apertium translator and translate the sentences into moksha. this way, we will have a parallel corpus of erzya-moksha sentences where the target side, namely erzya is of a high quality and the source side, namely moksha is synthetically generated using the rule-based system. apertium is not able to inflect all words, so it tags such words with different tags such as # and @. we remove these extra characters from the data. moksha input apertium our model gold standard минь карматама природать тонафнемонза. минь карматано *природать тонавтнемензэ. минек карматано природанть тонавтнемензэ. минь карматано природанть тонавтнеме. природать колга наукати мярьгихть естествознания. *природать содалмонтень мерить *естествознания. природантьдонть наукантень мерить естествознания. природадо наукантень мерить естествознания. естествознаниять тейнек пяк оцю значенияц. *естествознаниять тенек пек покш *значенияц. естествознаниять тенек пек покш значенияц. естествознаниянть значениязо миненек пек покш. сон лезды лац шарьхкодемс природать. сон лезды парсте чарькодемс *природать. сон лезды лазтнэнь чарькодиця природанть. сон лезды тенек природанть видестэ-парсте чарькодеме. this ensures a good and grammatical target representation. this is similar to the back translation methodology described by @xcite . the data is split randomly into 80 % training, 10 % validation and 10 % testing. the model is trained only on this apertium generated dataset. ## neural machine translation we use the nllb-200 model by meta @xcite to conduct our experiments. the model has been trained to support translation for over 200 languages. erzya and moksha are not supported by the model by default. in fact, the only uralic languages that are supported are finnish, estonian and hungarian -none of which are endangered. nllb-200 is a 54.5b parameter mixture of experts (moe) model that has been trained on a dataset containing more than 18 billion sentence pairs. on benchmark evaluations, nllb-200 outperforms other state-of-the-art models by up to 44%. the model's performance has been validated through extensive evaluations for each of the 200 languages it supports. we use transformers python library @xcite to fine-tune the 600m distilled nllb-200 model 1 for moksha to erzya translation. we add two new languages to the tokenizer: mdf_cyrl and myv_cyrl for moksha and erzya. we trained the model for 5 epochs. we used weight decay of 0.1 and an initial learning rate of 2e-5. the model was validated after each epoch using bleu as the validation metric. the final validation bleu score was 0.85. 1 https://huggingface.co/facebook/nllb-200-distilled600m ## results we have withheld 10% of the data for testing proposes. our model achieves a bleu score of 0.73. when looking at the results, we can perceive errors coming from the fact that we used apertium translator such as missing vocabulary resulting in wrong word choice in the translation output as well as minimal transfer rules to address diversity in verbal government and idiomatic expressions. this calls for further comparison of the model with apertium translations. we take a relatively small parallel corpus of a natural science book that has been translated into erzya and moksha from russian foot_0 . the corpus consists of a little over 2000 sentences of humanauthored translations. we test both our model and apertium on this dataset. neither of the models reaches very good performance when measured by bleu scores. the apertium translator gets a bleu score of 0.037 whereas our model gets a bleu of 0.095. it is important to note that our model got an improvement of 0.058 bleu score. the results can be seen in the biggest problem apertium generated data has is a lack of vocabulary coverage. our model gets the grammar and morphology correct more frequently than the rule-based apertium translator because of the good target representation from the high-quality erzya sentences used in training, although there are instances of multiexponence in morphology as in природанть+донть 'the na-ture+about the' which might be explained by incomplete transfer rules in the apertium translator. however, both apertium and our model struggle with vocabulary and many of the words are either not translated at all or are translated to wrong words. however, our result suggest that using apertium to generate data for an nmt model is a viable way of combining the rule-based tradition with latest neural models. in particular, the fact that the nmt model was able to produce better results makes this a worthwhile approach for any endangered language machine translation project. this is even more so in cases where the rule-based translation system has reached a higher level of maturity. ## conclusions in conclusion, our study has revealed significant insights into the performance of our neural machine translation (nmt) model when compared to the rule-based apertium translator in the context of translation between two endangered languages, moksha and erzya. the primary challenge observed in the apertiumgenerated translations was the limited vocabulary coverage. our nmt model, on the other hand, demonstrated better accuracy in terms of grammar and morphology due to the high-quality erzya sentences used during training. nevertheless, both apertium and our model struggled with vocabulary, resulting in untranslated or incorrectly translated words. in light of our findings, our results suggest that employing apertium to generate data for an nmt model represents a viable approach that combines the rule-based translation tradition with state-ofthe-art neural models. notably, our nmt model outperformed apertium, making this approach valuable for endangered language machine translation projects, particularly when the rule-based translation system has reached a higher level of maturity. this study highlights the potential for leveraging advanced technology to revitalize and preserve endangered languages through more accurate and efficient translation methods. the findings of this study open up several promising avenues for future research and development in the field of machine translation, particularly for endangered languages. addressing the challenge of limited vocabulary coverage observed in both the rule-based apertium translator and the nmt model is critical. future research could focus on methods to expand and enrich the vocabulary used in training data, possibly through the inclusion of additional linguistic resources or domain-specific terminology. augmenting the parallel corpus with more diverse and representative data, including various text genres and dialects, can contribute to better training nmt models. collecting and curating additional language resources could be a valuable step. furthermore, we could engage with the community of speakers and experts in erzya and moksha languages to gather feedback, improve resources, and establish collaborative efforts in language preservation and machine translation. these future work directions reflect the ongoing efforts needed to advance the field of machine translation, particularly in the context of preserving and revitalizing endangered languages, and hold the potential to significantly improve the quality and accessibility of translation services for these linguistic communities.