Paragraph-Level Machine Translation for Low-Resource Finno-Ugric Languages (2025)

Abstract: We develop paragraph-level machine translation for four low-resource Finno-Ugric languages: Proper Karelian, Livvi, Ludian, and Veps. The approach is based on sentence-level pre-trained translation models, which are fine-tuned with paragraph-parallel data. This allows the resulting model to develop a native ability to handle discource-level phenomena correctly, in particular translating from grammatically gender-neutral input in Finno-Ugric languages. We collect monolingual and parallel paragraph-level corpora for these languages. Our experiments show that paragraph-level translation models can translate sentences no worse than sentence-level systems, while handling discourse-level phenomena better. For evaluation, we manually translate part of FLORES-200 into these four languages. All our results, data, and models are released openly.

## introduction the existence of massively multilingual pre-trained translation models (e.g. m2m100, nllb, and madlad-400: @xcite @xcite has made work on machine translation significantly easier by eliminating the need for training large models from zero. nevertheless, even the largest of these models still leave many low-resource languages out-mainly due to lack of or difficulty to acquire textual data (monolingual or parallel) in those languages. moreover, these translation models approach translation by handling each sentence independently and thus do not handle discourse-level phenomena well foot_0 . ignoring the discourse-level phe-nomena has been shown to pose problems for translation quality and its assessment @xcite . even though decoderonly language models (e.g. @xcite are an easy way to approach document-level translation, the availability of pre-trained open multilingual language models and their language coverage are even narrower than for translation models. also, translation is more efficiently solved with sequence-to-sequence models when emergent abilities are not a requirement and the main purpose is to solve translation, not other tasks. in this paper, we focus on developing machine translation for the finno-ugric family of languages, which is a good fit for addressing both aforementioned issues, namely support for low-resource languages and discourse-level phenomena ignorance: we narrow down our scope to four underresourced members of the finno-ugric language family: proper karelian, livvi, ludian, and veps. all four are low-resource languages and are not included in m2m100, nllb, or madlad-400; they are also not supported by google translate foot_2 or deepl foot_3 , as of january 2025. but rather via monolingually denoising documents in several languages; translation is later taught to the model on sentence level. text in veps: naine tuli kodihe. hänen mašin jäi garažas. english translation: the woman came home. her car remained in the garage. figure 1 : example of translation challenges related to gender-neutral pronouns in finno-ugric languages: the veps text includes the pronoun hänen, which can be translated both as "her" and "his"; resolving this ambiguity requires looking at the first sentence and the word naine (woman) as the antecedent. with the issues listed above in mind, we collect paragraph-level corpora and develop paragraphlevel machine translation models by simply finetuning sequence-to-sequence models on parallel paragraph pairs, comparing the results to sentencelevel approach. in order to fit the paragraph into the context window of the model, we limit its length to five sentences at most-our experiments show that such a bounded context still allows the model to learn extrasentential dependencies. our key contributions are thus the following: the collected data foot_4 , trained models foot_5 , and created benchmarks foot_6 are released openly. next, we outline the related work in section 2 and present the methodology in section 3. currently, attempts have been made to incorporate context in the attention-based models' scope by modifying their architecture. the researchers offered methods such as hierarchical attention @xcite or memory networks @xcite among others. however, the most straightforward strategies, like passing an entire text to the model, proved also the most effective. @xcite trained the transformer model @xcite on documents, repeatedly dividing them into parts to vary input lengths. although this approach has shown a big leap in translation quality, it does not remedy another important problem: long processing times of large documents. the time and memory consumption of transformer-based systems scales quadratically with the input length. we try to avoid this issue by splitting documents into small, fixed-size paragraphs rather than translating documents fully. ## methodology in this chapter, we briefly describe our approach to dealing with paragraph-level data, ways to extract paragraphs from documents and evaluate paragraph-level translations. we chose madlad-400 as the basis for our experiments, since, in addition to being a small, powerful, and open-source model, it has the potential for paragraph-level translation as it was pre-trained with document-level monolingual data. ## data in this paper, we focus on (i) two dialects of karelian: livvi (olo) and proper karelian (krl) foot_8 ; (ii) ludian (lud), which is closely related to karelian, but is considered a language in its own right @xcite ; and (iii) veps (vep). these are all endangered finnic languages, mainly spoken in finland and russia. ## experimental setup to investigate the effect of paragraphs on the quality of translation of proper karelian, livvi, ludian, and veps, we fine-tune two madlad models: one on sentence-level data and the other on paragraphlevel data. we translate the languages into russian and vice versa. russian was chosen as a translation objective (among other high-resource languages available in madlad) because most of the openly available parallel texts were aligned with the russian language. to further improve the model, we perform back-translation making use of our monolingual data. we back-translate in a single direction-from finno-ugric languages to russian-and thus, enhance the quality of translation from russian to finno-ugric languages (otherwise quite low). we avoid back-translation between the four selected languages because low-quality synthetic data can harm the resulting performance instead of improving it @xcite . using the huggingface framework 12 , we finetune both the sentence-level and paragraphlevel model for 10 epochs under equal conditions. we set the hyperparameters of seq2seqtrainingarguments to their default values with the following exceptions: we perform fine-tuning on the lumi 13 supercomputer with amd instinct mi250x gpus. we use both models to translate paragraphs from the modified smugri flores benchmark. for generation, we set the standard beam size of 5. we evaluate translations with the bleu and chrf++ metrics, of which we use the sacrebleu @xcite implementations. when calculating chrf++, 12 https://huggingface.co/ 13 https://lumi-supercomputer.eu/ we count only word bigrams. to measure statistical significance and confidence intervals, we do bootstrap resampling with 1000 resamples. ## results in this section, we examine the obtained results, starting with a quantitative analysis that presents translations from proper karelian, livvi, ludian, and veps into russian, as well as from russian to these four languages. next, we conduct a brief qualitative analysis. after this, we compare our results with those generated by the online machine translation engine tartu nlp neurotõlge foot_12 . finally, we explore how well translation abilities transfer to the unseen case of english translation. ## conclusion in this paper, we developed a machine translation system for four low-resource finno-ugric languages: proper karelian, livvi, ludian, and veps. unlike previous mt systems that cover the same languages, ours is paragraph context-aware. the analysis showed that the model consistently translates names and terminology, though, it still encounters difficulties with coreference resolution. the developed system has been trained only on parallel corpora with russian. nevertheless, the system is also capable of translating to and from english, despite not being trained to do so, with paragraph-level abilities being successfully transferred to this case. additionaly, we presented a flores-based benchmark dataset for proper karelian (viena), ludian, and veps. the collected paragraph-level corpora are released as huggingface scripts that will allow one to re-collect the data. we leave for future work experiments with more finno-ugric languages, including creating a paragraph-level benchmark that enables a more thorough evaluation of discourse-level phenomena handling. it would also be interesting to compare our results with multilingual decoder-only models, as many of these are starting to emerge.