Language ver Y Rare for All (2025)

Abstract: In the quest to overcome language barriers, encoder-decoder models like NLLB have expanded machine translation to rare languages, with some models (e.g., NLLB 1.3B) even trainable on a single GPU. While general-purpose LLMs perform well in translation, open LLMs prove highly competitive when fine-tuned for specific tasks involving unknown corpora. We introduce LYRA (Language verY Rare for All), a novel approach that combines open LLM fine-tuning, retrieval-augmented generation (RAG), and transfer learning from related high-resource languages. This study is exclusively focused on single-GPU training to facilitate ease of adoption. Our study focuses on two-way translation between French and Monégasque — a rare language unsupported by existing translation tools due to limited corpus availability. Our results demonstrate LYRA’s effectiveness, frequently surpassing and consistently matching state-of-the-art encoder-decoder models in rare language translation.

## introduction machine translation has come a long way since its inception in the 1940s. the methodology evolved from the initial rule-based approach @xcite to statistical machine translation @xcite and most recently adopted neural systems as the defacto approach yielding superior results @xcite . an important breakthrough occurred with the advent of transformers @xcite whose attention-based architecture did not only allow for better translation but paved the way for an nlp revolution through llms @xcite @xcite . the considerable progress observed on a wide range of nlp tasks is the combined result of the ingenuous transformer neural architecture, the availability of large gpu compute resources and macroscopic amounts of training data. however, the uneven data amounts between different languages translate to varying performances on nlp tasks @xcite , including machine translation. thus, contrary to widespread languages for which large text corpora are available including parallel data, lesser known languages suffer from data scarcity which makes it difficult to train deep learning models @xcite . moreover, compensating this inequality by obtaining data for low resource languages is expensive and logistically challenging @xcite @xcite . this work is concerned with training a neural machine translator between the french and monégasque language. a very low resource language only spoken by around 5,000 people to date in the principality of monaco and which, to our knowledge, remains uncovered by any neural machine translator. we take on the task of creating a parallel french-monégasque dataset enabling the training of translators and language models on this language. we finetune multiple models on this task and present our methodology called lyra allowing to optimize results with limited data (about 10k parallel sentences and a dictionary). ## related works given the challenge it poses, the low-resource setting has received much attention in the literature @xcite @xcite . the proposed strategies include targeted data gathering @xcite , exploiting monolingual data @xcite , backtranslation @xcite , transfer learning @xcite and multilingual models @xcite . the most notable effort towards a model with high language coverage is nllb @xcite (no language left behind). the latter translator was trained for pairs among over 200 different languages using a sparsely gated mixture of experts architecture. for this purpose, the authors created the flores-200 dataset consisting of 3000 parallel sentences establishing a benchmark for multilingual machine translation. however, this effort did not include the monégasque language. while nllb uses an encoder-decoder architecture specifically intended for translation, decoderonly models also reached competitive performance on multiple tasks including translation @xcite @xcite . this motivated works to improve results with such models @xcite @xcite since they offer a far more interesting option due to their higher flexibility and impressive multitasking abilities @xcite @xcite . moreover, decoder-only models can leverage strategies like rag to improve performance and enjoy greater attention in the literature leading to faster progress. finally, these models hold the same potential for multilingual translation and transfer learning. nonetheless, these references did not consider low-resource languages. most recently, both model types were combined by gentranslate @xcite which uses a seq2seq model to sample translations that are fed into an llm to combine them into an improved answer. note however that this work assumes high compute resources with multiple gpus. in this work, nllb as well as a few open llms are finetuned using lyra on a newly created french-monégasque dataset using only a single gpu machine. we compare their performances on this translation task and showcase the benefits of lyra in the low-resource setting. ## data since we are unaware of any preexisting parallel corpus involving monégasque, we created a french-monégasque dataset using ocr from a few sources including: a french-monégasque dictio-nary, a monégasque grammar book, as well as a few literary works available in both languages. these include works such as the poem collection "lettres de mon moulin", the play "antigone" and some tintin comics. the acquired inputs were later combined into parallel entries via manual annotation. the dataset contains a total number of 10,794 parallel french-monégasque sentences in addition to 42,698 entries from the dictionary and the grammar book which includes verb conjugations and proverbs. the test set was constituted by selecting sentences with high quality translation in order to ensure a reliable basis for evaluation. the fact that this unique existing dataset has under 100k pairs makes the monégasque language a very low resource language based on the conventions adopted by @xcite . ## methods the lyra methodology, illustrated on figure 1 , aims to maximize translation quality in the low data context using three main strategies. ## experiments the impact of each strategy on translation quality is evaluated by testing them sequentially. the effect of data standardization is measured prior to testing the other strategies. performance is measured using the bleu score @xcite as well as meteor which is more correlated with human assessment @xcite . we also pro-vide evaluations using the chrf++ metric @xcite in appendix a. ## conclusion in this work, we presented lyra, a methodology to boost machine translation performance despite scarce data. we saw that enhancing data quality effectively improved results in general. rag also showed significant potential although some model specific adaptation may sometimes be necessary. finally, we have also seen that models can reach higher proficiency in a low resource language thanks to transfer learning. further gains will likely be possible by finetuning future higher performing llms. finally, data augmentation is another interesting research avenue to deal with the low-resource setting.