Rethinking Low-Resource MT: The Surprising Effectiveness of Fine-Tuned Multilingual Models in the LLM Age (2025)

Abstract: This study challenges the current paradigm shift in machine translation, where large language models (LLMs) are gaining prominence over traditional neural machine translation models, with a focus on English-to-Faroese translation. We compare the performance of various models, including fine-tuned multilingual models, LLMs (GPT-SW3, Llama 3.1), and closed-source models (Claude 3.5, GPT-4). Our findings show that a fine-tuned NLLB model outperforms most LLMs, including some larger models, in both automatic and human evaluations. We also demonstrate the effectiveness of using LLM-generated synthetic data for fine-tuning. While closed-source models like Claude 3.5 perform best overall, the competitive performance of smaller, fine-tuned models suggests a more nuanced approach to low-resource machine translation. Our results highlight the potential of specialized multilingual models and the importance of language-specific knowledge. We discuss implications for resource allocation in low-resource settings and suggest future directions for improving low-resource machine translation, including targeted data creation and more comprehensive evaluation methodologies.

## introduction the recent rise of llms has introduced new possibilities in machine translation @xcite . llms demonstrated impressive performance across various language pairs, often through the use of in-context learning @xcite . these new opportunities often come at a price in terms of computational resources: llms have massive requirements in terms of pretraining data and high-end hardware. hardware requirements can sometimes be mitigated by using closed-source llm apis (e.g., openai api). however, this approach introduces issues related to transparency and license limitations. these limitations and high requirements disproportionately affect low-resource languages and communities. for such languages, lack of resources can often extend beyond data scarcity and effectively imply lack of computational infrastructure and expertise, rendering the use of apis offered by tech giants the only available option. this is the case for faroese, an insular scandinavian language and official language of the faroe islands. neural machine translation (nmt) models are less demanding in terms of computational resources. however, due to their more limited reasoning capabilities compared to llms, they often underperform in low-resource settings. nonetheless, there are potential strategies to leverage the linguistic knowledge of an llm in conjunction with lightweight mt models to optimize performance while minimizing resource requirements. one such approach is to use llms to augment parallel datasets, allowing a lighter mt model to be trained on this synthetic data @xcite . in nlp, efficiency encompasses various factors like data requirements, model size, training costs, and performance metrics. this paper focuses on the relationship between model performance and size, a crucial consideration for real-world applications. we explore different approaches to english-to-faroese machine translation, investigating how various techniques balance translation quality with model compactness. our research aims to shed light on the trade-offs between performance and model size in this specific language pair. we will compare the following approaches, in the context of english to faroese mt: these strategies will be compared based on automatic and human evaluation.we will be comparing the following open-source llms: llama 3.1, (meta) @xcite in its 8b version, and gpt-sw3, a generative model for the nordic languages, primarily swedish, @xcite , in its 1.3, 6.7 and 40b version. their performance will be compared to closedsource models such as claude 3.5 sonnet (anthropic, 2024) by anthropic, gpt-4 turbo @xcite and gpt-4o (openai, 2024) by openai. we compare the llms with no language left behind (nllb) @xcite , an opensource nmt multilingual model covering, among other under-resourced languages, faroese. all new models produced via fine-tuning in this paper are now publicly available. foot_0 2 background and related work ## discussion our study on english to faroese machine translation reveals several important findings that provide new insights into the relative strengths of different approaches to low-resource language translation, including large language models and specialized multilingual models. surprisingly, the finetuned nllb model outperformed most llms, including gpt-4 and gpt-sw3 40b, in both automatic and human evaluations. this suggests that specialized multilingual models, when finetuned appropriately, can be highly effective, often achieving comparable or even superior performance to larger llms for specific language pairs. the success of nllb highlights the importance of domain-specific training and more compact, efficient models, which can be especially valuable in low-resource settings where computational power may be limited. furthermore, the performance of gpt-sw3, despite its smaller size compared to llama 3.1, underscores the critical role of language-specific knowledge in translation tasks. these findings have significant implications for resource allocation and model selection in lowresource language translation. while automatic and human evaluations generally aligned on model rankings, there were key differences in perceived quality. this reveals the limitations of relying solely on automatic metrics, especially for low-resource languages. human evaluations showed that while claude 3.5 sonnet and nllb 1.3b had similar error counts, claude performed better in content preservation and received a higher overall esa score, suggesting that evaluators may prioritize factors like fluency and naturalness beyond just error quantity. the improvement in nllb's performance when fine-tuned on both the sprotin corpus and llm-generated synthetic data (fo_en_synthetic) highlights the potential of leveraging llms to augment training data for low-resource languages @xcite . this strategy could enhance translation quality in resource-constrained settings. however, despite these gains, all evaluated models still exhibit significant errors, falling short of human-quality translation, which calls for further research. these findings suggest that fine-tuning smaller, specialized models may offer a more cost-effective solution than relying on large llms, and that targeted data creation, informed by common error patterns, could further boost performance. additionally, the discrepancies between automatic and human evaluations emphasize the need for more nuanced evaluation methods for low-resource language translation. future work should focus on iterative improvement techniques such as back-translation, exploring methods to distill knowledge from larger llms to smaller, more deployable models, and creating more diverse and representative parallel datasets for low-resource languages like faroese. ## conclusion our study on english to faroese machine translation offers a nuanced perspective on the effectiveness of different approaches to low-resource language pairs, highlighting how fine-tuned models like nllb can rival or outperform larger llms for low-resource languages. this suggests that focusing on fine-tuning smaller models and creating targeted synthetic datasets may be more effective and resource-efficient. despite improvements, all models still fall short of human-quality translation, emphasizing the need for further research on error patterns, data augmentation, and better evaluation methods. advancing low-resource translation likely calls for a tailored combination of specialized models with effective data augmentation strategies.