I rish-based Large Language Model with Extreme Low-Resource Settings in Machine Translation (2024)

Abstract: Large Language Models (LLMs) have demonstrated exceptional performances in a wide range of natural language processing tasks. However, their success does not always extend to machine translation, particularly in challenging scenarios such as translating low-resource languages. This study investigates the multilingual capability of LLMs, with a case study on Irish, an extremely low-resource language, focusing on translation tasks between English and Irish. We propose a dynamic, efficient language adaptation framework for English-centric LLMs, which involves layer-specific adjustments and subsequent fine-tuning for machine translation. Our findings highlight several key insights: (1) different layers in the LLM serve distinct functions such as language understanding and task reasoning, (2) effective translation requires extensive pre-training on both source and target languages, and (3) targeted fine-tuning for machine translation leads to significant improvements of 36.7% for English to Irish and 133.4% for Irish to English compared to the previous state-of-the-art.

## introduction large language models (llms) have recently revolutionized the field of natural language processing (nlp), demonstrating remarkable performance across a wide range of tasks. these models, built on the transformer architecture, leverage vast amounts of data to achieve exceptional levels of linguistic understanding. however, significant challenges remain, particularly in the domain of machine translation for low-resource languages @xcite . traditional approaches for neural machine translation (nmt) are often data-inefficient and rely on large numbers of parallel data pairs to obtain reliable performance, limiting their applicability in low-resource * corresponding author tasks @xcite . this paper seeks to explore the multilingual capabilities of llms, specifically focusing on irish, an extremely low-resource language, and the translation tasks between english and irish. irish, classified as an endangered language, poses unique challenges for machine translation. the limited availability of parallel corpora @xcite and the sparse representation in pre-training datasets @xcite make it a vital candidate for investigating the potential of llms in low-resource settings. llms, such as chatgpt @xcite , bloom @xcite , and the llama series @xcite , are predominantly english-centric, although pre-trained on multilingual datasets. the extent to which these models can effectively translate between low-resource languages remains an open question. our research identifies several key insights to successfully apply llm to the low-resource scenario, such as the requirement for the llms to be bilingual through extensive pre-training on both languages. we propose a novel framework for efficiently adapting english-centric llms to a novel unseen language, and further fine-tuning for the task of machine translation. our approach involves a two-stage training process: dynamic continued pre-training, where we selectively train layers of the llm based on their language capability, indicated by retrieval scores, and additional fine-tuning on specific machine translation datasets. by focusing on the layers responsible for language understanding and reasoning, we aim to enhance the bilingual capabilities of the llm while being efficient, requiring only a fraction of the model's total parameters for effective language adaptation. specifically, we achieve an improvement of up to 46.14 bleu score for irish to english translation and 13.22 bleu score for english to irish transla-tion compared to previous state-of-the-art methods on the loresmt-2021 dataset @xcite . our source code and model weights are made publicly available at https://github.com/ reml-ai/uccix for future research and benchmarking purposes. ## conclusion in this work, we investigate the application of llms to the domain of neural machine translation, particularly focusing on irish, an extremely low-resource language. we analyze the bilingual capabilities of llms and propose a dynamic language adaptation strategy aimed at balancing the model's performance across multiple languages. we hypothesize that certain layers of the llm serve as interface layers for language understanding, and reasoning layers, and we develop a novel, efficient training approach that dynamically identifies and trains only the relevant layers. our experimental results demonstrate the effectiveness of our approach, achieving balanced performance between languages. moreover, we show that leveraging large-scale pre-trained llms and further finetuning them on machine translation tasks with limited parallel data can significantly enhance translation performance in resource-constrained scenarios, with performance enhancement up to 46.14 bleu score. this highlights the potential of our method to improve machine translation tasks involving extremely low-resource languages.