Exploring Very Low-Resource Translation with LLM s: The U niversity of E dinburgh’s Submission to A mericas NLP 2024 Translation Task (2024)

Abstract: This paper describes the University of Edinburgh’s submission to the AmericasNLP 2024 shared task on the translation of Spanish into 11 indigenous American languages. We explore the ability of multilingual Large Language Models (LLMs) to model low-resource languages by continued pre-training with LoRA, and conduct instruction fine-tuning using a variety of datasets, demonstrating that this improves LLM performance. Furthermore, we demonstrate the efficacy of checkpoint averaging alongside decoding techniques like beam search and sampling, resulting in further improvements. We participate in all 11 translation directions.

## introduction we participated in americasnlp 2024's shared task on machine translation (mt). it requires participants to translate from spanish to 11 indigenous american languages: aymara (aym), bribri (bzd), ashaninka (cni), chatino (ctp), guarani (gn), huichol (hch), nahuatl (nhe), otomi (ote), quechua (quy), shipibo-konibo (shp) and tarahumara (tar). we adopted multilingual large language models (llms) and our workflow consists of data curation, continued pre-training, instruction tuning, and several decoding techniques. we submitted to all 11 translation directions. we study and report the feasibility of using llms for very low-resource machine translation tasks. llms have recently been the focus of recent research interest, and in machine translation, they have demonstrated competitive or better performance against traditional neural mt systems in high-resource languages @xcite @xcite . nonetheless, research has shown that these * denotes equal contribution models struggle in low-resource settings if used off-the-shelf @xcite , and there has been limited exploration of adapting llms to extremely low-resource mt. existing approaches rely on massively multilingual dictionaries @xcite or a series of complex grammatical and linguistic tools @xcite . despite their effectiveness, a pitfall of these approaches is that it can be hard to scale them up to build multilingual, low-resource llms. moreover, it is unclear how the (scarce) monolingual and parallel data available for these languages can be effectively utilised, and how recent developments in mt of high-resource languages @xcite scale to very low-resource settings. this work attempts to take a step towards answering these questions. we build multilingual llms for these indigenous american languages by finetuning llama-2 7b @xcite , mistral 7b @xcite and mala-500 @xcite . we explore continued pre-training with lora on various monolingual and parallel data sources. we then conduct instruction tuning using a variety of tasks and language pairs, and show this contributes to performance improvements in mt. we end this work by demonstrating how familiar techniques such as checkpoint averaging, beam search, and sampling help boost llm performance for low-resourced translation as well. ## approach to adapt llms for the task of translating indigenous american languages, we follow the 2-stage training paradigm proposed in related work @xcite and explore its effectiveness for low-resource languages.