Pipeline Analysis for Developing Instruct LLM s in Low-Resource Languages: A Case Study on B asque (2025)

Abstract: Large language models (LLMs) are typically optimized for resource-rich languages like English, exacerbating the gap between high-resource and underrepresented languages. This work presents a detailed analysis of strategies for developing a model capable of following instructions in a low-resource language, specifically Basque, by focusing on three key stages: pre-training, instruction tuning, and alignment with human preferences. Our findings demonstrate that continual pre-training with a high-quality Basque corpus of around 600 million words improves natural language understanding (NLU) of the foundational model by over 12 points. Moreover, instruction tuning and human preference alignment using automatically translated datasets proved highly effective, resulting in a 24-point improvement in instruction-following performance. The resulting models, Llama-eus-8B and Llama-eus-8B-instruct, establish a new state-of-the-art for Basque in the sub-10B parameter category.

## introduction large language models (llms) have revolutionized the field of natural language processing (nlp), significantly advancing the state of the art in a wide range of tasks, from language generation to language understanding. models such gpt-4 @xcite have had a particularly profound impact, showcasing the capabilities of llms in realworld applications, showcasing their broad utility across various domains. however, the proprietary nature of these models makes them impractical for many researchers and developers. in response, the open-source community has risen to the challenge, developing models that closely rival their proprietary counterparts, such as llama-3 @xcite , mixtral @xcite , qwen2 @xcite or gemma-2 @xcite . however, despite these advances, most of these models remain primarily op-figure 1 : comparison of basque performance between our llama-eus models and llama-3.1 baselines. this includes the foundational models' average accuracies on nlu tasks (see section 3) and the instruction-following accuracies of instructed models (see sections 4 and 5). for the instructed models, lighter colors denote partially correct responses while represent correct answers. open-source models, however, offer a unique opportunity to bridge this gap. by leveraging techniques such as transfer learning, it is now feasible to adapt llms originally trained on large, predominantly english datasets, and fine-tune them for underrepresented languages using smaller, specialized datasets @xcite @xcite . this approach opens the door for developing high-quality llms for languages with limited computational resources, ensuring more equitable access to these transformative technologies. the development of instructed llms typically involves three main stages: (a) pre-training a foundational llm, (b) instructing the llm, and (c) aligning the model to user preferences. in this paper, we propose strategies to address each of these stages tailored to low-resource languages, and we evaluate the importance of each step in the development of an llm for such languages. for the purpose of this work, we focus on basque, a low-resource language as one with minimal representation in the foundational llm, lacking a sufficiently large and high-quality corpus, and with insufficient data to fully execute the instruction and alignment phases. we conduct experiments using models with fewer than 10 billion parameters, specifically llama-3.1-8b, to focus on computationally lightweight models. the main contributions of this work are as follows: ## previous work adapting generative llms to other languages has gained significant attention in recent years, with various strategies explored to improve model performance while maintaining computational efficiency. broadly, the approaches address two main stages: continual pre-training with target language data and instruction tuning to align llm to user preferences @xcite . continual pre-training involves further training an llm on a large corpus of target language textual data. this approach has consistently demonstrated success in boosting language comprehension and generation capabilities @xcite @xcite . for instance, @xcite report significant improvement in their chinese-llama over the original llama's @xcite ability to understand and generate chinese content by further pre-training it on extensive chinese data. mala-500 @xcite is another notable initiative for adapting llms to low-resource languages. it utilizes continual pre-training on llama 2 with the glot500-c dataset @xcite to support 534 languages. one major challenge in adapting llms to new languages is catastrophic forgetting, where a model loses prior knowledge during new language training @xcite . to address this, recent studies have incorporated a portion of the original english corpus alongside the target language data during training, as seen in efforts to adapt models to japanese @xcite and estonian @xcite . in the context of basque, the latxa @xcite family of foundational models,which ranges from 7 to 70 billion parameters and is based on llama 2, represents a significant advancement in adapting llms to the basque language. through continual pre-training on a specialized basque corpus, these models have achieved substantial improvements in processing basque text. ## adapting a foundational model to basque in this section, we focus on adapting a foundational english-centric model to basque by continual pretraining with high-quality basque data. continual pre-training is a crucial technique for adapting large language models to new languages, where the goal is to incrementally refine the model's capabilities for the new language. ## instruction tuning the model in basque in this section, we present experiments aimed at enabling a foundational llm to follow instructions in basque. our focus is twofold: first, to assess whether using a foundational model specifically adapted to basque, as described in the previous chapter, offers any advantages; and second, to evaluate the feasibility of leveraging english instruction datasets translated into basque via automatic translation for the instruction fine-tuning process. ## alignment to human preferences in basque in this section, we present experiments focused on adapting instruction tuned models for basque to align with human preferences, with the goal of improving their ability to generate answers in basque. similar to the experiments described in section 4, we address two key aspects: first, the feasibility of achieving alignment using preference sets in basque, generated through the automatic translation of english datasets; and second, the potential benefits of adapting an instructed model specifically to basque, compared to using a general instructed model without explicit adaptation to the language. several algorithms have been developed to improve the alignment of language model responses with human preferences. among these, direct preference optimization (dpo) @xcite has emerged as a particularly promising approach due to its simplicity and effectiveness. dpo algorithm requires a dataset consisting of paired samples of "preferred" and "rejected" reponses to a given prompt. dpo leverages these preference pairs to directly optimize the model's output generation, focusing on improving the likelihood of the preferred responses compared to the rejected ones. unlike traditional reinforcement learning methods like ppo @xcite , dpo bypasses the need for reward models, simplifying the training process by operating directly on these preference rankings. consequently, we have chosen to implement this algorithm in our study. ## conclusions this work provides a comprehensive analysis of strategies for developing a model capable of following instructions in a low-resource language such as basque, focusing on three key stages: pre-training, instruction tuning, and alignment with human preferences. the results demonstrate that tailoring each of these stages to the target language significantly enhances model performance compared to baseline english-centric models based on llama 3.1 8b. in the pre-training stage, it has been shown that continual pre-training of a foundational englishcentric llm with a high-quality corpus of fewer than 1 billion words in the low-resource language can yield an average improvement of over 12 points in natural language understanding (nlu) tasks. in the instruction tuning and human preference alignment stages, automatic translation of english datasets proved effective for training models to follow instructions in basque, surpassing the instructed llama 3.1 8b baseline. using basqueadapted models as the training base further enhanced performance. following instruction tuning, the application of dpo with translated preference datasets improved model accuracy by up to 7 points. the experimental results establish llama-eus-8b as the most suitable foundational model for basque within its parameter scale, and llama-eus-8binstruct as the first model specifically instructiontuned for basque, achieving the highest performance in this language. however, despite these advances, the performance of both models in basque still lags behind llama 3.1's performance in english, underscoring a substantial performance gap and indicating considerable room for further enhancement. especially llama-eus-8b-instruct, still require further refinement to become competitive in real-world production environments.