Continued Pretraining and Interpretability-Based Evaluation for Low-Resource Languages: A G alician Case Study (2025)

Abstract: Recent advances in Large Language Models (LLMs) have led to remarkable improvements in language understanding and text generation. However, challenges remain in enhancing their performance for underrepresented languages, ensuring continual learning without catastrophic forgetting, and developing robust evaluation methodologies. This work addresses these issues by investigating the impact of Continued Pretraining (CPT) on multilingual models and proposing a comprehensive evaluation framework for LLMs, focusing on the case of Galician language. Our first contribution explores CPT strategies for languages with limited representation in multilingual models. We analyze how CPT with Galician corpora improves text generation while assessing the trade-offs between linguistic enrichment and task-solving capabilities. Our findings show that CPT with small, high-quality corpora and diverse instructions enhances both task performance and linguistic quality. Our second contribution is a structured evaluation framework based on distinguishing task-based and language-based assessments, leveraging existing and newly developed benchmarks for Galician. Additionally, we contribute new Galician LLMs, datasets for evaluation and instructions, and an evaluation framework.

## introduction as researchers continue to push the boundaries of what large language models (llms) can achieve, several key limitations have emerged that deserve further investigation: how to improve the ability of llms to handle underrepresented languages @xcite , how to ensure that they learn new knowledge and abilities without forgetting the knowledge and abilities they already possess @xcite , and how to enhance evaluation so as to measure their generalization capabilities across different tasks. our work addresses these challenges by exploring the impact of continued pretraining (cpt) and proposing a comprehensive framework for evaluating llms. figure 1 : the proposed approach involves three stages: corpus creation using a mix of instructions and raw text, model training via continual and instruction-based pretraining, and evaluation through automatic and human assessments. the first contribution of this work is an investigation into the impact of cpt on llm performance, particularly for languages that are underrepresented in the base multilingual model, such as galician. we will explore different strategies for doing cpt of galician corpora in multilingual base models that have little or no knowledge of galician. we hypothesize that cpt with text in such a minority language improves the ability of the base multilingual model to generate text in that language. however, if instructions are not incorporated into the training corpus during cpt, the model risks losing its ability to perform task-solving activities effectively. by comparing the effects of cpt with and without instructions, we aim to provide insights into balancing linguistic enrichment and task-solving capabilities. the second contribution of our work is a comprehensive framework for evaluating llms, simileval 1 , designed to assess their performance across the three dimensions introduced in @xcite : what to evaluate, how to evaluate, and where to evaluate. for the first question, what to evaluate, we distinguish between two different aspects: the tasks that the model performs and the language that the model generates. task-based evaluation measures the ability of a model to respond to questions, solve problems, and perform activities. complementing this, language-based evaluation assesses the quality of the generated text, emphasizing coherence, fluency, and grammaticality. these two aspects are in line with the distinction made in cognitive sciences between functional linguistic competence, that is, non-languagespecific skills required for real-life language use, and formal linguistic competence, that is, knowledge of linguistic rules and patterns @xcite . in the present work, we will evaluate several llms in galician to examine the relationship between task performance and language quality and determine whether correlations exist between these two aspects. to tackle the second question, how to evaluate, we will analyze and make use of different strategies and metrics, including log-likelihood-based evaluation, which evaluates performance based on the probability assigned to correct answers, similarity-based evaluation, which compares generated outputs against expected responses using metrics such as cosine similarity or semantic overlap, and human-based evaluation. all these 'hows' are essential to have a holistic view of the quality of the evaluated models. finally, for the where to evaluate, we will explore all the benchmarks that are freely available to evaluate llms in galician language. some of them have been developed for the present study, as they were not available until now. by addressing these two main contributions, our work offers insights into the interplay between cpt, instruction pre-training, and evaluation methodologies. more precisely, we found that cpt on small, high-quality corpora provided with diverse instructions achieves competitive performance in task-based evaluations while enhancing linguistic quality in generative tasks. although this study focuses on galician, the proposed methodology (figure 1 ) and evaluation framework are languageagnostic and can be applied to other underrepresented languages with similar resource constraints, providing a general approach to developing more robust and adaptable llms with improved linguistic competence and task-solving capabilities. other relevant specific contributions are the following: i) the development of new generative llms with 8b parameters for the galician language with cpt, ii) three new galician datasets for llm evaluation, iii) synthetic datasets instructions in galician, and iv) an evaluation framework, called simil-eval, with a set of similarity-based metrics. all contributions are released under free and open-source licences. this work is organized as follows. section 2 describes, on the one hand, studies on the relationship between cpt and instructions and, on the other, the characteristics and challenges of the main evaluation frameworks. section 3 introduces the main features of the models we have elaborated with cpt, our similarity-based evaluation framework, and the datasets we have used and elaborated for evaluation along with the human evaluation protocols. then, in section 4, we describe the experiments carried out, show the results and discuss the main findings. we finish in section 5 with some concluding remarks, future work, and limitations. ## related work in this section we will review the most relevant studies on the relationship between cpt and the use of instructions and will analyze different methods to evaluate llms. ## experiments the objective is to evaluate language models with a significant proportion of galician in their training data, focusing exclusively on pretrained models and excluding those with instruction tuning. the selected models vary in size and linguistic scope, from monolingual galician models to multilingual models covering several european languages: cpt models: this group includes our models described in section 3.1; llama-3.1-8b: the base model for cpt training, which is a multilingual model covering eight languages, including spanish and brazilian portuguese; salamandra-2b 10 and salamandra-7b 11 : multilingual models focused on european languages including galician, making them particularly relevant for assessing galician language capabilities; carballo-bloom 12 : a monolingual 1.7b galician model, offering insights into the perfor-mance of a galician-specific model compared to larger multilingual ones. ## conclusions in this work we evaluated the impact of cpt on multilingual language models, comparing their performance using both automatic and human evaluation metrics. larger base multilingual models outperform models that were adapted to a specific language with cpt in task-based evaluation, though cpt models trained with small and high-quality corpora, as well as with diverse instructions, show competitive results in task-based evaluation while they improve the linguistic quality in generative tasks. some discrepancies between automatic and human evaluations indicate the need to refine the current metrics, both those that measure task performance and those that assess linguistic quality. in the current state of evaluation metrics, we must conclude that humans are still necessary to assess the results of llms more fairly.