Sub-1 B Language Models for Low-Resource Languages: Training Strategies and Insights for B asque (2025)

Abstract: This work investigates the effectiveness of small autoregressive language models (SLMs) with up to one billion parameters (sub-1B) for natural language processing (NLP) tasks in low-resource languages, focusing on Basque. We analyze optimal training strategies by comparing training from scratch and continual pre-training using state-of-the-art SLM architectures. Our analysis considers factors such as model size and the extent of Basque presence in the pre-training corpus. To assess linguistic capabilities, models are evaluated on 12 NLP tasks using the Harness framework. We also conduct a manual evaluation of fine-tuned models on three downstream natural language generation (NLG) tasks: question answering (QA), summarization, and machine translation (MT). Our findings indicate that continual pre-training on a multilingual SLM substantially enhances linguistic performance compared to training from scratch, particularly in low-resource language settings where available corpora typically contain fewer than one billion words. Additionally, the presence of Basque during the pre-training and larger model sizes contribute positively to performance in NLG tasks.

## introduction in recent years, we have witnessed a growing interest in small language models (slms) that can run efficiently on-device with low energy and memory consumption, as well as fast response times, such as mobillama @xcite , openelm @xcite or smollm2 @xcite . leading research labs are also releasing smaller versions of their flagship models, namely llama3.2 1b @xcite , deepseek-r1 1.5b @xcite and qwen3-5 0.6b @xcite , to reach users and use cases with computational constraints. this work focuses on slms with up to one billion parameters (sub-1b), specifically exploring their effectiveness for performing nlp tasks in low-resource languages, which struggle to collect over 1b word datasets foot_0 , with basque (see appendix a for details about the language) as a primary case study. we aim to identify the most effective training strategy for these models. to address this, we investigate several key aspects of the training process using state-of-the-art slm architectures @xcite : is it more efficient to train models from scratch, or should we start with pre-trained models from other languages? does prior exposure to the target language during pre-training provide any advantages? and how does model size influence performance? the results of our experiments show the following (see summary in figure 1 ): as part of the experimentation, the first sub-1b models foot_2 for basque have been created, along with fine-tuned versions for qa, summarization, and mt tasks. furthermore, two new basque datasets have been developed for qa and summarization tasks, namely closebookqa-eu foot_3 and samsum-eu foot_4 . ## model training we selected two competitive english-centric models: openelm @xcite and llama3.2 @xcite . specifically, we included openelm-270m, openelm-1b, and llama3.2-1b. this selection enables comparisons between two model sizes (270m vs 1b) and across models with varying levels of exposure to basque during the pre-training phase (see to determine the most effective training strategy for sub-1b slms, we explored two approaches: training models from scratch @xcite and continual pre-training of multilingual models @xcite @xcite @xcite . from-scratch models -applied exclusively to the openelm architecture-were pre-trained on zelaihandi @xcite , the largest freely available basque text corpus, comprising 521 million words. continual models were trained using an 80-20 mix of zelaihandi and a fineweb @xcite subset, following prior works @xcite @xcite to avoid catastrophic forgetting, as english results reported in appendix f indicate. for full pre-training detail see appendix c. continual pre-trained models retained their base model's tokenizer, while from-scratch models used a native 32k llama3 tokenizer trained on zelaihandi, resulting in a 50% reduction in the tokens-per-word ratio and a tokenization scheme more closely aligned with the morphological structure of basque (for a more in-depth analysis of the tokenizers see section 4.3). ## analysis of language priors in slms in this section, we examine the extent to which the models described in section 2 were exposed to basque during pre-training. to this end, we compare the base and continual 1b versions of llama-3.2-1b and openelm-1b. following previous work @xcite , we sampled 512 generations per model using only the beginning-of-sequence token as input. to examine how language priors shift with stronger language cues, we also assessed model generations when provided with partial prompts of varying lengths in basque foot_5 . each generation had a maximum length of 300 tokens and was produced with a temperature of 1.0. the primary language of each generated sequence was determined automatically using fasttext @xcite . ## evaluation we conducted an intrinsic evaluation of the linguistic competences of both from-scratch and continual pre-trained models and compared them to the original model using the harness evaluation framework. while harness offers an automatic and cost-effective method for assessing the potential linguistic performance of slms, it does not fully reflect real-world performance, as the scores are based on the system selecting the most appropriate answers from multiple-choice questions. to better evaluate the models in realistic settings, we also fine-tuned and manually evaluated them on downstream nlg tasks. in addition, we explore how native tokenizers contribute to more efficient and linguistically aligned from-scratch models. ## conclusions this work examines the effectiveness of slms with up to 1b parameters for nlp tasks in low-resource languages, focusing on basque. our findings show that continual pre-training of a multilingual slm notably improves performance compared to training from scratch, with larger model sizes and the presence of basque during pre-training further enhancing the performance on nlg tasks.