Efficient Continual Pre-training of LLM s for Low-resource Languages (2025)

Abstract: Open-source large language models (Os-LLMs) propel the democratization of natural language research by giving the flexibility to augment or update model parameters for performance improvement. Nevertheless, like proprietary LLMs, Os-LLMs offer poorer performance on low-resource languages (LRLs) than high-resource languages (HRLs), owing to smaller amounts of training data and underrepresented vocabulary. On the other hand, continual pre-training (CPT) with large amounts of language-specific data is a costly proposition in terms of data acquisition and computational resources. Our goal is to drastically reduce CPT cost.To that end, we first develop a new algorithm to select a subset of texts from a larger corpus. We show the effectiveness of our technique using very little CPT data. In search of further improvement, we design a new algorithm to select tokens to include in the LLM vocabulary.We experiment with the recent Llama-3 model and nine Indian languages with diverse scripts and extent of resource availability.For evaluation, we use IndicGenBench, a generation task benchmark dataset for Indic languages. We experiment with various CPT corpora and augmented vocabulary size and offer insights across language families.

## introduction large language models (llms) like @xcite , chatgpt, llama-2 @xcite , llama-3 @xcite , palm @xcite , inter alia, are opening up new possibilities for low-resource languages (lrls). until recently, collecting sufficient labeled lrl data to finetune llms for classification and generation tasks used to be challenging. today, llms give decent performance with zero/few-shot inference. having said that, there is still a substantial performance gap between high-resource languages (hrls) and lrls avg. tokens per word odia malayalam kannada santali telugu tamil gujrati assamese punjabi bengali sanskrit bodo marathi konkani urdu hindi english 0 20 40 60 80 llama-3* mistral gemma phi-3 gpt-4o for llms @xcite . this is since lrls like indic languages are still under-represented by recent llms, as shown in figure 1 : compared to english, the average number of tokens required to generate a lrl word by these llms is substantially higher. the inability to represent a word with a single token may lead to suboptimal learning of context thus potentially affecting llm's performance for lrl tasks. a feasible way to overcome such shortcoming is to initiate continual pre-training (cpt), specifically with lrl text. cpt can help llms learn domains/languages that are un/under-explored in the pre-training stage. while this is a viable option to improve llm's performance, training such gigantic models consumes expensive gpu resources and time, which makes it less feasible in resource-constrained setups. to address these issues and harness cpt's potential, we propose a two-pronged approach. first, we introduce a score-based method to select a small set of high-quality, language-specific training data. concurrently, we implement a strategy to expand the token vocabulary in llms. this vocabulary augmentation improves the understanding of important words in low-resource languages, leading to further performance gains. the strategies and the rigorous experiments undertaken are detailed next. ## related work language models use diverse subword tokenization algorithms like byte-pair encoding (bpe) @xcite , sentence-piece @xcite , word-piece @xcite , and unigram @xcite . due to the limited size of an llm's token vocabulary, over-fragmentation @xcite is a common problem, especially for multilingual models where not all languages get equal representation. apart from task performance degradation @xcite , over-fragmentation leads to slow inference @xcite and increased training and inference/generation cost @xcite . various mitigation methods have been proposed, including vocabulary expansion @xcite and replacing existing tokens in the vocabulary with new ones @xcite . in recent work such as chinesellama @xcite and tamilllama @xcite , the authors add new language-specific tokens and then pretrain the model with large amounts of training data. more recently, @xcite and @xcite explore cpt of llms while varying the corpus, additional vocabulary and embedding initialization techniques. however, they do not focus on strategies to select corpus and vocabulary. in contrast, in this work, we propose a global + local joint rank-based system to first select the small-scale training corpus and then augment the llm's vocabulary with additional languagespecific tokens for cpt. with a small amount of informative training data and added vocabulary, we show substantial llm performance improvement for indic languages. ## proposed method in this work, we design a two-stage approach to improve llm's performance with reduced resource requirements. in the first stage, we select a subset of the available lrl corpus, and in the next stage, we select prospective new tokens for vocabulary augmentation. these two together are used for the purpose of cpt as shown in figure 2 . as the figure shows, the second stage (token selection) is optional. section 3.1 and 3.2 describe the corpus and vocabulary selection algorithms, respectively. algorithm 1 corpus selection for cpt. ## experiment and results to check the effectiveness of our two-stage cpt method, we use indicgenbench @xcite , a generation task benchmark dataset for indic languages covering cross-lingual summarization, machine translation (mt) and question-answering (qa) tasks (see figure 4 in the appendix for dataset overview). for mt and qa tasks, there are two variants: one where the target language is one of the indic languages (flores(@xmath0, xorqa(xx)), and the other where the target language is english (flores(@xmath1, xorqa(en)). for summarization, it is only from english to indic languages (crosssum). we experiment with nine indic languages covering six (devanagari, bengali, arabic, telugu, olchiki, gurumukhi) different scripts and three (high/mid/low) types of resource availability as described in the existing work @xcite and use the llama-3-8b parameter model as our base llm. we perform all our experiments in zero-shot set- ## conclusion this work proposes a technique to select a compact cpt corpus and a method to augment the llm vocabulary with a small set of new tokens. experiments on indicgenbench, covering nine indian languages with diverse scripts and resources, show that a small cpt corpus improves performance, with additional gains possible through limited vocabulary augmentation. however, improvements vary by script, and excessive token addition or a larger cpt corpus may not always help. we also observed that language-specific cpt can negatively impact english generation. our findings offer valuable insights for leveraging llms in lrls.