Adapting Multilingual LLM s to Low-Resource Languages using Continued Pre-training and Synthetic Corpus: A Case Study for H indi LLM s (2025)

Abstract: Multilingual LLMs support a variety of languages; however, their performance is suboptimal for low-resource languages. In this work, we emphasize the importance of continued pre-training of multilingual LLMs and the use of translation-based synthetic pre-training corpora for improving LLMs in low-resource languages. We conduct our study in the context of the low-resource Indic language Hindi. We introduce Nemotron-Mini-Hindi 4B, a bilingual SLM supporting both Hindi and English, based on Nemotron-Mini 4B. The model is trained using a mix of real and synthetic Hindi + English tokens, with continuous pre-training performed on 400B tokens. We demonstrate that both the base and instruct models achieve state-of-the-art results on Hindi benchmarks while remaining competitive on English tasks. Additionally, we observe that the continued pre-training approach enhances the model’s overall factual accuracy.

## introduction the accuracy and utility of large language models (llms) have continuously improved over time. both closed and open-source llms have demonstrated strong performance in english and several other languages. open models such as nemotron @xcite , gemma @xcite , and llama @xcite are inherently multilingual. for instance, the nemotron-4 15b model was pre-trained on 8 trillion tokens, of which 15% were multilingual @xcite . however, the proportion of multilingual data is limited, which in turn affects the accuracy of these models on non-english languages. the model's performance further diminishes as we move from high-resource to low-resource languages. in this work, we specifically focus on the indic language hindi as our target low-resource language. out of the 8 trillion tokens used to train the nemotron-4 models, only 20 billion tokens are in hindi. as a result, while the model can understand and generate hindi content to a reasonable extent, the usability of such a multilingual llm for specific low-resource languages remains questionable. frequent hallucinations, meaningless sentences, and mixing of english content often occur when responding to purely hindi queries in the devanagari script. there is a strong need to adapt multilingual llms to target languages to enhance their usability. recently, in the context of indic languages, target language supervised fine-tuning (sft) has become a common practice to adapt llms to specific languages @xcite . however, it remains to be studied whether language-specific sft tuning improves llms' understanding in regional contexts. some studies suggest that sft can introduce llms to new domain knowledge, though it is typically used to enhance the model's instructionfollowing capability @xcite . sft on translated english instruction tuning data is widely used to develop regional llms for indic languages. while this may improve instructionfollowing in the target language, it may not enhance llms' understanding of regional contexts @xcite . another approach to updating llm knowledge is continued pre-training, but the limited availability of tokens for low-resource languages makes this both infeasible and prone to overfitting. in this work, we focus on a continued pretraining approach using a mix of real and synthetic corpora. we demonstrate that a robust base model can be adapted to the target language with a small continued pre-training corpus. this approach is particularly relevant for low-resource languages, where the amount of training data is limited. the synthetic pre-training dataset is curated by translating high-quality generic english corpora into the target language. to further expand the corpus and support roman script queries in the target language, the text is transliterated into roman script and used for pre-training. the base model is then aligned using supervised fine-tuning (sft), followed by preference tuning with direct preference optimization (dpo). we observe that the continued pre-training approach is particularly useful for reducing hallucinations, improving regional knowledge of llms, and enhancing response capabilities in the target language. the high-level process is outlined in figure 1 , based on this approach, we present nemotron-mini-hindi-4b-base 1 and nemotron-mini-hindi-4b-instruct 23 , state-of-the-art small language models (slms) for the hindi language. these slms support hindi, english, and hinglish. the hindi models are based on the multilingual nemotron-mini-4b (also known as minitron-4b), adapted with continued pre-training on 400 billion hindi and english tokens. the data blend used equal proportions of both languages. the instruct version of the model was developed using sft and dpo techniques. the model outperforms all similarly sized models on various indicxtreme, indicnlg benchmark tasks and popular translated english benchmarks such as mmlu, hellaswag, arc-c, and arc-e @xcite . we also perform llm-based evaluations using the benchmark datasets indicquest @xcite and in-house subjectiveeval, with gpt-4 serving as the judge llm. this is the first study to present and 1 https://huggingface.co/nvidia/ nemotron-4-mini-hindi-4b-base 2 https://huggingface.co/nvidia/ nemotron-4-mini-hindi-4b-instruct 3 https://build.nvidia.com/nvidia/ nemotron-4-mini-hindi-4b-instruct evaluate bilingual language models of this nature. we provide a thorough study of the models in both languages. ## related work in this section, we review various approaches for adapting llms to different languages. several efforts have focused on adapting llama models to indic languages. a common method involves extending the vocabulary, followed by sft or peft (lora) using translated and available sft corpora in indic languages. examples of such work include openhathi, airavata @xcite , tamil-llama @xcite , navarasa foot_0 , ambari, malayallm, and marathi-gemma @xcite . notably, some of these efforts employ bilingual next-word prediction, alternating between english and the target language in the pre-training corpus. airavata also introduced an evaluation framework foot_1 for indic llms, which we leverage to evaluate nemotron-mini-hindi 4b and other multilingual models. apart from indic languages, similar efforts have been made for other languages, including chinese llama @xcite , llamaturk (toraman, 2024), @xcite , and redwhale @xcite for chinese, turkish, finnish, and korean, respectively. these llms use one or more techniques such as tokenizer extension, secondary pretraining, and supervised finetuning. the key distinction of our work lies in its emphasis on developing bilingual llms, whereas the aforementioned efforts concentrate on creating monolingual llms. @xcite show that large language models can learn low-resource languages effectively using in-context learning and few-shot examples, improving performance through crosslingual contexts without extensive tuning. @xcite enhance multilingual llms for low-resource languages by using adapters with data from conceptnet, boosting performance in sentiment analysis and named entity recognition. ## methodology in this section, we describe our methodology for adapting multilingual llms to target languages to improve performance in those languages. specifically, we build a bilingual slm that supports both hindi and english. we conduct our adaptation experiments using the multilingual nemotron-mini-4b model (also known as minitron-4b). the model undergoes continuous pre-training with an equal mixture of hindi and english data, consisting of 200b tokens per language. the original nemotron-4b model was primarily trained on english tokens and had seen only 20b hindi tokens. given the limited amount of hindi data, adapting an existing multilingual model rather than training from scratch is an effective strategy, allowing us to leverage the knowledge learned from the pre-trained model. additionally, as nemotron-4b employs a large 256k tokenizer, we did not need to extend the tokenizer. the fertility ratio for hindi text is 1.7, which is better than that of its llama (2.64) and gemma (1.98) counterparts. ## results and discussion we observe similar results with the instruct model on indicxtreme, indicnlg, and translated english benchmarks. the results are presented in ## conclusion we present nemotron-mini-hindi-4b-base and nemotron-mini-hindi-4b-instruct, state-of-the-art slms primarily designed for the hindi language. these models have been continuously pre-trained and aligned using a combination of hindi and english data. the hindi corpus includes both real and synthetic data, with the synthetic data generated through translation. the models outperform similarly sized models on various hindi benchmarks, as assessed through reference-based and llm-as-ajudge evaluations. they also perform competitively on english benchmarks. we emphasize the importance of pre-training to reduce hallucinations and enhance the factuality of the models.