DIPL om A : Efficient Adaptation of Instructed LLM s to Low-Resource Languages via Post-Training Delta Merging (2025)

Abstract: This paper investigates how open-weight instruction-tuned large language models (LLMs) can be efficiently adapted to low-resource languages without requiring costly large-scale post-training. We introduce DIPLomA (Decoupled Instruction-Preserving Language Adaptation), a lightweight delta-based transfer strategy that provides a practical and effective solution for this scenario. DIPLomA decouples language adaptation from post-training alignment by first continually pretraining a foundational LLM on a modest amount of monolingual target-language data while anchoring on English replay, and then injecting instruction-following capabilities via delta-based weight merging from the instructed counterpart of the base LLM. We evaluate DIPLomA on Basque and validate its generality on Welsh and Swahili, demonstrating consistent and substantial gains in instruction-following, linguistic proficiency, and safety. Compared to strong baselines, our method achieves average relative improvements of 50 points in Basque, 63 in Welsh, and 51 in Swahili, while preserving the original model’s multilingual performance. These results highlight DIPLomA as an effective, resource-efficient strategy for bringing high-quality instruction alignment to underrepresented languages at scale.

## introduction the development of instruction-following large language models (llms) typically follows a multi-stage pipeline @xcite @xcite . first, language model pre-training is conducted on massive multilingual corpora where the model learns linguistic competence and world knowledge via next-token prediction. second, a post-training stage where the model learns to follow human instructions with several iterations of supervised fine-tuning on curated instruction datasets and reinforcement-based methods to align to human preferences. these stages often integrate additional capabilities such as tool use and implement safety mitigations. while this pipeline has yielded highly capable models with remarkable performance across a wide range of tasks in english and other high-resource languages, it depends critically on access to large-scale, high-quality data and human annotations at each stage-resources that remain scarce or costly for low-resource and underrepresented languages. this creates a barrier to building instruction-aligned models for many of the world's languages. in this work, we propose diploma (decoupled instruction-preserving language adaptation), a scalable and efficient approach to extend instruction-following capabilities of open-weight instructed llms to low-resource languages without the need for any instruction or human preference data, reducing the cost of developing these models. our approach, conceptually aligned with recent methods @xcite , is based on two key observations: (1) continual pre-training on monolingual data significantly enhances a model's fluency and linguistic competence in the target language, and (2) instruction-following and human preference alignment are largely encoded in the delta-the weight differences-between a base pre-trained model and its instruction-tuned counterpart. building on this insight, we first continually pre-train the base llm on monolingual data in a low-resource language to improve its linguistic capacity. then, instead of post-training from scratch, we merge the post-training delta into the language-adapted model via weight merging. this simple yet effective method allows us to transfer not only instruction-following capabilities, but also human preference alignment-such as safety control learned during large-scale post-training into the language-adapted model. we validate our approach on three linguistically diverse low-resource languages: basque, welsh, and swahili, where we conduct a comprehensive suite of evaluations covering instruction-following, linguistic fluency, and safety. we compare diploma against several baselines, including models trained with explicit instruction tuning on translated data. results show that diploma yields substantial gains in instruction-following accuracy, linguistic proficiency and safety for low-resource languages, while maintaining its multilingual capabilities (see figure 1 ). our contributions are as follows: ## related work recent work in multilingual nlp has focused on adapting llms to new languages beyond english. core strategies include continual pre-training on target-language corpora, multilingual instruction tuning and parameter-efficient fine-tuning. continual pre-training enhances llm performance by exposing english-centric models to extensive target-language data. for instance, @xcite improve chinese modeling through tokenizer extension and continued training on chinese corpora. @xcite adapt llama-2 to japanese similarly, preserving english capabilities. @xcite demonstrate that even limited estonian pre-training, followed by instruction tuning, yields significant gains. in basque, @xcite and @xcite show that continual pre-training on large basque corpora leads to substantial improvements. multilingual instruction tuning has become a key component in aligning llms with user intent across languages. this process involves fine-tuning models using instruction-following data in the target language, often by translating existing english datasets. for example, @xcite demonstrate the effectiveness of translating english instructions to estonian and similarly @xcite find that translating both instructions and human preference annotations into basque effectively boosts the instruction-following capabilities of llama-eus. the sambalingo project also shows that even small bilingual instruction datasets can be highly beneficial when paired with tokenizer and vocabulary updates @xcite . although promising, @xcite report that a substantial gap remains between high-resource languages like english and low-resource languages. this disparity is largely attributed to the extensive post-training and alignment stages employed by state-of-the-art models @xcite @xcite , which rely on massive instruction datasets and human preference annotations. while these pipelines are highly effective, reproducing them for low-resource languages is often infeasible due to the limited availability of high-quality instruction data and the significant cost of manual annotation. to address these limitations, recent work has explored whether alignment can be transferred more efficiently through parameter deltas, the differences between an instructed model and its base checkpoint, rather than through costly retraining. huang et al. (2024) introduce the "chat vector" and demonstrate that adding it to a continually-pretrained model imparts conversationality and instruction-following across languages. @xcite also explore delta-transfer for model merging and knowledge fusion. more recently, @xcite proposed param∆, a training-free method for english-centric models that transfers post-training knowledge via parameter deltas. while these approaches demonstrate the potential of delta-based transfer, their improvements are modest and they do not address low-resource languages, which is the focus of our work. diploma is conceptually related to these methods but differs in scope and goals. we systematically investigate the feasibility of delta-based transfer in truly low-resource scenarios, targeting basque, welsh, and swahili with corpora of around 500m words. to stabilize adaptation, we introduce an anchor english corpus during continual pre-training, and we validate the approach across multiple llm families. in addition, we conduct extensive ablations on interpolation and pre-training strategies. finally, our study also provides a comprehensive evaluation by combining automatic benchmarks with manual analysis to assess instruction-following, safety, multilinguality, and linguistic proficiency in underrepresented languages. ## diploma: decoupled instruction-preserving language adaptation our approach, diploma (decoupled instruction-preserving language adaptation), addresses these costs and limitations by transferring instruction-following capabilities and human preference alignment into foundational llms that have already been adapted to low-resource languages. this allows us to bypass the need to replicate large-scale post-training pipelines in each target language while preserving the original model's capabilities. the method consists of two stages: for this method to be applied, two models are required: a base pre-trained model and the instruction-tuned variant of that model. the former provides the foundational linguistic competences and other foundational skills, while the latter captures the instruction-following capabilities and human preference alignment to be transferred. first continual pre-training is performed on monolingual data d mono to adapt the base model to the target language: next, delta weights between the instruction-tuned model and the base foundational model are computed: these delta represent the changes introduced during large-scale post-training. we then inject this capabilities into the language adapted model as: ## evaluation to thoroughly assess the effectiveness of our method, we evaluate performance across four key dimensions relevant to instruction-following llms: ## experimental setup we base our experiments on the llama-3 models @xcite , which are particularly well-suited for our method, as both the foundational llama-3.1-8b and the instruction-tuned llama-3.1-8b-instruct variants are publicly available and released under the llama 3.1 community license agreement. this setup aligns with the requirements of our technique, which necessitates access to the weights of both the base and instruction-tuned models. for basque language adaptation, we leverage the publicly available llama-eus model @xcite , a version of llama-3.1-8b continually pre-trained on high-quality basque corpora. this model was trained on zelaihandi (san vicente et al., 2024), a domain-diverse and freely licensed high-quality corpus specifically curated for llm pre-training in basque. the zelaihandi dataset comprises 531 million words. following the methodology described in section 3, we extract the post-training capabilities of the llama-3.1-8b-instruct model by computing the parameter delta with respect to its foundational counterpart, llama-3.1-8b. this delta encapsulates the changes introduced by instruction tuning and alignment with human preferences. subsection 6.3 includes further details on the values and distribution of this delta. we then apply this delta to the llama-eus model, resulting in a new model that retains strong linguistic competence in basque while acquiring instruction-following abilities-despite never having been explicitly instruction-tuned in this language. ## results we evaluate each system on the dimensions explained in section 4 across a mix of human and automatic evaluations. we also evaluate the multilingual robustness on english. inference details in appendix e. the results in in the instruction-following dimension for basque, diploma achieves a 54 point improvement over the original llama-3.1-instruct and significantly outperforming both baselines fine-tuned with model basque english inst. follow. ling. prof safety inst. follow. safety ## conclusions we presented diploma, a straightforward yet powerful and efficient application of delta-based weight merging for adapting instruction-following llms to low-resource languages without the need for costly, full retraining of the instruction-tuning pipeline. by decoupling linguistic adaptation from instruction alignment, our approach first continually pre-trains a base llm on monolingual target-language data to build strong linguistic competence, then integrates post-training capabilities via parameter delta merging from an english-centric llm. this two-step process effectively preserves alignment and safety properties learned in english, while at the same time notably improving performance in the target language. our extensive experiments on basque, complemented by validation on welsh and swahili-three typologically diverse, underrepresented languages-demonstrate that diploma achieves substantial improvements in instruction-following accuracy, linguistic proficiency, and safety behavior relative to strong baselines and prior methods. crucially, it maintains robust performance in english, thereby helping to avoid catastrophic forgetting and enabling true multilingual instruction capabilities.