Extracting General-use Transformers for Low-resource Languages via Knowledge Distillation (2025)

Abstract: In this paper, we propose the use of simple knowledge distillation to produce smaller and more efficient single-language transformers from Massively Multilingual Transformers (MMTs) to alleviate tradeoffs associated with the use of such in low-resource settings. Using Tagalog as a case study, we show that these smaller single-language models perform on-par with strong baselines in a variety of benchmark tasks in a much more efficient manner. Furthermore, we investigate additional steps during the distillation process that improves the soft-supervision of the target language, and provide a number of analyses and ablations to show the efficacy of the proposed method.

## introduction to curb the detrimental effects of pretraining with very little pretraining data in a low-resource language, most works opt to use pretrained massively multilingual transformers (mmts) such as mbert @xcite and mdeberta @xcite instead. however, this comes with a number of tradeoffs. finetuning in only one language causes negative interference in a model that compresses many languages within a limited parameter budget @xcite . this would mean that an mmt, in theory, would perform worse than using a transformer pretrained in one specific language @xcite . additionally, mmts are unnecessarily costly as most researchers who use them are only interested in one language among many -this is most especially the case in low-resource language research communities that also suffer from a lack of computational resources @xcite . 1 code can be found in the following repository: https: //github.com/jcblaisecruz02/nlp805-distillation in this work, we propose the use of simple knowledge distillation to extract robust and efficient single-language pretrained transformers from an mmt. we study a number of intermediate steps that improve the distillation method, such as targetlanguage conditioning and student initialization. we then compare the performance of our extracted models on strong baselines on a variety of benchmark tasks and perform ablations and analyses to pinpoint the sources of strong performance from our simple method. ## results and discussion a summary of the results can be found on we can see that our models perform strongly across the three benchmark tasks. for the hate speech classification and nli tasks, our dbert base model outperforms its teacher mbert as well as the distilled distilmbert version with an almost 2x speedup in terms of training time. this shows that the method, albeit simple, works well to produce general-use transformers for these tasks. performance lags slightly behind on ner, which we assume is a harder task for an extracted model as there are a lot of named entities in the vocabulary from other languages that are not completely removed and present a significant amount of negative interference. we investigate these behaviors further in ablations. the dbert tiny variant showed strong results that came close to the baselines on hate speech classification but lags behind the other models in all other tasks. we hypothesize that this is due to the size of the model not having enough capacity to fully capture the teacher's representation of the target language given that the source representation space is extremely large due to the presence of other languages. unsurprisingly, roberta tagalog performs the best in all three tasks given that it is a full-sized bert-type model that is trained solely in tagalog. the mbert and distilmbert models are likewise strong performers but are much slower during training than the dbert models which has a significant impact on research in low-resource languages where computing is often scarce. overall, this provides empirical evidence that distilling a general-purpose transformer from a larger mmt yields robust results despite the method's relative simplicity. ## related work knowledge distillation is an established tool in modern nlp research, especially after the release of bert in 2018. most works such as distilbert and tinybert @xcite aim to distill the full model while retaining all languages that may be incorporated in the original training data. these models perform well across a number of crosslingual benchmarks such as xnli @xcite , but represent a challenge in real-world use especially for low-resource languages. recent works have begun to use knowledge distillation for smaller, targeted use-case models. wibowo et al. (2024) 2023 ) distills smaller models for the goal of efficiently producing stronger task-based models via further distillation. however, most of these works focus directly on the end task, instead of creating a general-use case student model that is targeted for one language specifically. ## future work the current method provides a strong way to distill a language-specific general-use model from a much larger mmt, while being flexible enough to function as the base for more targeted tasks. for future work, the following may be explored as an augmentation to the current method: extrapolating to an unseen language -much like in bloom+1 (yong et al., 2023), we could explore teacher conditioning to add an unseen language to an existing language model. general purpose llms -moving beyond small pretrained models, we can explore the use of the same method for general purpose multilingual llms such as aya @xcite and bloomz @xcite to see if we can transfer learned instruction-following performance on a language-specific student model. ## conclusion in this work, we present an extremely simple method of extracting general-use language-specific transformers from pretrained mmts that retain the robust performance of the original teacher models. these models and the process of obtaining them are both ideal for research in low-resource languages as both the compute resources and the data available for researchers in these areas are often very scarce. for future work, we present a number of augmentations that can be explored from this relatively simple method, such as unseen language extrapolation, and extension to large language models.