KDA : Knowledge Distillation Adapter for Cross-Lingual Transfer (2025)

Abstract: State-of-the-art cross-lingual transfer often relies on massive multilingual models, but their prohibitive size and computational cost limit their practicality for low-resource languages. An alternative is to adapt powerful, task-specialized monolingual models, but this presents challenges in bridging the vocabulary and structural gaps between languages. To address this, we propose KDA, a Knowledge Distillation Adapter framework that efficiently adapts a fine-tuned, high-resource monolingual model to a low-resource target language. KDA utilizes knowledge distillation to transfer the source model’s task-solving capabilities to the target language in a parameter-efficient manner. In addition, we introduce a novel adapter architecture that integrates source-language token embeddings while learning new positional embeddings, directly mitigating cross-lingual representational mismatches. Our empirical results on zero-shot transfer for Vietnamese Sentiment Analysis demonstrate that KDA significantly outperforms existing methods, offering a new, effective, and computationally efficient pathway for cross-lingual transfer.

## introduction cross-lingual transfer (clt) is a critical subfield of natural language processing (nlp) dedicated to leveraging knowledge from high-resource languages, typically english, to perform tasks in lowresource languages. the primary goal is to circumvent the expensive data annotation process required for each new language. the dominant and most successful paradigm for clt has been large-scale multilingual pre-training. although these models naturally develop some degree of unified multilingual representations (pires et al., 2019; conneau et al., 1 the adapter weights are publicly available at https:// huggingface.co/haiimphuong/kda-roberta-twitter 2020; @xcite , a dedicated line of work has focused on further adapting them to languages with different scripts or morphological structures not well-represented in the shared vocabulary, using methods like language-specific adapters @xcite @xcite . despite their effectiveness, these approaches are all constrained by a core limitation of the multilingual backbone: their massive parameter count leads to prohibitive computational costs, creating a substantial barrier for many researchers and practitioners. these challenges motivate the exploration of more flexible, resource-efficient alternatives, leading to a compelling research question: can we achieve effective clt without relying on a massive, pre-trained multilingual model? a promising avenue is to adapt high-performing, readily available monolingual models. prevailing approaches in this area include vocabulary adaptation, which modifies a model to use a new vocabulary @xcite @xcite , and representation alignment methods like monolingual embedding transfer @xcite @xcite . however, these methods share a common shortcoming: they are not directly optimized for the final task in the target language. instead, the adaptation phase optimizes a general objective such as masked language modeling, lexicon mapping, or an auxiliary alignment loss. even when a downstream task loss is used @xcite , direct supervision is only applied on the source language data. consequently, these methods primarily endow the model with a general cross-lingual ability, rather than tailoring it for optimal performance on a specific end task. to address this gap, we propose kda, a novel knowledge distillation adapter framework for direct, task-specific cross-lingual transfer. kda transfers knowledge from a high-performing source-language teacher model to a student model that retains the same architecture but incorporates a new target-language embedding layer and a lightweight adapter, while reusing all other pretrained components. the distillation is further guided by a small parallel corpus to align crosslingual representations effectively. as illustrated in figure 1 , only the adapter's parameters are updated during training. the adapter is optimized to align the student's output with the teacher's. specifically, for a given source sentence fed to the teacher, the adapter learns to make the student produce an identical task-specific output distribution when given the corresponding target sentence. this approach efficiently adapts the model to the new language and task by updating only a small fraction of its total parameters. to validate our approach, we demonstrate the effectiveness of kda on a cross-lingual sentiment analysis task. specifically, we transfer knowledge from a fine-tuned english sentiment model to perform sentiment analysis in vietnamese without requiring any annotated vietnamese data. our experiments show that kda outperforms both large multilingual models and recent monolingual adaptation methods. notably, kda achieves this superior performance while using a smaller backbone language model, highlighting the efficiency and effectiveness of optimizing directly on the downstream task in the target language. ## results and discussion the comprehensive performance of our kda method in comparison to all baselines is summarized in ## conclusion in this work, we propose kda, a novel and parameter-efficient framework for cross-lingual transfer that enables the use of monolingual pretrained models in new target languages without requiring large multilingual backbones or extensive cross-lingual resources. by combining a knowledge distillation process with a novel, embedding-aware adapter architecture, kda offers a parameter-efficient pathway for adapting high-resource models to low-resource languages. through comprehensive experiments on vietnamese sentiment analysis, kda demonstrates substantial improvements over multilingual fine-tuning and translation-based baselines, achieving competitive performance with a fraction of the trainable parameters. these results underscore the novelty and practicality of kda as a scalable solution for low-resource language adaptation. flecting practical computational constraints. while this setup demonstrates the efficiency and effectiveness of kda in resource-constrained environments, further evaluation on larger-scale models remains an important avenue for future research. such experiments may provide deeper insights into the scalability and upper-bound performance of the framework. additionally, kda is currently designed to operate in a per-language-pair setting, requiring a dedicated adapter for each source-target pair. this design introduces a trade-off between scalability and task specialization. although less scalable than approaches that fine-tune a single multilingual model across multiple languages, kda offers a more focused and optimized solution for specific transfer directions. this aligns with real-world scenarios where maximizing performance for a particular low-resource language is the primary goal.