Encoder-Aware Sequence-Level Knowledge Distillation for Low-Resource Neural Machine Translation (2025)

Abstract: Domain adaptation in Neural Machine Translation (NMT) is commonly achieved through fine-tuning, but this approach becomes inefficient as the number of domains increases. Knowledge distillation (KD) provides a scalable alternative by training a compact model on distilled data from a larger model. However, we hypothesize that vanilla sequence-level KD primarily distills the decoder while neglecting encoder knowledge, leading to suboptimal knowledge transfer and limiting its effectiveness in low-resource settings, where both data and computational resources are constrained. To address this, we propose an improved sequence-level KD method that enhances encoder knowledge transfer through a cosine-based alignment loss. Our approach first trains a large model on a mixed-domain dataset and generates a Distilled Mixed Dataset (DMD). A small model is then trained on this dataset via sequence-level KD with encoder alignment. Experiments in a low-resource setting validate our hypothesis, demonstrating that our approach outperforms vanilla sequence-level KD, improves generalization to out-of-domain data, and facilitates efficient domain adaptation while reducing model size and computational cost.

## introduction domain adaptation in neural machine translation (nmt) has been extensively studied @xcite , but significant gaps remain, particularly for low-resource languages @xcite . a common approach involves fine-tuning generaldomain pre-trained models on specific target domains @xcite . while effective, this approach becomes cumbersome as the number of domains increases, requiring separate models for each domain. this not only increases the space complexity of the machine translation (mt) system but also incurs higher maintenance and system costs. moreover, it fails to exploit the shared information inherent across domains, limiting its efficiency. to address these challenges, research has increasingly focused on developing a single model capable of handling multiple domains @xcite @xcite . one promising direction is knowledge distillation (kd) @xcite , where domain-specific teacher models (expert models), typically deep neural networks, generate distilled target-side data for each domain. @xcite demonstrated that a student model, usually a shallower neural network, can be trained on a mixture of the original and distilled data, effectively capturing domain knowledge in a compact form. this data-centric approach not only reduces the need for maintaining multiple models but also leverages domain similarities, making it both effective and easy to adopt. however, kd-based domain adaptation methods may not generalize well to low-resource languages, as training domain-specific teacher models with limited data is inherently suboptimal. furthermore, existing approaches heavily rely on sequence-level distillation @xcite , but its effectiveness in low-resource settings remains unclear. we hypothesize that sequence-level kd primarily distills the decoder while transferring limited encoder knowledge, leading to suboptimal knowledge transfer in encoder-decoder architectures @xcite @xcite . if this hypothesis holds, overlooking encoder knowledge may hinder the adaptability of distilled models across multiple domains. to address this, we propose a cosinebased alignment between the teacher and student encoders to enhance the effectiveness of distillation. building on the methodologies of @xcite and @xcite , we propose a mixeddomain kd approach for low-resource nmt. @xcite demonstrated that training a single model on a mixture of domains can facilitate domain adaptation in low-resource scenarios. inspired by this, we first train a single teacher model (with six encoder and decoder layers) on a mixeddomain dataset. the teacher then generates distilled target text by decoding the source text from the original dataset. combining these distilled sentence pairs with the original dataset, we construct a distilled mixed dataset (dmd), following the approach of @xcite . using this dmd, we train a randomly initialized student model (with three encoder and decoder layers) through sequence-level distillation, augmented with our proposed teacher-student encoder alignment (stage 1). in stage 2, we fine-tune the distilled models on two new domains to assess their adaptability. our contributions are as follows: ## results and discussion to evaluate the effectiveness of our proposed approach, we conduct experiments in two distinct settings. the primary evaluation is carried out on the german-english language pair in a simulated low-resource setting ( §5.1), where we have access to multiple domains and can systematically control the amount of training data. this setup allows us to isolate and analyze the core contributions of our method under controlled conditions. additionally, we conduct an ablation study on the english-sinhala language pair to examine the effect of the weighting parameter α under a bona fide low-resource scenario with limited parallel data and minimal domain diversity ( §5.2). this analysis helps assess the robustness of our method when applied to real-world low-resource language settings. ## conclusion our experiments validate the hypothesis that sequence-level distillation primarily distills the decoder, potentially leading to suboptimal knowledge transfer, and demonstrate the efficacy of our proposed methodology in addressing this limitation. we show that vanilla sequence-level distillation often produces underperforming models, making it unsuitable for resource-constrained settings involving low-resource languages and compute-poor environments. in contrast, models trained using our approach exhibit superior generalization to out-ofdomain data and demonstrate enhanced capabilities as domain adapters. by conducting all experiments on a single machine under constrained resources, we highlight the practicality of our method and aim to inspire fur-ther research on domain adaptation through knowledge distillation, even in resource-limited settings. our findings underline the potential of knowledge distillation as a viable strategy for achieving effective multi-domain adaptation. furthermore, our ablation study on the alignment loss weighting (α) in the english-sinhala direction reveals that the optimal setting is dataset-dependent, with each language direction benefiting from a different choice of α.