A Self-Distillation Recipe for Neural Machine Translation (2025)

Abstract: Self-distillation distills the deeper sub-networks to the shallower sub-networks without using an extra teacher model, and has been proven effective in improving the performance of a series of computer vision tasks. In this paper, we study the representation-based self-distillation methods for Neural Machine Translation (NMT) considering the efficiency issue with a large vocabulary. We present a rank-order augmented Pearson correlation loss and an iterative distillation method to prevent the discrepancy of predictions between the student and a stronger teacher from disturbing the training. To prevent the teacher from misleading the student’s learning, we utilize a warm-up strategy and present a gradient adaption method to scale down or zero the Knowledge Distillation (KD) gradients which are opposite to the translation. Experiments show that our method can lead to significant improvements over the strong Transformer baseline on low/middle/high-resource tasks, obtaining comparable performance to previous MT KD studies without pre-training a teacher. Deeper Transformer experiments show that our method can lead to comparable or better performance with fewer layers.

## introduction the transformer translation model @xcite has greatly improved the performance of neural machine translation (nmt) @xcite @xcite , especially when empowered by deeper structures @xcite @xcite @xcite @xcite @xcite @xcite @xcite . knowledge distillation (kd) @xcite that first trains a teacher model and then transfers the teacher's knowledge to the student network by mimicking the output of the teacher, is among the most promising solutions to ensure the efficiency and to improve the performance of machine translation @xcite @xcite @xcite @xcite . self-distillation @xcite performs knowledge distillation inside the same neural network, normally distilling deeper sub-networks to shallower ones. it only requires one-stage training and does not need to search for the most proper teacher model for knowledge distillation, thus can significantly reduce training overhead and facilitate efficient and effective knowledge distillation for computer vision tasks. @xcite show that self-distillation can lead to strong results without an additional teacher model on a number of computer vision tasks. in this paper, we study self-distillation methods for neural machine translation. to address the efficiency issue with a large vocabulary in neural machine translation, we distill deeper representations to shallower layers. we present a rank-order augmented pearson correlation loss and an iterative distillation method to prevent potential training disturbance due to the performance discrepancy between the students and the stronger teacher @xcite @xcite , and a gradient adaptation method together with a warm-up strategy to avoid possible misleading during knowledge distillation. our main contributions are as follows: • we study efficient self-distillation methods for neural machine translation, present an iterative distillation method to reduce the capacity gap between teachers and students, propose a gradient adaptation method and adopt a warm-up strategy to prevent potential negative impacts of knowledge distillation on machine translation, and present a rank-order augmented pearson correlation loss to further ensure the effectiveness of representationbased knowledge distillation. ## our method as nmt usually has a large vocabulary size, which leads to high computation and memory costs to distill between prediction probability distributions, we study efficient self-distillation based on the hidden representations produced during the forward propagation of the model. since both the teachers and students are trained together from random initialization with selfdistillation, the teachers may be less likely to provide reliable guidance for the training of students at the initial stage. we adopt a warm-up strategy disabling the knowledge distillation and to train only with the mt loss at the beginning. zhang et al. (2019c, 2022 ) distill the output predictions of the deepest layer to shallow layers by default, under the expectation that the deepest layer may generally offer the best performance than shallower layers, as shown in figure 1 (a) . ## analysis most of our analyses were conducted on the wmt 14 en-de task, with newstest 2013 and 2014 as the development set and test set respectively. ## conclusion in this paper, we study efficient representationbased self-distillation methods for neural machine translation. we present an iterative distillation method to prevent potential adverse impacts of knowledge distillation due to the capacity gap between teachers and students, propose a gradient adaptation method and adopt a warm-up strategy to ensure that the knowledge distillation gradients are consistent with the machine translation gradients, and present a rank-order augmented pearson correlation loss to further ensure the effectiveness of representation-based knowledge distillation. experiments on low/middle/high-resource tasks show that our method can lead to significant improvements over the strong baselines, obtaining comparable performance to previous machine translation knowledge distillation studies without pretraining a teacher model. experiments with shallower and deeper transformers show that our method can obtain better performance efficiently with fewer layers. our method is also effective in the challenging multilingual translation setting or with recurrent decoder.