Selective Knowledge Distillation for Neural Machine Translation (2021)

Abstract: Neural Machine Translation (NMT) models achieve state-of-the-art performance on many translation benchmarks. As an active research field in NMT, knowledge distillation is widely applied to enhance the model’s performance by transferring teacher model’s knowledge on each training sample. However, previous work rarely discusses the different impacts and connections among these samples, which serve as the medium for transferring teacher knowledge. In this paper, we design a novel protocol that can effectively analyze the different impacts of samples by comparing various samples’ partitions. Based on above protocol, we conduct extensive experiments and find that the teacher’s knowledge is not the more, the better. Knowledge over specific samples may even hurt the whole performance of knowledge distillation. Finally, to address these issues, we propose two simple yet effective strategies, i.e., batch-level and global-level selections, to pick suitable samples for distillation. We evaluate our approaches on two large-scale machine translation tasks, WMT’14 English-German and WMT’19 Chinese-English. Experimental results show that our approaches yield up to +1.28 and +0.89 BLEU points improvements over the Transformer baseline, respectively.

## introduction machine translation has made great progress recently by using sequence-to-sequence models @xcite @xcite @xcite . recently, some knowledge distillation methods @xcite @xcite @xcite @xcite are proposed in the machine translation to help improve model performance by transferring knowledge from a teacher model. these methods can be divided into two categories: word-level and sequence-level, by the granularity of teacher information. in their researches, the model learns from teacher models by minimizing gaps between their outputs on every training word/sentence (i.e., corresponding training sample) without distinction. despite their promising results, previous studies mainly focus on finding what to teach and rarely investigate how these words/sentences (i.e., samples), which serve as the medium or carrier for transferring teacher knowledge, participate in the knowledge distillation. several questions remain unsolved for these samples: which part of all samples shows more impact in knowledge distillation? intuitively, we may regard that longer sentences are hard to translate and might carry more teacher knowledge. but are there more of these criteria that can identify these more important/suitable samples for distillation? further, what are the connections among these samples? are they all guiding the student model to the same direction? by investigating the carrier of teacher knowledge, we can shed light on finding the most effective kd method. hence, in this paper, we aim to investigate the impacts and differences among all samples. however, it is non-trivial to analyze each of them. therefore, we propose a novel analytical protocol by partitioning the samples into two halves with a specific criterion (e.g., sentence length or word cross-entropy) and study the gap between performance. extensive empirical experiments are conducted to analyze the most suitable sample for transferring knowledge. we find that different samples differ in transferring knowledge for a substantial margin. more interestingly, with some partitions, especially the student model's word cross-entropy, the model with half of the knowledge even shows better performance than the model using all distill knowledge. the benefit of the distillation of two halves cannot collaborate. this phenomenon reveals that the distillation of two halves cannot collaborate, even hurt the whole performance. hence, a more sophisticated selective strategy is necessary for kd methods. next, we propose two simple yet effective methods to address the observed phenomenon according to word cross-entropy (word ce), which we find is the most distinguishable criterion. we first propose a batch-level selection strategy that chooses words with higher word ce within the current batch's distribution. further, to step forward from local (batch) distribution to global distribution, we use a global-level fifo queue to approximate the optimal global selection strategy, which caches the word ce distributions across several steps. we evaluate our proposed method on two large-scale machine translation datasets: wmt'14 english-german and wmt'19 chinese-english. experimental results show that our approach yields an improvement of +1.28 and + 0.89 bleu points over the transformer baseline. in summary, our contributions are as follows: ## related work knowledge distillation approach @xcite aims to transfer knowledge from teacher model to student model. recently, many knowledge distillation methods @xcite @xcite @xcite @xcite @xcite have been used to get effective student model in the field of natural language processing by using teacher model's outputs or hidden states as knowledge. as for neural machine translation (nmt), knowledge distillation methods commonly focus on bet-ter improving the student model and learning from the teacher model. @xcite first applied knowledge distillation to nmt and proposed the sequence-level knowledge distillation that lets student model mimic the sequence distribution generated by the teacher model. it was explained as a kind of data augmentation and regularization by @xcite . further, @xcite improved the quality of distillation information by using an ensemble model as the teacher model. @xcite improved non-autoregressive model performance by learning distillation information from the autoregressive model. @xcite proposed a layer-wise distillation method to be suitable for the deep neural network. @xcite let translation model learn from language model to help the generation of machine translation. to the best of our knowledge, there is no previous work in nmt concerning the selection of suitable samples for distillation. the few related ones mainly focus on selecting appropriate teachers for the student model to learn. for instance, @xcite let the student model only learn from the individual teacher model whose performance surpasses it. @xcite proposed an online knowledge distillation method that let the model selectively learn from history checkpoints. unlike the above approaches, we explore the effective selective distillation strategy from sample perspective and let each sample determine learning content and degree. ## selective knowledge distillation for nmt as mentioned above, there exist un-suitable mediums/samples that hurt the performance of knowledge distillation. in this section, we address this problem by using two simple yet effective strategy of selecting useful samples. in section 4, we find that word ce is the most distinguishable criterion. hence, we continue to use the word ce as the measure in our methods. as the word cross-entropy is a direct measure of how the student model agrees with the golden label, we refer to words with relatively large cross-entropy as difficult words, and words with relatively small cross-entropy as easy words, in the following parts. this is to keep the notation different from previous analysis. then, we only need to define what is "relatively large". here, we introduce two ce-based selective strategies: batch-level selection (bls). given a minibatch b of sentence pairs with m target words, we sort all words in the current batch with their word ce in descending order and select the top r percent of all words to distill teacher knowledge. more formally, let a denote the word ce set, which contains the word ce of each word in batch b. we define s @xmath0, and s easy is its complementary part. for those words in s hard , we let them get extra supervision signal from teacher model's distillation information. therefore, the knowledge distillation objective in equation 3 can be be re-formulated as: ## experiments we carry out experiments on two large-scale machine translation tasks: wmt'14 english-german (en-de) and wmt'19 chinese-english (zh-en). ## conclusion in this work, we conduct an extensive study to analyze the impact of different words/sentences as the carrier in knowledge distillation. analytic results show that distillation benefits have a substantial margin, and these benefits may not collaborate with their complementary parts and even hurt the performance. to address this problem, we propose two simple yet effective strategies, namely the batch-level selection and global-level selection. the experiment results show that our approaches can achieve consistent improvements on different sized datasets across different language pairs.