Sampling and Filtering of Neural Machine Translation Distillation Data (2021)

Abstract: In most of neural machine translation distillation or stealing scenarios, the highest-scoring hypothesis of the target model (teacher) is used to train a new model (student). If reference translations are also available, then better hypotheses (with respect to the references) can be oversampled and poor hypotheses either removed or undersampled. This paper explores the sampling method landscape (pruning, hypothesis oversampling and undersampling, deduplication and their combination) with English to Czech and English to German MT models using standard MT evaluation metrics. We show that careful oversampling and combination with the original data leads to better performance when compared to training only on the original or synthesized data or their direct combination.

## introduction model distillation is a process of transferring the knowledge of one or more, usually larger, model(s) into another, usually smaller, model @xcite . a variation of this is training a new model in a way that its performance is similar to that of the already trained one. this is achieved by making use of either teacher predictions (black-box) or other products of the workings of the teacher, such as attention-score or decoder score (grey/glass-box). assuming we have access to a parallel corpus, we focus on sampling the translation hypotheses and making use not only of the teacher scores but also of their comparison to the reference. there are various possible motivations for model distillation. the student model can be much smaller than the teacher model, which has the benefit of faster inference speed @xcite . it can also be used for model stealing, where an adversary tries to copy the teacher functionality. this is a practical concern for production-level mt systems @xcite . one of the approaches for knowledge distillation is to use the teacher model to generate a new dataset for the student model to train on. having access to a trained teacher model, this approach does not require parallel data and can leverage large monolingual corpora. reference translations, however, help with determining which of the teacher's translations are good and which are of low quality. we focus on this approach and propose and compare several importance sampling approaches to prepare training data for student models that leverage access to reference translations. these include pruning, upsampling and undersampling, deduplication and their combination. we show that a combination of these methods improves the student performance over just using the reference or the best hypothesis (by the decoder score), which is a common distillation practice. the experiment code is available open-source. foot_0 ## methods the evaluation of every sampling method follows the following three-step process. first, the specific parallel corpus (section 2.1) is translated by the teacher model (section 2.2) for the considered translation direction. new datasets based on metrics are then created. the reference is taken into consideration during the hypothesis selection. we train new models (students) on these datasets and measure their performance. there are 12 hypotheses (default in marian nmt) provided by the teacher using beam search for every source sentence which we consider when composing a new dataset. ## results baseline. training on the original data leads to better results than training on the best scoring hypotheses. training on all hypotheses results in slightly lower bleu performance. this may be caused by the small amount of training data available in which case taking all hypotheses just improves the vocabulary and language modelling capacity. ## summary although widely used, taking only the highestscoring sentence (with respect to the decoder score or any reference-based metrics, such as bleu) does not lead to the best results. in the context of the proposed experiments, these are achieved by a combination of top hypotheses and the original data, such as s 4,3,2,1 bleu +4×original (upsampling according to bleu and joining with the original data duplicated four times). here, an improvement of an average +2 bleu points against t 1 score + original was achieved. the choice of the sampling metric does not significantly influence the results, especially in cases where more than the top one hypothesis is sampled. because of this, in most scenarios the decoder score can be used instead, reducing the need for translation references. future work. we worked with only two upsampling schemes: s 4,3,2,1 and s 2,2,1,1 . however, the two vectors are arbitrary and more of the vast vector space should be explored, especially with more than the top four hypotheses considered or more skewed towards the best hypothesis. more sophisticated methods based on the value of the metric instead of just the ordering could also be tried out. the effects of large models (both teacher and student) and data access should be explored to verify the transferability of the results of the current setup. specifically, the teacher model should not be a distilled model itself. the robustness of the training should also be established. even though this paper focused solely on mt, the importance sampling methods could also be applied and verified on other nlp tasks, possibly even on more general machine learning problems.