Speed-optimized, Compact Student Models that Distill Knowledge from a Larger Teacher Model: the UEDIN - CUNI Submission to the WMT 2020 News Translation Task (2020)

Abstract: We describe the joint submission of the University of Edinburgh and Charles University, Prague, to the Czech/English track in the WMT 2020 Shared Task on News Translation. Our fast and compact student models distill knowledge from a larger, slower teacher. They are designed to offer a good trade-off between translation quality and inference efficiency. On the WMT 2020 Czech ↔ English test sets, they achieve translation speeds of over 700 whitespace-delimited source words per second on a single CPU thread, thus making neural translation feasible on consumer hardware without a GPU.

## introduction the conventional set-up of the wmt shared tasks on news translation emphasizes translation quality (however measured) above all else. constraints on the data that may be used for training in the 'constrained' track establish a level playing field in terms of the information available to the translation model and its training process, but there are no constraints on the computational power and effort spent to achieve the results. in contrast, the wngt shared task on efficient translation @xcite encourages participants to submit systems that are both accurate and efficient during inference (i.e., translation). so far, there has been little interaction between the two tasks. with our joint submission between the university of edinburgh (uedin) and charles university, prague (cuni), we strive to bridge this gap. we submitted small, efficient systems that distilled knowledge from a more powerful teacher model via sequence-level knowledge distillation (kim and 1 bogoychev et al. (2020) report translation speeds of up to 3135 source words per second on a single cpu thread; the actual throughput depends not only on the computer hardware used for translation but also on the distribution of translation segment lengths in the test set. rush, 2016). in a nutshell, the procedure can be described as follows: while the computational effort to first train a teacher model and then distill its knowledge into a student model is considerably greater than just training the teacher -in addition to training the teacher, we have to translate the training data and then train a student on that data -, the advantage is at inference time. even the larger of the two student models we present in this paper can translate on a single cpu core at an acceptable speed (cf. tab. 4 ). translation can further be sped up by quantizing parameters to 8 bits of precision and using integer general matrix multiplication (intgemm) for inference. even though our submissions to the wmt 2020 shared task on news translation were produced by unquantized floating point models, we report performance numbers for quantized models as well to demonstrate their efficacy and show that they can speed up translation by about 10% with negligible trade-offs in terms of bleu score over unquantized models. all models used in this work are based on the transformer architecture @xcite . details are discussed in the sections below; the hyper-parameter settings for each model are listed in tab. 1. the student models described in this paper can be obtained via https://github.com/ browsermt/students . ## teacher models the teacher models were produced by cuni, using the tensor2tensor deep-learning toolkit @xcite . 2 the teachers were trained on the full czeng 2.0 dataset @xcite ), 3 consisting of genuine (authentic) parallel data as well as monolingual news data translated by cuni's transformer systems from wmt 2018 @xcite to generate back-translated synthetic training data @xcite . rather than shuffling and mixing authentic and synthetic training data, the teacher models were trained on alternating blocks of authentic and synthetic data ("block-regime backtranslation" (block-bt); popel et al., 2020), spending about 10 hours of training time on each block. the model parameters for the final teacher models were obtained by checkpoint averaging over the last 8 checkpoints of the training process, saved in hourly intervals. the @xmath0; popel, 2020). however, the cuni submission used a beam size of 4 instead of 8 as used in this work, resulting in a bleu score on the wmt 2020 @xmath1.2 lower than the bleu score reported in tab. 4 . the @xmath2, resulting in a bleu score on the wmt 2020 test set that is 1.0 bleu points lower than the score achieved by the model used for cuni's primary submission. ## student models the smaller, more efficient student models were trained by uedin with the marian nmt toolkit 3 http://ufal.mff.cuni.cz/czeng @xcite . 4 the students were trained on artificial training data produced by knowledge distillation @xcite , where the target side of the parallel data is the teacher model's translation of the source side. the basic idea is that the teacher guides the student towards translations that can be achieved with the teacher's knowledge. ## results in text segments in the wmt 2020 cs↔en test sets are aligned at the paragraph level; we therefore split the provided segments into individual sentences prior to translation with moses-style sentence splitting foot_6 and restored paragraphs afterwards. all bleu scores were computed with sacrebleu. 10 for the @xmath3, removing repetitions and adapting quotation marks to czech spelling conventions boosted the bleu score by 1.6 bleu points; for student models, this post-processing is not necessary. having been trained on post-processed teacher output, the student models have learned this correctly. except where indicated, translation was with a greedy search (beam size 1) and a shortlist of 50 translation candidates per source word. due to resource congestion, we were not able to fully train the models by the submission deadline; our submissions are based on the systems with the best validation bleu score available at the time. for validation, we used the concatenation of all parallel data for the respective translation direction from the wmt test sets from the years 2008 through 2019 where the original language of the data is the source language for the translation direction in question. in terms of bleu score on the wmt 2020 test set, the submitted primary system for @xmath4. 0.5 bleu points below the final system; the @xmath5.5 bleu points, as shown in tab. 4. ## conclusion we presented student models that distill knowledge from a larger teacher model without loss in bleu performance. (in fact, for the wmt 2020 test set, our larger student models technically outperform the teacher in terms of bleu, but we consider that difference accidental.) at the same time, they are significantly faster and do not require a gpu for inference, making it possible to perform neural machine translation on consumer-grade hardware without the use of a gpu.