Knowledge Distillation for Sustainable Neural Machine Translation (2022)

Abstract: Knowledge distillation (KD) can be used to reduce model size and training time, without significant loss in performance. However, the process of distilling knowledge requires translation of sizeable data sets, and the translation is usually performed using large cumbersome models (teacher models). Producing such translations for KD is expensive in terms of both time and cost, which is a significant concern for translation service providers. On top of that, this process can be the cause of higher carbon footprints. In this work, we tested different variants of a teacher model for KD, tracked the power consumption of the GPUs used during translation, recorded overall translation time, estimated translation cost, and measured the accuracy of the student models. The findings of our investigation demonstrate to the translation industry a cost-effective, high-quality alternative to the standard KD training methods.

## introduction deep neural networks (dnns) underpin state-of-the-art applications of artificial intelligence in almost all fields, such as image @xcite , speech @xcite and natural language processing (nlp) @xcite . however, dnn architectures @xcite are often data-, compute-, space-, power-and energy-hungry, typically requiring powerful gpus or large-scale clusters to train and deploy, which has been viewed as a "nongreen" technology @xcite . furthermore, often the best-performing models are ensembles of hundreds or thousands of base-level models, which require large amounts of space and time for storage and execution @xcite @xcite . neural mt (nmt) @xcite @xcite systems have greatly improved mt compared to statistical mt (smt) @xcite systems. these state-of-the-art nmt systems, however, require much more computing power and data than smt systems @xcite , and if the effects of climate change are to be controlled they are unsustainable in the long run. these systems are also largely unsuitable for training engines for low-resource languages and scenarios, where the data simply does not exist in the amounts required for high quality results to be achieved. to some extent, model compression, more specifically knowledge distillation techniques, can remedy this. knowledge distillation @xcite can be used to transfer the knowledge from a teacher network (a large model) to a student network (a smaller model). this is generally done by using a smaller, fast model to approximate the function learned by a much larger and slower model with better performance @xcite . the methods described by @xcite and @xcite can be used for word-level knowledge distillation, since nmt models make use of multi-class prediction at the word-level. these models, however, need to predict complete sequences that are dependent on previous predictions as well. @xcite propose sequence-level kd wherein a new training set is generated by translating a data set with the teacher model using beam search. the newly generated training set is then used to train a smaller student model. the data sets often contain millions of sentences and thus translating the training set by a large-scale teacher model for kd training @xcite can be a cumbersome task and computationally expensive process. thus, kd training in mt is responsible for a considerable amount of co 2 emissions. this is a concerning matter for the environment. besides, this is also a concern for tsps who want to increase their margins in translation productivity by offering translations by smaller student models to their clients. more specifically, the standard and computationally expensive kd training process can negatively impact the translation productivity gain in industry. the standard kd training methods use large-scale teacher models. we tested a number of variants of a teacher model for translating the source sentences of our training data. for example, we investigated the effects of changing the beam size and using quantisation @xcite while translating the training data. more specifically, we tested the approaches of both @xcite and @xcite , who focused on quantisation during inference to reduce the size of their student models and increase the speed at which these models translate sentences. the quality of translations by a quantised teacher model would naturally be worse than that of the translations by non-quantized teacher model. the same is true when one uses a very small batch size (e.g. 1) at decoding. as a result, the translations by these fast decoders would naturally impact the quality the resultant student model. in other words, you are likely to obtain a worse student model when you use a quantised teacher model to distil knowledge. our investigation focused on examining the magnitude of quality drop of the student models when using the different variants of the teacher models for kd, and in return how faster, cheaper and environmental friendly the kd training process would be. we considered english-to-german for our investigation, and recorded a number of parameters (i.e. co 2 emissions, translation time, accuracy) including power consumption of the gpus used for translation. we empirically demonstrated that our proposed kd training methods are computationally less expensive in comparison to the standard kd methods, and more importantly, they do not deteriorate the accuracy of the student models much. as far as we are aware, related work on model efficiency and knowledge distillation @xcite @xcite @xcite focus on performance during inference whereas in this paper we focus on the effects of quantisation for kd training, in order to make the process as a whole more efficient rather than just reducing the size and increasing the speed of the student models. ## experimental setup in this section we describe the various aspects of our experiments. we first discuss the data and how it was preprocessed and then move on to describe how our mt systems are trained. lastly, we describe how we evaluate the quality of these mt systems. ## results and discussion figure 2 : the co 2 emissions and cost in usd of translation when using 1,2 or 4 gpus during the distillation process. taking the power draw and co 2 emissions into account, the effects that these methods of speeding up decoding have on the accuracy of the student models are shown in when considering the cost for tsps, as seen in figure 2 provides a summary of the most notable results in ## conclusions and future work we described various methods in which the distillation of knowledge can be made more efficient and in turn more sustainable. most significantly, the impact of batch sizes when using quantisation and a smaller beam size result in a less than 1 bleu point drop in accuracy, while at the same time reducing decoding time hby at least 10 hours compared to the original method. in terms of efficiency, the combined setup is found to be the best method for distilling knowledge from the teacher to student models. the co 2 emissions of our combined setup is on average 10kg less than the original setup while accuracy decreases only slightly. the environmental impact of distilling knowledge from a teacher model to a student model are encouraging, and we contend that more importance should be given to this issue since inefficient methods emit on average 10 times more co 2 than optimised methods, yet the cost in accuracy of the student models is minimal. taking only the end result (student model) into account is not sustainable and more consideration needs to be put into the whole process. we have shown that during the process of distilling knowledge from a teacher model to a student model, using just 2 gpus can result in the fastest translation time while using 1 gpu is the most cost-effective and in most cases the most environmentally friendly as well. interestingly, from our results it is clear that when taking co 2 emissions and cost into account, using 4 gpus is much less efficient compared to using only 1 gpu. in future we will investigate the efficiency of using cpus during the distillation process as well as during inference when the student models are deployed. we also aim to develop a composite metric that takes carbon emissions, accuracy and access to resources into account in order to rate the performance of mt systems. furthermore, we aim to investigate to what extent these decoding methods work on different language pairs, especially their effect on lowresource languages where access to data is considerably more problematic.