Continual Knowledge Distillation for Neural Machine Translation (2023)

Abstract: While many parallel corpora are not publicly accessible for data copyright, data privacy and competitive differentiation reasons, trained translation models are increasingly available on open platforms. In this work, we propose a method called continual knowledge distillation to take advantage of existing translation models to improve one model of interest. The basic idea is to sequentially transfer knowledge from each trained model to the distilled model. Extensive experiments on Chinese-English and German-English datasets show that our method achieves significant and consistent improvements over strong baselines under both homogeneous and heterogeneous trained model settings and is robust to malicious models.

## introduction current neural machine translation (nmt) systems often face such a situation: parallel corpora are not publicly accessible but trained models are more readily available. on the one hand, many data owners are usually unwilling to share their parallel corpora with the public for data copyright, data privacy and competitive differentiation reasons, leading to recent interests in federated learning for nmt @xcite . on the other hand, trained nmt models are increasingly available on platforms such as hugginface ( https://huggingface.co ) and opus-mt ( https://opus.nlpl.eu/opus-mt ) since these models can be directly used without public access to the original training data. as a result, a question naturally arises: can we take advantage of increasingly available trained nmt models to enhance one nmt model of interest? in this work, we propose a method called continual knowledge distillation (ckd) to address @xmath0 figure 1 : continual knowledge distillation for neural machine translation. knowledge is continually distilled from a sequence of teacher models to one student model. at each time step, the current student model (i.e., ϕ t ) fuses the knowledge transferred from both the current teacher model (i.e., θ * t ) and the previous student model (i.e., ϕ t-1 ). all teacher models are frozen and the student model is trainable. different models are highlighted in different colors. as its name suggests, ckd is an intersection of knowledge distillation @xcite and continual learning @xcite . on the one hand, ckd differs from standard knowledge distillation in that the knowledge is transferred from teacher models to the student model asynchronously instead of synchronously. as a result, the knowledge transferred to the student model from previous teacher models can be overridden by an incoming teacher model, which is often referred to as the catastrophic forgetting problem @xcite . the situation aggravates when not all teacher models convey knowledge benefi-cial to the student model. on the other hand, ckd is different from conventional continual learning methods by focusing on learning one task (i.e., enhancing the student model) rather than learning many different tasks. the learning process is still very challenging as compared with standard continual learning because the original training data of teacher models is inaccessible to the student model. consequently, we have to resort to knowledge distillation at each time step to make the most of teacher models. to address these aforementioned challenges, we propose to fuse two knowledge sources for the student model at each time step: filtering the new knowledge from the current teacher model (i.e., knowledge filtration) and inheriting the old knowledge from the previous student model (i.e., knowledge inheritance) simultaneously. experimental results show that our method significantly and consistently outperforms strong baselines under both homogeneous and heterogeneous teacher settings for chinese-to-english and german-to-english translation. and it is also robust to malicious teachers. ## related work knowledge distillation. knowledge distillation (kd) is the most widely used technique for transferring knowledge between models @xcite . despite of their effectiveness, conventional kd methods usually implicitly assume that the teacher model is superior or complementary to the student model @xcite . although recently @xcite allow a big model to learn from small models, they still require that the small models are better than the big model for the given tasks and datasets. however, the assumption does not necessarily hold in our scenario due to the diversity of teacher models. multi-teacher kd @xcite @xcite @xcite , which distills knowledge from multiple teachers simultaneously, is highly related to this work. generally, multi-teacher kd requires all teachers to be available at the same time, which will result in enormous extra memory consummation as the number of teachers grows. more importantly, new teachers may be released constantly @xcite , which can not be seen in advance. therefore, multi-teacher kd methods are not feasible to our scenario. l2kd @xcite leverages sequential kd to continually learn new tasks, having different goal and challenges compared with our scenario. another line of related work is selective distillation @xcite @xcite , which selects data and losses to accelerate kd or enhance model robustness. in contrast, we select data for conducting different ways of distillation in our proposed method. continual learning. continual learning (cl) for neural machine translation (nmt) aims at learning knowledge of new domains @xcite @xcite or languages @xcite @xcite without forgetting old knowledge. our scenario also requires learning new knowledge but focuses on improving performance of the student on its test set instead. moreover, alleviating the negative impact of the less beneficial knowledge conveyed by "weak" teachers is essential in our scenario, which is hardly explored in cl for nmt. while our scenario is a multi-step process, multi-step cl is less explored in nmt @xcite . @xcite address a similar task of adapting from multiple out-of-domain models to a single in-domain model. nevertheless, they assume the training data for the out-of-domain models are available, which is inaccessible in our scenario. besides, leveraging highresource language nmt models to improve lowresource language translation has also attracted intensive efforts @xcite @xcite , which can be a future extension of our method. ## conclusion and future work to take advantage of increasingly available trained neural machine translation (nmt) models to improve one model of interest, we propose a novel method named continual knowledge distillation. specially, knowledge from the trained models is transferred to the interested model via knowledge distillation in a sequential manner. extensive experiments on two language pairs under homogeneous, heterogeneous, and malicious teacher settings show the effectiveness of our proposed method. in the future, we will further explore the effect of the teacher model order. it is also worth involving more sophisticated methods in knowledge filtration, such as gradient-based and meta-learning-based methods. moreover, it is also a promising research direction to exchange knowledge among all the models such that all of them achieve improvement.