Continual Learning with Semi-supervised Contrastive Distillation for Incremental Neural Machine Translation (2024)

Abstract: Incrementally expanding the capability of an existing translation model to solve new domain tasks over time is a fundamental and practical problem, which usually suffers from catastrophic forgetting. Generally, multi-domain learning can be seen as a good solution. However, there are two drawbacks: 1) it requires having the training data for all domains available at the same time, which may be unrealistic due to storage or privacy concerns; 2) it requires re-training the model on the data of all domains from scratch when adding a new domain and this is time-consuming and computationally expensive. To address these issues, we present a semi-supervised contrastive distillation framework for incremental neural machine translation. Specifically, to avoid catastrophic forgetting, we propose to exploit unlabeled data from the same distributions of the older domains through knowledge distillation. Further, to ensure the distinct domain characteristics in the model as the number of domains increases, we devise a cross-domain contrastive objective to enhance the distilled knowledge. Extensive experiments on domain translation benchmarks show that our approach, without accessing any previous training data or re-training on all domains from scratch, can significantly prevent the model from forgetting previously learned knowledge while obtaining good performance on the incrementally added domains. The code and data with step-by-step instructions will be released upon acceptance.

## introduction in the real scenario, translating an out-of-domain sentence is a common situation while it usually cannot work well due to domain discrepancy. an effective solution is to incrementally expand the capability of the existing translation model, i.e., continual learning @xcite . however, the biggest challenge is catastrophic forgetting when * work was done when liang was interning at pattern recognition center, wechat ai, tencent inc, china. the model learns new knowledge and it would forget the previously acquired knowledge @xcite . a theoretically good technique is multi-domain learning, which usually requires having all the training data available at the same time and re-training the model on all domains from scratch @xcite . nevertheless, in practice, it may be unfeasible because we sometimes cannot access the previous data due to storage or privacy concerns, and retraining would bring more training and resource consumption. to overcome these drawbacks, many efforts have been devoted that fall into three categories, i.e., constructing pseudo data of previous domains/tasks, adding task-specific adapters, and regularizationbased learning. (i) the first category aims to create pseudo data of the previous task and mix them with the new task data for joint training @xcite @xcite . although intuitive and effective, they generally require obtaining a large training data of previous tasks and are not flexible in practice. (ii) the second category is to add additional task-specific layers for new tasks and only optimizes these parameters with the new task data, having achieved impressive performance @xcite @xcite @xcite @xcite . however, the task-specific adapters may increase the difficulty of the model to be aware of which tasks the input belongs to and thus neglect the distinct task characteristics, which limits its application in practice. (iii) the third category essentially searches a trade-off between the new task and the previous ones through multi-objective training with an extra penalty item (e.g., l2 or ewc regularization) on the parameters @xcite . therefore, previous methods usually lead to under-or over-constraint problems and achieve a suboptimal performance. be-sides, they typically require the parallel data of the previous tasks/domains @xcite and the time and space cost for computing the penalty item is expansive, especially with new tasks/domains appearing @xcite . in this paper, to address the above issues, we present a semi-supervised contrastive distillation (named scd) framework for incremental neural machine translation. specifically, to memorize the learned knowledge from previous domains, we propose to exploit unlabeled data from the same distributions of the older domains through knowledge distillation. to this end, we utilize the sourceside data related to the previous domains, e.g., the source-side data of validation set 1 , which is smallscale and easy to obtain compared to requiring parallel data. furthermore, to guarantee distinct domain characteristics in the model as new domains appear, we devise a cross-domain contrastive objective to enhance the distilled knowledge, which encourages the model to learn to keep different domain characteristics and thus benefits translation for various domains. we validate our proposed scd framework on the commonly-used machine translation benchmark (aharoni and goldberg, 2020), which contains five domains. we incrementally add a single domain at each time to simulate the real-world situation. extensive experiments show that our model effectively addresses the catastrophic forgetting issue and significantly outperforms related strong methods in terms of bleu @xcite scores, demonstrating its effectiveness. in summary, our main contributions are 2 : copy performance only with a handful of unlabeled data than that using a large of parallel data. ## methods in this section, we first describe the problem definition § 2.1. secondly, we introduce the proposed semi-supervised distillation method in § 2.2, which prevents the model from forgetting the previously learned knowledge. then, to further ensure the domain characteristics, we present a cross-domain contrastive objective to enhance the distilled knowledge § 2.3. finally, we elaborate on the training and inference in § 2.4. ## related work continual learning of translation. recent studies on continual learning of machine translation mainly includes data memory-based method, taskspecific adapters, and regularization-based method. specifically, (1) the data memory-based methods @xcite @xcite usually require maintaining part or all of the training data of the previous domains/task, which is not flexible in practice and maybe not realistic in the real world due to data privacy. for example, @xcite produce many mixed-language sentences via a bilingual dictionary. (2) the task-specific adapter methods @xcite @xcite @xcite @xcite typically require assigning additional model parameters to different domains/tasks, which requires the model to know which task the input comes from. ## conclusion in this paper, we propose a new continual learning framework for incremental neural machine translation without accessing any previous training data or re-training on all domains from scratch. to maintain the performance of the previous domain, we propose to utilize small-scale source-side development data of the previous domain via knowledge distillation. to further ensure distinct domain characteristics in a model, we devise a cross-domain contrastive objective to enhance the distilled knowledge. extensive experiments on a more general scenario show that our method can achieve significant improvements over several strong baselines. in the future, we would like to investigate our method for other tasks, e.g., cross-lingual summarization @xcite .