Lifelong Language Knowledge Distillation (2020)

Abstract: It is challenging to perform lifelong language learning (LLL) on a stream of different tasks without any performance degradation comparing to the multi-task counterparts. To address this issue, we present Lifelong Language Knowledge Distillation (L2KD), a simple but efficient method that can be easily applied to existing LLL architectures in order to mitigate the degradation. Specifically, when the LLL model is trained on a new task, we assign a teacher model to first learn the new task, and pass the knowledge to the LLL model via knowledge distillation. Therefore, the LLL model can better adapt to the new task while keeping the previously learned knowledge. Experiments show that the proposed L2KD consistently improves previous state-of-the-art models, and the degradation comparing to multi-task models in LLL tasks is well mitigated for both sequence generation and text classification tasks.

## introduction training a single model to learn a stream of different tasks sequentially usually faces the catastrophic forgetting problem @xcite ): after learning a new task, the model forgets how to handle the samples from previous tasks. lifelong learning manages to accumulate the knowledge and retain the performance of previously learned tasks. it is important especially for real-world natural language processing (nlp) applications, because these applications need to interact with many users from different domains everyday, and the language usage also evolves from time to time. hence, various nlp tasks have been studied for lifelong learning in the previous work, including sentiment analysis @xcite , task 1 task 2 task 3 … task 1 lll model task 2 task 3 … lll model (a) normal lifelong language learning. task 1 teacher 1 lll model task 2 teacher 2 task 3 teacher 3 … lll model (b) lifelong language knowledge distillation. 2017), conversational agents @xcite , word and sentence representation learning @xcite ), text classification, and question answering (d' @xcite . in recent, lamol @xcite improved the performance of lll by a general framework: 1) it followed the idea about considering many nlp tasks as question answering (qa) @xcite and adapted all tasks into the language modeling (lm) form. in the unified framework, it can perform lll on many nlp tasks by generating answers based on the contexts and the questions using a single language model, and 2) it outperformed the previous methods by a considerable margin and is only 2%-3% worse than the multi-tasking upper bound, which jointly learns all tasks in a mixed dataset. this paper further improves lll by introducing lifelong language knowledge distillation (l2kd), which can be flexibly applied upon the lamol architecture or other lll methods for sequence generation learning. the motivation of our work mainly comes from how to efficiently compress the knowledge under a lifelong language learning framework. if the model can learn a new task in an efficient way, the previously learned knowledge may not be affected and thus the problem of catastrophic forgetting can be mitigated. inspired by knowledge distillation @xcite @xcite , in which a student (smaller) model is trained to imitate the behavior of a teacher (larger) model in order to reach the performance closer to the teacher model, the lll model in l2kd can be seen as a weak learner that needs to compress knowledge from different tasks into a compact single model. thus lll can benefit from the similar procedure of knowledge distillation, although the model size is equal to its teacher model. the similar idea about distilling knowledge from equal-size models has also been studied in born-again neural network @xcite , multitask learning @xcite and lifelong computer vision learning @xcite , but never been explored in lifelong language learning research. in l2kd, we train a new teacher model when facing a new task, and the lll model imitates the behavior of its teacher at each training stage, as illustrated in figure 1 . this method only needs a little extra time to train a disposable teacher model for each new task, and the teacher model can be discarded when learning the next task; therefore, there is no extra memory or model capacity required for the target lll model, making the proposed model more memory-efficient for real-world usage. ## proposed approach before describing how l2kd works, in section 2.1 we briefly introduce the architecture of lamol @xcite , which l2kd is built upon. then we introduce different knowledge distillation strategies in section 2.2, and how to apply them to l2kd in section 2.3. ## experimental setup to evaluate the proposed method, we conduct a set of experiments detailed below. ## results and discussion we discuss the results for three settings: 1) different sequence generation tasks, 2) same tasks in different domains, and 3) different text classification tasks in order to validate the effectiveness of the proposed approach. ## conclusion this paper presents lifelong language knowledge distillation (l2kd), a simple method that effectively help lifelong language learning models to maintain good performance comparable to its multitask upper bounds. the experiments show the consistent improvement achieved by l2kd for three different settings, indicating the effectiveness of the proposed method to train robust lll models. in addition, the proposed approach only requires a little extra time for training the teacher without extra memory or capacity needed, showing the potential of being applied to the practical scenarios.