Distill and Replay for Continual Language Learning (2020)

Abstract: Accumulating knowledge to tackle new tasks without necessarily forgetting the old ones is a hallmark of human-like intelligence. But the current dominant paradigm of machine learning is still to train a model that works well on static datasets. When learning tasks in a stream where data distribution may fluctuate, fitting on new tasks often leads to forgetting on the previous ones. We propose a simple yet effective framework that continually learns natural language understanding tasks with one model. Our framework distills knowledge and replays experience from previous tasks when fitting on a new task, thus named DnR (distill and replay). The framework is based on language models and can be smoothly built with different language model architectures. Experimental results demonstrate that DnR outperfoms previous state-of-the-art models in continually learning tasks of the same type but from different domains, as well as tasks of different types. With the distillation method, we further show that it’s possible for DnR to incrementally compress the model size while still outperforming most of the baselines. We hope that DnR could promote the empirical application of continual language learning, and contribute to building human-level language intelligence minimally bothered by catastrophic forgetting.

## introduction humans and many advanced animals can learn new tasks without necessarily forgetting the old ones @xcite . this ability to continuously learn, accumulate knowledge and reuse them to tackle new challenges through the lifespan is a critical requirement for human-like intelligence. however, the currently dominant paradigm for machine learning is still to train a model on a static dataset, where state-of-the-art methods deliver impressive performance on that particular task. but when learning tasks in a stream where data distribution may shift, the models generally fail to isolate acquired knowledge and forget previously learned tasks. such phenomenon is known as catastrophic forgetting. there are mainly two stretches of methods in overcoming catastrophic forgetting. one is data-based method that achieves continual learning through reproducing the data distribution of old tasks @xcite . either storing real or generating pseudo examples, the model replays the training data of experienced tasks when learning a new one. the other stretch is model-based that enables continual learning with modification to the model architecture @xcite . for example, some methods @xcite regularize the loss function to constrain the updates of weights, especially those important for solving tasks. in the continual learning settings of the above methods, the tasks in a stream are mostly in essence of the same type, just different in domains. and most of the methods have only been applied to solve computer vision tasks. the emergence and rapid development of large-scale pre-trained language models make it possible to solve essentially different types of natural language understanding (nlu) tasks with one base architecture. but these over-parameterized models are also in risk of over-fitting, more prone to forgetting when sequentially learning diverse data distributions @xcite . to this end, we propose a framework this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . ## related work tackling new tasks without necessarily losing the knowledge learned in previous tasks is one fundamental requirement for a human-like intelligent system @xcite . however, the currently dominant paradigm of machine language learning is still training a model on a static dataset to achieve satisfactory performance on that particular task @xcite . most of these methods, especially the deep neural network-based ones, do not fare well in the continual learning scenarios. in continual learning, a model is required to fit on a stream of tasks where data distribution may not be uniform. for example, a network fitted on a first task tends to forget how to perform on it after trained on a sequential new task @xcite . this problem, namely catastrophic forgetting, poses a severe challenge in building a general language intelligent system with lifelong learning capacity. efforts have been made in recent years to overcome the catastrophic forgetting of deep neural networks. there are mainly two stretches of methods, differentiated by the way of isolating and reusing the accumulated knowledge. one stretch of method works by reproducing the data distribution of witnessed tasks. some members of this family select and store informative samples in an explicit memory module. the memorized samples will be re-trained when learning future tasks. for example, mbpa from de masson d' @xcite uses such an episodic memory module for sparse experience replay and local adaptation to allow continual learning. due to the efficiency and memory limit of storing raw examples, some methods supplant the explicit memory with a generator. lamol @xcite , for example, is trained to simultaneously solve a task and imitate training examples, thus saving the distribution of previous tasks. the other stretch of method regularizes model structures to remember old tasks while fitting on the new ones. instead of preserving informative samples, these methods seek to protect important weights of the network for performing a particular task. for example, elastic weight consolidation (ewc) by @xcite constrains change of informative weights for previous tasks identified by fisher's information. instead of relying on such parametric regularization, @xcite proposed to split a base network into dedicated parameter subspaces for different tasks to allow continual learning. worth noting that there are also hybrid approaches that integrate the above two stretches of methods to overcome catastrophic forgetting. gradient episodic memory (gem) @xcite and its imrproved version agem @xcite sample and store a subset of real samples from previous tasks and use them to constrain parameter gradients when optimizing on the following new tasks. ## results in this section, we will first give an overview of different models' continual learning ability and evaluate if they are robust to the variation of the task order in a sequence. we then compare specifically how well different models overcome the catastrophic forgetting while examining two hyper-parameters that may largely influence dnr's performance in continual learning. in these two subsections, we observe dnr's advantage over the baselines. in the last subsection, we discuss if the distillation approach of dnr can further compress the model size while still outperform other tested models.