Efficient Meta Lifelong-Learning with Limited Memory (2020)

Abstract: Current natural language processing models work well on a single task, yet they often fail to continuously learn new tasks without forgetting previous ones as they are re-trained throughout their lifetime, a challenge known as lifelong learning. State-of-the-art lifelong language learning methods store past examples in episodic memory and replay them at both training and inference time. However, as we show later in our experiments, there are three significant impediments: (1) needing unrealistically large memory module to achieve good performance, (2) suffering from negative transfer, (3) requiring multiple local adaptation steps for each test example that significantly slows down the inference speed. In this paper, we identify three common principles of lifelong learning methods and propose an efficient meta-lifelong framework that combines them in a synergistic fashion. To achieve sample efficiency, our method trains the model in a manner that it learns a better initialization for local adaptation. Extensive experiments on text classification and question answering benchmarks demonstrate the effectiveness of our framework by achieving state-of-the-art performance using merely 1% memory size and narrowing the gap with multi-task learning. We further show that our method alleviates both catastrophic forgetting and negative transfer at the same time.

## introduction humans learn throughout their lifetime, quickly adapting to new environments and acquiring new skills by leveraging past experiences, while retaining old skills and continuously accumulating knowledge. however, state-of-the-art machine learning models rely on the data distribution being stationary and struggle in learning diverse tasks * equal contribution, name order decided by coin flip. in such a lifelong learning setting (parisi et al., 2019) (see section 2 for a formal definition). in particular, they fail to either effectively reuse previously acquired knowledge to help learn new tasks, or they forget prior skills when learning new onesthese two phenomena are known as negative transfer @xcite and catastrophic forgetting @xcite , respectively. these downsides limit applications of existing models to real-world environments that dynamically evolve. due to its potential practical applications, there is a surge of research interest in the lifelong learning, especially in the vision domain @xcite @xcite @xcite @xcite . however, its application to language learning has been relatively less studied. while progress in large-scale unsupervised pretraining @xcite @xcite @xcite has recently driven significant advances in the field of natural language processing (nlp), these models require large amounts of in-domain training data and are prone to catastrophic forgetting when trained on new tasks @xcite , hindering their deployment in industry or other realistic setups where new tasks/domains continuously emerge. one successful approach to achieving lifelong learning has been augmenting the learning model with an episodic memory module @xcite . the underlying idea is to first store previously seen training examples in memory, and later use them to perform experience replay @xcite or to derive optimization constraints @xcite while training on new tasks. @xcite propose to use such a memory module for sparse experience replay and local adaptation in the language domain, achieving state-of-the-art results for lifelong learning on text classification and question answering tasks. despite its success, the method has three critical downsides, which we demonstrate later in our experiments: in this paper, we address these limitations and tackle the problem of efficient lifelong language learning. that is, we focus on storing limited training examples in memory. our contributions are three-fold: first, we identify three common principles underlying lifelong learning methods. we seek to characterize them in language learning and glean insights on overlooked downsides of the existing method. second, stemming from this analysis, we propose a meta-lifelong framework that unifies these three principles. our approach is a direct extension of d' @xcite and it explicitly meta-learns the model as a better initialization for local adaptation. finally, we conduct extensive experiments to demonstrate that our proposed approach can use the identified three principles to achieve efficient lifelong language learning. we find that our framework outperforms prior methods while using 100 times less memory storage. moreover, we demonstrate that our method can effectively alleviate catastrophic forgetting and negative transfer, closing the performance gap with the multi-task learning upper bound. it can also potentially obtain 22 times faster inference speed. ## background: principles of lifelong language learning following prior work (d'autume et al., 2019), we consider the lifelong learning setting where a model needs to learn multiple tasks in a sequential order via a stream of training examples without a task descriptor, i.e. the model does not know which task an example comes from during both training and testing. this setup is ubiquitous in practice, as environments consistently evolve without sending an explicit signal. formally, during training, the model makes a single pass over the training example stream consisting of n tasks in an ordered sequence, with ℓ being the specific task loss. notice that while the average risk is most commonly evaluated after the model has seen all tasks, we can also evaluate a specific task at different stages to demonstrate the model's training behavior, and evaluate its robustness against catastrophic forgetting and negative transfer. while different methods have been developed to optimize eq.( 1 ), we abstract away from their specific assumptions and instead focus on identifying common principles, among which we stress the following three points that are most relevant to language learning: generic representation. stemming from transfer learning @xcite , a key idea of transferring knowledge across diverse tasks is to learn a generic representation (such as a neural network encoder) that is able to encode useful information for all tasks. for instance, regularization based lifelong learning methods @xcite @xcite add an extra constraint to prevent the model parameter θ from drastically deviating when training on new tasks, thereby learning a generic model for old tasks as well. in the language domain, as language models have proven success to generate highly generic representation for many language understanding tasks @xcite , both d' @xcite and @xcite propose utilizing a pretrained language model @xcite to initialize parameters, and further training the model on d train . experience rehearsal. motivated by the complementary learning systems (cls) theory @xcite that humans rely on episodic memory to store past experiences and conduct experience rehearsal, we can also retrain lifelong learning models on previously seen tasks to reduce forgetting. while prior methods use memory to define optimization constraints @xcite @xcite , recent work use either stored examples @xcite or generated synthetic data @xcite to perform experience replay. further, d' @xcite shows that a sparse 1% rate of replaying to learning new examples is sufficient for lifelong language learning. task-specific finetuning. in multi-task learning, injecting task-specific parameters and finetuning on individual task have proven effective for different language understanding tasks @xcite or even diverse languages @xcite . prior work @xcite exploit this idea to expand model parameters for new tasks in lifelong learning setting. however, all these methods require a task descriptor in order to know when to add new parameters. when no such signal exists, local adaptation @xcite uses k stored nearest neighbors of each test example to perform extra finetuning at inference time. recent work @xcite demonstrate that the sentence embeddings produced by pretrained models can be used to effectively measure query similarity and that local adaptation can improve performance on text classification, question answering and language modelling. ## proposed framework with these principles in mind, we next turn to the problem of how to achieve efficient lifelong learning. to motivate our proposed framework, we first review the state-of-the-art method, improved mbpa (d' @xcite , and show how these principles help us to identify the limitation. ## experiments 4.1 evaluation dataset to evaluate the proposed framework, we conduct experiments on text classification and question answering tasks (see appendix a for details). following prior work, we consider each dataset as a separate task and the model needs to sequentially learn several tasks of the same category (e.g. all text classification tasks). as pointed out in @xcite , many nlp tasks can be formulated as question answering and thus our setup is general. ## conclusion in this work, we identify three principles underlying different lifelong language learning methods and show how to unify them in a meta-lifelong framework. our experiments demonstrate the effectiveness of the proposed framework on text classification and question answering tasks. we report new state-of-the-art results while using 100 times less memory space. these results illustrate that it is possible to achieve efficient lifelong learning by establishing complementary learning systems. our analysis also shows that negative transfer is an overlooked factor that could cause sub-optimal performance, and we highlight the importance of balancing the trade-off tween catastrophic forgetting and negative transfer for future work.