How Relevant is Selective Memory Population in Lifelong Language Learning? (2022)

Abstract: Lifelong language learning seeks to have models continuously learn multiple tasks in a sequential order without suffering from catastrophic forgetting. State-of-the-art approaches rely on sparse experience replay as the primary approach to prevent forgetting. Experience replay usually adopts sampling methods for the memory population; however, the effect of the chosen sampling strategy on model performance has not yet been studied. In this paper, we investigate how relevant the selective memory population is in the lifelong learning process of text classification and question-answering tasks. We found that methods that randomly store a uniform number of samples from the entire data stream lead to high performances, especially for low memory size, which is consistent with computer vision studies.

## introduction while humans learn throughout their lifetime, current deep learning models are restricted to a bounded environment, where the input distribution is fixed. when those models are sequentially learning new tasks, they suffer from catastrophic forgetting @xcite because the input distribution changes. several methods have been proposed to address catastrophic forgetting, mainly for computer vision (cv) @xcite and few others for natural language processing (nlp) @xcite . in both, one of the prominent approaches is experience replay with episodic memory @xcite , which aims to store previously seen training examples and later use them to perform gradient updates while training on new tasks. in the experience replay approach, random sampling is the de facto method for the memory population, as it has shown good results in cv @xcite @xcite . in contrast, other works have shown that memory selection is relevant for deep reinforcement learning @xcite , image classification @xcite , and analogical reasoning @xcite . however, no previous work has explored nlp tasks, which raises the question of whether memory selection is necessary for lifelong language learning. in this paper, we adopt and evaluate seven memory population methods under a lifelong language learning setup with sparse experience replay. we conducted experiments with text classification and question answering tasks. we find that methods that obtain memory with a random sample from the global data distribution for text classification provide the best results in both high and low memory regimes. conversely, for the question answering task, a method that provides a balanced memory composition per task performs better. ## related work lifelong learning in nlp. rather than training a language model on a fixed dataset, lifelong (continual) language learning setups consist of a stream of tasks (e.g., text classification). in this setup, a model aims to retain the most relevant information to prevent catastrophic forgetting. existing approaches for nlp include purely replay-based methods @xcite @xcite , meta-learning based methods @xcite and generative replay-based methods @xcite . ## lifelong language learning setup we consider the lifelong language learning setting proposed by d' @xcite , in which a model learns multiple tasks in sequential order from a stream of training examples foot_0 . in this setup, each example is only allowed to be viewed once. this setup adopts sparse experience replay, which performs a gradient update at a certain interval during training. we leverage this method, as d' @xcite have shown that a sparse 1% rate of replaying to learning new examples is sufficient for lifelong language learning. this setting also includes local adaptation @xcite , which is a process that retrieves k-nearest neighbors examples from memory to update model parameters used to predict a particular test example. however, recent works have tried to reduce its use @xcite or even avoid it @xcite because it significantly slows down the inference speed. we do not use this mechanism in our main experimentation because our goal is to analyze the effect of selective memory on the generalization of the model. nevertheless, section 6 briefly shows how resulting memory composition influences local adaptation. ## selective episodic memory for the previously described lifelong learning setup, we extend a replay model (see section 5) with the following seven memory population methods: naive random. a basic method for memory population. it samples a percentage of elements of each task. in our experiments, the percentage value is the same as the memory capacity, and we sample the elements on the fly from the current batch. reservoir. a reservoir @xcite allows sampling elements from a stream without knowing how many elements to expect. it samples each element with a probability m n where n is the number of elements observed so far and m is the memory size. this way, it acts randomly to maintain a uniform sample from the already seen stream. ring buffer. similar to @xcite , this method allocates m c elements for each class c of the task in memory. the strategy is a fifo buffer, so the memory is always filled with the latest task observations. if the total number of classes is unknown, the value of m is gradually reduced as new tasks are observed. surprise. unexpected events have been shown to influence episodic memory in humans @xcite . one way to measure surprise is by computing the entropy of the output distribution of an input batch. analogous to @xcite , we use the time difference between the current entropy value and that of the previous batch to sample high-surprise elements. minimum margin. similar to @xcite , who introduced a margin-based method for cv replay models, we define the margin as the difference between the probability of the true class and the probability of the other most likely class. we store the most uncertain examples, that is, those with the smallest margin for which the probability of the true class is only marginally different from the probability of the other most likely class. ## results performance. text classification and question answering results are shown in for the question answering problem, ring buffer memory performed best. next, the naive random, reservoir, surprise and min. margin methods performed similarly. compared to the text classification results, the differences in average performance across models and the standard deviations are substantially smaller. this difference could be due to the more homogeneous nature of the question answering tasks (i.e., start and end span predictions), contrary to the heterogeneous set of classes used in a stream of text classification tasks. overall, the max. loss and surprise method results in lower returns, which is inconsistent with previous findings from cv @xcite . for the mof approach, we were not able to replicate the improvement in performance @xcite in this nlp-specific application. we suspect that this is caused by the unsuitability of the [cls] token for semantic similarity purposes (reimers and gurevych, 2019). finally, reservoir leads to the best results as it maintains a random sample over a global distribution that is not known in advance. this supports previous work on cv @xcite , which defaults to the reservoir sampling due to its simplicity and efficiency. we were able to confirm that the reservoir and naive random methods are indeed the most efficient in terms of their required training time, together with max. loss and ring buffer (see ## conclusion in this work, we studied memory population methods for episodic memory in the context of lifelong language learning. our empirical analysis shows that simple methods such as naive random and reservoir are the best choice for text classification and question answering because they randomly sample the global distribution. however, in the case of question answering, a balanced memory in terms of tasks leads to better results.