Generative Replay Inspired by Hippocampal Memory Indexing for Continual Language Learning (2023)

Abstract: Continual learning aims to accumulate knowledge to solve new tasks without catastrophic forgetting for previously learned tasks. Research on continual learning has led to the development of generative replay, which prevents catastrophic forgetting by generating pseudo-samples for previous tasks and learning them together with new tasks. Inspired by the biological brain, we propose the hippocampal memory indexing to enhance the generative replay by controlling sample generation using compressed features of previous training samples. It enables the generation of a specific training sample from previous tasks, thus improving the balance and quality of generated replay samples. Experimental results indicate that our method effectively controls the sample generation and consistently outperforms the performance of current generative replay methods.

## introduction humans and intelligent animals continually acquire new knowledge and skills throughout their lifetime. this ability, called continual learning (cl) or lifelong learning, is a fundamental requirement for human-like general intelligence @xcite . cl is also crucial for practical applications, as new data and tasks to train models appear every day in the real world. it is especially important for natural language processing (nlp) systems, in which vocabulary and language usage change over time. however, most neural network based models are trained with a static dataset. when learning different tasks sequentially, performance on the previously learned tasks tends to significantly degrade, referred to as catastrophic forgetting @xcite . learning new tasks without catastrophic forgetting has been a long-standing challenge in machine learning and neural networks. replay is an approach to alleviate catastrophic forgetting by retraining with previous tasks' data when training a new task. although replay-based methods are effective in most cl scenarios, it is necessary to retain training data for all previous tasks, which may cause problems with storage requirements and data privacy. therefore, generative replay was developed @xcite , which uses pseudo-samples generated from generation models instead of real samples. in the nlp domain, lamol @xcite was proposed as a generative replay framework, where a single language model simultaneously learns to solve tasks and to generate pseudo-samples (fig. 1a ). although the generative replay does not require any previous task's data, it typically underperforms actual sample replay because of the balance and quality problems in sample generation. regarding the balance problem, generation models tend to generate a large number of samples for recently learned tasks, that results in forgetting older tasks. this is due to the difficulty in controlling sample generation and catastrophic forgetting occurring in generation models. regarding the quality problem, generative replay methods commonly assume to generate even unseen samples which is not included in the past training datasets. however, since generating such unseen samples is more difficult than generating previously trained samples, it may cause the degradation of the quality of replay samples. to prevent the catastrophic forgetting, it is sufficient to generate only previously learned samples. to address these issues, we refer to the memory retrieval mechanism in the biological brain that achieves cl. according to the hippocampal indexing theory @xcite , the hippocampus encodes memory engrams for new neocortical activity patterns and uses them as memory indexes to recall past experiences. inspired by this hippocampal mechanism, we propose hippocam- @xcite context question answer @xcite [eos] we evaluated hmi on two different cl scenarios using the original lamol as a baseline. the first scenario is a sequence of different types of tasks, for which we used five natural language understanding (nlu) tasks from decanlp @xcite . the other scenario is a sequence of different domains in the same task, for which we used five text classification datasets and single-pass setting, which is considered as an ideal scenario for cl. the results indicate that hmi consistently outperforms lamol and improves robustness to training task order and amount of replay samples. we also investigated the balance of previously learned tasks in generated samples and found that hmi enables the generation of even old task samples, which indicates the controllability of sample generation with hmi. furthermore, we explored the potential of further improvement of hmi with different sample selection strategies for replay. regularization-based methods aim to constrain changes in model parameters important for previous tasks. various methods have been proposed to estimate the importance of each parameter. for example, elastic weight consolidation (ewc) @xcite uses the fisher information matrix. synaptic intelligence (si) @xcite estimates importance from the contribution to loss changes. memory-aware synapses (mas) @xcite computes the sensitivity of parameters on the basis of the gradient of model outputs. architecture-based methods dynamically change the network structure to assign model parameters for each task. progressive neural networks (pnn) @xcite freeze the current parameters and add a new column of the network when training a new task. instead of extending the network, packnet @xcite applies network pruning using dynamic filters to separate the neurons used for each task. replay-based methods mitigate catastrophic forgetting by retraining for previous tasks when training for a new one. mbpa++ (de masson d'autume et al., 2019) introduces an episodic memory that stores real samples of previous tasks to use for experience replay and local adaptation. meta-mbpa @xcite applies a meta-learning algorithm to improve mbpa++. to enhance the replaybased methods with a limited amount of samples, wang et al. (2020) and huang et al. (2021) also investigated effective selection strategies other than random sampling. instead of keeping real samples for replay, @xcite proposed generative replay, which trains a model to generate pseudosamples. @xcite proposed lamol as a generative replay method for nlp tasks. lamol uses gpt-2 @xcite to simultaneously learn a variety of nlp tasks and pseudosample generation. l2kd @xcite and dnr @xcite use knowledge distillation to extend lamol. mfk-lamol @xcite makes replay more efficient by using more forgotten pseudo-samples in generative replay. rational-lamol @xcite uses critical freezing guided by supervised or unsupervised rationale. rvae-lamol @xcite enhances lamol by mapping different tasks into a limited unified feature space. current generative replay methods have problems on the balance and quality of generated samples. to address these issues, we propose a samplegeneration control with the hmi method, inspired by the biological brain. in contrast to the previous work, our approach prevents low quality samples by using the assumption that a model generates only previously learned samples. hmi also achieves balanced sample generation by strong sample-level conditioning rather than task-level conditioning. although our hmi can be applied to most of the existing generative replay methods, similar to other recent work, we build hmi upon lamol, which is a simple generative replay baseline for cl in nlp and whose implementation code is available. ## lamol: language modeling for lifelong language learning before describing hmi, we briefly explain lamol @xcite , on which we propose our hmi, in this section. lamol is a generative replay framework using a single gpt-2 to solve different types of nlp tasks and generate pseudo-samples. in lamol, all training samples are fed into gpt-2 as a sequence of context, question, and answer. as illustrated at the top of figure 1a , gpt-2 learns each task in a qa manner, predicting the answer part on the basis of the given context and question. as well as training qa, gpt-2 learns language modeling (lm) to generate the whole sequence of the context, question, and answer, as illustrated at the bottom of figure 1a . during the training step, the parameters of gpt-2, θ gpt-2 , are optimized to minimize the qa loss l qa and the lm loss l lm together as @xmath0, where λ is a hyperparameter. when training for a new task, lamol generates pseudo-samples for previous tasks to use for replay. assume a stream of tasks {t 1 , t 2 , . . . t t } to train a model with lamol, where the number of tasks t may be unknown. before training a new task t i (@xmath1, gpt-2 generates pseudo-samples by top-k sampling from the first token @xcite . the number of pseudo-samples is γ|t i |, where γ is the sampling ratio and |t i | is the number of training samples in t i . defective samples, which do not have a unique @xcite token that indicates the start position of the answer, are discarded, and the others are mixed with t i to alleviate forgetting for @xmath2. when using the same @xcite for all tasks, the ratio for old tasks in the generated samples decreases exponentially in theory @xcite . therefore, @xcite proposed to replace @xcite with a task-specific token [task] (e.g., "__sst__") to control gpt-2 to generate pseudosamples belonging to the specific task. in the beginning of training for t i , γ i-1 |t i | pseudo-samples for each previous task, t 1 , . . . , t i-1 , are generated using the corresponding task-specific token. ## hippocampal memory indexing (hmi) in this section, we introduce our hippocampal memory indexing (hmi) that can suppress the problems of unbalanced and low-quality generation in the replay by accessing compressed features for previous training samples. ## analysis 6.1 balance of replay samples to validate the controllability in sample generation in hmi-lamol, we investigated the balance of each task in the replay samples generated in the experiments on the five nlu tasks, described in § 5.2.1. to classify the replay samples, we used the bert classifier model trained with the same training data as the experiments of cl. figure 3 shows the portion of each task in the generated replay samples for each replay step in the cl experiments. we first find that lamol generated many samples from more recently learned tasks rather than from older tasks. it became more evident in the smaller γ, which is consistent with the performance trend in cl. although task-specific tokens alleviate this problem to some extent, when γ = 0.01, almost all of the generated samples were from the most recently learned task in all replay steps. these results indicate that the task-specific tokens are insufficient to tie the generated samples to each task. however, hmi-lamol successfully controls the sample generation for each previous task. in contrast to lamol, it enables the generation of samples for older tasks even with the small γ. ## conclusion we proposed hippocampal memory indexing (hmi), inspired by the biological brain, that controls generative replay by conditioning sample generation with compressed representations of previous training samples. experimental results indicated that hmi successfully controls the sample generation of generative replay and consistently improves the cl performance and robustness. hmi is expected to be further improved by exploring better selection strategies for generating samples.