Gradient Based Memory Editing for Task-Free Continual Learning
Abstract
Prior work on continual learning often operate in a “task-aware” manner, by assuming that the task boundaries and identiﬁes of the data examples are known at all times.
While in practice, it is rarely the case that such information are exposed to the methods ( i.e. , thus called “taskfree”)–a setting that is relatively underexplored.
Recent attempts on task-free continual learning build on previous memory replay methods and focus on developing memory construction and replay strategies such that model performance over previously seen examples can be best retained.
In this paper, looking from a complementary angle, we propose a novel approach to “edit” memory examples so that the edited memory can better retain past performance when they are replayed.
We use gradient updates to edit memory examples so that they are more likely to be “forgotten” in the future.
Experiments on ﬁve benchmark datasets show the proposed method can be seamlessly combined with baselines to signiﬁcantly improve the performance.
Code has been released at https://github.com/INK-USC/GMED .
1. Introduction
Accumulating past knowledge and adapting to evolving environments are one of the key traits in human intelligence (McClelland et al., 1995).
While contemporary deep neural networks have achieved impressive results in a range of machine learning tasks (sometimes at the level of human performance), they haven’t yet to manifest the ability of continually learning over evolving data streams (Ratcliff, 1990).
These models are observed to suffer from catastrophic forgetting (McCloskey & Cohen, 1989; Robins, 1995) when trained in an online fashion– i e. , performance drop over previously seen examples during the sequential learning process.
To this end continual learning (CL) methods are developed to alleviate catastrophic forgetting issue when models are trained on non-stationary data streams (Goodfellow et al., 2013; Kirkpatrick et al., 2017).
Most existing work on continual learning pose an assumption that, when models are trained on a stream of tasks sequentially, the task speciﬁcations such as task boundaries or identities are exposed to the models ( i.e. , incremental task assumption).
These task-aware CL methods make explicit use of task information to avoid catastrophic forgetting issue, including consolidating important parameters on previous tasks (Kirkpatrick et al., 2017; Zenke et al., 2017; Nguyen et al., 2018), distilling knowledge from previous tasks (Li & Hoiem, 2017; Rannen et al., 2017), or separating taskspeciﬁc model parameters (Rusu et al., 2016;
Serr ` a et al., 2018).
However, in practice, it is likely the case that the data comes in a sequential, non-stationary fashion without task identity or boundary—a setting that is commonly termed as task-free (online) continual learning (Aljundi et al., 2018).
To tackle this challenging setting, recent attempts on taskfree CL methods have been made (Aljundi et al., 2018;
Zeno et al., 2018; Lee et al., 2020).
Regularization and architecture based approaches rely on inferring task boundaries or identities (Aljundi et al., 2018;
Lee et al., 2020), or estimating importance of parameters online (Zeno et al., 2018) to consolidate or separate model parameters.
On the other as memory-based CL methods have achieved strong results, which store a small set of previously seen instances in a ﬁx-sized memory, and utilize them for replay (Robins, 1995;
Rolnick et al., 2019) or regularization (Lopez-Paz &
Ranzato, 2017; Chaudhry et al., 2019a).
The core problem in memory-based CL methods is about how to manage the memory instances and replay them given the limited computation budget ( e.g. , number of instances to update each time) so that the model performance can be maximally preserved or enhanced.
This problem has been studied mainly from two dimensions in previous work:
1) what instances to include in the memory from the data stream (Aljundi et al., 2019b; Rebufﬁ et al., 2017; Chaudhry et al., 2019b); and 2) which instances in the memory need to be replay at a certain point (Aljundi et al., 2019a).
In this paper, we provide a new angle to solving the memory management problem in task-free continual learning and study how to make gradient updates on the memory examples.
We develop a novel memory editing algorithm which are complementary to existing memory replay methods and data sampling strategies for memory update, and can cope with them to form different integrated approaches.
It is challenging, as it asks for a principled and sound optimization objective of editing.
Our proposed method, named Gradientbased Memory EDiting ( GMED ), edits stored examples with gradient-based updates so that they are more likely to be forgotten.
Speciﬁcally, we estimate the forgetting of a stored example by its loss increase in the upcoming one online model update, which is differentiable regarding the store examples.
We then perform gradient ascent on stored examples so that they are more likely to be forgotten.
Experiments show that our algorithm consistently improves over baselines on four out of ﬁve benchmark datasets under various memory sizes.
Our ablation studies show the proposed editing mechanism outperforms alternative approaches such as random editing.
Notably, our algorithm can be seamlessly combined with best performing baselines to further improve performance.
The algorithm is general and can be coped with other models and datasets.
2. Related Works
Task-aware Continual Learning.
Most of continual learning algorithms are studied under “task-aware” settings, where the model visits a sequence of clearly separated “tasks”.
A great portion of algorithms make explicit use of task boundaries (Kirkpatrick et al., 2017; Rusu et al., 2016; Lopez-Paz & Ranzato, 2017), by learning separate parameters for each task, or discourage changes of parameters that are important to old tasks.
Existing continual learning algorithms can be summarized into three categories: regularization-based, architecture-based and databased approaches.
Regularization based approaches (Kirkpatrick et al., 2017; Zenke et al., 2017; Nguyen et al., 2018;
Adel et al., 2020) discourage the change of parameters that are important to previous data.
Architecture-based approaches (Rusu et al., 2016; Serr`a et al., 2018; Li et al., 2019) allows expansion of model architecture to separate parameters for previous and current data.
Data-based approaches (Robins, 1995; Shin et al., 2017; Lopez-Paz &
Ranzato, 2017) replay or constrain model updates with real or synthetic examples.
Task-free Continual Learning.
Recently, task-free continual learning (Aljundi et al., 2018) have drawn increasing interest, where we do not assume knowledge about task boundaries.
To the best of our knowledge, only a handful number of regularization based (Zeno et al., 2018; Aljundi et al., 2018), architecture based (Lee et al., 2020), generative replay based (Rao et al., 2019) approaches are applicable in the task-free CL setting.
Meanwhile, most memory [Figure] based continual learning algorithms are applicable to the task-free setting (Aljundi et al., 2019a;b).
Memory-based
CL algorithms store a subset of examples in a ﬁx-sized replay memory and utilize them later at training to alleviate forgetting.
Experience Replay (ER) (Robins, 1995) is a simple yet surprisingly powerful (Chaudhry et al., 2019b) approach in this category.
The algorithm stores a subset of examples in the memory and regularly draw a small batch from the memory to retrain on them later at training.
Recent research has studied online strategies to improve the performance gain when examples get replayed from two dimensions: in terms of which examples to store , and which examples to replay .
Figure 1 shows an categorization of memory-based task-free continual learning.
In terms of deciding which examples to store , one solution is to randomly select examples to store with reservoir sampling (Vitter, 1985), which ensures each example in the stream has an equal probability of being stored in the memory.
In addition to reservoir sampling, recent works also study storing prototypes (Rebufﬁ et al., 2017) or examples that contribute to the diversity of the stored examples, such as Gradient based Sample Selection (GSS)
(Aljundi et al., 2019b).
In terms of deciding which examples to replay , existing algorithms usually rely on random sampling from the replay memory.
Maximally Interfering Retrieval (MIR) (Aljundi et al., 2019a) select examples with the largest estimated forgetting.
In particular, a task-aware approach, Hindsight Anchor Learning (HAL) (Chaudhry et al., 2020), shares the same assumption that forgettable examples should be prioritized more.
However, HAL only applies to task-aware settings and requires extra memory storage to keep track of the learned anchors.
3. Background
In this section we ﬁrst present the problem formulation of task-free continual learning and then introduce preliminaries on memory-based continual learning methods.
3.1. Problem formulation
In task-free continual learning, we consider a (potentially inﬁnite) stream of data examples D , which have nonstationary data distribution.
At each time step t , the model receives one or a mini batch of examples ( x t , y t ) from the data stream D .
For simplicity of discussion, here we assume that example ( x t , y t ) from D is generated by:
ﬁrst sampling a latent “task” z ∼ P ( z ; t ) , and then followed by sampling a data example from a joint data distribution P ( x, y | z ; t ) that is conditioned on task z , i.e. , ( x t , y t ) ∼ P ( x, y | z ; t ) .
Here
P ( z ; t ) is non-i.i.d and depends on time; and P ( x, y | z ; t ) also changes over time.
The goal of task-free online continual learning is to seek a classiﬁcation model f ( x ; θ ) , parameterized by θ , over new example(s) ( x, y ) from the data stream D that minimizes a predeﬁned loss (cid:96) ( x, y ; θ ) while not increasing the loss on previously seen examples.
Such capability is evaluated by testing the model over a held-out set which consists of new examples that follows the task distribution of the stream D .
3.2. Memory-based CL Methods
In a nutshell, memory-based CL algorithms maintain a ﬁxsized replay memory M which is used for storing (subset of) previously seen examples ( x t , y t ) from the stream D .
When the memory is full, the algorithm needs to either identify a memory examples ( x, y ) to be replaced by new example, or to discard the new example it just received.
Following the same setup in previous memory-based CL methods, our experiments use reservoir sampling (Vitter, 1985) to determine how the memory will be updated with new examples received from stream D : every time the model receives a new example, it draws an integer j between 0 and N randomly, where N is the number of examples visited so far.
If j < | M | ( i.e. , the memory size or budget), it replace the example at the j -th position in the memory with the new example; otherwise, this newly received example will be discarded.
Reservoir sampling ensures at each time step each visited example is kept with an equal probability | M | /N .
At each time step t , the algorithm also needs to determine which of the memory examples will be used for replay.
Same as previous methods, we randomly sample one or a mini-batch of examples ( x, y ) from the memory M .
As an alternative replay strategy, MIR (Aljundi et al., 2019a) identiﬁes a subset of memory examples based on a predeﬁned optimization objective ( i.e, perform one step of training on ( x, y ) .), and then replays the selected examples.
4. Gradient Based Memory Editing
We propose Gradient based Memory Editing (GMED), a novel algorithm for updating the stored memory examples in an online fashion.
We state our hypothesis about which examples may help to retain performance in Sec. 4.1.
We then formulate an online optimization objective for example editing in Sec. 4.2.
In Sec. 4.3 and 4.4, we introduce algorithmic details of GMED and its integration with MIR.
4.1. Hypothesis for Memory Editing
Given that there is no prior knowledge about the forthcoming examples in the data stream D , previous task-free CL methods usually impose (implicit) assumptions regarding what kinds of examples may improve model’s test performance after replaying these examples.
For example,
GSS (Aljundi et al., 2019b) assumes that the diversity of memory examples contributes to the model performance;
MIR (Aljundi et al., 2019a) and HAL (Chaudhry et al., 2020) assume that replaying examples that are likely to be “forgotten” can beneﬁt the performance.
Empirical study by Toneva et al.
(2019) also show that there are constantly forgotten examples, which beneﬁt overall performance when they get replayed compared to other examples.
Our work is based on a similar hypothesis that: replaying examples that are likely to be forgotten by the current model helps retain its test performance .
However, rather than determining which memory examples will be replayed next, our work studies a complementary aspect on how to edit the memory examples, based on current model’s behaviors, to make the examples more ”forgettable”.
Formally, we use the loss (cid:96) ( x, y ; θ t ) to measure the model’s performance over an arbitrary example ( x, y ) at the time step t , where θ t denotes the model parameters at t .
Suppose we train the model on D until the time step T , the “forgetting” measurement of an example ( x, y ) for the model at t , denoted by d t : T ( x, y ) , is deﬁned as the “loss increase” at time T compared to that at time t , shown as follows.
[Equation]
where θ t is the model parameters at the current time step t .
A larger d t : T ( x, y ) indicates that the example is more likely to be forgotten by the model at the end of training.
Our hypothesis assumes replaying memory examples that [Figure] have larger forgetting measurement helps model to generalize.
The hypothesis is formally stated as follows.
Hypothesis 1.
Given a budget of C examples to replay at time t , in order to minimize the loss over new examples from task k
(e.g., in test set), an ideal strategy is to replay the most forgettable examples, denoted as S k , selected from the training examples of task k , denoted as D k .
Following Hypothesis 1, the set of examples S k can be obtained by solving the following optimization problem.
[Equation]
where D k is the training examples of the latent task k .
Unfortunately, the sample selection problem in Eq.
2 cannot be solved in an online setting, even when we know task identities, as one can only evaluate the objective function in
Eq. 2 at a future time step T .
Therefore, approximations and modiﬁcations are necessary to enable online optimization.
4.2. Online Optimization for Memory Editing
We propose an optimization problem that is tractable in an online fashion, which shares the same goal of making memory examples used for replay more likely to be forgotten.
We modify Eq.
2 where we (1) edit individual examples stored in memory instead of selecting the most forgettable examples from D k as shown in Eq. 2, and (2) estimate the forgetting measure in an online fashion, and (3) relax the constraint S k ⊆ D k for selecting examples.
Formally, suppose that the model is at the t -th time step and that ( x, y ) is an example from the a certain task k .
In order to retain the performance on test examples from the same task k , we propose the following objective:
[Equation]
where d ( · ) is the forgetting deﬁned in Eq.
4 and (cid:96) ( x, y ; θ t ) is the loss of the example ( x, y ) at the current time step t .
β is a trade-off hyper-parameter deciding the regularization strength.
The optimal ( x ∗ , y ) has the same label y
as the original example ( x, y ) .
Speciﬁcally, we discuss three main differences in Eq.compared to Eq.
2 in the Hypothesis 1 as follows.
Editing instead of Selection .
Eq.
2 describes a combinatorial optimization problem of selecting k most forgettable examples.
In contrast, Eq. 3 describes an optimization problem over an continuous input space with the objective of maximizing forgetting of each individual example.
Estimating Forgetting Online .
We maximize the forgetting of the example ( x, y ) in the upcoming one update at time step t + 1 , noted as d t : t +1 ( x, y ) , instead of the forgetting when the model get evaluated, i.e., d t : T ( x, y ) .
The former can be evaluated online efﬁciently without any overhead on replay memory.
Relaxed Constraints .
Eq.
3 do not constrain ( x, y ) ∈ D k .
It allows ( x ∗ , y ) to be an arbitrary example in the input space without being a real example from D k .
In practice, ( x, y ) is initialized as different input examples, and we perform only one or a few gradient updates each time it is drawn from the memory for replay.
We also add a regularization term β(cid:96)
( x, y ; θ t ) to discourage the loss increase on the example.
The editing is made conservative, so that the edited ( x (cid:48) , y ) is still likely to be an example from its original latent task.
The objective in Eq.
3 is differientiable with respect to x , allowing us to update x with gradient ascent.
In the rest of this section, we introduce algorithmic details of GMED.
4.3. The GMED Algorithm
Given the hypothesis and the optimization formulations, we introduce the algorithmic details in GMED.
We start by combining GMED with ER.
It introduces an additional “editing” step before replaying examples drawn the memory.
For simplicity of discussion, we assume the batch size is 1, but the method also applies with a larger batch size.
We assume at time step t the model receives a stream example ( x ( D ) , y ( D ) ) from the training stream D , and randomly draws a memory example ( x, y ) from the memory M .
We ﬁrst compute the forgetting ( i.e., loss increase) on the memory example ( x, y ) when the model performs one gradient update on parameters with the stream example ( x ( D ) , y ( [Equation]
[Equation]
where θ t and θ (cid:48) t are model parameters before and after the gradient update respectively.
Figure 2(a) visualize the steps to compute forgetting.
Following the optimization objective proposed in Eq.
2, we perform a gradient update on x to increase its forgetting, while using a regularization term to discourage the loss increase on the example at the current time step.
[Equation]
where α is a hyperparameter for the stride of the update.
Figure 2(b) visualize the editing step.
The algorithm then discards the updated parameter θ (cid:48) t , and updates model parameters θ t with the updated memory example ( x (cid:48) , y ) and the stream example ( x ( D ) , y ( D ) ) , in a similar way to ER.
[Equation]
We replace the original examples in the memory with the edited example.
In this way, we continuously edit examples stored in the memory alongside training.
Algorithmsummarize the proposed ER+GMED algorithm.
4.4. Hybrid Methods
GMED is studied from a complementary direction compared to most prior approaches.
Therefore, we can combine
GMED with existing memory-based CL algorithms without much effort.
We illustrate the point by proposing a hybrid approach of GMED and MIR.
At each time step, MIR retrieves the most forgettable examples from the memory with the forgetting deﬁned as Eq. 4.
We do not edit the selected examples directly; instead, we additionally draw another random mini-batch from the memory to apply editing.
The motivation is that examples drawn by MIR are already most forgettable ones; if we directly perform editing on them, we would fall into a loop that letting forgettable examples more forgettable, which is not desired.
5. Experiments
We compare the performance of GMED against state-of-theart CL algorithms on ﬁve benchmark datasets.
We introduce our experimental setup and discuss our results on comparisons with baselines and performance analysis.
Input: learning rate τ , edit stride α , regularization strength β , model parameters θ Receives : stream example ( x ( D ) , y ( D ) ) Initialize : replay memory M for t = 1 to T do ( x, y ) ∼ M (cid:96) before ← loss ( x, y, θ t ) (cid:96) stream ← loss ( x ( D ) , y ( D ) , θ t ) //update model parameters with stream examples, discarded later θ (cid:48)
t ← SGD ( (cid:96) stream , θ t , τ ) //evaluate forgetting of memory examples (cid:96) after ← loss ( x, y, θ (cid:48) t ) d ← (cid:96) after − (cid:96) before //edit memory examples x (cid:48) ← x + α ∇ x ( d − β(cid:96) before ) (cid:96) = loss (( x (cid:48) , y ) ∪ ( x ( D ) , y ( D ) ) , θ t )
θ t +1 ← SGD ( (cid:96), θ t , τ ) replace ( x, y ) with ( x (cid:48) , y ) in M reservoir update ( x ( D ) , y ( D ) , M )
We consider ﬁve public CL datasets in our experiments.
Split MNIST (Goodfellow et al., 2013) splits the
MNIST (LeCun et al., 1998) dataset of handwritten digit classiﬁcation into 5 disjoint subsets by their labels as different tasks.
The dataset consists of 5 tasks, where each task consists of 1,000 training examples.
The goal is to classify over all 10 digits when the training ends.
Permuted MNIST (Goodfellow et al., 2013) applies a ﬁxed random pixel permutation to the MNIST dataset as different tasks.
The dataset consists of 10 tasks with 1,000 training examples each.
The goal is to classify over 10 digits without knowing the permutation applied.
Rotated MNIST (Lopez-Paz & Ranzato, 2017) applies a ﬁxed image rotation between 0 to 180 degree to the MNIST dataset.
At k -th task, the degree is randomly selected from the interval [ 180( k − 1) K , 180 kK ] , where k ∈ { 1 , 2 , ..., K } and
K is the number of tasks.
The dataset consists of 20 tasks with 1,000 examples each.
The models classify overdigits without knowing the rotation when the training ends.
Split CIFAR-10 (Zenke et al., 2017) splits the CIFAR10 (Krizhevsky, 2009) image classiﬁcation dataset intodisjoint subsets by their labels.
The dataset consists oftasks, where each task consists of 10,000 training examples.
The models classify over all 10 classes.
Split mini-ImageNet (Aljundi et al., 2019a) splits the mini- [Table] ImageNet (Deng et al., ; Vinyals
et al., 2016) classi ﬁcation dataset to 20 disjoint subsets by their labels.
The dataset consists of 20 tasks, where each task consists of 1,250 examples in total from 5 classes.
The models classify over all 100 classes.
We do not provide information about task identities or task boundaries to the model at both training and test time.
Following deﬁnitions in prior literature (van de Ven & Tolias, 2019), our Split MNIST, Split CIFAR-10, and Split miniImageNet experiments are under the class-incremental setup, while Permuted MNIST and Rotated MNIST experiments are under the domain-incremental setup.
5.2. Compared Methods
For our methods, we report the performance of ER + GMED and MIR + GMED, where we build GMED upon ER or
MIR as introduced in section 4.3.
We compare with several memory based continual learning methods, but do not compare with regularization based approaches, as most of them are known to perform poorly in the class-incremental setup (van de Ven & Tolias, 2019).
We include task-aware approaches for comparison, but it should be noted that these approaches make use of extra information compared to taskfree approaches.
• Experience Replay (ER) (Robins, 1995; Rolnick et al., 2019)
stores examples in a ﬁx-sized memory for future replay.
We use reservoir sampling to decide which examples to
store and replace.
Following prior works (Aljundi et
al., ; 2019a; Chaudhry et al , at each time step we draw the same number of examples as the batch size from the memory to replay, which are both set to 10.
The algorithm applies to the task-free scenario.
• Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato, 2017) also stores examples in a memory.
Before each model parameter update, GEM project gradients of model parameters so that the update does not incur loss increase on any previous task.
The approach is not task-free.
• Averaged Gradient Episodic Memory (AGEM)
(Chaudhry et al., 2019a) prevents the average loss increase on a randomly drawn subsets of examples from the memory.
We draw 256 examples to compute the regularization at each iteration.
The approach is task-free.
• Bayesian Gradient Descent
(BGD)
(Zeno et al., 2018) is a regularization-based continual learning algorithm.
It adjust learning rate for parameters by estimating their certainty, which notes for their importance to previous data.
The approach is task-free.
• Gradient based Sample Selection (GSS) (Aljundi et al., 2019b) builds upon ER by encouraging the diversity of stored examples.
We use GSS-Greedy, which is the best performing variant in the paper.
The approach is task-free.
• Hindsight Anchor Learning (HAL) (Chaudhry et al., 2020) learns an pseudo “anchor” example per task per class in addition to the replay memory by maximizing its estimated forgetting, and tries to ﬁx model outputs on the anchors at training.
However, unlike GMED, they estimate forgetting with loss increase on examples when the model train for a pass on the replay memory (and thus forgetting is estimated with “hindsight”).
The approach is not task-free.
• Maximally Interfering Retrieval
(MIR)
(Aljundi et al., 2019a) improves ER by selecting top forgettable examples from the memory for replay.
Following the ofﬁcial implementation, we evaluate forgetting on a candidate set ofexamples for mini-ImageNet dataset, and 50 examples for others.
While the approach is task-free, the ofﬁcial implementation ﬁlter out memory examples that belong to the same task as the current data stream, which assumes knowledge about tasks boundaries.
We remove this operation to adapt the method to the task-free setup.
Therefore, our results are not directly comparable to the ofﬁcial results.
Besides, we also report the following results for reference.
• Fine tuning performs online updates on model parameters without applying continual learning algorithms.
• iid Online .
We randomly shufﬂe the data stream, so that the model visits an i.i
d. stream of examples.
• iid Ofﬂine is similar to iid-Online, but we allow multiple pass over the data.
We also consider two variants of memory editing strategies for ablation study:
• Random Edit is an ablation study for GMED.
It edit examples to a random direction with a ﬁxed stride.
The stride is tuned by validation, similar to GMED.
• Hindsight Edit estimates example forgetting in a similar way to HAL.
The model obtain temporal updated parameters θ (cid:48) t by training on a random mini-batch from the memory instead of stream examples.
5.3. Experiment Setup
We mostly follow the training setup in (Aljundi et al., 2019a).
For three MNIST datasets, we use a MLP classiﬁer with [Table] [Table] hidden layers with 400 hidden units each.
For Split CIFAR10 and Split mini-ImageNet datasets, we use a ResNetclassiﬁer.
We use SGD optimizer with a learning rate of 0.05 for MNIST and 0.1 for Split CIFAR-10 and Split mini-ImageNet datasets.
We perform three steps of model parameter updates for each example we visit in the Split mini-ImageNet dataset, and one step for others, following (Aljundi et al., 2019a).
Similarly, we perform three steps of memory editing in GMED in Split mini-ImageNet dataset, and one for others.
We use a batch size ofthroughout the experiment.
By default, we set the size of replay memory as 10,000 for split mini-ImageNet, andfor all other datasets.
We also report performance under different memory sizes.
Hyperparameter Selection for GMED.
GMED introduces two hyperparameters: stride of the editing α , and the regularization strength β .
Following (Chaudhry et al., 2019a), we tune the hyperparameters with only the training and validation set of ﬁrst three tasks.
The models are trained until convergence before they proceed to the next task.
The tasks used for hyperparameter search are included for reporting ﬁnal accuracy, following (Ebrahimi et al., 2020).
We perform a grid search over all combinations of α and β and select the one with the best [Figure] validation performance on the ﬁrst three tasks.
We select α from [0 . 01 , 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 , 5 . 0 , 10 . 0] , and select β from [0 , 10 − 3 , 10 − 2 , 10 − 1 , 1] .
We tune hyperparameters for ER-GMED and apply the same hyperparameters on
MIR+GMED.
Table 1 show the optimal hyperparameters selected for each dataset.
5.4. Results and Performance Analysis
We report the ﬁnal accuracy achieved by different methods in Table 2.
We summarize the following ﬁndings.
Overall Performance.
From the results, we see MIR +
GMED achieves best performance among memory based continual learning algorithms.
The improvement of GMED differs by datasets.
ER+GMED clearly improves over ER by an absolute margin of 1.72%, 1.67%, and 1.79% accuracy respectively in Split MNIST, Split CIFAR-10, and Split mini-ImageNet datasets.
On Rotated MNIST the improvement is less signiﬁcant, with an absolute accuracy improvement of 0.94%, while we do not see a meaningful improvement on Permuted MNIST dataset.
MIR+GMED improves performance on all the datasets.
Similarly, the improvement is clear on Split MNIST, Split CIFAR-10, and
Split mini-ImageNet with an absolute accuracy improvement of 2.98%, 1.07%, and 1.28%, and less signicifant on
[Figure]
[Figure]
Comparison with Random and Hindsight Editing .
Table 3 compares between GMED, random editing and hindsight editing.
We see ER+Random Edit performs worse than ER + GMED in all cases.
However, we see ER+Random
Edit outperforms ER on split CIFAR-10.
We cojecture the reason is that the random editing alleviates the overﬁtting to memory examples.
We also observe GMED outperforms ER + Hindsight Editing in all cases.
Effect of the Regularization Term Figure 3 compare the performance between the tuned regularization strength β and 0 in two the Rotated MNIST and split mini-ImageNet datasets.
We see the regularization term improves the performance on two datasets.
Performance Under Various Memory Sizes.
Figureshow the performance under various memory sizes.
We see in Split MNIST, Rotated MNIST, Split CIFAR-10 and
Split mini-ImageNet, the improvement of ER+GMED over
ER is consistent under various memory sizes.
5.5. Case Study and Discussion
Visualization of Edited Examples.
Figure 5 visualize the editing on memory examples.
We show examples from ﬁrst two task (0/1, 2/3) in the Split MNIST dataset.
The ﬁrst and second rows show the original and edited examples, noted as x before and x after .
The third row shows the difference
[Figure]
between two ∆ x = x after − x before .
We see no signiﬁcant visual differences between original and edited examples.
However, by looking at the difference ∆ x , we see there are examples whose contours get exaggerated, e.g. , examplesand 12, and some get blurred, e.g. , examples 2, 3, 5, and 6.
Intuitively, to make an ambiguous example more forgettable, the editing should exaggerate its features; while to make a typical example more forgettable, the editing should blur its features.
Our visualizations align with the intuition above: examples 1 and 12 are not typically written digits, while examples like 2, 3, 5, and 6 are typical.
Visualization of Editing Directions .
In Figure 6, we show the t-SNE (Maaten & Hinton, 2008) visualization of the editing vector ∆ x
= x after − x before for examples from ﬁrst 2 tasks in Split MNIST.
We see the editing vectors cluster by the labels of the examples.
It implies the editing performed is correlated with the labels and is clearly not random.
6. Conclusion
In this paper, we propose Gradient based Memory Editing for task-free continual learning.
The approach estimates forgetting of stored examples online and edit them so that they are more likely to be forgotten in upcoming updates.
Experiments on benchmark datasets show our method can be combined with existing approaches to signiﬁcant improve over baselines on several benchmark datasets.
Our analysis further show the method is robust under various memory sizes, and outperforms alternative editing methods.