L EARNING WITH L ONG - TERM R EMEMBERING : F OLLOWING THE L EAD OF M IXED S TOCHASTIC G RADIENT
Anonymous authors
Paper under double-blind review
A BSTRACT
Current deep neural networks can achieve remarkable performance on a single task.
However, when the deep neural network is continually trained on a sequence of tasks, it seems to gradually forget the previous learned knowledge.
This phenomenon is referred to as catastrophic forgetting and motivates the ﬁeld called lifelong learning.
The central question in lifelong learning is how to enable deep neural networks to maintain performance on old tasks while learning a new task.
In this paper, we introduce a novel and effective lifelong learning algorithm, called
MixEd stochastic GrAdient (MEGA), which allows deep neural networks to acquire the ability of retaining performance on old tasks while learning new tasks.
MEGA modulates the balance between old tasks and the new task by integrating the current gradient with the gradient computed on a small reference episodic memory .
Extensive experimental results show that the proposed MEGA algorithm signiﬁcantly advances the state-of-the-art on all four commonly used lifelong learning benchmarks, reducing the error by up to 18%.
I NTRODUCTION
A signiﬁcant step towards artiﬁcial general intelligence (AGI) is to enable the learning agent to acquire the ability of remembering past experiences while being trained on a continuum of tasks.
Current deep neural networks are capable of achieving remarkable performance on a single task (Goodfellow et al., 2016).
However when the network is retrained on a new task, its performance drops drastically on previously trained tasks, a phenomenon which is referred to as catastrophic forgetting (Ratcliff, 1990; Robins, 1995; French, 1999; Kirkpatrick et al., 2017).
In stark contrast, human cognitive system is capable of acquiring new knowledge without damaging previously learned experiences.
It is thus of great importance to develop algorithms to allow deep neural networks to achieve continual learning capability (i.e., avoiding catastrophic forgetting).
The problem of catastrophic forgetting motivates the ﬁeld called lifelong learning (Thrun &
Mitchell, 1995; Kirkpatrick et al., 2017; Parisi et al., 2019).
A central dilemma in lifelong learning is how to achieve a balance between the performance on old tasks and the new task (Robins, 1995; Kirkpatrick et al., 2017).
During the process of learning the new task, the originally learned knowledge will typically be disrupted, which leads to catastrophic forgetting.
On the other hand, a learning algorithm biasing towards old tasks will interfere with the learning of the new task.
Several lines of methods are proposed recently to address this issue.
Examples include regularization based methods (Kirkpatrick et al., 2017; Zenke et al., 2017), knowledge transfer based methods (Rusu et al., 2016), episodic memory based methods (Lopez-Paz et al., 2017; Chaudhry et al., 2018b;
Riemer et al., 2018).
However, the existing methods require over-parameterized neural networks (Kirkpatrick et al., 2017; Chaudhry et al., 2018a) or are not ﬂexible enough handle the stochastic nature of the learning process (Lopez-Paz et al., 2017; Chaudhry et al., 2018b) since they did not explicitly consider the model’s performance on the current task and old tasks in the learning process.
In this paper, we propose a novel and effective lifelong learning algorithm, called MixEd stochastic
GrAdient (MEGA), to address the catastrophic forgetting problem.
We cast the problem of balancing the performance on old tasks and the new task as an optimization problem with composite objective.
Our formulation is general and closely related to several recently proposed lifelong learning algorithms (Lopez-Paz et al., 2017; Chaudhry et al., 2018b; Riemer et al., 2018).
We approximately solve the optimization problem using one-step stochastic gradient descent with the standard gradient replaced by the proposed mixed stochastic gradient.
The mixed stochastic gradient is derived from the gradients computed on the data of the current task and an episodic memory which stores a small subset of observed examples from old tasks (Lopez-Paz et al., 2017; Chaudhry et al., 2018b; Riemer et al., 2018).
Based on our derivation, the direction of the mixed stochastic gradient balances the loss on old tasks and the new task in an adaptive manner.
Therefore, the proposed MEGA algorithm allows deep neural networks to learn new tasks while avoiding catastrophic forgetting.
Our contributions are as follows.
(1) We propose a novel and effective algorithm, called MEGA, for lifelong learning problems.
(2) We extensively evaluate our algorithm using conventional lifelong learning benchmark datasets, and the results show that the proposed MEGA algorithm signiﬁcantly advances the state-of-the-art performance across all the datasets.
MEGA achieves an average accuracy of 91.21 ± 0.10% on Permuted MNIST , which is 2% better than the previous state-of-the-art model.
On Split CIFAR , our proposed MEGA achieves an average accuracy of 66.12 ± 1.93%, which is about 5% better than the state-of-the-art method.
Specially, on the Split
CUB dataset, MEGA achieves an average accuracy of 80.58 ± 1.94%, which surpasses the multi-task baseline which is previously believed as an upper bound performance of lifelong learning algorithms (Chaudhry et al., 2018b).
(3) Finally, we also show that the proposed MEGA algorithm can handle increasingly non-stationary settings when the number of tasks becomes signiﬁcantly larger.  R ELATED W ORK
Improving the continual learning ability of neural network is a prerequisite to extending it to more practical vision tasks.
Several lines of lifelong learning methods are proposed recently, we categorize them into different types based on the methodology,
Regularization based approaches: EWC (Kirkpatrick et al., 2017) adopts Fisher information matrix to prevent important weights for old tasks from changing drastically.
In PI (Zenke et al., 2017), the authors introduce intelligent synapses which endows each individual synapse with a local measure of “importance” to avoid old memories from being overwritten.
RWALK (Chaudhry et al., 2018a) utilizes a KL-divergence based regularization for preserving knowledge of old tasks. While in MAS (Aljundi et al., 2018) the importance measure for each parameter of the network is computed based on how sensitive the predicted output function is to a change in this parameter. Knowledge transfer based methods: PROG-NN (Rusu et al., 2016) is a representative knowledge transfer based lifelong method.
In PROG-NN, a new “column” with lateral connections to previous hidden layers is added for each new task.
The lateral connections allow the new task to leverage the knowledge extracted from the old tasks.
Recently, in Lee et al.
(2019), the authors proposed a method to leverage unlabeled data in the wild to avoid catastrophic forgetting using knowledge distillation.
The authors introduce global distillation for incorporating unlabeled data in the learning process.
Episodic memory based approaches:
In episodic memory based lifelong learning methods, a small episodic memory is used for storing a subset of the examples from old tasks.
Different episodic memory based approaches differ in the way of computing the reference gradients of the episodic memory.
GEM (Lopez-Paz et al., 2017) computes the reference gradients using each individual previous tasks while in AGEM (Chaudhry et al., 2018b)
the reference gradient is computed on the episodic memory by randomly sampling a batch of examples.
MER a (Riemer et al., 2018) is recently proposed lifelong learning algorithm which maintains an experience replay style memory with reservoir sampling and employs a meta-learning style training strategy.
Several methods are proposed recently to improve the episodic memory based approaches.
In (Aljundi et al., 2019), a line of methods are proposed to select important samples to store in the memory in order to reduce memory size.
Instead of storing samples, in (Farajtabar et al., 2019) the authors proposed Orthogonal
Gradient Descent (OGD) which projects the gradients on the new task onto a subspace in which the projected gradient will not affect the model’s output on old tasks and is still useful for learning the new task.
There are also some other lifelong learning methods which do not fall in above categories.
In (He & Jaeger, 2018), the authors proposed a variant of the standard back-propagation algorithm called conceptor aided backprop that shields gradients against degradation of performance on old task.
Zeng et al.
(2019) proposed orthogonal weights modiﬁcation (OWM) to enables networks to continually learn different mapping rules in a context-dependent way.
Our proposed method belongs to episodic memory based appraoch and is most related to Chaudhry et al.
(2018b).
Our work differs from Chaudhry et al.
(2018b) in two aspects.
First, we introduce a more effective to modify the direction of the current gradient which put equal emphasis on both old tasks and the new task.
Second, we explicitly consider the performance of the model on old tasks and the new task in the process of modifying the gradient direction.
Our method is also related to several multi-task learning works (Sener & Koltun, 2018;
Kendall et al., 2018; Chen et al., 2017).
In (Sener & Koltun, 2018; Kendall et al., 2018), the authors aim at achieving a good balance between different tasks by learning to weigh the loss on each task .
In contrast, our approach directly leverages loss information in the context of lifelong learning for overcoming catastrophic forgetting.
Compared with (Chen et al., 2017), instead of using the gradient norm information, our method and Lopez-Paz et al.
; Chaudhry et al.
(2018b) focus on modifying the direction of the current gradient.
L IFELONG L EARNING 3. P ROBLEM S TATEMENT
Lifelong learning (LLL) (Rusu et al., 2016; Kirkpatrick et al., 2017; Lopez-Paz et al., 2017;
Chaudhry et al., 2018b) considers the problem of learning a new task without degrading performance on old tasks, i.e., to avoid catastrophic forgetting (French, 1999; Kirkpatrick et al., 2017).
Suppose there are T tasks which are characterized by T datasets: { D 1 , D 2 , .., D T } .
Each dataset
D t consists of a list of triplets ( x i , y i , t ) , where y i is the label of i -th example x i , and t is a task descriptor that indicates which task the example comes from.
Similar to supervised learning, each dataset D t is split into a training set D trt and a test set D tet .
In the learning protocol introduced in Chaudhry et al.
(2018b), the tasks are separated into D CV = { D 1 , D 2 , ..., D T CV } and D EV =
{ D T CV +1 , D T CV +2 , ..., D T } .
D CV is used for cross-validation to search for hyperparameters.
D EV is used for actual training and evaluation.
As pointed out in Chaudhry et al.
(2018b), some regularization-based lifelong learning algorithms, e.g.,
Elastic
Weight Consolidation (Kirkpatrick et al., 2017), are sensitive to the choice of the regularization parameters.
Introducing D CV can help ﬁnd the best regularization parameter without exposing the actual training and evaluation data.
While searching for the hyperparameters, we can have multiple passes over the examples in D CV , the training is performed on D EV with only a single pass over the examples (Lopez-Paz et al., 2017; Chaudhry et al., 2018b).
In lifelong learning, a given model f ( x ; w ) is trained sequentially on a series of tasks { D T CV +1 ,
D T CV +2 , ..., D T } .
When the model f ( x ; w ) is trained on task D t , the goal is to predict the labels of the examples in D tet by minimizing the empirical loss (cid:96) t ( w ) on D trt in an online fashion without suffering accuracy drop on { D te T CV +1 , D te T CV +2 , ..., D tet } . 3. E VALUATION M ETRICS
Average Accuracy and Forgetting Measure (Chaudhry et al., 2018a) are common used metrics for evaluating performance of lifelong learning algorithms.
In Chaudhry et al.
(2018b), the authors introduce another metric, called Learning Curve Area (LCA), to assess the learning speed of different lifelong learning algorithms.
In this paper, we further introduce a new evaluation metric, called Long-term Remembering (LTR), to characterize the ability of lifelong learning algorithms for remembering the performance of tasks trained in the far past.
Suppose there are M k mini-batches in the training set of task D k .
Similar to Chaudhry et al.
(2018b), we deﬁne a k,i,j as the accuracy on the test set of task D
j after the model is trained on the i -th minibatch of task D k .
Generally, suppose the model f ( x ; w ) is trained on a sequence of T tasks { D 1 ,
D 2 , ..., D T } .
Average Accuracy and Forgetting Measure after the model is trained on the task D k are deﬁned as
[Equation]
where f kj = max l ∈{ 1 , 2 ,..,k − 1 } a l,M l ,j − a k,M k ,j .
Clearly, A T is the average test accuracy and F T assesses the degree of accuracy drop on old tasks after the model is trained on all the T tasks.
Learning Curve Area (LCA) (Chaudhry et al., 2018b) at β is deﬁned as, LCA [Equation] =
[Equation]
(2) where Z b = 1 T (cid:80) Tk =1 a k,b,k .
Intuitively, LCA measures the learning speed of different lifelong learning algorithms.
A higher value of LCA indicates that the model learns quickly.
We refer the readers to Chaudhry et al.
(2018b)
for more details about LCA.
All the metrics introduced above fail to capture one important aspect of lifelong learning algorithms, that is, the ability to retain performance on the tasks trained in the far past.
In this paper we introduce a new metric, called Long-Term Remembering (LTR), which is deﬁned as
[Equation]
After the model is trained on all the T tasks, LTR quantiﬁes the accuracy drop on task D j relative to a j,M j ,j .
The coefﬁcient T − j emphasizes more on the tasks trained earlier.
Different algorithms can have the same average accuracy but very different LTR based on their ability to maintain the performance on the past tasks (a.k.a, long-term remembering ).
M IXED S TOCHASTIC G RADIENT
In this section, we introduce the proposed Mixed Stochastic Gradient (MEGA) algorithm.
Following previous works (Lopez-Paz et al., 2017; Chaudhry et al., 2018b), when the model is trained on the t -th task, an episodic memory M is used for storing a subset of the examples from all the old tasks k < t .
The main idea of MEGA is to minimize the loss on the episodic memory and the t -th task by iteratively moving in the direction of the proposed mixed stochastic gradient.
In the lifelong learning setting, the learning of task t is conducted over a single pass of the training examples in an online fashion.
To establish the tradeoff between the performance on old tasks and the t -th task, we consider the following optimization problem with composite objective:
[Equation]
where w ∈ R d is the parameter of the model, ξ, ζ are random variables with ﬁnite support, (cid:96) t ( w ) is the expected training loss of the t -th task, (cid:96) ref ( w ) is the expected loss calculated on the data stored in the episodic memory, α 1 and α 2 are hyperparameters which control the relative importance of (cid:96) t ( w ) and (cid:96) ref ( w ) .
Intuitively, a larger (cid:96) ref ( w ) signiﬁes catastrophic forgetting.
Note that during the learning process of each task, every data sample is i.i.d., and hence the current example of task t could be viewed as a random sample due to online-to-batch conversion argument (Cesa-Bianchi et al., 2004).
In the lifelong learning setting, (cid:96) t ( w ; ξ ) is the training loss calculated on the current mini-batch (controlled by ξ ) of task t , (cid:96) ref ( w ; ζ ) is the loss calculated on a random mini-batch (controlled by ζ ) sampled from the episodic memory.
In the traditional online learning, the loss is calculated only based on the current received example but not on historical samples, and hence α 1 = 1 , α 2 = 0 .
In this case, the weights are only optimized for the current task while ignoring previous tasks which leads to catastrophic forgetting.
If α 2 > 0 , this formulation naturally involves examples from old tasks.
In lifelong learning, since we need to consider performance on both old tasks and the current task, we typically do not consider the degenerate case when α 2 = 0 .
We use w tk to denote the weight when the model is being trained on the k -th mini-batch of task t .
Clearly, both (cid:96) t ( w ) and (cid:96) ref ( w ) are determined by w tk during training.
This implies that the relative value of (cid:96) t ( w ) and (cid:96) ref ( w ) is changing between mini-batches.
Therefore, α 1 and α 2 should be adjusted adaptively based on w tk in order to achieve a good balance between old tasks and the current task.
To this end, with a little abuse of notation, we deﬁne two parameter-dependent functions α 1 , α 2 : R d (cid:55)→ R + to characterize the relative importance of the current task and old tasks.
Mathematically, we propose to use the following update:
[Equation]
Solve optimization problem in Eq.
to
Appendix A.1.
Set Update Discard the samples added initially if M is full. 15: end for 16: end for where α 1 , α 2 : R d (cid:55)→ R + are real-valued mappings.
We employ ﬁrst-order methods (e.g., stochastic gradient descent) to approximately solve the optimization problem (5 which naturally motivates us to design the MixEd stochastic GrAdient (MEGA) algorithm.
MEGA is doing update (6) to approximately solve (5), where one-step stochastic gradient descent is performed with the initial point set to be w tk :
[Equation]
where η is the learning rate, ξ tk and ζ tk are random variables with ﬁnite support, ∇ (cid:96) t ( w tk ; ξ tk ) and ∇ (cid:96) ref ( w tk ; ζ tk ) are unbiased estimators of ∇ (cid:96) t ( w tk ) and ∇ (cid:96) ref ( w tk ) respectively, and α 1 ( w tk ) ∇ (cid:96) t ( w tk ; ξ tk ) + α 2 ( w tk ) ∇ (cid:96) ref ( w tk ; ζ tk ) is referred to as the mixed stochastic gradient .
The main difﬁculty of the update (6) is to deﬁne well-behaved mappings
α 1 ( · ) and α 2 ( · ) which are consistent with the goal of lifelong learning.
To this end, we introduce two approaches, angle-based approach (Section 4.1) and a direct approach (Section 4.2) to address this difﬁculty.
It is worth mentioning that several recent advances on lifelong learning (Lopez-Paz et al., 2017; Chaudhry et al., 2018b; Riemer et al., 2018) have close relationship with the angle-based approach in our
MEGA framework which will be illustrated in Section 4.1.
4. A NGLE -B ASED A PPROACH
Note that the mixed stochastic gradient is a linear combination of ∇ (cid:96) t ( w tk ; ξ tk ) and ∇ (cid:96) ref ( w tk ; ζ tk ) .
While keeping the magnitude the same as ∇ (cid:96) t ( w tk ; ξ tk ) , geometrically the mixed stochastic gradient can be viewed as an appropriate rotation of ∇ (cid:96) t ( w tk ; ξ tk ) by a desired angle θ .
This perspective leads to the angle-based approach in our framework.
The key idea of the angle-based approach is to ﬁrst appropriately rotate the stochastic gradient calculated on the current task (i.e., ∇ (cid:96) t ( w tk ; ξ tk ) ) by an angle θ tk , and then use the rotated vector as the mixed stochastic gradient to conduct the update (6) in each mini-batch.
For simplicity, we omit the subscript k and superscript t later on unless speciﬁed.
We use g mix to denote the desired mixed stochastic gradient which has the same magnitude as ∇ (cid:96) t ( w ; ξ ) .
Speciﬁcally, we look for the mixed stochastic gradient g mix which direction aligns well with both ∇ (cid:96) t ( w ; ξ ) and ∇ (cid:96) ref ( w ; ζ ) .
Mathematically, we want to maximize
[Equation]
which is equivalent to ﬁnd an angle θ such that
[Equation]
where ˜ θ ∈ [0 , π ] is the angle between ∇ (cid:96) t ( w ; ξ ) and ∇ (cid:96) ref ( w ; ζ ) .
To capture the relative importance of the current task and old tasks which is crucial for lifelong learning, we introduce (cid:96) t ( w ; ξ ) and (cid:96) ref ( w ; ζ ) into (8),
[Equation]
Here we give some discussions of several special cases of Eq.
• When (cid:96) ref ( w ; ζ ) = 0 , then θ = 0 , and in this case α 1 ( w ) = 1 , α 2 ( w ) = 0 in (6), which means the mixed stochastic gradient reduces to ∇ (cid:96) t ( w ; ξ ) .
In the lifelong learning setting, (cid:96) ref ( w ; ζ ) = 0 implies that there is almost no catastrophic forgetting, and hence we can update the model parameters exclusively for the current task by moving in the direction of ∇ (cid:96) t ( w ; ξ ) .
• When (cid:96) t ( w ; ξ ) = 0 , then θ = ˜ θ , and in this case α 1 ( w ) = 0 , α 2 ( w ) =
(cid:107)∇ (cid:96) t ( w ; ξ ) (cid:107) 2 / (cid:107)∇ (cid:96) ref ( w ; ζ ) (cid:107) 2 , provided that (cid:107)∇ (cid:96) ref ( w ; ζ ) (cid:107)
(cid:54) = 0 (deﬁne 0/0=0).
This means the direction of the mixed stochastic gradient is the same as the stochastic gradient calculated on the data in the episodic memory (i.e., (cid:96) ref ( w ; ζ ) ).
In the lifelong learning setting, this update can help improve the performance on old tasks, i.e., avoid catastrophic forgetting.
In the general case, we assume (cid:96) ref ( w ; ζ ) and (cid:96) t ( w ; ξ ) are both positive (the edge cases are covered in the above discussion).
Since the optimization problem (9) is possibly nonconvex, we propose to use projected gradient ascent to approximately solve it.
Mathematically, we do multiple updates using the following formula,
[Equation]
where Π is the projection operator, g ( β )
= − sin( β )
+ (cid:96) ref [Equation] w ; ζ ) [Equation]
sin(˜ θ − β ) , r = [Equation] ref
[Equation]
; ζ )
[Equation]
t ( w ; ξ ) . It is not difﬁcult to show that the smoothness parameter of the function g (cid:48) ( β ) = cos
[Equation]
+
[Equation]
is 1+ r , and hence projected gradient ascent can converge to a stationary point (Nesterov, 1998).
To avoid getting stuck on saddle point or local minima or local maxima, we can use multiple random starting points and select the one which achieves the largest function value.
This strategy is proven to be successful in our experiments.
After we ﬁnd the desired angle θ , it is easy to obtain α 1 ( w ) and α 2 ( w ) in Eq.
For details, please refer to Appendix A.1.
Please see Algorithm 1 for the summary of the algorithm.
Comparison with Existing Works There are several existing works setting θ in different manners.
In Lopez-Paz et al.
(2017) and Chaudhry et al.
(2018b),
θ = ˜ θ if ˜ θ ≤ π 2 , and θ = ˜ θ − π 2 if ˜ θ ≥ π 2 .
Note that ∇ (cid:96) ref ( w ; ζ ) is deﬁned differently in Lopez-Paz et al.
(2017) and Chaudhry et al.
(2018a).
In Lopez-Paz et al.
(2017), ∇ (cid:96) ref ( w ; ζ ) is calculated on the data of each task separately stored in the episodic memory.
While in Chaudhry et al.
(2018a), ∇ (cid:96) ref ( w ; ζ ) is computed on a random mini-batch sampled from the episodic memory.
Our work differs from Lopez-Paz et al.
; Chaudhry et al. (2018b) in two aspects. First, we explicitly maximize the correlation between the mixed stochastic gradient g mix and the current gradient ∇ (cid:96) t ( w ; ξ ) (calculated on current data), and the correlation between the mixed stochastic gradient g mix and the reference gradient ∇ (cid:96) ref ( w ; ζ ) (calculated on memory data) as in Eq (7). Intuitively, the direction of g mix will not bias towards the current task or old tasks (since it is easy to see that the angle between g mix and ∇ (cid:96) t ( w ; ξ ) , and the angle between g mix and ∇ (cid:96) ref ( w ; ζ ) are both acute (except the edge case where the angle between ∇ (cid:96) t ( w ; ξ ) and ∇ (cid:96) ref ( w ; ζ ) is 180 degree).
By following g mix , each update tends to degrade the loss on both current task and old tasks.
While in previous works (Lopez-Paz et al., 2017; Chaudhry et al., 2018b), the corresponding g mix is found by projecting the current gradient to be perpendicular to ∇ (cid:96) ref ( w ; ζ ) when the angle between them is obtuse (in this case, g mix · ∇ (cid:96) ref ( w ; ζ ) = 0). Intuitively, both GEM and A-GEM put more emphasis on the current task (when g mix is perpendicular to ∇ (cid:96) ref ( w ; ζ ) , the update following the direction of g mix will not directly reduce the loss on old tasks). Second, we introduce the loss on the current batch and on the sampled batch from episodic memory.
This allows us to better balance the performance on the current task and old tasks. Since it is preferable to put more emphasis on the tasks (task) which achieve(s) low performance.
4. D IRECT A PPROACH
We also introduce MEGA-D which is a direct approach for implementing MEGA.
In MEGA-D, instead of rotating the stochastic gradient computed on the data in the current mini-batch as in the angle-based approach, we deﬁne α 1 ( · ) and α 2 ( · ) in the deﬁnition of mixed stochastic gradient in a direct manner.
Speciﬁcally, in the update of (6), we set α 1 = 1 , α 2 = (cid:96) ref ( w ; ζ ) /(cid:96)
t ( w ; ξ ) if (cid:96) t ( w ; ξ ) > 0 , and α 1 = 0 , α 2 = 1 if (cid:96) t ( w ; ξ )
= 0 .
Intuitively, if (cid:96) t ( w ; ξ ) = 0 , then it means that the model performs well on the current task and can focus on improving performance on the data stored in the episodic memory, and hence α 1 = 0 , α 2 = 1 .
Otherwise, we keep the balance of the two terms of mixed stochastic gradient according to (cid:96) t ( w ; ξ ) and (cid:96) ref ( w ; ζ ) .
MEGA-D is a simple variant of MEGA which balances current task and old tasks by only leveraging loss information.
E XPERIMENTS 5. D ATASETS
In the experiments, we consider the following four conventional lifelong learning benchmarks,
• Permuted MNIST (Kirkpatrick et al., 2017): this is a variant of standard MNIST dataset (LeCun et al., 1998) of handwritten digits with 20 tasks.
Each task has a ﬁxed random permutation of the input pixels which is applied to all the images of that task.
• Split CIFAR (Zenke et al., 2017): this dataset consists of 20 disjoint subsets of CIFARdataset (Krizhevsky et al., 2009), where each subset is formed by randomly sampling 5 classes without replacement from the original 100 classes.
• Split CUB (Chaudhry et al., 2018b): the CUB dataset (Wah et al., 2011) is split into 20 disjoint subsets by randomly sampling 10 classes without replacement from the original 200 classes.
• Split AWA (Chaudhry et al., 2018b): this dataset consists of 20 subsets of the AWA dataset (Lampert et al., 2009).
Each subset is constructed by sampling 5 classes with replacement from a total of 50 classes.
Note that the same class can appear in different subsets.
As in Chaudhry et al.
(2018b), in order to guarantee that each training example only appears once in the learning process, based on the occurrences in different subsets the training data of each class is split into disjoint sets.
We also include Many Permutations which is a variant of Permuted MNIST to introduce more non-stationality into the learning process.
In Many Permutations , there are a total 100 tasks with 200 examples per task.
The way to generate the tasks is the same as in Permuted MNIST , that is, a ﬁxed random permutation of input pixels is applied to all the examples for a particular task.
5. N ETWORK A RCHITECTURES
To be consistent with the previous works (Lopez-Paz et al., 2017; Chaudhry et al., 2018b), for
Permuted MNIST we adopt a standard fully-connected network with two hidden layers.
Each layer has 256 units with ReLU activation.
For Split CIFAR we use a reduced ResNet18.
For Split
CUB and Split AWA , we use a standard ResNet18 (He et al., 2016).
5. B ASELINES AND E XPERIMENTAL S ETTINGS
We compare the proposed MEGA with several state-of-the-art lifelong learning methods,
• VAN: in VAN, a single network is trained continuously on a sequence of tasks in a standard supervised learning manner.
[Figure]
• MULTI-TASK: in MULTI-TASK, a single network is trained on the shufﬂed data from all the tasks with a single pass.
• Episodic memory based approach: GEM (Lopez-Paz et al., 2017) and AGEM (Chaudhry et al., 2018b) are episodic memory based approaches which modify the current gradient when its angle between the gradient computed on the episodic memory is obtuse.
MER (Riemer et al., 2018) is another recently proposed episodic memory based approach which maintains an experience replay style memory with reservoir sampling and employs a meta-learning style training strategy.
• Regularization-based approaches: EWC (Kirkpatrick et al., 2017), PI (Zenke et al., 2017), RWALK (Chaudhry et al., 2018a) and MAS (Aljundi et al., 2018) are regularization-based approaches which prevent the important weights of the old tasks from changing too much.
• Knowledge transfer based approach: in PROG-NN (Rusu et al., 2016), a new column with lateral connections with previous hidden layers is added for each new task.
This allows knowledge transfer between old tasks and the new task.
To be consistent with Chaudhry et al.
(2018b), for episodic memory based approaches, the episodic memory size for each task is 250, 65, 50, and 100, and the batch size for computing the gradients on the episodic memory (if needed) is 256, 1300, 128 and 128 for MNIST, CIFAR, CUB and AWA, respectively.
To ﬁll the episodic memory, the examples are chosen uniformly at random for each task as in Chaudhry et al.
(2018b).
For each dataset, 17 tasks are used for training and 3 tasks are used for hyperparameter search.
For MEGA and MER Riemer et al. (2018), we do not conduct exhaustive hyperparameter search and reuse the hyperparameters of AGEM (Chaudhry et al., 2018b).
For other baselines, we use best hyperparameters found by Chaudhry et al.
(2018b).
For the detailed hyperparameters, please see Appendix G of Chaudhry et al. (2018b).
In MEGA, we solve Eq.9 three times with different random starting points and the update in Eq.10 is repeated for ten iterations. [Figure] A v g A cc u r a c y
[Figure] A v [Figure] A cc u
[Figure]
a c y
[Figure]
R ESULTS 6. MEGA VS B ASELINES
In Fig.
1 we show the results across different measures on all the benchmark datasets.
We have the following observations based on the results.
First, the proposed MEGA outperforms all baselines across the benchmarks, except that PROG-NN achieves a slightly higher accuracy on Permuted
MNIST .
As we can see from the memory comparison, PROG-NN is very memory inefﬁcient since it allocates a new network for each task, thus the number of parameters grows super-linearly with the number of tasks.
This becomes problematic when large networks are being used.
For example,
PROG-NN runs out of memory on Split CUB and Split AWA which prevents it from scaling up to real-life problems.
On other datasets, MEGA consistently performs better than all the baselines, from Fig.
2 we can see that on Split CUB , MEGA even surpasses the multi-task baseline which is previously believed as an upper bound performance of lifelong learning algorithms (Chaudhry et al., 2018b).
Second, MEGA achieves the lowest Forgetting Measure across all the datasets which indicates its ability to overcome catastrophic forgetting.
Third, the proposed MEGA also obtains a high LCA across all the datasets which shows that MEGA also learns quickly.
The evolution of
LCA in the ﬁrst ten mini-batches across all the datasets is shown in Fig.
Last, compared with
AGEM (Chaudhry et al., 2018b), which is the state-of-the-art method for lifelong learning, MEGA has the same memory cost and similar time complexity.
For detailed results, please refer to Tableand Table 5 in Appendix A.2.
In Fig.
2 we show the evolution of average accuracy during the lifelong learning process.
As more tasks are added, while the average accuracy of the baselines generally drops due to catastrophic forgetting, MEGA can maintain and even improve its performance.
This shows MEGA has a clear advantage over the state-of-the-art lifelong learning methods.
6. L ONG - TERM R EMEMBERING
In Table 1 we show the results of Long-term Remembering (LTR) of some representative lifelong learning methods on different datasets.
As stated before, an algorithm with low LTR indicates that it can maintain the performance on the tasks trained initially.
From Table 1 we can see that the proposed MEGA algorithm achieves the lowest LTR across all the datasets.
This demonstrates that
MEGA can learn tasks in succession without forgetting the initial tasks which is crucial for realworld lifelong learning applications. [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
6. D IRECT A PPROACH
We show the comparison of MEGA and MEGA-D in Table 2.
The performance of MEGA-D is on par with MEGA across all the datasets.
This shows that it is important to explicitly consider the loss on the episodic memory in order to overcome catastrophic forgetting.
[Table]
6. M ANY P ERMUTATION
We show the results on Many Permutation in Table 3.
Compared with Permuted MNIST , Many
Permutation has 5 times more tasks (100) and much fewer examples per task (200).
This introduces more non-stationarity into the learning process.
Nevertheless, the proposed MEGA achieves competitive results in this setting.
Compared with MER which achieves similar results to MEGA,
MEGA is much more time efﬁcient since it does not rely on the meta-learning procedure.
Update
MEGA-D result in the table.
[Table]
C ONCLUSION
In this paper
[Table]
we present a lifelong learning algorithm called MEGA which achieves the state-ofthe-art performance across all the benchmark datasets.
In MEGA, we cast the lifelong learning problem as an optimization problem with composite objective and solve it with the proposed mixed stochastic gradient.
We also propose an important lifelong learning metric called LTR to characterize the ability of lifelong learning algorithms to maintain performance on the tasks trained in the far past.
Extensive experimental results show that the proposed MEGA achieves superior results across all the considered metrics and establishes the new state-of-the
A A PPENDIX A. S OME D ERIVATIONS
For notation simplicity, we use g , ˆ g , a , b to replace ∇ (cid:96) t ( w ; ξ ) , ∇ (cid:96) ref ( w ; ζ ) , α 1 ( w ) , α 2 ( w ) respectively.
If g = ˆ g , then a = 1 , b = 0 .
Otherwise, the goal is to solve
[Equation]
The solution of (11) is
[Equation]
A. RESULT TABLES
with different random seeds.
[Table] [Table]
A. A BLATION STUDIES
In this section, we include ablation studies to why the proposed method can improve upon A-GEM
Chaudhry et al.
(2018b).
We only consider maximizing the correlation between the mixed stochastic gradient g mix and the current gradient ∇ (cid:96) t ( w ; ξ ) (calculated on current data), and the correlation between the mixed stochastic gradient g mix and the reference gradient ∇ (cid:96) ref ( w ; ζ ) (calculated on memory data) as in Eq (7).
We call this method MEGA-C. The only difference between MEGA-C and A-GEM is the way of modifying the gradient direction. We compare MEGA-C to MEGA and A-GEM in the Table 6.
The experimental settings are the same as in Section 5.3.
[Table]
From Table 6 we observe that except on Split CIFAR
[Table]
MEGA-C outperforms A-GEM on all the datasets.
This demonstrate the beneﬁts of the proposed approach for modifying gradient direction.
By considering the loss information as in MEGA, we further improve the results on all the datasets.
This shows that both of the two components (gradient direction modiﬁcation and loss information) contribute to the improvement of the proposed approach.
A. L EARNING PROCESS
In this section we report the result matrices for MEGA and AGEM (Chaudhry et al., 2018b) on each dataset.
The entry ( i , j ) of the matrix is the test accuracy of the j -th task after the model is trained on the i -th task.
A.4. P ERMUTED MNIST
[Table]
[Table]
A.4.
S PLIT CUB
MEGA
[Table] [Table]
A.4. S PLIT AWA MEGA
[Table]
A-GEM
[Table] [Table]