La-MAML: Look-ahead Meta Learning for Continual Learning
Abstract
The continual learning problem involves training models with limited capacity to perform well on a set of an unknown number of sequentially arriving tasks.
While meta-learning shows great potential for reducing interference between old and new tasks, the current training procedures tend to be either slow or ofﬂine, and sensitive to many hyper-parameters.
In this work, we propose Look-ahead MAML (La-MAML) , a fast optimisation-based meta-learning algorithm for online -continual learning, aided by a small episodic memory.
Our proposed modulation of per-parameter learning rates in our meta-learning update allows us to draw connections to prior work on hypergradients and meta-descent .
This provides a more ﬂexible and efﬁcient way to mitigate catastrophic forgetting compared to conventional prior-based methods.
La-MAML achieves performance superior to other replay-based, priorbased and meta-learning based approaches for continual learning on real-world visual classiﬁcation benchmarks.
1. Introduction
Embodied or interactive agents that accumulate knowledge and skills over time must possess the ability to continually learn.
Catastrophic forgetting (French, 1999; Mcclelland et al., 1995), one of the biggest challenges in this setup, can occur when the i.i d. sampling conditions required by stochastic gradient descent (SGD) are violated as the data belonging to different tasks to be learnt arrives sequentially.
Algorithms for continual learning (CL) must also use their limited model capacity efﬁciently since the number of future tasks is unknown.
Ensuring gradient-alignment across tasks is therefore essential, to make shared progress on their objectives.
Gradient Episodic Memory (GEM) (Lopez-Paz &
Ranzato, 2017) investigated the connection between weight sharing and forgetting in CL and developed an algorithm that explicitly tried to minimise gradient interference .
This is an objective that meta-learning algorithms implicitly optimise for (refer to (Nichol et al., 2018) for derivations of the effective parameter update made in ﬁrst and second order meta learning algorithms).
Meta Experience Replay (MER)
(Riemer et al., 2019) formalized the transfer-interference trade-off and showed that the gradient alignment objective of GEM coincide with the objective optimised by the ﬁrst order meta-learning algorithm Reptile (Nichol et al., 2018).
Besides aligning gradients, meta-learning algorithms show promise for CL since they can directly use the metaobjective to inﬂuence model optimisation and improve on auxiliary objectives like generalisation or transfer.
This avoids having to deﬁne heuristic incentives like sparsity (Le et al., 2017) for better CL.
The downside is that they are usually slow and hard to tune, effectively rendering them more suitable for ofﬂine continual learning (Javed & White, 2019; Riemer et al., 2019).
In this work, we overcome these difﬁculties and develop a gradient-based meta-learning algorithm for efﬁcient, online continual learning.
We ﬁrst propose a base algorithm for continual meta-learning referred to as Continual-MAML (C-MAML) that utilizes a replay-buffer and optimizes a meta-objective that mitigates forgetting.
Subsequently, we propose a modiﬁcation to C-MAML, La-MAML, which incorporates modulation of per-parameter learning rates (LRs) to pace the learning of a model across tasks and time.
Finally, we show that the algorithm is scalable, robust and achieves favourable performance on several benchmarks of varying complexity.
2. Related work
Relevant CL approaches can be roughly categorized into replay-based, regularisation-based (or prior-based) and meta-learning-based approaches.
In order to circumvent the issue of catastrophic forgetting, replay-based methods maintain a collection of samples from previous tasks in memory.
Approaches utilising an episodicbuffer (Castro et al., 2018; Rebufﬁ et al., 2017) uniformly sample old data points to mimic the i.i.d.
setup within continual learning.
Approaches proposing Generative-replay (Shin et al., 2017) train generative models to replay the data distribution of past samples. However, they currently face scalability issues arising from the difﬁculty of modeling complex non-stationary distributions.
GEM (Lopez-Paz &
Ranzato, 2017) and A-GEM (Chaudhry et al., 2019) take memory samples into account to determine altered lowinterference gradients for updating parameters.
Regularisation-based methods avoid using replay at all by constraining the network weights according to heuristics intended to ensure that performance on previous tasks is preserved.
This involves penalising changes to weights deemed important for old tasks (Kirkpatrick et al., 2017) or enforcing weight or representational sparsity (Aljundi et al., 2019) to ensure that only a subset of neurons remain active (and thus receiving gradients) at any point of time.
Meta-Learning-based approaches are fairly recent and have shown impressive results on small benchmarks like Omniglot and MNIST.
MER
(Riemer et al., 2019), inspired by GEM(Lopez-Paz & Ranzato, 2017), utilises replay to incentivise alignment of gradients between old and new tasks.
Online-aware Meta Learning (OML) (Javed & White, 2019) introduces a meta-objective for a pre-training algorithm to learn an optimal representation ofﬂine , which is subsequently frozen and used for CL.
The the orthogonal setup of online learning is investigated in (Al-Shedivat et al., 2018;
Finn et al., 2019; Nagabandi et al., 2019), where a learning agent uses all previously seen data to adapt quickly to an incoming stream of data, thereby ignoring the problem of catastrophic forgetting.
Our motivation lies in developing a scalable , online algorithm capable of learning from limited cycles through streaming data with reduced interference on old samples.
In the following sections, we review background concepts, outline our proposed algorithm, and note connections to prior work not directly pertaining to CL.
3. Preliminaries
We consider a setting where a sequence of T tasks [ τ 1 , τ 2 , ..τ T ] is learnt by observing their training data [ D 1 , D 2 , ..
D T ] sequentially.
We deﬁne X i , Y i = {
( X in , Y in ) } N i n =0 as the set of N i input-label pairs randomly drawn from D i .
An any time-step j during online learning, we aim to minimize the empirical risk of the model on all the t tasks seen so far ( τ 1: t ), given limited access to data ( X i , Y i ) from previous tasks τ
i ( i < t ). We refer to this objective as the cumulative risk , given by:
[Equation]
where (cid:96) i is the loss on τ
i and f i is a learnt, possibly taskspeciﬁc mapping from inputs to outputs using parameters θ .
L t = (cid:80) ti =1 (cid:96) i is the sum of all task-wise losses for tasks τ 1: t .
T is unknown beforehand
and we merely use it to specify an upper bound in our summations.
Let θ j 0 denote the model’s parameters at time j , and (cid:96) denote a loss objective to be minimised.
Then the SGD operator acting on parameters θ j 0 , denoted by U ( θ j 0 ) , is deﬁned as:
[Equation]
where g j 0 = ∇ θ j 0 (cid:96) ( θ j 0 ) .
f U can be composed for k updates as U k (cid:16) θ j 0 (cid:17) =
U... ◦
U ◦ U ( θ j 0 )
= θ
jk .
α is a scalar or a vector LR.
U ( · , x ) implies gradient updates are made on data sample x .
We now introduce the MAML (Finn et al., 2017) and OML (Javed & White, 2019) algorithms, that we build upon in Section 4.
Model-Agnostic
Meta-Learning (MAML) :
Metalearning (Schmidhuber, 1987), or learning-to-learn (Thrun & Pratt, 1998) has emerged as a popular approach for training models amenable to fast adaptation on limited data.
MAML (Finn et al., 2017) proposed optimising model parameters to learn a set of tasks while improving on auxiliary objectives like few-shot generalisation within their task distributions.
We review some terminology used in gradient-based meta-learning: 1) at time-step j during training, model parameters θ j 0 (or θ 0 for simplicity), are often referred to as an initialisation , since MAML ﬁnds an ideal starting point for few-shot gradient-based adaptation on unseen data.
2) Fast updates , refer to gradient-based updates made to a copy of θ 0 , optimising an inner objective (in this case, (cid:96) i for some τ i ).
A meta-update involves the trajectory of fast updates from θ 0 to θ k , followed by making a permanent slow-update to θ 0 .
This slow-update is computed by evaluating an auxiliary objective (or meta-loss L meta ) on θ k , and differentiating through the trajectory to obtain ∇ θ 0 L meta ( θ k ) .
MAML thus optimises θ j 0 to perform optimally on tasks in { τ 1: t } after undergoing a few gradient updates on their samples by optimsiing the objective:
[Equation]
Equivalence of Meta-Learning and Gradient Alignment
Objectives : The ﬁrst-order meta-learning algorithm Reptile has been shown to be approximately equivalence to the
MAML objective (Nichol et al., 2018).
MER
(Riemer et al., 2019) then showed that their CL objective of minimising loss on tasks τ 1: t seen till time j , while aligning gradients between them:
[Equation]
is equivalent to the Reptile objective i.e.:
[Equation]
where the meta-loss L t = (cid:80) ti =1 (cid:96) i is evaluated on samples from tasks τ 1: t .
We will show how a version of this objective coincides with our proposed multi-step MAML-based algorithm.
Note that this implies that the procedure to learn an initialisation through meta-learning, coincides with the procedure to learn optimal parameters for CL.
Online-aware Meta-Learning (OML) :
(Javed & White, 2019) proposed to meta-learn a Representation-Learning
Network (RLN) to provide a representation suitable for CL to a Task-Learning Network (TLN) .
The RLN’s representation is learnt in an ofﬂine phase, where it is trained using catastrophic forgetting as the learning signal .
Data from a ﬁxed set of tasks ( τ val ), is repeatedly used to evaluate the
RLN and TLN as the TLN undergoes temporally correlated updates.
In every meta-update ’s inner loop, the TLN undergoes fast updates on streaming task data with a frozen RLN.
The RLN and updated TLN are then evaluated through a meta-loss computed on data from τ val along with the current task.
This tests how the performance of the model has changed on τ val in the process of trying to learn the streaming task.
The meta-loss is then differentiated to get gradients for slow updates to the TLN and RLN.
This composition of two losses to simulate CL in the inner loop and test forgetting in the outer loop, is referred to as the OML objective .
The slow updates to the RLN lead to it eventually providing a better representation to the TLN for CL.
4. Proposed approach
In the previous section, we described how the OML objective can directly regulate CL behaviour, and how MER exploits the equivalence of the objectives of Reptile and CL to construct a CL algorithm that uses Reptile updates.
We noted that OML is for ofﬂine pre-training of a static representation and that MER’s online algorithm is still prohibitively slow.
We observe that we can adapt the OML objective to be optimised continuously online with respect to all of the model parameters similar to a standard k -step MAML update.
We then prove the equivalence of this MAML objective to that of a standard CL objective, to be able to use this algorithm for continual learning in a principled manner.
In this section, we describe Continual-MAML (C-MAML), the base algorithm that we propose for online continual learning.
We then detail an extension to C-MAML, referred to as Look-Ahead MAML (La-MAML), outlined in Algorithm 1.
4.1. C-MAML
C-MAML aims to optimise the OML objective online , so that learning on the current task doesn’t lead to forgetting on previously seen tasks.
We deﬁne this objective, adapted to optimise a model’s parameters θ instead of a representation at time-step j , as:
[Equation]
where S jk is a stream of data tuples (cid:16) X tj + l , Y tj + l (cid:17) k l =1 from the current task τ t that is seen by the model at time j .
The meta-loss L t = (cid:80) t i =1 (cid:96) i is evaluated on θ jk =
U k
( θ j 0 , S jk ) .
It evaluates the ﬁtness of θ jk for the continual learning prediction task deﬁned in Eq.
1 until τ t .
We omit the implied data argument ( x i , y j ) ∼ ( X i , Y i ) that is the input to each loss (cid:96) i in L t for any task τ
i .
We will show in Appendix B that optimising our objective in Eq. 6 through the k -step MAML update in C-MAML:
[Equation]
coincides with optimising the CL objective of AGEM (Chaudhry et al., 2019):
[Equation]
This differs from Eq.
4’s objective by being asymmetric : it focuses on aligning the gradients of τ t and the average gradient of τ 1: t instead of aligning all the pair-wise gradients between tasks τ 1: t .
Incentivizing alignment of all τ 1: t with the common τ t indirectly incentivises alignment amongst τ 1: t .
This results in a drastic speedup over MER’s objective (Eq. 4) which tries to align all τ 1: t equally, and therefore has to resample incoming samples s ∼ τ t to form a uniformly distributed batch over τ 1: t .
Since each s then has 1 t -th the contribution in gradient updates, it becomes necessary for MER to take multiple passes over many such uniform batches including s .
To evaluate the meta-loss L t ( θ jk ) which indicates performance of θ jk on all tasks seen till time j , old tasks’ data is sampled from a replay-buffer R .
R is populated through reservoir sampling on the incoming data stream as in (Riemer et al., 2019).
The SGD-based inner-updates starting from initial weights θ j 0 are carried out using data from the task being streamed at the time of meta-update step j .
At the start of every meta-update step, a batch b is sampled from the current task and is combined with a batch sampled from R to form the meta-batch , b m .
θ j 0 is updated for k steps by seeing samples from b one at a time.
The outer loss is evaluated on b m .
4.2. La-MAML
Despite the fact that meta-learning incentivises the alignment of within-task and across-task gradients, there can still be some interference between the gradients of old and new tasks, τ 1: t − 1 and τ t respectively.
This would lead to forgetting on τ 1: t − 1 , since its data is no longer fully available to us.
This is especially true at the beginning of training a new task, when its gradients aren’t necessarily aligned with the old ones.
A mechanism is thus needed to ensure that meta-updates are conservative with respect to τ 1: t − 1 , so as to avoid negative transfer on them.
The magnitude and direction of the meta-update needs to be regulated, guided by how the loss on τ 1: t − 1 would be affected by the update.
In La-MAML , we include a set of learnable per-parameter learning rates (LRs) to be used in the inner updates , as depicted in Figure 1.
This is motivated by our observation that the expression for the gradient of Eq.
with respect to the inner loop’s LRs directly reﬂects the alignment between the old and new tasks.
The augmented learning objective and its gradient with respect to the LR vector α , denoted as g MAML ( α ) is then given as:
[Equation]
We provide the full derivation in the Appendix A, and simply state the expression for a ﬁrst-order approximation (Finn et al., 2017) of g MAML ( α ) here.
The ﬁrst term in g MAML ( α ) corresponds to the gradient of the meta-loss on batch b m : g meta .
The second term indicates the cumulative gradient from the inner-update trajectory: g traj .
This expression indicates that the gradient of the LRs will be negative when the inner product between g meta and g traj is high, ie.
the two are aligned.
Similarly, it should be zero when the two are orthogonal (not interfering) and positive when there is disagreement between the two, leading to interference.
Negative (positive) LR gradients would pull up (down) the LR magnitude.
We depict this visually in Figure 2.
Algorithm 1 La- Input: Network weights θ , LRs α , inner objective (cid:96) , meta objective L , learning rate for α : η
R ← {} (cid:46) replay-buffer for t = 1 to T do
Data t ← Sample ( D t ) for ep
= 1 to num ep do for b in Data t do k = sizeof ( b ) b m ← Sample ( R, b )
Initialize θ 00 , α 0 ← θ, α for n = 0 to k − 1 do
Push b [ n ] to R with reservoir sampling θ 0 n +1 = θ
[Equation]
end for
[Equation]
end for end for end for
We propose updating the network weights and LRs asynchronously in the meta-update.
Let α j +1 be the updated LR vector obtained by taking an SGD step with the LR gradient from Eq.
10 at time j .
We then update the weights as:
[Equation]
where k is the number of steps taken in the inner-loop.
The
LRs α j +1 are clipped to positive values to avoid ascending the gradient, and also to avoid making interfering parameterupdates, thus mitigating catastrophic forgetting.
The metaobjective thus conservatively modulates the pace and direc- tion of learning to achieve quicker learning progress on a new task while facilitating transfer on old tasks.
Algorithm 1 illustrates this procedure.
Lines (a), (b) are the only difference between C-MAML and La-MAML, with C-MAML using a ﬁxed scalar LR α for the meta-update to θ j 0 instead of α j +1 .
Our meta-learning based algorithm incorporates concepts from both prior-based and replay-based approaches.
Our
LRs modulate the parameter updates in an entirely data driven manner, guided by the interplay between the gradients of the replay samples and the streaming task.
However, since LRs evolve with every meta-update, their decay is temporary.
This is unlike many prior-based approaches, where penalties on the change in parameters gradually become so high that the network capacity saturates (Kirkpatrick et al., 2017).
Our learnable LRs can be modulated to high and low values as tasks arrive, thus being a simpler, ﬂexible and elegant way to constrain weights.
This asynchronous update resembles trust-region optimisation (Yuan, 1999) since the LRs are evolved in a manner similar to look-ahead search , which adjusts step-sizes based on the loss incurred on adapted parameters.
Our LR update is also analogous to the heuristic uncertainty-based LR update schemes of UCB (Ebrahimi et al., 2020) or BGD (Zeno et al., 2018), the latter of which we will include in our comparative analysis in Section 5.3.
4.3. Connections to Work Outside Continual Learning
Stochastic Meta-Descent (SMD) :
When learning over a non-stationary data distribution, using decaying learning rate schedules is not common.
Strictly diminishing learning rate schedules aim for closer and faster convergence to a ﬁxed mimima of a stationary distribution, which is at odds with the goal of online and continual learning.
In many Continual Learning scenarios it is impossible to manually tune the schedule since the extent of the data distribution is unknown.
However, adaptivity in learning rates is still highly desired to better adapt to the optimisation landscape and accelerate learning.
Another reason to desire adaptivity in CL is to modulate the degree of adaptation of certain parameters, to reduce catastrophic forgetting.
Our adaptive learning rates can be connected to work on meta-descent (Baydin et al., 2018; Schraudolph, 1999) in standard ofﬂine supervised learning (OSL).
While several variations of metadescent exist, the core idea behind these variations and our approach is the same: gain adaptation , analogous to gain adjustment in a Kalman Filter (KF).
In a KF, the gain is a quantity that signiﬁes how much trust we place in a proposed belief or parameter update. While in our case, we want to check the correlation between old and new task gradients to adapt the gain so that we can make the most shared progress on old and new tasks, in the case of (Baydin et al., 2018; Schraudolph, 1999) the correlation between two successive stochastic gradients on the same data distribution is used to converge faster. HD (Baydin et al., 2018) proposes analytically, asynchronously updating learning rates during optimization. This is done by differentiating the update rule at any time-step j with respect to the learning rates at time-step j − 1 .
We instead rely on the meta-objective’s differentiability with respect to the LRs, to obtain these learning rate hypergradients automatically.
Learning LRs in meta-learning : Meta-SGD (Li et al., 2017) ﬁrst proposed learning the per-parameter learning rates used within the inner-loop of MAML in the few-shot learning setting.
Some notable differences between their update and ours exist.
They synchronously update the weights and learning rates while our asynchronous update to the learning rates serves to carry out a more conservative update to the weights.
The intuition for our update stems from the need to mitigate gradient interference and its connection to the transfer-interference trade-off ubiquitous in continual learning.
Similarly, α -MAML (Singh Behl et al., 2019) proposed analytically updating the two scalar learning rates used in the inner and outer MAML update for more adaptive few-shot learning.
In contrast, our per-parameter learning rates are modulated implicitly through back-propagation, to regulate change in parameters based on their alignment across tasks, providing our model with a more powerful degree of adaptability in the CL domain.
5. Experiments
In this section we evaluate La-MAML in task incremental settings, where the model learns a set of sequentially streaming classiﬁcation tasks.
Experiments are performed on the
MNIST, CIFAR and TinyImagenet (tin) datasets. Similar
[Table] to (Riemer et al., 2019) [Table] we use the retained accuracy (RA) metric to compare various approaches.
RA is the average accuracy of the model across tasks at the end of training.
We also report the backward-transfer and interference (BTI) values which measure the average change in the accuracy of each task from when it was learnt to the end of the last task.
A smaller BTI implies lesser forgetting during training.
Efﬁcient Lifelong Learning (LLL) : Formalized in (Chaudhry et al., 2019), the setup of efﬁcient lifelong learning assumes that incoming data for every task has to be processed in only one single pass: once processed, data samples are not accessible anymore unless they were added to a replay memory.
We evaluate our algorithm on this challenging (Single-Pass) setup as well as the standard (Multiple-Pass) setup where ofﬂine training-until-convergence is performed for every task, once we have access to its data.
[Table]
5.1. Continual learning benchmarks
First, we carry out experiments on the toy continual learning benchmarks proposed in prior CL works.
MNIST Rotations , introduced in (Lopez-Paz & Ranzato, 2017), comprises tasks to classify MNIST digits rotated by a different common angle in [0, 180] degrees in each task.
In MNIST
Permutations , tasks are generated by shufﬂing the image pixels by a ﬁxed random permutation.
Unlike Rotations, the input distribution of each task is unrelated here, leading to less positive transfer between tasks.
Many Permutations , a more complex version of Permutations, has ﬁve times more tasks (100 tasks) and ﬁve times less training data (200 images per task).
We use the same architecture and experimental settings as in MER (Riemer et al., 2019), allowing us to compare directly with their results.
We use the cross-entropy loss as the inner and outer objectives during meta-training.
Similar to (Nichol et al., 2018), we see improved performance when evaluating and summing the meta-loss at all steps of the inner updates as opposed to just the last one.
We compare our method in the Single-Pass setup against multiple baselines including Online , Independent , EWC (Kirkpatrick et al., 2017), GEM (Lopez-Paz & Ranzato, 2017) and MER
(Riemer et al., 2019), as well as ablations detailed in Appendix E.
In Table 1, we see that La-MAML achieves comparable or better performance than the baselines on all benchmarks.
Table 2 shows that La-MAML matches the performance of MER in less than 20% of the training time, owing to its sample-efﬁcient objective which allows it to make make more learning progress per iteration.
This also allows us to scale it to real-world visual recognition problems as described next.
5.2. Real-world classiﬁcation
While La-MAML fares well on the MNIST benchmarks, we are interested in understanding its capabilities on more complex visual classiﬁcation benchmarks.
We conduct experiments on the CIFAR-100 dataset in a task-incremental manner (Lopez-Paz & Ranzato, 2017).
20 tasks comprising of disjoint 5-way classiﬁcation problems are streamed.
We also evaluate on the TinyImagenet-200 dataset by partitioning its 200 classes into 40 5-way classiﬁcation tasks.
Experiments are carried out in both the Single-Pass and Multiple-Pass settings, where in the latter we allow training for up to a maximum of 10 epochs.
Each method has a replay-buffer containing 200 and 400 samples for CIFAR100 and TinyImagenet respectively.
We provide further details about the baselines and evaluation setup in Appendix D.
Table 4 reports the results of these experiments.
We consistently observe superior performance of La-MAML as compared to other CL baselines on both datasets across setups.
While the iCarl baseline attains lower BTI in some setups, it achieves that at the cost of much lower perfor- mance throughout learning.
Among the high-performing approaches, La-MAML has the lowest BTI.
Recent work (Chaudhry et al., 2019; Riemer et al., 2019) noted that Experience Replay (ER) is often a very strong baseline that closely matches the performance of the proposed algorithms.
We highlight the fact that meta-learning and LR modulation combined show an improvement of more than 10 and 18% (as the number of tasks increase from CIFAR to Imagenet) over the ER baseline in our case, with limited replay.
Overall, we see that our method is better-performing under both the standard and LLL setups of CL which come with different kinds of challenges.
Many CL methods (Ebrahimi et al., 2020; Serra et al., 2018) are suitable for only one of the two setups.
As shown in Figure 3, our model evolves to become resistant to forgetting as training progresses.
This means that beyond a point, it can keep making gradient updates on a small window of incoming samples without needing to do meta-updates .
[Table]
5.3. Evaluation of La-MAML’s learning rate modulation
To capture the gains from learning the LRs, we compare LaMAML with our base algorithm, C-MAML .
We ablate our choice of updating LRs asynchronously by constructing a version of C-MAML where per-parameter learnable LRs are used in the inner updates while the meta-update still uses a constant scalar LR during training.
We refer to it as Sync-LaMAML or Sync since it has synchronously updated LRs that don’t modulate the meta-update.
While only minor gains are seen on the MNIST benchmarks from asynchronous LR modulation, the performance gap increases as the tasks get harder.
On CIFAR-100 and TinyImagenet, we see a trend in the RA of our variants with La-MAML performing best followed by Sync .
This shows that optimising the LRs aids learning and our asynchronous update helps in knowledge consolidation by enforcing conservative updates to mitigate interference.
To test our LR modulation against an alternative bayesian modulation scheme proposed in BGD (Zeno et al., 2018), we deﬁne a baseline called Meta-BGD where per-parameter variances are modulated instead of LRs.
This is described in further detail in Appendix E. Meta-BGD emerges as a strong baseline and matches the performance of C-MAML given enough Monte Carlo iterations m , implying m times more computation than C-MAML.
Additionally, Meta-BGD was found to be sensitive to hyperparameters and required extensive tuning.
We present a discussion of the robustness of our approach in Appendix C.
We also compare the gradient alignment of our three variants along with ER in Table 3 by calculating the cosine similarity between the gradients of the replay samples and newly arriving data samples.
As previously stated, the aim of many
CL algorithms is to achieve high gradient alignment across tasks to allow parameter-sharing between them.
We see that our variants achieve an order of magnitude higher cosine similarity compared to ER, verifying that our objective promotes gradient alignment.
the beginning, inner updates lead to catastrophic forgetting (CF) since the weights are not suitable for CL yet, but eventually become resistant when trained to retain old knowledge while learning on a stream of correlated data.
We also see that RA maintains its value even as more tasks are added indicating that the model is successful at learning new tasks without sacriﬁcing performance on old ones.
M ETHOD
[Table]
6. Conclusion
We introduced La-MAML, an efﬁcient meta-learning algorithm that leverages replay to avoid forgetting and favors positive backward transfer by learning the weights and LRs in an asynchronous manner.
It is capable of learning online on a non-stationary stream of data and scales to vision tasks.
We presented results that showed better performance against the state-of-the-art in the setup of efﬁcient lifelong learning (LLL) (Chaudhry et al., 2019), as well as the standard continual learning setting.
In the future, more work on analysing and producing good optimizers for CL is needed, since many of our standard go-to optimizers like Adam (Kingma & Ba, 2014) are primarily aimed at ensuring faster convergence in stationary supervised learning setups.
Another interesting direction is to explore how the connections to meta-descent can lead to more stable training procedures for meta-learning. Acknowledgements
The authors are grateful to Matt Reimer, Sharath Chandra
Raparthy, Alexander Zimin and the anonymous reviewers for proof-reading the paper and suggesting improvements.
This research was enabled in part by support provided by
Compute Canada ( www.computecanada.ca ).
A. Hypergradient Derivation for La-MAML
We derive the gradient of the weights θ j 0 and LRs α j at time-step j under the k -step MAML objective, with L t = (cid:80) ti =
(cid:96) i as the meta-loss and (cid:96) t as the inner-objective :
[Equation]
as a constant w.r.t α j to get the FO-MAML approximation)
[Equation]
Where (a) is obtained by recursively expanding and differentiating the update function U () as done in the step before it.
(b) is obtained by assuming that the initial weight in the meta-update at time j : θ j 0 , is constant with respect to α j .
Similarly we can derive the MAML gradient for the weights θ j 0 , denoted as g MAML ( θ j 0 ) as:
[Equation]
(repeatedly applying chain rule and using θ jk = U
[Equation]
θ
[Equation]
[Equation]
(cid:0) using U (cid:48)
( θ jn )
=
I − α(cid:96)
(cid:48)(cid:48) [Equation] [Equation] jn )
[Equation]
( (cid:48) implies derivative with respect to argument)
[Equation]
Setting all ﬁrst-order gradient terms as constants to ignore second-order derivatives, we get the ﬁrst order approximation as:
[Equation]
In Appendix B, we show the equivalence of the C-MAML and CL objectives in Eq.
by showing that the gradient of the former ( g MAML ( θ j 0 ) ) is equivalent to the gradient of the latter.
B. Equivalence of Objectives
It is straightforward to show that when we optimise the OML objective through the k -step MAML update, as proposed in
C-MAML in Eq.
[Equation]
where the inner-updates are taken using data from the streaming task τ t , and the meta-loss L t ( θ )
= (cid:80) ti =1 (cid:96) i ( θ ) is computed on the data from all tasks seen so far, it will correspond to minimising the following surrogate loss used in CL :
[Equation]
We show the equivalence for the case when k
= 1 , for higher k the form gets more complicated but essentially has a similar set of terms.
Reptile (Nichol et al., 2018) showed that the k -step MAML gradient for the weights θ j 0 at time j , denoted as g MAML ( θ j 0 ) is of the form:
[Equation]
(putting k = 1 )
Expressing the terms as derivatives, and using ∂ ∂θ j
[Equation]
1 , we get :
[Equation]
which is the same as the gradient of Eq. 13 .
where:
[Equation]
( gradient of the meta-loss evaluated at the initial point )
[Equation]
( for i < k ) ( gradients of the inner-updates evaluated at the initial point )
[Equation]
(sequence of parameter vectors)
[Equation]
(Hessian of the meta-loss evaluated at the initial point)
[Equation]
( for
[Equation]
k
[Equation]
(Hessian of the inner-objective evaluated at the initial point)
And, in our case:
[Equation]
Bias in the objective : We can see in Eq. 13 that the gradient alignment term introduces some bias, which means that we don’t exactly converge to the minimiser of the losses on all tasks.
This has been acceptable in CL for the following reason:
We never aim to reach the minimiser of some stationary distribution anyway in the CL regime (as also mentioned in Section 4.3).
If we did converge to the minimiser of say t tasks at some time j , this minimiser would no longer be optimal as soon as we see the new task τ t +1 .
Therefore, in the limit of inﬁnite tasks or time, gradient alignment between tasks will pay off more as opposed to being able to converge to the exact minima, by allowing us to make shared progress on both previous and incoming tasks.
C. Robustness
Learning rate is one of the most crucial hyper-parameters during training and it often has to be tuned extensively for each experiment.
In this section we analyse the robustness of our proposed variants to their LR-related hyper-parameters on the
CIFAR-100 dataset.
Our three variants have different sets of these hyper-parameters which are speciﬁed as follows:
• C-MAML : Inner and outer update LR (scalar) for the weights ( α and β ) • Sync La-MAML : Inner loop initialization value for the vector LRs ( α 0 ), scalar learning rate of LRs ( η ) and scalar learning rate for the weights in the outer update ( β ) • La-MAML : Scalar initialization value for the vector LRs ( α 0 ) and scalar learning rate of LRs ( η )
La-MAML is considerably more robust to tuning compared to its variants, as can be seen in Figure 4c.
We empirically observe that it only requires tuning of the initial value of the LR, while being relatively insensitive to the learning rate of the LR ( η ).
We see a consistent trend where the increase in η leads to an increase in the ﬁnal accuracy of the model.
The increase is very gradual, across a wide range of LRs varying over 2 orders of magnitude (from 0.003 to 0.3), the difference in RA is only 6%.
This means that even without tuning this parameter ( η ), La-MAML would have outperformed most baselines at their optimally tuned values.
[Figure]
As seen in Figure 4a, C-MAML sees considerable performance variation with the tweaking of both the inner and outer LR.
We also see that the effects of the variations of the inner and outer LR follow very similar trends and their optimal values
[Figure]
ﬁnally selected are also identical.
This means that we could potentially tune them by doing just a 1D search over them together instead of varying both independently through a 2D grid search.
The Sync version of La-MAML (Figure 4b), while being relatively insensitive to the scalar initial value α 0 and the η , sees considerable performance variation as the outer learning rate for the weights: β is varied.
This variant has the most hyper-parameters and only exists for the purpose of ablation.
Fig.
5 shows the result of 2D grid-searches over sets of the above-mentioned hyper-parameters for C-MAML and La-MAML for a better overview.
D. Experimental
We carry out hyperparameter tuning for all the approaches by performing a search over the range [0.0001 - 0.3] for hyper-parameters related to the learning-rate.
Note that in the Single-Pass (LLL) setup, since we only see the data in one pass without the option of revisiting, seeing each data-point for multiple glances is essential. We deﬁne a hyper-parameter called glances , which indicates the number of gradient updates made on each data-point, with the performance of the algorithms increasing with the increase in the number of glances up to a certain point. This is replaced by the parameter indicating the number of epochs that the model is trained for, for each task in the Multiple-Pass setup.
Table 5 lists the optimal hyperparameters for all the compared approaches.
All setups used the SGD optimiser since it was found to preform better than Adam (Kingma & Ba, 2014) (possibly due to reasons stated in Section 4.3 regarding the CL setup).
To avoid exploding gradients, we clip the gradient values of all approaches at a norm of 2.0.
Class divisions across different tasks vary with the random seeds with which the experiments were run.
Overall, we did not see much variability across different class splits, with the variation being within 0.5-2% of the mean reported result as can be seen from Table
For all our baselines, we use a constant batch-size of 10 samples from the streaming task.
This batch is augmented with 10 samples from the replay buffer for the replay-based approaches.
La-MAML and its variants split the batch from the streaming task into a sequence of smaller disjoint sets to take k = 5 gradient steps in the inner-loop .
In MER, each sample from the incoming task is augmented with a batch of 10 replay samples to form the batch used for the meta-update.
We found very small performance gaps between the ﬁrst and second-order versions of our proposed variants with performance differences in the range of 1-2% for RA.
This is in line with the observation that deep neural networks have near-zero hessians since the ReLU non-linearity is linear almost everywhere ( ? ).
MNIST Benchmarks : On the MNIST continual learning benchmarks, images of size 28x28 are ﬂattened to create a 1x array.
This array is passed on to a fully-connected neural network having two layers with 100 nodes each.
Each layer uses
ReLU non-linearity.
These experiments use a modest replay buffer of size 200 for MNIST Rotations and Permutation and size 500 for Many Permutations.
Real-world visual classiﬁcation : For Cifar and Imagenet we used a CNN having 3 and 4 conv layers respectively with [Table] 0.1 η 0. [Table]
[Table]
0.3 Epochs/Glances    3x3 ﬁlters.
The output from the ﬁnal convolution layer is ﬂattened and is passed through 2 fully connected layers having 320 and 640 units respectively.
All the layers are succeeded by ReLU nonlinearity.
For CIFAR and Imagenet we allow a replay buffer of size 200 and 400 respectively which implies that each class in these dataset gets roughly about 1-2 samples in the buffer.
For Tiny-Imagenet, we split the validation set into val and test splits, since the labels in the actual test set are not released.
E. Baselines
On the MNIST benchmarks, we compare our algorithm against the baselines used in (Riemer et al., 2019), which are as follows:
• Online:
A baseline for the LLL setup, where a single network is trained one example at a time with SGD.
• EWC (Kirkpatrick et al., 2017): Elastic Weight Consolidation is a regularisation based method which constraints the weights important for the previous tasks to avoid catastrophic forgetting.
• GEM (Lopez-Paz & Ranzato, 2017):
Gradient Episodic Memory does constrained optimisation by solving a quadratic program on the gradients of new and replay samples, trying to make sure that these gradients do not alter the past tasks’ knowledge.
• MER
(Riemer et al., 2019): Meta Experience Replay samples i.i.d data from a replay memory to meta-learn model parameters that show increased gradient alignment between old and current samples.
We evaluate against this baseline in the LLL setup.
On the real-world visual classiﬁcation datasets we carry out experiments on GEM, MER along with:-
• ER: Experience Replay uses a small replay buffer to store old data using reservoir sampling.
This stored data is then replayed again along with the new data samples.
• IID:
Network gets the data from all tasks in an independent and identically distributed manner, thus bypassing the issue of catastrophic forgetting completely.
• iCARL (Rebufﬁ et al., 2017): iCarl is from the family of class incremental learners, which learns to classify images in the metric space.
It prevents catastrophic forgetting by using a memory of exemplar samples to perform distillation from the old network weights.
• A-GEM
(Chaudhry et al., 2019):
Averaged Gradient Episodic Memory proposed to project gradients of the new task to a direction such as to avoid interference with respect to the average gradient of the old samples in the buffer.
• Meta-BGD:
Bayesian Gradient Descent (Zeno et al., 2018) proposes training a bayesian neural network for CL where the learning rate for the parameters (the means) are derived from their variances.
We construct this baseline by equipping C-MAML with bayesian training, where each parameter in θ is now sampled from a gaussian distribution with a certain mean and variance.
The inner-loop stays as in C-MAML(constant LR), but the magnitude of the meta-update to the parameters in θ is now inﬂuenced by their associated variances.
The variance updates themselves have a closed form expression which depends on m monte-carlo samples of the meta-loss , thus implying m forward passes of the inner-and-outer loops (each time with a newly sampled θ ) to get m meta-gradients.