Achieving a Better Stability-Plasticity Trade-off via Auxiliary Networks in Continual Learning
Abstract
In contrast to the natural capabilities of humans to learn new tasks in a sequential fashion, neural networks are known to suffer from catastrophic forgetting , where the model’s performances drop dramatically after being optimized for a new task.
Since then, the continual learning (CL) community has proposed several solutions to equip the neural network with the ability to learn the current task ( plasticity ) while still achieving high accuracy on the old tasks ( stability ).
Despite remarkable improvements, the plasticity-stability trade-off is still far from being solved and its underlying mechanism is poorly understood.
In this work, we propose Auxiliary Network Continual Learning (ANCL), a new method that regularizes the continually learned model with an additional auxiliary network that is solely optimized on the new task.
More concretely, the proposed framework materializes in a regularizer that naturally interpolates between plasticity and stability, surpassing strong baselines on image classification datasets.
By analyzing the solutions of several CL methods, we further propose a new technique for hyperparamter search technique which dynamically adjusts the regularization parameter to achieve better stability-plasticity trade-off.
Introduction
The main objective of continual learning (CL) is to investigate how neural networks can learn from a sequence of task without progressively forgetting what has already been learned, a problem known in the connectionist approach to cognitive science as catastrophic forgetting [32, 11, 28].
To overcome the issue, various methods have been proposed which can be roughly categorized into regularization-based approaches [18, 2, 6, 40], distillation-based approaches [23, 17, 41, 7], structure-based approaches [1, 38, 25, 26], and replay-based approaches [33, 5, 4, 39].
Based on these approaches, recent works utilize an auxiliary network or module in various ways [36, 41, 25, 24] (Appendix A).
Notably, Active Forgetting with synaptic Expansion-Convergence (AFEC) [36] adds a regularization term calculated on the auxiliary network that allows the model to actively forget conflicting knowledge and retain the information that can be re-used from previous tasks, aiming at achieving a proper plasticity-stability trade-off.
Despite the significant advances, the underlying mechanism of this trade-off has received relatively less attention.
Thus, in this work, we formally define the adaptation of the auxiliary network and investigate how it affects the stability-plasticity trade-off.
More concretely, our main contributions are as follows:
We generalize and formalize the framework of Auxiliary Network Continual Learning (ANCL) that can apply the auxiliary network to a variety of continual learning (CL) approaches as a plug-in method (Section 3).
We empirically show that ANCL can achieve better stability-plasticity trade-off than the existing CL baselines on CIFAR-100 [20] and Tiny ImageNet [22] (Section 4).
Based on ANCL, we suggest a new method called Adaptive ANCL (Adap-ANCL) that estimates the regularization hyperparameter in a task-dependent fashion and show that it further boosts the performance (Section 4 and Appendix G).
Related Work
We consider the standard problem of learning sequentially T tasks using a neural network f θ , where θ ∈ R P denotes the learnable weights.
In the standard continual learning (CL) framework, when presented with task t the user has an access to the previous network weights θ ∗ 1: t − 1 ∈ R P , which are the result of the continual learning from task 1 to t − 1 .
It is well known that simply starting optimization from the weights θ ∗ 1: t − 1 to obtain the weights θ ∗ 1: t using the new data from the task t results in catastrophic forgetting [28] of the old tasks.
A common way to mitigate catastrophic forgetting is to include a regularization term, which binds the dynamics of each network parameter θ i ( i ∈ 1 , . .
. , P ) to the corresponding old network parameter θ 1: t − 1 ,i through a regularization term Ω 1: t − 1 ,i > 0 .
The new optimization problem then returns:
[Equation]
where in classification problems L CE ( θ ) is cross-entropy loss on the data of the task t and λ is the regularization strength and is usually selected via a grid search procedure.
Various regularizationbased methods chose different ways to estimate the parameter Ω i .
For example, Elastic Weight
Consolidation (EWC) [18] calculates Ω i through the approximation of Fisher Information Matrix (FIM), and Memory Aware Synapses (MAS) [2] measures Ω i as a function of the magnitude of the updates on parameter i .
In Appendix B, we further detail these and other CL frameworks that we build upon.
Our approach stems from Wang et al.
[36], who uses a biologically inspired argument to propose
Active Forgetting with synaptic Expansion-Convergence (AFEC).
AFEC adds an additional regularization term to the EWC loss to promote active forgetting , derived from a Bayesian analysis:
[Equation]
The first two terms are the same as above, while the last term promotes active forgetting by regularizing the weights θ ∈ R P towards the biologically inspired expanded weights ˆ θ t ∈ R P (see precise definition in [36]) through the FIM F t on the task t .
The last term in Eqn.
(2) can be applied to other regularization-based methods as a plug-and-play method.
Method
Motivated by AFEC [36], we formally define Auxiliary Network Continual Learning (ANCL) which applies the auxiliary network trained on the current task to the continually learned previous network in order to resolve the stability-plasticity dilemma.
Fig.
1 illustrates the conceptual difference between
CL and ANCL where CL can be any continual learning method that includes a regularizer via the old network.
Original CL approaches mainly focus on retaining the old knowledge obtained from the previous tasks by preventing weight updates that take away from the previous weight θ ∗ 1: t − 1 .
However, this might harmfully restrict the model’s ability to learn the new knowledge, which will ruin the right balance between stability and plasticity.
On the contrary, ANCL maintains the two types of network to maintain this balance: (1) the auxiliary network θ ∗ t , which is optimized solely on the current task t allowing for forgetting ( plasticity ) and (2) the old network θ ∗ 1: t − 1 that has been sequentially trained until task t − 1 ( stability ).
Then, both networks regularize together the optimization process of the main network weight θ ANCLt −where the stability-plasticity balance is adjusted through λ and λ a .
For example, ANCL can be applied to EWC [18] and MAS [2] to build Auxiliary Network EWC (A-EWC) and Auxiliary Network MAS (A-MAS) accordingly.
Then, the new loss of A-EWC and
A-MAS can be written as follows:
[Equation]
where the importance Ω t of the auxiliary parameter θ ∗ t,i is calculated following the original methods (EWC or MAS).
The first two terms are equal to Eqn.
(1) and A-EWC is equivalent to AFEC 1 except that the importance Ω t of the auxiliary network in ANCL is calculated only once 2 before the training of the new task and then fixed afterward.
Similarly, ANCL can be extended to distillation-based methods such as Learning without Forgetting (LwF) [23] or less-forgetting learning (LFL) [17].
Given the data x j , LwF regularizes the logit y j of the current model to be similar to the logit y old,j of the old network, while LFL drifts the activation f ( x j ) before the last layer of the main network towards the activation f old ( x j ) of the old network (detailed in Appendix B).
By applying ANCL to LwF and LFL, the new loss of Auxiliary Network
LwF (A-LwF) and Auxiliary Network LFL (A-LFL) is written as follows:
[Equation]
In Eqn. (4), y aux represents the temperature-scaled logit of the auxiliary network and C 1: t denotes the total number of classes until task t . The new regularizer is double summated over the class position c and the data x j .
In Eqn. (5), f aux ( x ) is the normalized and centered activation of the auxiliary network.
The auxiliary network θ ∗ t of ANCL works similarly to the expanded parameter ˆ θ t of AFEC with respect to adding an extra loss term, but ANCL uses the method-dependent regularizer compared to the fixed and independent regularizer of AFEC based on FIM.[-In other words, while AFEC plugs
 However, A--]
In other words, while AFEC plugs in the same loss term of the expanded parameter to every method, ANCL generates the loss term of the auxiliary network in the same way as the original CL where ANCL is applied.
This is a more natural way to implicitly embrace two networks because the stability-plasticity trade-off can be easily controlled by the scaling hyperparameters ( λ and λ a in Eqn. (3)) of two regularizers in the same type.
This is mathematically shown in Appendix C, where we analyze the gradients of ANCL.
If the two regularizers are in different type, each regularizer will change in different magnitude and consequently make it hard to arrive at the equilibrium. Methods [Table]
Experiment
CIFAR-100 [20] and Tiny ImageNet [22] are chosen to form two benchmarks after divided intotasks each: (1) CIFAR-100/10 and (2) Tinyimagenet-200/10. Benchmark (1) and (2) containand 20 classes per task respectively. Resnet32 [14] with multi-head at the last layer is used for all experiments.
Our experiment is classified as task incremental learning [34], where the task identity is given during both the training session and inference.
In addition, every experiment is conductedtimes, and we report the averaged metric.
We evaluate our ANCL approaches in Table 1: A-EWC, A-MAS, A-LwF, and A-LFL.
Fine-tuning is the naive approach that the model is fine-tuned on a sequence of task which is regarded as a lowerbound.
Joint uses the whole datasets to train the model, which then becomes an upperbound.
In all methods, applying ANCL gives an extra boost in accuracy compared to standard CL.
This coincides with the theoretical analysis of the ANCL loss in Appendix C, supporting that ANCL is more capable of balancing stability and plasticity.
The grid search is performed on both λ and λ a (detailed in Appendix E), and the accuracy of all tasks can be seen in Appendix D.
Moreover, we observed that the model favors more stability over plasticity during the training of later tasks through extensive analyses of the stability-plasticity trade-off (Appendix F).
One possible reason is that the model should remember more previous knowledge as the training progresses.
Based on this finding, we propose to employ the task-dependent hyperparameter λ a,t instead of λ a called Adaptive ANCL (Adap-ANCL) which is defined as follows: [Equation]
[Equation]
C t denotes the number of classes at task t and C 1: t represents all classes accumulated until task t . λ a,init is an initial value determined by grid search.
Thus, the adaptive parameter λ a,t decreases as the model learns more task.
In such manner, ANCL is dynamically optimized toward the better balance between stability and plasticity in every task.
Adap-ANCL further improves the accuracy of ANCL, which can be seen in Table 1.
Conclusion
In this paper, we propose a novel CL scheme called ANCL which effectively employ the auxiliary network to reach better stability-plasticity trade-off as a plug-in method.
It is shown theoretically and empirically that ANCL has a high potential and is actually able to effectively merge the old and auxiliary networks.
Based on ANCL, we finally propose Adap-ANCL which actively modifies a key hyperparameter depending on task.
A Continual Learning Methods using Auxiliary Network
In the last few years, several papers have proposed to use an auxiliary network or an extra module which is trained on a new task.
They tried to combine it with a previous network or module which has been trained on old tasks using existing continual learning techniques.
Elastic Weight Consolidation (EWC) [18] is one of the initial works among weight regularization methods that regularize weights related to the previous tasks.
After training each task, EWC copies and freezes the current network as an old network.
Then, it estimates the importance of each parameter in the old network so that the weights with the high importance remain unchanged when the model is optimized on future task.
Based on the EWC, Wang et al.
[36] proposes Active Forgetting with synaptic Expansion-Convergence (AFEC) which further regularizes the weights relevant to the current task through expanded parameters.
The expanded parameters are solely trained on the dataset of the new task initialized by the old network and are allowed to forget the previous tasks.
As a result,
AFEC can reduce potential negative transfer in continual learning by selectively merging the old parameters with the expanded parameter.
The stability-plasticity balance in AFEC is adjusted via hyperparameters which scale the regularization terms for remembering the old tasks and learning the new tasks, respectively.
Learning without Forgetting (LwF) [23] prevents forgetting using knowledge distillation [3, 15].
They apply a distillation loss so that the model can learn soft targets generated by the old model instead of typical one-hot targets.
The old model is copied and freezed before the training of the current task like EWC.
Deep Model Consolidation (DMC) [41] is another distillation method built upon LwF.
DMC proposes double distillation loss where the soft targets are generated using both the old model and a new model.
The new model in DMC is basically the same concept as the expanded parameter in AFEC which is optimized on the current task.
Then, the logits of new classes from the new model and the logits of old classes from the old model are concatenated to build the final soft targets.
Then, the stability-plasticity balance is achieved by penalizing the model to generate the same output as the old model for old classes and the same output as the new model for new classes.
Adaptive Aggregation Networks (AANet) [25] explicitly expands ResNet [14] to have the two types of residual blocks at each residual level: the one for retaining old knowledge and the other for learning new knowledge.
The outputs from the two residual blocks are linearly combined by aggregation weights and then proceeded to the next-level layer.
They train AANets through bilevel optimization.
In the first level, the parameters of the two residual blocks are trained.
In the second level, the aggregation weights are adapted which decide the balance between stability and plasticity within the ResNet.
Recent work by Mirzadeh et al.
[29] observes that multitask and continual solutions are connected by very simple curves with a low error in weight space, which is called Linear Mode Connectivity .
They empirically prove that this connectivity is a linear path if the multitask learning and the continual learning share same initialization weight.
Based on this observation, Lin et al.
[24] proposes a simple linear connector that linearly add the weights of two networks following this linear path to emulate the multitask solution: the one model remembering the old tasks and the other model learning the new tasks.
The stability-plasticity balance is preserved by combining the two networks.
The above methods [36, 41, 25, 24] all share the property that the auxiliary model or module is used to solve the stability-plasticity dilemma in continual learning.
Consequently, these methods are able to learn the current task better than the original method while still retaining the knowledge of the previous tasks.
However, the underlying mechanism of the interaction between the previous model and the auxiliary model is not widely studied.
Therefore, in this paper, we first formalize the framework of continual learning that adopts the auxiliary network called Auxiliary Network Continual
Learning (ANCL). Given this environment, we investigate the stability-plasticity trade-off from both a theoretical and empirical point of view and perform various analyses to better understand it. B Detail Explanation of Existing Continual Learning approaches B.1 Weight Regularization Methods Regularization-based methods in continual learning regularize the weight of the main network toward the old network so that the current weight does not deviate a lot from the previous optimal weight.
Various approaches choose different ways to calculate the regularizing parameter Ω i in Eqn.
For example, Elastic Weight Consolidation (EWC)
[18] calculates Ω i through the approximation of
Fisher Information Matrix (FIM).
The diagonal elements of FIM quantifies how curved the likelihood of each parameter is and can be approximated by a Hessian matrix near the optimum.
In other words, the larger the change in the gradient, the more relevant the corresponding parameter is to the previous tasks.
FIM is calculated after training each task, which means that R i of EWC cannot fully reflect the learning trajectory of each network weight.
Compared to EWC, Memory Aware Synapses (MAS) [2] proposes accumulating the changes of each parameter throughout the update history.
R i is measured through the magnitude of the updates on each parameter according to the change in output.
In other words, if the small change of specific parameter in a network causes the huge change of an output, that parameter should be memorized first.
Based on the above regularization-based methods, Wang et al.
[36] suggests a biologically inspired argument to propose Active Forgetting with synaptic Expansion-Convergence (AFEC) where an additional regularization term is added on EWC loss.
They argue that this term stimulates the active forgetting of previous knowledge that interferes with the new knowledge, whereas the existing regularization-based approaches highly concentrate on retaining the old weight.
The new loss of
AFEC can be seen in Eqn.
The last term in Eqn.
(2) promotes active forgetting by regularizing the parameters θ ∈ R P towards the biologically inspired expanded parameters ˆ θ t ∈ R P (see precise definition in [36]) through FIM F t on task t .
The expanded parameter is solely trained on the current dataset allowing the forgetting of the old datasets and λ a is a hyperparameter determined by grid search.
As a result, the main network parameter θ efficiently learns from both the old parameters θ ∗ 1: t − 1 and the expanded parameters ˆ θ t .
Moreover, the last term of Eqn.
(2) can be applied to other regularization-based methods as a plug-and-play method.
The expanded parameters ˆ θ t in AFEC are computed as follows: at the beginning of each task, the expanded parameter is initialized by the old parameters, Then, at each epoch, it is trained on task t without regularization, returning the network parameters ˆ θ t .
For the current epoch of the original network, these weights are then taken into account to modify the regularized dynamics.
This procedure is repeated every epoch, and hence for each epoch the dynamics of SGD is augmented with a freshly computed regularizer.
Distillation-based approaches prevent forgetting through knowledge distillation [3, 15] which was originally devised to train a smaller student network from a larger teacher network.
In this way, the main network can emulate the activation or logit of the previous network while learning the current task.
Learning without Forgetting (LwF) [23] propose following loss to make current network to distill the knowledge of the old network via imitating the logit:
[Equation]
Similar to Eqn.
(1), θ is the weight of the main network that is trained on a sequence of tasks so far and L CE ( θ ) is cross-entropy loss on current task t .
y old and y are temperature-scaled logits of the old network and the current network, respectively.
The main network that continuously learns data from task 1 to task t − 1 is freezed and save as the old network.
C 1: t − 1 denotes the total number of classes until task t − 1 .
Thus, y cj refers to the c th output of the logit of input x j .
The logit scaled by temperature τ on the c th class position is calculated as follows:
[Equation]
where o ( x j ) and o old ( x j ) are the output of the current network and the old network before softmax is applied.
Another distillation method, less-forgetting learning (LFL) [17], penalizes the differences of activations before last layer instead of logits:
[Equation]
f ( x j ) and f old ( x j ) are centered and normalized activation from the main and old network each.
The knowledge of the old network is transfered via the difference of two activations which lead the activation of the current model to the counterpart of the previous model.
C The Role of Auxiliary Network
In this section, we look into our CL and ANCL loss closely by analyzing its gradient.
According to each gradient, we compare how CL and ANCL methods converge to optimal weight in the point of stability-plasticity balance.
First, we start with EWC loss [18]:
[Equation]
where F 1: t − 1 ,i represents the accumulated fisher information matrix of parameter θ ∗ 1: t − 1 ,i until task t − 1 .
For simplicity, let’s assume importance F is fixed among all parameters of network.
Then, we can simplify the EWC loss as follows:
[Equation]
[Equation]
where we take the derivative with respect to the weight of the model θ .
Then, we have
[Equation]
The gradient consists of two terms: the first term that updates θ towards the minima of the current task through cross-entropy loss and the second term which regularizes θ to be as close as possible to θ ∗ 1: t − 1 .
If λ is gradually increased, the second term becomes more and more dominant in the gradient so that θ converges near θ ∗ 1: t − 1 at the end of the training.
In this case, stability is achieved but plasticity is neglected.
Next, we analyze with A-EWC loss in the same way.
We also assume here that importance F is fixed among all parameters of network.
Then, Eqn.
(3) can be rewritten as
[Equation]
Take the derivative with respect to θ :
[Equation]
In the last equality, α = λF 1: t − 1 and β = λ a F t are used.
The gradient of A-EWC also consists of two parts.
The first term is the same as before and the second term penalizes θ to move toward the interpolation of θ ∗ 1: t − 1 and θ ∗ t .
This linear interpolation fully depends on the hyperparameter λ and λ a as we fixed the importance.
Therefore, θ will converge to the weight between θ ∗ 1: t − 1 and θ ∗ t at the end, which turns out to be the region with high accuracy by mode connectivity figures in Appendix F.4.
In Eqn.
(13), the balance between stability and plasticity can be adjusted more easily through the ratio between λ and λ a while it is solely dependent on λ in Eqn.
For MAS [2] and A-MAS, similar analaysis can be applied as above if we simply replace the importance of EWC F with the importance of MAS Ω .
Among distillation losses, we first looked into LFL [17] loss:
[Equation]
Again, take the derivative with respect to θ :
[Equation]
The gradient of LFL loss can be divided into two parts: the first part that drives θ toward the minima of the current task and the second part that regularizes the difference of activations between the current model and the old model.
Compared to Eqn.
(12), LFL has more flexibility to memorize previous knowledge as it does not directly regularize its weight but only its activations.
The model trained with too high λ will mainly focus on generating exactly same activations as old model while neglecting learning current task.
Next, we investigate A-LFL loss in Eqn.
[Equation]
Same derivative is applied as follows:
[Equation]
Which can be organized as
[Equation]
The second term of the gradient of A-LFL derives the activation of the main network to be equal to the interpolated activation of the auxiliary and old network.
The interpolation is decided by the ratio of λ and λ a .
It is shown in Fig.
11 that this leads the weight to achieve the better trade-off in accuracy landscape overcoming simple linear interpolation between the old and auxiliary weight
Moreover, it is worth noting that Eqn.
(20) become quite similar form as Eqn.
(15) if we assume a one-layer network where f old ( x j ) =
W old x j , f aux ( x j ) =
W aux x j , and f ( x j ) =
Wx j hold such that W old , W aux , W ∈ R p × d and x j ∈ R d :
[Equation]
Now, the second term drift model weight W (except the last layer) toward the interpolation of the old weight and the auxiliary weight like that of the A-EWC gradient.
Again, we strike a balance between stability and plasticity balance through λ and λ a .
Next distillation loss is LwF [23] as follows:
[Equation]
For simplicity, we assume that y old and y are softmax logits scaled by temperature τ such as
[Equation]
Then, referring to Hinton et al.
[15], the gradient of LwF with respect to logits θ can be calculated as follows:
[Equation]
When τ is sufficiently large, then softmax can be approximated using exp ( o ( x j ) /τ )
≈
+ o ( x j ) /τ :
[Equation]
We further assume that logits of main network and old network are zero mean, i.e., (cid:80) k
o ( x j ) k
=and (cid:80) k
o ( x j ) kold = 0 .
Finally, we get the following approximation:
[Equation]
This demonstrates that the gradient of LwF is in equivalent form as the gradient of LFL under sufficiently large temperature and zero-mean logits from both main and old network.
One should also note that o ( x j ) in LwF is logit and f ( x j ) in LFL is activation before last layer.
Similarly, we can approximate the gradient of A-LwF:
[Equation]
Then, we get the final form as follows:
[Equation]
Similar to A-LFL, the second part of gradient shift the logit from main network toward the interpolation of the logits from old and auxiliary network.
The stability-plasticity dilemma is solved by directly controlling λ and λ a .
D Detailed Results on Experiments
In this section, we plot the final accuracy of the sequential 10 task of CIFAR-100 in Figure 2.
In
Figure 2, the accuracy for most tasks of ANCL approaches is higher than that of CL approaches.
In particular, ANCL achieves better performance in the later task compared to CL at the cost of losing some of the earlier task accuracy, which is well shown in Figure 2 (b) and (c).
This is because ANCL methods are able to learn new tasks better than CL through the extra regularization term as explained in Section C.
However, ANCL loses a bit of ability to remember initial knowledge as a trade-off, but ANCL is still comparable to CL in earlier tasks.
E Training details
Benchmark: CIFAR-100 [20] and Tiny ImageNet [22] are used to evalulate our ANCL methods.
CIFAR-100 contains 60,000 colored images from 100 classes with the size of 32 × 32 and is divided into 50,000 and 10,000 images to form a train set and a test set.
Then, the 10% of the train set is randomly sampled to build a validation set.
CIFAR-100 is divided into 10 tasks of 10 classes each to construct a benchmark (1) CIFAR-100/10 .
Tiny ImageNet consists of 100,000 train images and 10,000 test images and the 10% of the train set is again used as a validation set. It includes 200 classes and all data are 64 × 64 colored images which are resized as 32 × 32 for both training and inference. We equally divide Tiny ImageNet into 10 to build a benchmark (2) TinyImagenet-200/10 . Architecture: Resnet32 [14] are selected to conduct all experiments which are commonly used by serveral works in the literature of continual learning [33], [37], [16], [42], [8].
Multi-head layer is deployed at the last layer of Resnet32 to generate output with task identity.
Thus, total number of head is equal to the number of tasks.
Note that the heads of previous tasks are freezed while the model learns the current task, which helps to prevent forgetting compared to baselines updating every parameter [27].
Implementation: We build our ANCL methods based on code implementation of Framework for
Analysis of Class-Incremental Learning (FACIL) [27] which supports several baselines and existing
CL methods.
SGD Optimizer with 0.9 momentum and the mini-batch size of 128 is applied to all experiments.
The initial learning rate of each task is determined by grid search and the learning rate of the first task is slightly larger than that of the following task as our model is trained from scratch.
Then, the learning rate is decreased by factor 3 whenever there is no improvement in validation loss for 5 epochs. If the learning rate reaches minimum threshold, the model stops training. The model can be trained for maximum 200 epochs per task. Baseline: First, we do not apply any memory buffer that has an access to the previous datasets.
Task identity is given during both training and inference, which is regarded as a task incremental learning scenario.
In addition, every experiment train the model from scratch and conducted 3 times to generate averaged accuracy.
Finetuning is the naive approach that model is finetuned on each task, which is regarded as a lowerbound.
Joint use the whole dataset to train model, which becomes an upperbound.
For weight regularization methods, we evaluate EWC [18] and MAS [2].
For distillation methods, we implement
LwF [23], and LFL [17].
Based on these CL methods, we also compare our ANCL approaches:
A-EWC, A-MAS, A-LwF, and A-LFL.
Evaluation Metric:
In our paper, averaged accuracy ( AAC ) for T task is reported after the training of all tasks:
[Equation]
where A j,k is the test accuracy of task k after the continual learning of task j .
Resources: We conduct all our experiments using gpu "NVIDIA GeForce GTX 1080 Ti".
Algorithm:
Detailed pseudocode in Algorithm 1 explains how we implement ANCL approaches.
This is valid for all ANCL methods (A-EWC, A-MAS, A-LwF, and A-LFL) if appropriate ANCL loss is applied.
If lines 9-12 are skipped and "ANCL loss" in line 14 is replaced with "CL loss",
Algorithm 1 becomes the original CL algorithm.
In this section, three analyses were performed to deeply understand how stability-plasticity dilemma is solved through ANCL. It is worth to note that the specific learning regime described in Appendix
Input: Main network weight θ , Auxiliary network weight θ t , Old network weight θ 1: t − 1 ,
Hyperparameters λ , λ a
Output: Optimal weight θ ∗ 1 for task t = 1, 2, .., N do if t = 1 then //
Train main network on first task for epoch
e = 1, 2, .., E do
Train θ with L CE ( θ ) to obtain θ ∗ on task
Set θ ∗ 1:1 = θ ∗ (Calculate the importance of θ ∗ 1:1 if needed) //
Save old network weight
Freeze and save θ ∗ 1:  else //
Train auxiliary network θ t = copy( θ ∗ 1: t − 1 )
Train θ t with L CE ( θ t ) to obtain θ ∗ t (Calculate the importance of θ ∗ t if needed) //
Save auxiliary network weight
Freeze and save θ ∗ t //
Train main network for epoch e = 1, 2, .., E do
Train θ with ANCL loss Eqn. (3, 4, 5) to obtain θ ∗ on task t Set θ ∗ 1: t − 1 = θ ∗ (Calculate the importance of θ ∗ 1: t − 1 if needed) //
Save main network weight as old network weight Freeze and save θ ∗ 1: t −
F.1 Fig.
3 was adopted in all following analyses which is built upon the setting in Mirzadeh et al.
F.1 Training Regime for Trade-off Analysis
All analyses in this paper are based on the specific learning scheme described in Fig.
On task t, every model starts training from multitask weight θ multit − 1 which is trained on combined dataset D 1: t [-− 
 [Table] before-]−before (i.e., train on D 1 ∪ D 2 ∪· · ·∪ D t − 1 ).
If we fine-tune the model on data D t , it becomes auxiliary network θ ∗ t which regularizes ANCL later.
ANCL and CL approach is applied to get corresponding weitht θ ANCLt and θ CLt each.
Here, the weight of multiple tasks θ multit −on task t − 1 works as the old network weight θ ∗ 1: t − 1 to regularize ANCL and CL.
Finally, the initial weight is trained on the data
D 1: t to train the next multitask model θ multit and it becomes the next initialization(or old network) for the task t
+ 1 .
We used a fixed multitask weight as an initialization at the start of each task for a fair comparison among all methods.
Otherwise, every method will have different old and auxiliary network which end up confusing following precise analyses.
Especially, this training scheme is essential to visualize mode connectivity figure as Mirzadeh et al.
[29] empirically shows that linear mode connectivity is valid when models share same initialization.
F.2 Weight Distance
In this section, we measure Euclidean Distance from the weight of ANCL models to the weight of the previous model and the auxiliary model respectively.
If the parameter changes less, it is reasonable to expect that less forgetting will happen.
According to Mirzadeh et al.
[30], forgetting F 1 on task 1 is bounded using Taylor expansion of the loss as follow:
[Equation]
[Equation]
(31) where L 1 is the empirical loss on task 1 and ∇ 2 L 1 (ˆ θ 1 ) is the Hessian for L 1 at ˆ θ 1 .
λ maxis the maximum eigenvalue of ∇ 2 L 1 (ˆ θ 1 ) .
Above inequality implies that the forgetting F 1 is bounded by the norm of the difference between two weights near the minima of task 1.
Provided that our model learns task t now, the previous model refers to freezed and saved optimal network that has been trained until task t − 1 and the auxiliary model is the network that is initialized by the weight of the previous model and trained on task t .
Then, we can measure two types of weight distance (WD):
[Equation]
WD old and WD aux calculate the distance from the parameter of ANCL network θ ANCLt to the parameter of old network θ ∗ 1: t − 1 and the parameter of auxiliary network θ ∗ t respectively in the same setting as Fig.
The weight distance of ANCL approaches is shown in Fig.
We calculate the weight distance with different λ a in Eqn.
(3, 4, 5) which directly adjusting the trade-off between memorizing old tasks and learning new tasks.
The model weight remains close to the old weight when λ a is small, which can be seen in the left side of all figures.
In (a) A-EWC and (b) A-MAS, WD aux decreases and WD old increases as λ a grows larger.
This agrees with the analysis of ANCL gradient in Appendix C that the model will converge to the auxiliary network if last term in Eqn.
(3) becomes dominant within the loss.
In (c) A-LwF and (d) A-LFL, WD aux becomes relatively lower than WD old with increasing λ a but overall distance become larger for both WD old and WD aux .
It is reasonable to think that this is due to the difference between regularization-based methods and distillation-based methods.
Unlike EWC and MAS which directly regularize weight itself, LwF, and LFL choose rather indirect measure, applying loss term based on activation or logit.
Thus, in the distillation approaches, the model weight tends to move closer to the auxiliary weight with increasing λ a but not directly toward it like EWC and MAS.
F.3 Centered Kernel Alignment
Centered Kernel Alignment (CKA) [19] measures the similarity of two representation on the same set of data.
Given N data and p neurons, the layer activation matrices R 1 ∈ R N × p and R 2 ∈ R N × p are generated by two layers from two different networks.
Then, CKA is defined as:
[Equation]
HSIC stands for Hilbert-Schmidt Independence Criterion [13].We use linear HSIC to implement
CKA.
Ramasesh et al.
[31] measured the layer-wise CKA similarity scores to study catastrophic forgetting on CIFAR datasets.
They argue that forgetting happens at deeper layers close to output, as there is a severe drop in CKA score in deeper layers compared to initial layers.
In this analysis, we measure three CKA similarity following training regime in Fig.
[Equation]
where CKA is calculated over all layers l in resnet32 and averaged.
R ANCLt , R ∗ 1: t − 1 and R ∗ t are the activation matrices of the ANCL, old, and auxiliary networks, respectively.
R multit is the activation output from multitask model trained on whole data D 1: t until task t .
Thus, if CKA multi is high, the model is likely to perform well on all task like multitask model.
In Fig.
5, three types of CKA ( CKA old , CKA aux , and CKA multi ) are calculated with different λ a that adjusts the stability-plasticity balance.
CKA old , CKA aux , and CKA multi measures the CKA similarity from ANCL model to old, auxiliary, and multitask model respectively.
In all methods, increasing λ a results in higher CKA aux and lower CKA old , which means that the representation of the ANCL network becomes more similar to that of the auxiliary network and less similar to that of the old network.
We can clearly see stability-plasticity trade-off is made by simply controlling λ a .
On the other hand, if CKA multi is high, it can be interpreted that the model tends to generate representation similar to the multitask model, which is the goal of continual learning.
Thus, the model is highly likely to be at the best trade-off where λ a achieve the highest CKA multi .
For example, (b)
A-MAS and (d) A-LFL reach the best trade-off at λ a = 50 and λ a = 100 respectively.
In addition, one might ask how the best trade-off changes in a sequence of tasks.
To answer this question, we also plot CKA figures on different tasks in Fig.
6 and 7.
Here, we focus on the optimal λ a with the highest CKA multi which is denoted as λ ∗ a inside the figure.
In Fig.
6, λ ∗ a changes as 50 , 10 and 5 on the task 2 - 4 and in Fig.
7, λ ∗ a changes as 100 , 50 and 10 .
One promising reason for this shrinking λ ∗ a is that there are more things to remember as we increase the number of task.
In other words, optimal model favors more stability over plasticity in later tasks with respect to the trade-off.
This phenomenon is again seen by the analysis of mode connectivity in Fig.
10 and 11, which motivates our new approach of Adap-ANCL afterward in Appendix G.
F.4 Mode Connectivity
Recent works [9], [12] find a simple curves between two local optima of deep neural networks (DNN) such that train loss and test error remain low along these curve.
This simple linear path with low error can be visualized in the loss surface of DNN using gradient-based optimization methods.
This path called Mode Connectivity has been studied empirically and theoretically with some assumptions in different papers.
Venturi et al.
[35] theoretically studied mode connectivity when the number of neurons is greater than the number of training data points in one-hidden-layer neural networks.
Then, Kuditipudi et al.
[21] investigated whether mode connectivity still holds in more realistic setting such as overparameterized networks and noise stable networks.
Frankle et al.
[10] analyze the effect of SGD noise on gradeintbased neural networks and linearly connected minimum was used to determine the outcome of optimization.
Mode connectivity in continual learning is first studied by Mirzadeh et al.
They empirically showed that the multitask solution and the continual solutions are linear connected in loss landscape assuming that both start from same initialization.
We also follow Mirzadeh et al.
to visualize the mode connectivity in a two-dimensional plot.
In Fig. 8, we can actually check that continual learning solutions ( θ multiand θ ∗ 2 ) are linearly connected to multitask solution ( θ multi) in both loss and accuracy landscape.
In the left and center columns, multitask weight achieves higher accuracy and low loss compared to continual learning solutions.
This is because every weight is fine-tuned on the dataset and the multitask model has higher discriminating ability due to its full access to previous datasets.
In the right column, we visualize the mean loss and mean accuracy of task 1 and 2.
Then, it is clear that higher accuracy and lower loss is achieved somewhere in the middle of θ multiand θ ∗ 2 .
We follow the training scheme in Fig.
3 and θ multiis equivalent to the model trained on D 1 only.
Next, we project CL (EWC, MAS, LwF and LFL) and ANCL (A-EWC, A-MAS, A-LwF and A-LFL) solutions on the previous graph and picture it as Fig.
We change λ a to see how the ANCL weight changes in the weight space.
In (a) A-EWC and (b) A-MAS, it is clearly seen in the figure that λ a adjusts the interpolation between θ CL and θ ∗ 2 .
Large λ a drift the ANCL weight θ ANCL directly toward the auxiliary weight θ ∗ 2 and the ANCL achieves the highest accuracy on the interpolation as it is in the higher contour.
Similarly in (c) A-LwF and (d) A-LFL, the ANCL weight θ ANCL starts near the CL weight θ CL and tends to move toward auxiliary weight θ ∗ 2 .
As distillation methods have more flexibility in balancing stability and plasticity, its optimum is able to climb up to the higher contour of mean accuracy than the ANCL applied to regularization-based methods.
As a result, the best trade-off is made at somewhere between the old weight and the auxiliary weight.
Again we want to emphasize that this is the projection of weight in the two-dimensional space built by three weights ( θ multi [Equation] and θ multi).
In other words, it approximates the relative position of CL and ANCL weight but does not show exact position of them in weight space.
Similar to Fig.
6 and 7, we demonstrate how optimal λ a changes in weight space as training is proceeded.
In Fig.
10 and 11, the mean accuracy plot on different tasks is drawn with projected CL and ANCL weight.
As we train our model on task 2, 3, and 4, the contour of mean accuracy becomes biased toward the old weight θ multit .
Thus, λ ∗ a with the highest mean accuracy is becoming smaller as the optimal weight prefers the interpolation close to the old weight.
This coincides with the results observed in Fig.
6 and 7 that λ a shrinks as the model learns more tasks.
In conclusion, smaller λ a achieves better trade-off in later tasks.
Motivated by these observations, we propose a new technique called Adaptive ANCL (Adap-ANCL).
Instead of using a fixed independent parameter, Adap-ANCL measures λ a in a task-dependent manner.
G Adaptive ANCL
ANCL methods solve the stability-plasticity dilemma by adjusting the ratio between λ and λ a , which is chosen by grid search.
However, fixed λ a can results in worse balance between retaining old knowledge and learning new knowledge because previous knowledge is accumulated as training is proceeded.
Thus, it is reasonable to diminish λ a to put more weight on remembering previous tasks compared to learning a new tasks.
We defined our new λ a,t on task t as follows:
[Equation]
C t denotes the number of classes at task t and C 1: t means all classes until task t .
λ a,init is initial λ a determined by grid search.
Thus, our adaptive parameter λ a,t reduces according to the ratio between the number of current classes and the total classes that have been trained so far.
In this way, ANCL is able to find a better balance between stability and plasticity in every task.
In Table 1, ANCL with adaptive λ a gives extra boost accuracy in all methods.
This means that adap-ANCL can dynamically find optimal weight by applying flexible hyperparameter.