CPR: Classiﬁer-Projection Regularization for Continual Learning
Abstract
We propose a general, yet simple patch that can be applied to existing regularization-based continual learning methods called classiﬁer-projection regularization (CPR).
Inspired by both recent results on neural networks with wide local minima and information theory, CPR adds an additional regularization term that maximizes the entropy of a classiﬁer’s output probability.
We demonstrate that this additional term can be interpreted as a projection of the conditional probability given by a classiﬁer’s output to the uniform distribution.
By applying the Pythagorean theorem for KL divergence, we then prove that this projection may (in theory) improve the performance of continual learning methods.
In our extensive experimental results, we apply CPR to several state-of-the-art regularization-based continual learning methods and benchmark performance on popular image recognition datasets.
Our results demonstrate that
CPR indeed promotes a wide local minima and signiﬁcantly improves both accuracy and plasticity while simultaneously mitigating the catastrophic forgetting of baseline continual learning methods.
1. Introduction
Catastrophic forgetting (McCloskey & Cohen, 1989) is a central challenge in continual learning (CL): when training a model on a new task, there may be a loss of performance (e g., decrease in accuracy) when applying the updated model to previous tasks.
At the heart of catastrophic forgetting is the stability-plasticity dilemma (Carpenter &
Grossberg, 1987; Mermillod et al., 2013), where a model exhibits high stability on previously trained tasks, but suffers from low plasticity for the integration of new knowledge (and vice-versa).
Attempts to overcome this challenge in neural network-based CL can be grouped into three main strategies: regularization methods (Li & Hoiem, 2017; Kirkpatrick et al., 2017; Zenke et al., 2017; Nguyen et al., 2018;
Ahn et al., 2019; Aljundi et al., 2018b), memory replay (Lopez-Paz & Ranzato, 2017; Shin et al., 2017; Rebufﬁ et al., 2017; Kemker & Kanan, 2017), and dynamic network architecture (Rusu et al., 2016; Yoon et al., 2018;
Golkar et al., 2019).
In particular, regularization methods that control model weights bear the longest history due to its simplicity and efﬁciency to control the trade-off for a ﬁxed model capacity.
In parallel, several recent methods seek to improve the generalization of neural network models trained on a single task by promoting wide local minima (Keskar et al., 2016;
Chaudhari et al., 2019; Pereyra et al., 2017; Zhang et al., 2018).
Broadly speaking, these efforts have experimentally shown that models trained with wide local minima-promoting regularizers achieve better generalization and higher accuracy (Keskar et al., 2016;
Pereyra et al., 2017; Chaudhari et al., 2019; Zhang et al., 2018), are better calibrated (Pereyra et al., 2017), and can be more robust to weight perturbations (Zhang et al., 2018) when compared to usual training methods.
Despite the promising results, methods that promote wide local minima have yet to be applied to CL.
In this paper, we make a novel connection between wide local minima in neural networks and regularization-based
CL methods.
The typical regularization-based CL aims to preserve important weight parameters used in past tasks by penalizing large deviations when learning new tasks.
As shown in the top of Fig.
1, a popular geometric intuition (as ﬁrst given in EWC (Kirkpatrick et al., 2017)) for such CL methods is to consider the (uncertainty) ellipsoid of parameters around the local minima.
When learning new tasks, parameter updates are selected in order to not signiﬁcantly hinder model performance on past tasks.
Our intuition is that promoting a wide local minima—which conceptually stands for local minima having a ﬂat , rounded uncertainty ellipsoid—can be particularly beneﬁcial for regularizationbased CL methods by facilitating diverse update directions for the new tasks ( i.e. , improves plasticity) while not hurting the past tasks ( i.e. , retains stability).
As shown in the bottom of Fig.
1, when the ellipsoid containing the parameters with 𝜃 !
∗ 𝜃 # ∗ 𝜃 !
∗ 𝜃 $ ∗ 𝜃 # ∗ 𝜃 !
∗
Regularization-based CL with wide local minima
Typical regularization-based CL 𝜃 !
{+∗ 𝜃 # ∗ 𝜽"𝜃 ! ∗ 𝜃 $ ∗ 𝜃 # ∗ 𝜃 ! ∗ 𝜽"
 Low error for Task+}
Low error for Task
Low error for Task Parameter update 𝜽"𝜽"𝜽" [Figure] low-error is wider, i.e. , when the wide local minima exists, there is more ﬂexibility in ﬁnding a parameter that performs well for all tasks after learning a sequence of new tasks.
We provide further details in Section 2.1.
Based on the above intuition, we propose a general, yet simple patch that can be applied to existing regularization-based
CL methods dubbed as Classiﬁer-Projection Regularization (CPR).
Our method implements an additional regularization term that promotes wide local minima by maximizing the entropy of the classiﬁer’s output distribution.
Furthermore, from a theory standpoint, we make an observation that our CPR term can be further interpreted in terms of information projection (I-projection) formulations (Cover & Thomas, 2012; Murphy, 2012; Csiszár & Matus, 2003;
Walsh & Regalia, 2010; Amari et al., 2001; Csiszár & Matus, 2003; Csiszár & Shields, 2004) found in information theory.
Namely, we argue that applying CPR corresponds to projecting a classiﬁer’s output onto a Kullback-Leibler (KL) divergence ball of ﬁnite radius centered around the uniform distribution.
By applying the Pythagorean theorem for KL divergence, we then prove that this projection may (in theory) improve the performance of continual learning methods.
Through extensive experiments on several benchmark datasets, we demonstrate that applying CPR can significantly improve the performance of the state-of-the-art regularization-based CL: using our simple patch improves both the stability and plasticity and, hence, achieves better average accuracy almost uniformly across the tested algorithms and datasets—conﬁrming our intuition of wide local minima in Fig.
Furthermore, we use a feature map visualization that compares methods trained with and without
CPR to further corroborate the effectiveness of our method.
Related work Several methods have been recently proposed to reduce catastrophic forgetting (see (Parisi et al., 2018) for a survey).
In this paper, we mainly focus on regularizationbased CL methods (Li & Hoiem, 2017;
Kirkpatrick et al., 2017; Aljundi et al., 2018a; Chaudhry et al., 2018; Zenke et al., 2017; Nguyen et al., 2018;
Ahn et al., 2019; Aljundi et al., 2018b).
Broadly speaking, the motivation behind regularization-based CL is to measure the importance of model parameters in previous tasks.
This measure is then used in a regularization term for overcoming catastrophic forgetting when training for new tasks.
Consequently, the main research focus of regularization-based CL is creating metrics for quantifying weight importance on previous tasks (e.g., (Kirkpatrick et al., 2017; Aljundi et al., 2018a;
Chaudhry et al., 2018; Zenke et al., 2017; Nguyen et al., 2018; Ahn et al., 2019)).
In contrast, here we focus on developing a general method for augmenting regularizationbased CL instead of proposing (yet another) new metric for weight importance.
The method introduced here, namely
CPR, can be applied to any regularization-based CL method to simultaneously improve both plasticity and stability.
The work closest to ours is (Aljundi et al., 2018b), which encourages sparsity of representations for each task by adding an additional regularizer to regularization-based CL methods.
Note that the motivation of (Aljundi et al., 2018b)— imposing sparsity of neuron activations—is considerably different from ours, which is to promote wide local minima.
Moreover, whereas (Aljundi et al., 2018b) focuses on average accuracy, we carefully evaluate in our experiments the advantage of the added CPR regularization in terms of increasing both plasticity and stability of CL in addition to accuracy.
Several papers have recently proposed methods that promote wide local minima in neural networks in order to improve single-task generalization, including using small mini-batch size (Keskar et al., 2016), regularizing the output of the softmax layer in neural networks (Szegedy et al., 2016; Pereyra et al., 2017), using a newly proposed optimizer which constructs a local-entropy-based objective function (Pereyra et al., 2017) and distilling knowledge from other models (Zhang et al., 2018).
We expand upon this prior work and investigate here the role of wide local minima in CL.
Our objective is to train neural networks that converge to wide local minima for each task, and subsequently benchmark the advantage of wide local minima in CL through numerous experiments.
To the best of our knowledge, this is the ﬁrst paper to study the role of wide local minima in CL.
2. CPR: Classiﬁer-Projection Regularization for Wide Local Minimum
In this section, we elaborate in detail the core motivation outlined in Fig.
1, then formalize CPR as the combination of two regularization terms: one stemming from prior regularization-based CL methods, and the other that promotes a wide local minima.
Moreover, we provide an information-geometric interpretation (Csiszár, 1984; Cover & Thomas, 2012; Murphy, 2012) for the observed gain in performance when applying CPR to CL.
We consider continual learning of T classiﬁcation tasks (with known task boundaries), where each task contains N training sample-label pairs { ( x tn , y tn ) }
Nn =1 , t ∈ [1 , · · · , T ] with x tn ∈ R d , and the labels of each task has M t classes, i.e. , y tn ∈ [1 , · · · , M t ] .
We denote f θ : R d → ∆ M as a neural network-based classiﬁcation model with softmax output layer parametrized by θ .
2.1. Motivation: Introducing wide local minima in continual learning
Consider the setting of typical regularization-based CL (top of Fig. 1).
We denote θ ∗ i as a local minima of task i .
From the shape of the low-error ellipsoids, after learning task 2 , an appropriate regularization strength can make the parameter update from θ ∗ 1 to ˆ θ 2 since it can achieve low-errors on both tasks 1 and 2.
However, while learning task 3 , the ellipsoids may not overlap enough, and it becomes infeasible to obtain a parameter that performs well on all three tasks.
In this case, regularization-based CL determines the trade-off between stability and plasticity in terms of its regularization strength; namely, the larger strength (direction 1 ) results in a parameter with more stability, ˆ θ 13 , so that less forgetting on tasks 1 and 2 is achieved, whereas the smaller strength (direction 2 ) leads to more plasticity so that the updated parameter ˆ θ 23 performs well on more recent tasks ( 2 and 3 ) at the cost of compromising the performance for task 1 .
In contrast, when the wide local minima exists for each task (bottom of Fig. 1), it is more likely that the ellipsoids will have non-empty intersections.
A regularization-based
CL may therefore more easily ﬁnd a parameter, ˆ θ 3 , that is simultaneously close to the the local minimas for each task, i.e. , { θ ∗ i } 3 i =1 .
This intuition suggests that once we promote the wide local minima of neural networks during continual learning, both the stability and plasticity will improve and result in higher accuracy—which is precisely what we verify in our experimental results for CPR (see Sec. 3).
2.2. Classiﬁer projection regularization for continual learning
Regularization-
continual learning Typical regularization-based CL methods attach a regularization term that penalizes the deviation of important parameters learned from past tasks in order to mitigate catastrophic forgetting.
The general loss form for these methods when learning task t is
[Equation]
where L t CE ( θ ) is the ordinary cross-entropy loss function for task t , λ is the dimensionless regularization strength, Ω t − 1 =
{ Ω t − 1 i } is the set of estimates of the weight importance, and { θ t − 1 i } is the parameter learned until task t − 1 .
A variety of previous work, e.g. , EWC (Kirkpatrick et al., 2017), SI (Zenke et al., 2017), MAS (Aljundi et al., 2018a), and RWalk (Chaudhry et al., 2018), proposed different ways of calculating Ω t − 1 to measure weight importance.
Single-task wide local minima Several recent schemes have been proposed (Pereyra et al., 2017; Szegedy et al., 2016; Zhang et al., 2018) to promote wide local minima of a neural network for solving a single task.
These approaches can be uniﬁed by the following common loss form
[Equation]
in which g is some probability distribution in ∆ M that regularizes the classiﬁer output f θ , β is a trade-off parameter, and D KL ( ·(cid:107)· ) is the KL divergence (Cover & Thomas, 2012).
Notice, for example, when g is the uniform distribution P U in ∆ M , the regularization term corresponds to maximizing the entropy as in (Pereyra et al., 2017), and when g is another classiﬁer’s output f θ (cid:48) , then (2) becomes equivalent to the loss function proposed in (Zhang et al., 2018).
CPR: Achieving wide local minima in continual learn- ing Combining the above two regularization terms, we propose the CPR as the following loss form for learning task t :
[Equation]
where λ and β are the regularization parameters.
The ﬁrst regularization term promotes the wide local minima while learning task t by using P U as the regularizing distribution g in (2), and the second term is from the typical regularization-based CL.
Note that this formulation is oblivious to Ω t − 1 and, hence, it can be applied to any state-of-theart regularization-based CL methods.
In our experiments, we show that the simple addition of the KL-term can signiﬁcantly boost the performance of several representative state-of-the-art methods, conﬁrming our intuition on wide local minima for CL given in Section 2.1 and Fig 1.
Furthermore, we show in the next section that the KL-term can be geometrically interpreted in terms of I-projections (Csiszár, 1984; Cover & Thomas, 2012; Murphy, 2012), providing an additional argument (besides promoting wide local minima) for the beneﬁt of using CPR in continual learning.
2.3. Interpretation by information projection
Given a distribution P and a convex set of distributions Q in the probability simplex ∆ m (cid:44) { p ∈ [ [Equation] 1] m | (cid:80)
[Equation]
= [Equation] p
[Equation]
= 1 } , information projection (I-projection) aims to ﬁnd P ∗ in Q such that the KL divergence between P ∗ and P is minimized, i.e. ,
[Equation]
The above quantity has several operational interpretations in information theory (e.g., in universal source coding (Cover & Thomas, 2012)).
The I-projection enables a “geometric” interpretation of KL divergence, where D KL ( Q (cid:107) P ) behaves as the squared Euclidean distance, ( Q, P ∗ , P ) form a “right triangle,” and the following lemma resembles the KL divergence equivalent of the Pythagorean theorem (not satisﬁed in general by the KL divergence)
(Cover & Thomas, 2012).
Lemma 1.
Suppose ∃ P ∗ ∈
Q such that D KL
[Equation] P [Equation]
= min Q ∈Q D KL ( Q (cid:107) P ) , then
[Equation]
A natural extension of the I-projection is to seek the conditional distribution Q Y | X in a set C that is closest (measured by the KL divergence) to a given conditional distribution
P Y | X .
Viewing a classiﬁer ( e.g. , a neural network with a softmax output layer) as a conditional probability distribution P Y | X , where Y is the class label and X is the input, we call this extension as the classiﬁer projection .
Formally, given a convex set C of conditional distributions, the classiﬁer projection is deﬁned as
[Equation]
We consider a simple CL setting with single head and ﬁxed number of classes.
Then, we pick the set of possible classiﬁers C to be a KL divergence ball centered at the uniform distribution P U , i.e. ,
[Equation]
We select P U since it is the centroid of ∆ M and, hence, the worst-case divergence between any distribution and P U is at most log M .
The following proposition is a direct consequence of Lemma 1.
Proposition 1.
For any classiﬁer P t − 1 ∗ Y | X ∈ C ( P U , (cid:15) ) for task t − 1 with data distribution P t − 1 X , and let any classiﬁer for task t be P tY | X / ∈ C
( P U , (cid:15) ) and P t ∗ Y |
X be the projected classiﬁer by (6) , then
[Equation]
Proposition 1 indicates that when evaluated on the previous task, the classiﬁer of the current task is more similar (in terms of cross-entropy) to each other after projection, thus guaranteeing a smaller change in training loss and accuracy.
From the vantage point of classiﬁer projection, the CPR regularization term in (3) can be viewed as the Lagrange dual of the constraint Q Y | X ∈ C
( P U , (cid:15) ) —the term that projects the classiﬁer of individual tasks towards the uniform distribution in order to minimize changes when training sequential tasks (See Fig. 2).
[Figure]
3. Experimental Results
We apply CPR to four regularization-based supervised CL methods: EWC (Kirkpatrick et al., 2017), SI (Zenke et al., 2017), MAS (Aljundi et al., 2018a) [Figure] and RWalk (Chaudhry et al., 2018), and further analyze CPR via ablation studies and feature map visualizations [Figure]
3.1. Data and evaluation metrics
We select CIFAR-100 (Krizhevsky et al., 2009), CIFAR10/100 (Krizhevsky et al., 2009), Omniglot (Lake et al., 2015), and CUB200 (Welinder et al., 2010) as benchmark datasets.
Note that we ignore the permuted-MNIST dataset (LeCun et al., 1998) since most state-of-the-art algorithms can already achieve near perfect accuracy on it.
CIFAR100 is divided into 10 tasks where each task has 10 classes.
CIFAR-10/100 additionally uses CIFAR-10 for pre-training before learning tasks from CIFAR-100.
Omniglot hastasks, where each task is a binary image classiﬁcation on a given alphabet.
For these datasets, we used a simple feedforward convolutional neural network (CNN) architecture.
For the more challenging CUB200 dataset, which hastasks with 20 classes for each task, we used a pre-trained
ResNet-18 (He et al., 2016) as the initial model.
Training details, model architectures, hyperparameters tuning, and source codes are available in the Supplementary Material (SM).
For evaluation, we ﬁrst let a k,j ∈ [0 , 1] be the j -th task accuracy after training the k -th task ( j ≤ k ) .
Then, we used the following three metrics to measure the continual learning performance:
• Average Accuracy (A) is the average accuracy A k on the ﬁrst k tasks after training the k -th task, i.e. , A k =k (cid:80) kj =1 a k,j .
While being a natural metric, Average Accuracy fails to explicitly measure the plasticity and stability of a CL method.
• Forgetting Measure (F) evaluates stability.
Namely, we deﬁne the forgetting measure f jk of the j -th task after training k -th task as f jk = max l ∈
j,..., k − 1 } a l,
j − a k,j , ∀ j < k , and the average forgetting measure F k of a CL method as F k = 1 k − 1 (cid:80) k − 1 j =1 f jk
• Intransigence Measure (I) measures the plasticity.
Let a (cid:63)j be accuracy of a model trained by ﬁne-tuning for the j -the task without applying any regularization.
The intransigence measure I s,k is then deﬁned as
I s,k =
[Equation]
, where i [Equation] =
a [Equation] The F and I metrics were originally proposed in (Chaudhry et al., 2018), and we slightly modiﬁed their deﬁnitions for our usage.
Note that a low F k and I 1 ,k implies high stability (low forgetting) and high plasticity (good forward transfer) of a CL method, respectively.
3.2. Quantifying the role of wide local minima regularization
We ﬁrst demonstrate the effect of applying CPR with varying trade-off parameter β in (3) by taking EWC (Kirkpatrick et al., 2017) trained on CIFAR-100 as a running example.
Fig. 3(a) shows how the aforementioned metrics varies as β changes over [0 . 1 , . .
First, we observe that Acertainly increases as β increases.
Moreover, we can break down the gain in terms of I 1 , 10 and F 10 ; we observe I 1 ,monotonically decreases as β increases, but F 10 does not show the similar monotonicity although it also certainly decreases with β .
This suggests that enlarged wide local minima is indeed helpful for improving both plasticity and stability.
In the subsequent experiments, we selected β using validation sets by considering all three metrics; among the β ’s that achieve sufﬁciently high A 10 , we chose one that can reduce F 10 more than reducing I 1 , 10 , since it turns out improving the stability seems more challenging.
(In fact, in some experiments, when we simply consider A 10 , the chosen β will result in the lowest I 1 , 10 but with even higher
F 10 than the case without CPR.)
For comparison purposes,
[Table]
[Figure]
we also provide experiments using Deep Mutual Learning (Zhang et al., 2018) and Label Smoothing (Szegedy et al., 2016) regularizer for achieving the wide local minima in the
SM; their performance was slightly worse than CPR.
With the best β in hand, Fig. 3(b) experimentally veriﬁes whether using CPR indeed makes the local minima wide.
Following the methodology in (Zhang et al., 2018), we perturb the network parameters, after learning the ﬁnal task, of EWC and EWC+CPR by adding Gaussian noise with increasing σ , then measure the increase in test loss for each task.
From the ﬁgure, we clearly observe that EWC+CPR has a smoother increase in test loss compared with EWC (without CPR) in each task.
This result empirically conﬁrms that CPR indeed promotes wide local minima for each task in CL settings and validates our initial intuition given in Sec.
In the SM, we repeat the same experiment with MAS (Aljundi et al., 2018a).
3.3. Comparison with state-of-the-art
Next, we apply CPR to the state-of-the-art regularizationbased CL on the benchmark datasets and measure the performance improvement with the three metrics in Section 3.1.
For the regularization strengths, we ﬁrst select the best λ without CPR, then choose β according to the procedure in
Section 3.2.
The results in Table 1 are averaged over 10 repeated experiments with different random initialization and task sequence using the chosen ( λ, β ) .
The hyperparameters are reported in the SM.
CIFAR-100 and CIFAR-10/
In Table 1 and
Fig. 4(a), we observe that CPR consistently improves all regularization-based methods for all tested datasets in terms of increasing A 10 and decreasing I 1 , 10 , and also consistently decreases F 10 except for MAS in CIFAR-100.
Additionally, we ﬁnd that for CIFAR-10/100, the orders of the 10 tasks in CIFAR-100 and CIFAR-10 affect the performance of the
CPR; namely, in the SM, we show that when CIFARtasks are positioned in different positions rather than at the beginning, the gain due to CPR got much bigger.
Omniglot This dataset is well-suited to evaluate CL with long task sequences (50 tasks).
In Table 1, it is clear that the CPR considerably increases both plasticity and stability in long task sequences.
In particular, CPR signiﬁcantly decreases F 10 for EWC and leads to a huge improvement in A 10 .
Interestingly, unlike the previous datasets, I 1 , 10 is
[Figure]
negative , implying that past tasks help in learning new tasks for the Omniglot dataset; when applying CPR, the gains in
I 1 , 10 are even better.
Furthermore, Fig. 4(b) indicates that applying CPR leads to less variation in A t curves.
CUB200 The results in Table 1 and Fig. 4(c) show that CPR is also effective when using a pre-trained ResNet model for all methods and metrics, except for EWC.
Here, CPR significantly increases A 10 and reduces I 1 , 10 when compared to
EWC, whereas F 10 is slightly increased for EWC + CPR.
3.4. Ablation study
We study the ablation of the CPR on the regularizationbased methods using CIFAR-100 with the best ( λ , β ) found previously, and report the averaged results over 5 random initializations and task sequences in Fig.
The ablation is performed in two cases: (i) using CPR only at task t , denoted as EWC + CPR (only t -th task), and (ii) using
CPR except task t , denoted as EWC + CPR (w/o t -th task).
Fig. 5(a) shows f t 10 , the amount of forgetting for task t after learning the task 10, and Fig.
5(b) shows I t +1 , 10 , the amount of gap with ﬁne-tuning after task t .
In Fig. 5(a), we observe that CPR helps to decrease f t 10 for each task whenever it is used (except for task 3 ), but f t 10 of EWC +
CPR (w/o t -th task) shows a more random tendency.
On average, EWC + CPR does reduce forgetting in all tasks, demonstrating the effectiveness of applying CPR to all tasks.
Notably in Fig.
5(b), I t +1 , 10 of EWC + CPR (only t -th task) is lower than that of EWC + CPR (w/o t -th task) only when t = 1 ; this indicates that CPR is most beneﬁcial in terms of plasticity when CPR is applied as early as possible to the learning sequence.
EWC + CPR again achieves the lowest (i.e., most favorable)
I t +1 , 10 .
Fig. 5(c), as a further evidence, also suggests that applying CPR for t = 1 gives a better accuracy.
Moreover, the accuracy of EWC + CPR (w/o t -th task) gets closer to the optimal EWC + CPR, which is consistent with the decreasing difference of I
t +1 , 10 between
EWC + CPR (w/o t -th task) and EWC + CPR in Fig. 5(b).
The EWC + CPR still gives the best A 10 and individual a t,accuracy.
We emphasize that model converging to a wide local minima from the ﬁrst task onwards considerably helps the training of future tasks as well, i.e. , a signiﬁcant increase in the plasticity can be achieved.
By using this ﬁnding, we conducted an experiment on the case where CPR have to learn unscheduled additional tasks and got the impressive experimental result which is reported in SM.
3.5. Feature map visualization using UMAP
We present next two-dimensional UMAP (McInnes et al., 2018) embeddings to visualize the impact of CPR on learnt representations.
We compare representations produced by models trained on CIFAR-100 in two cases: (i) an oracle model which learns from the ﬁrst and the t -th task at training time t , and (ii) sequential CL using EWC and EWC + CPR.
We sample 30% of the test data for producing the visualization.
Details and parameters for UMAP are provided in the
SM.
We ﬁrst visualize O t, 1 , deﬁned as the output feature map of the ﬁrst output layer given the ﬁrst task’s test data after training the t -th task.
The ﬁrst row of Fig.
6 displays the respective embeddings, where c t corresponds to the center point of the cluster for the t -th task.
In the ideal case (in terms of stability), there would be little to no change in
O t, 1 during CL.
This is evident in the embeddings for the joint model, which show that each cluster O t, 1 is almost perfectly centered.
In contrast, the resulting embedding from
EWC has a slightly scattered c t when compared to the joint (oracle) model.
This indicates that, whenever the model is trained on a new task, feature maps of the output layer may drift despite EWC’s regularization for previous task parameters.
EWC + CPR, in turn, display more centered c t than EWC, indicating that by applying CPR to EWC model parameters become more robust to change after training future tasks.
In order to provide further evidence that CPR provides better plasticity on new tasks, we visualized h t , deﬁned as the embedding for the feature map of the last hidden lay-
Joint
EWC
EWC + CPR
V i s
ua li
z a t i on o f O ! , #
V i s
ua li
z a t i on o f h !
ers given t -th test data after training the t -th task.
In the second row of Fig. 6, Joint and EWC + CPR show closer feature embeddings.
EWC, in turn, has a ﬁrst and second task feature maps divided from other tasks.
Strikingly, the feature embeddings for the ﬁrst task are completely separated.
Therefore, we believe that CPR helps the model share feature representations from the start of training, potentially explaining the improvement of the intransigence measure observed in Sec 3.4.
e are unaware of prior work that makes use of feature embedding to identify reasons for catastrophic forgetting and limited plasticity of CL methods, and hope that such feature map visualizations become a useful tool for the ﬁeld.
Additional visualizations on different random initializations, different task sequences and MAS (Aljundi et al., 2018a) are reported in the SM.
4. Conclusion
We proposed a simple classiﬁer-projection regularization (CPR) which can be combined with any regularizationbased continual learning (CL) method.
Through extensive experiments, we demonstrated that, by converging to a wide local minima at each task, CPR can signiﬁcantly increase the plasticity and stability of CL.
These encouraging results indicate that wide local minima-promoting regularizers have a critical role in successful CL.
Moreover, we observed the impact of CPR through feature map visualizations—a practice that we hope will become more common in future analysis of CL methods.
As a theoretical interpretation, we argue that the additional term found in CPR can be understood as a projection of the conditional probability given by a classiﬁer’s output onto a ball centered around the uniform distribution.
1. Mathematical Proofs
{+Lemma 
+}
[Cover & Thomas, 2012, Theorem 11.6.1]
If D KL ( Q (cid:107) P ) is unbounded, then the inequality holds.
Assume that D KL ( Q (cid:107) P ) is bounded, then it implies D KL ( Q ∗ (cid:107) P ) =
min Q ∈Q D KL ( Q (cid:107) P ) is also bounded.
Since Q is a convex set, we consider a convex combination Q θ of Q ∗ and Q , i.e.,
Q θ =
(1 − θ ) Q ∗ + θQ ∈ Q , where θ ∈ [0 , 1] .
Since Q ∗ is the minimizer of D KL ( Q (cid:107) P ) , we have
[Equation]
(S.1)
[Equation]
[Equation]
where the facts that the exchange of derivatives and integrals is guaranteed by the dominated convergence theorem and that the integrals (cid:82) Q ∗ = (cid:82) Q = 1 .
Therefore
[Equation]
we have D KL ( Q (cid:107) P ) ≥ D ( Q (cid:107) Q ∗ )
+ D ( Q ∗ (cid:107) P ) , the desired result.
1.2. Proposition
Note that C
( P U , (cid:15) ) is a convex set by deﬁnition since the KL divergence is convex, and hence Lemma 1 applies.
By Lemmaand the information inequality (i
[Equation]
e., the KL divergence is always non-negative),
[Equation]
Therefore
[Equation]
we have
[Equation]
where the inequality comes from (S.10).
2. Experimental details of Section 3.
For training models on CIFAR100, CIFAR10/100 and Omniglot, we used the Adam (Kingma & Ba, 2015) optimizer with initial learning rate 0.001 for 100 epochs.
For training CUB200, we set the initial learning rate as 0.0005 and trained the model for 50 epochs.
Here we also used the learning rate scheduler which drops the learning rate by half when validation error is not decreased.
All experiments was implemented in PyTorch 1.2.0 with CUDA 9.2 on NVIDIA 1080Ti
GPU.
Following (Ahn et al., 2019)
[Table]
we use a simple CNN model for training CL benchmark dataset except for CUB200 and details of an architecture is in Table 1 and
[Table] [Table]
[Table]
3. Additional Experimental Results of Section 3. 3.1. Experimental Results of Wide Local Minima using Training Data
[Figure]
Figure 1 shows the experimental result of Section 3.2 using training data.
We clearly see that training loss of EWC + CPR slowly increases than EWC in all tasks.
3.2. Experimental Results on MAS (Aljundi
et al., 2018)
and Deep Mutual Learning (
Zhang et al., 2018)
We did the same experiments of Section 3.2 using MAS (Aljundi et al., 2018), and Figure 2 shows the results.
In Figure 2(a), we observe that MAS shows a clear trade-off between F 10 and I 1 , 10 as β increases, unlike the result of EWC in the manuscript. (We note SI (Zenke et al., 2017) and RWalk (Chaudhry et al., 2018) showed similar trend as EWC (Kirkpatrick et al., 2017) in the manuscript).
MAS + CPR achieves the highest accuracy in the range of 0 .
5 ≤ β ≤ 0 .
but we can see that β
= { 0 . 7 , 0 . 9 } shows a worse F 10 compared with MAS.
Therefore, we can select β = 0 .
5 as the best hyperparameter using the criteria for selecting β proposed in Section 3.2 of the manuscript.
We also experimented Deep Mutual Learning (DML) (Zhang et al., 2018) as the regularization for converging wide local minima.
We used β = 1 only because DML reports the best result (with β = 1 ) which is converging to a better wide local minima compared to Entropy Maximization (Pereyra et al., 2017).
In our experiment, DML shows an increased A 10 and decreased F 10 , I 1 , 10 but it is not as effective as our CPR.
Most decisively, DML requires training at least more than two models so we excluded DML from our consideration.
Figure 2(b) shows the experimental result on adding Gaussian noise to the parameters which is trained on CIFAR-100.
We clearly observe that test loss of each task more slowly increases by applying CPR to MAS.
We believe this is another evidence that CPR can be generally applied to regularization-based CL methods, promoting the wide-local minima. [Figure]
4. Selected Best Hyperparameters
[Table]
RWalk 8 / 0.
For each dataset [Table] we ﬁrstly searched best λ for each regularization-based CL method and then we selected best β for CPR.
All best hyperparameters are proposed in Figure 3.
5. Experimental Results on CIFAR100/10, CIFAR50/10/
As an additional experiments of Section 3.3 in the manuscript
[Table]
we experimented on CIFAR100/10 and CIFAR50/10/50, which are the different versions of CIFAR10/100.
Namely, we changed the order of the tasks and varied the location for which CIFAR-10 task is inserted.
Table 4 and Figure 5 show the results.
We can achieve better relative improvements on all metrics compared to CIFAR-10/
[Table]
[-[Figure] +0.0376 (+6.5%) 0.0304 0.0238 -0.
 [Figure] +0.0216 (+3.5%) 0.0681 0.-][Table]
[Figure]
6. Hyperapameter Settings and Visualization Details of UMAP
From several visualizations, we found out that best hyperparameters for UMAP(McInnes et al., 2018) as {n_neighbors = 200, min_dist = 0.1, n_components = 2} and we got all visualization results with these hyperparameters.
We used raw features of O t, 1 as a input of UMAP, however, for visualizing h t , we reduced the dimension of h t to 50 by using PCA.
7. Additional Feature Map Visualizations
Joint
[Equation]
V i s
ua li
z a t i on o f
O ! , #
V i s
ua li
z a t i on o f h !
We visualize O t, 1 and h t of Joint, EWC (Kirkpatrick et al., 2017), EWC (Kirkpatrick et al., 2017)
+ CPR with a different seed and visualizations are shown in Figure 7.
We hold the experimental settings and we can see the similar pattern of O t,and h t , which is already shown in Section 3.5 of the manuscript.
Especially, O t, 1 of EWC showed clearly divided clusters compared with the visualization result in the manuscript, nevertheless, we conﬁrm that the feature maps become to be more shared and centered by applying CPR to EWC.
We also did same visualization using MAS (Aljundi et al., 2018) and the results are shown in Figure 7.
We checked the similar results of O t, 1 and h t , and we could see that, by applying CPR to MAS, O t, 1 and h t are more centered than before.
From these additional visualizations, we want to emphasize that the pattern of O t, 1 and h t is a general phenomenon of regularization-based CL methods, and these can show why the typical regularization-based CL methods still suffer from the stability-plasticity dilemma at the feature map level.
Also, we could check again that CPR increases the stability and plasticity of the regularizaion-based CL methods by alleviating this phenomenon.
Joint
MAS
MAS + CPR
V i s
ua li
z a t i on o f
O ! , #
V i s
ua li
z a t i on o f h !
[Figure]
Joint
MAS
MAS + CPR
V i s
ua li
z a t i on o f
O ! , #
V i s
ua li
z a t i on o f h !
[Figure]
[Table]
8. Experiments on additional tasks
From Section 3.4 in the manuscript, we demonstrated the critical role of CPR in terms of increasing the plasticity.
From this result, we thought that CPR might helps to learn additional future tasks well without the hyperparameter search for new whole tasks.
To verify our hypothesis, we designed a new task sequence made up of 20 tasks, CIFAR100(10 tasks)
+ SVHN (Netzer et al., 2011)(5 tasks)
+ Synthetic MNIST
(Roy et al., 2018)(5 tasks) and each task of SVHN and Synthetic MNIST is a binary image classiﬁcation.
Table 5 shows experimental results of EWC (Kirkpatrick et al., 2017) on additional tasks.
We divide the experimental setting as two different cases.
The ﬁrst case is that we newly search the best hyperparameter for all 20 tasks (denoted as all tasks), and in the second case, we just use the best hyperparameter got from CIFAR-100 (denoted as CIFAR-100).
EWC + CPR (CIFAR-100) shows a low I 10 , 20 compared with EWC (all tasks), as a result, EWC + CPR (CIFAR-100) achieve the higher A 20 than EWC (all tasks).
Also, we observe that, if we ﬁnd the best hyperparameters ( λ , β ) for all 20 tasks again, EWC + CPR (all tasks) still achieves the best result in all metrics.
In conclusion, we believe that this is a remarkable result, and it shows the effect of wide local minimum in CL continues in additional tasks.