S LICED
C RAM ´ ER
S YNAPTIC
C ONSOLIDATION
FOR
P RESERVING D EEPLY L EARNED R EPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep neural networks suffer from the inability to preserve the learned data representation (i.e., catastrophic forgetting) in domains where the input data distribution is non-stationary, and it changes during training.
Various selective synaptic plasticity approaches have been recently proposed to preserve network parameters, which are crucial for previously learned tasks while learning new tasks.
We explore such selective synaptic plasticity approaches through a unifying lens of memory replay and show the close relationship between methods like Elastic
Weight Consolidation (EWC) and Memory-Aware-Synapses (MAS).
We then propose a fundamentally different class of preservation methods that aim at preserving the distribution of the network’s output at an arbitrary layer for previous tasks while learning a new one.
We propose the sliced Cram ´ er distance as a suitable choice for such preservation and evaluate our Sliced Cram ´ er Preservation (SCP) algorithm through extensive empirical investigations on various network architectures in both supervised and unsupervised learning settings.
We show that SCP consistently utilizes the learning capacity of the network better than online-EWC and MAS methods on various incremental learning tasks.
I NTRODUCTION
Incremental learning without catastrophic forgetting is one of the core characteristics of a lifelong learning machine (L2M) and has recently gained renewed attention from the machine learning community.
In real-world applications, the input distribution of the data (e.g., sensory inputs) is prone to constant changes due to environmental variations (e.g., seasonal changes), exposure to new situations (e.g., change in the surface friction), sensory malfunction (e.g., water droplets on a camera), among others.
It is therefore desirable to continue to train the base computational model only on the new data/task and incrementally accumulate knowledge to improve the performance of the system over time, as opposed to retraining the model on the composition of old and new data.
The existing computational models, for instance deep convolutional neural networks (CNNs), face two fundamental issues regarding incremental learning, 1) catastrophic forgetting (McCloskey &
Cohen, 1989), which refers to the forgetting of previously acquired knowledge when learning new tasks as a result of interference between the old and new tasks, and 2) intransigence , which refers to the inability to acquire new knowledge while trying to preserve old knowledge (e.g., reducing the learning rate) Chaudhry et al.
Note that we use the term ‘knowledge’ here to indicate the input/output behavior of the computational model as in (Hinton et al., 2015).
A successful incremental learner should be able to overcome both forgetting and intransigence.
The commonly used strategies to overcome catastrophic forgetting include:
selective synaptic plasticity to preserve learned knowledge (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi et al., 2018; Chaudhry et al., 2018), which is rooted in the idea of homeostatic plasticity in neuroscience, 2.
additional neural resource allocation to learn new knowledge and preserve old knowledge, (Rusu et al., 2016; Lee et al., 2017; Li & Hoiem, 2017; Rannen et al., 2017; Schwarz et al., 2018; Li et al., 2019), which is similar to neurogenesis in the hippocampus, 3. memory and experience replay (Rebufﬁ et al., 2017; Shin et al., 2017; Wu et al., 2018;
Hu et al., 2019; Rostami et al., 2019), which is based on the well-established theory of complementary learning system (CLS) (McClelland et al., 1995)
Each framework has its advantages and disadvantages, and Parisi et al.
(2019) provide an excellent survey of these methods.
We note that the term “synaptic weights” refers to the strength of a connection between two nodes.
The term ‘plasticity’ is used analogously to ‘neural plasticity’ in the human brain, which refers to the ability of the neurons to change their synaptic weights.
The term “selective plasticity” refers to the desired capability of a network to selectively increase or decrease the plasticity of individual synapses throughout the neural architecture.
Our focus in this paper is on selective synaptic plasticity.
The standard deep neural network architectures are uniformly plastic; hence, all neurons are prone to changes during training, and this powerful capability is also the demise of these networks and leads to catastrophic forgetting.
The idea of selective synaptic plasticity is to partially preserve synapses that are critical for previously learned tasks by rigidifying those synapses (i.e., to enforce critical synapses to change less).
Rigidifying the network over time leads to a loss of learning capability for future tasks, which is known as ‘intransigence’ in the literature.
Selective synaptic plasticity, by itself, could not fully overcome intransigence.
A combination of strategies like efﬁcient memory replay for reconsolidation, neurogenesis, and selective synaptic plasticity could lead to superior methods that defeat both catastrophic forgetting and intransigence.
Chaudhry et al.
(2018), for instance, provide such a combination of memory replay and selective synaptic plasticity.
Also, there have been various efforts toward making the idea of neural resource allocation scalable, the progress and compress work Schwarz et al.
(2018), and the incremental moment matching Lee et al.
(2017) work fall under this category.
In this paper, we focus on selective synaptic plasticity to preserve learned representations in a deep neural network.
Inspired by (Chaudhry et al., 2018), we take a geometric view and devise a new method for selective synaptic plasticity.
The proposed method is fundamentally different from the previous approaches like Elastic Weight Consolidation (EWC)
(Kirkpatrick et al., 2017) and Memory Aware Synapses (MAS)
(Aljundi et al., 2018), which we indicate as sample-based approaches.
Instead, we focus on identifying synaptic importance parameters that preserve the ‘ distribution ’ of the latent representation of a task.
Focusing on preserving the distribution of the latent representation of a neural network at an arbitrary layer, as opposed to the expected change in network’s response for individual samples, enforces a less restrict regularization on the network, and enables a better utilization of the network learning capacity.
The primary concept of sample-based versus distribution-based regularizations are visualized in Figure 1.
Similar to the MAS framework, our proposed method denoted as Sliced Cram ´ er Preservation (SCP) is also able to preserve a task representation in any layer of a neural network, hence, enabling its application to various unsupervised or self-supervised learning settings.
Our speciﬁc contributions in this paper are:
• Introducing a distribution-based regularization using the sliced Cram ´ er distance (aka, Cram´er Wold distance (Tabor et al., 2018)) for selective synaptic plasticity that preserves the distributions of the representations of previously seen tasks at an arbitrary layer of a deep neural network while learning a new task.
• Providing a geometric interpretation for the MAS algorithm (Aljundi et al., 2018) that further gives insight into the otherwise heuristic choices made in the approach.
• Comparing the proposed method to online-EWC and MAS on the benchmark permuted MNIST dataset, sequential unsupervised learning with auto-encoders, and the more interesting problem of semantic segmentation of driving scenes, and demonstrating signiﬁcant improvements in overcoming catastrophic forgetting and intransigence over these methods.
P ROBLEM S ET - UP AND P RELIMINARIES
Consider data from a stream of tasks X t = { x ti ∼ p tX } n t i =1 , where p tX is the probability density function (PDF) for task t deﬁned on X ⊂ R d .
We consider both supervised and unsupervised tasks, where in the supervised case the input sample, x ti , is accompanied with the corresponding label y ti ∈ R K .
Let φ ( · ; θ ) : R d → R K denote a parametric function (e.g., a neural network) that is to be optimized to solve the stream of tasks.
In the supervised learning setting, we consider φ to be the mapping to the logits prior to applying the softmax layer.
[Figure]
E x ⇠ p AX ⇥
d (  ( x ; ✓ ) , ( x ; ✓ ⇤ A ))
⇤ d p  p AZ ( ·| ✓ ) ,p AZ ( ·| ✓ ⇤ A )
(cid:0) p AZ
| θ ) ,
AZ ( ·| θ ∗ A ) (cid:1)
2. A G EOMETRIC V IEW OF E LASTIC W EIGHT C ONSOLIDATION
In their seminal work
Kirkpatrick et al.
(2017) considered the problem of overcoming catastrophic forgetting in supervised and also reinforcement learning scenarios where a supervisory signal y exists, whether in the form of labels/annotations or environmental rewards, respectively.
Here we reiterate the geometric interpretation of the EWC framework following the work of (Chaudhry et al., 2018) which we will then adapt to deﬁne our generic consolidation framework.
Let p θ ( y | x ) =
softmax( φ ( x ; θ )) , where [ p θ ( y | x )]
j is the softmax probability of the j-th class.
For simplicity, let us consider the case where we want to learn only two tasks consecutively, i.e., tasks ‘A’ and ‘B.’ Then, EWC ensures that while learning task ‘B,’ the conditional likelihood p θ ( y | x A ) does not drift far from the optimal conditional likelihood p θ ∗ A ( y | x A ) , where θ ∗ A are the parameters initially optimized for task A:
[Equation]
where λ is the regularization coefﬁcient.
Note that equation 1 is essentially an optimization with a memory replay regularizer.
Here, while learning task B, samples from task A (i.e., a memory buffer from this task) are fed through the network and the conditional likelihood is constantly checked against the optimal conditional likelihood for task A, p θ ∗ A ( y | x A ) , to ensure a minimal deviation from those parameters.
The key question answered by the EWC framework is on how to avoid memory replay and yet achieve a similar result, i.e., not forget the knowledge from old task (task A).
The answer lies in the second-order Taylor expansion of the regularizer around the parameters optimized for the old task, θ ∗ A .
It is straightforward to show (Chaudhry et al., 2018) that the second-order Taylor expansion of the regularizer is of the form:
[Equation]
where F θ is the Fisher Information Matrix (FIM) and is deﬁned as:
[Equation]
see supplementary material for complete derivations.
Therefore, when δ θ → 0 the KL-divergence regularizer enforces closeness of θ to θ ∗ A in a Riemmanian pseudo-manifold induced by the FIM.
Given that the number of parameters could easily reach several million in standard deep neural networks, it is practically infeasible to store and use the FIM matrix, F θ .
Therefore, Kirkpatrick et al.
(2017) assume that F θ is diagonal and further approximate the KL-divergence with:
[Equation]
where M is the total number of parameters in the neural network.
This leads to the main equation in the EWC framework:
[Equation]
From our point of view, the critical aspect of these derivations is the connection between memory replay and structural plasticity.
Note that we started with equation 1, which uses the idea of a memory replay regularizer.
Then by assuming δ θ → 0 , using the second-order Taylor expansion of the KL-divergence around θ ∗ A , and assuming that FIM is a diagonal matrix we arrived at equation 5, which provides the idea of synaptic importance parameters and is an embodiment of the selective synaptic plasticity framework.
Next, we use this critical aspect and develop an analogous regularizer (i.e., based on memory replay) for the MAS algorithm.
2. G ENERALIZING TO U NSUPERVISED L EARNING
The EWC framework as explained in the previous section preserves the softmax probability of samples from Task A while learning
Task B.
This limits the applicability of the method to networks with outputs living on a K-dimensional simplex, e.g., supervised learning and reinforcement learning where the network outputs a probability over a ﬁnite set of actions.
More recently, Aljundi et al.
(2018) presented their Memory-Aware-Synapses (MAS) framework, which lifts the requirement for
EWC outputs to live on a simplex, and enables calculation of the synaptic importance parameters even in unsupervised learning and also during testing.
While the method is exciting and practically very useful, there is no geometric motivation behind the algorithm.
Here we reverse engineer the importance term used in MAS and show a simple regularizer that leads to the MAS algorithm and more importantly provides a geometric interpretation for the algorithm.
Let the regularizer for Task B, be the expected absolute difference between squared (cid:96) 2 norms of the output of the network for samples from Task A, i.e.:
[Equation]
It is straightforward (see Supplementary material) to show that using the second-order Taylor expansion of the above regularizer leads to:
[Equation]
where [Ω] m,n = E x ∼ p AX
[Equation]
, which is the importance parameter used by
Aljundi et al.
From a geometric perspective, MAS preserves the norms of the samples from
Task A while learning
Task B.
In other words; MAS enforces closeness of θ to θ ∗ A in a Riemmanian pseudo-manifold induced by matrix Ω .
The general idea of using the expected value of a “suitable” distance/divergence between samples,
E x ∼ p
[Equation]
x ; θ ) , φ
[Equation]
x ; θ
[Equation]
)) and leveraging its second-order Taylor expansion to obtain synaptic importance values is crucial here and could lead to various undiscovered algorithms based on new distances/divergences.
We refer to these approaches as sample-based regularization methods.
2. P RESERVING D ISTRIBUTION OF AN A RBITRARY L AYER
We approach the problem of overcoming catastrophic forgetting from the angle of preserving
Task A’s distribution at an arbitrary layer of the neural network, when learning Task B. Let z Ai = φ ( x Ai
; θ ) ∈ R K be the output of the network, for a sample from Task A, at the target layer (e.g., output logits in a NN classiﬁer, or reconstructed image of an autoencoder).
The distribution of the random variable z A in the target layer follows from the Random Variable Transform (RVT) theorem (Gillespie, 1983): [Equation]
[Equation]
Then we propose the following general regularization to overcome forgetting when learning task B:
[Equation]
[Equation] where d ( · , · ) is a discrepancy measure between the two probability distributions deﬁned in R K .
Note that equation 9 could be performed with any discrepancy measure or distance, e.g., the Wasserstein distance (Villani, 2008; Kolouri et al., 2017), using a memory replay strategy.
In what follows, we describe a ‘suitable’ distance d ( · , · ) that: 1) respects the underlying geometry of the space, and 2) enables a similar strategy to that of the EWC framework to provide importance parameters.
S LICED -C RAM ´ ER D ISTANCE FOR S TRUCTURAL P LASTICITY 3. C RAM ´ ER D ISTANCE
The p-Cram´er distance (Cram´er, 1928; Sz´ekely & Rizzo, 2013) between two one-dimensional probability density functions p 0 and p 1 is deﬁned as the (cid:96) p -norm between their cumulative distribution functions.
Note that here we avoid any measure theoretic notations for simplicity.
Let q i ( t ) =
(cid:82) t
−∞ p i ( τ ) dτ denote the cumulative distribution function for p i , then the p-Cram ´ er distance is deﬁned as:
[Equation]
for p ≥ 1 .
Similar to the Wasserstein distance and unlike the KL-divergence and its symmetric form Jensen-Shannon distance (i.e., the square root of the Jensen-Shannon divergence), the Cram´er distance respects the underlying geometry of the space.
Moreover, the Cram´er distance provides unbiased sample gradients (Bellemare et al., 2017), and for p = 1 , C 1 is equivalent to the 1Wasserstein distance, W 1 .
In addition, and similar to the Wasserstein metric, the dual of the Cram´er distance is of the form of an integral probability metric (IPM) (Dedecker & Merlev`ede, 2007).
To further demonstrate the favorable characteristics of this distance, consider the following parametric distribution matching in one-dimension, where the target distribution, p , is a box distribution deﬁned on R and p τ ( t )
= p ( t − τ ) is the shifted version of p and the goal is to optimize τ to minimize the distance between p τ and p ( τ ∗ = 0 ).
For this simple setting, we calculate the energy landscape (i.e., the distance between p τ and p ) as a function of τ for the Jensen-Shannon distance, W p , and
C p for p = 1 , 2 .
Figure 2 shows the distributions p and p τ on the left and the energy landscape as a function of τ on the right.
It can be clearly seen that the Wasserstein and Cram´er distances respect the underlying geometry of the problem, while the Jensen-Shannon (JS) distance fails to do so.
To extend the Cram´er distance to higher-dimensional distributions, we utilize the idea of distribution slicing used in various recent publications (Kolouri et al., 2016; 2019; 2018).
We note that the slicedCram´er distance (also known as the Cram´er-Wold distance) was recently used in Tabor et al.
(2018) for generative modeling.
We brieﬂy describe the idea in the following section.
3. S LICED -C RAM ´ ER D ISTANCE
The idea of slicing a higher-dimensional distribution has roots in the Radon transform that is commonly used in computational tomography.
The idea is to represent a high-dimensional distribution via the inﬁnite set of its marginal distributions.
In short, let p 0 and p 1 be d-dimensional probability density functions deﬁned on X ⊂ R d , then their Radon transform is deﬁned as:
[Equation]
for ∀ t ∈ R and ∀ ξ ∈ S d − 1 where S d − 1 denotes the d-dimensional unit sphere.
Note that R p i ( · , ξ ) is a so called slice of p i , which is a one-dimensional marginal distribution of p i .
Let R q i ( · , ξ ) be the corresponding cumulative distribution function of R p i ( · , [Equation]
[Equation]
then the sliced-Cram ´ er distance between p 0 and p 1 is the expected value of the Cram ´ er distance between their one-dimensional slices, i.e., R p i ( · , ξ )
when ξ ∼ U S
d − 1 for U S d − 1 being the uniform distribution on the d-dimensional unit sphere.
In other words, the sliced Cram ´ er distance is deﬁned as:
[Equation]
Now we are ready to propose our method for overcoming representation forgetting.
3. O VERCOMING REPRESENTATION FORGETTING
The critical point here is that the sample-based regularizers could over-estimate the importance of synapses, leading to intransigence faster.
More importantly, a substantial expected change in the network’s output (over individual samples) would not necessarily mean catastrophic forgetting.
Changes in the network’s output within a mode (e.g., in supervised classiﬁcation within the distribution of a particular class) are harmless, so long as the representation of the data is not signiﬁcantly changing.
Our proposed regularizer tolerates such changes and therefore has the potential for better utilization of the network’s learning capacity.
We propose the following regularization for overcoming catastrophic forgetting:
[Equation]
[Equation] equation 14 requires memory replay from the old task(s) while learning the new one.
To transition from memory replay to selective synaptic plasticity, we derive the second-order Taylor expansion of the regularizer SC 22 around the optimal parameters for the previous tasks, θ ∗ A .
Assuming that θ = θ ∗ A + δ θ where δ θ → 0 , it is straightforward to show that:
[Equation]
(15) where Γ θ ∗ A is deﬁned as:
[Equation]
See supplementary materials for the detailed derivations.
Note that similar to F θ ∗ A , Γ θ ∗ A is also positive-semi-deﬁnite (PSD) and therefore our Sliced-Cram´er regularizer enforces closeness of θ and θ ∗ in a Riemannian pseudo-manifold induced by the PSD matrix Γ θ .
While the deﬁnition of Γ in equation 16, regardless of its similarity to equation 3, may seem intimidating, it leads to a straightforward empirical algorithm, which we discuss in the next section.
Before that, we point out that similar to the FIM, calculating Γ is also practically infeasible, and we approximate Γ with a diagonal matrix that simpliﬁes the regularization into:
[Equation]
Finally, we emphasize that while equation 17 is similar to equation 5, it enforces a very different constraint on the neural network.
equation 5 enforces conditional class likelihoods to be preserved, which is sample-based; however, our proposed formulation in equation 17 preserves the distribution of network’s outputs at a particular layer for old tasks, i.e., it preserves the distribution of the previously learned representations.
P ROPOSED A LGORITHM
Here we derive the algorithmic steps required to calculate Γ empirically, as shown in Algorithm 1.
The empirical distribution at the network’s output can be written as, p A Z ( z | θ )
≈ 1 N
[Equation]
N n =1 δ
( z − φ ( x An ; θ )) , and the slices of this empirical distribution are deﬁned as:
[Equation]
Let u ( · ) denote the step function, which is the cumulative distribution of the Dirac delta function.
Then the cumulative distribution of R p AZ ( t, ξ | θ ) can be written as, R q AZ ( t, ξ | θ )
≈ 1 N
[Equation]
Nn =1 u ( t − ξ · φ ( x An ; θ )) .
Therefore we have:
[Equation]
substituting equation 18 into equation 16 and using a Monte-Carlo approximation of the integration of S d − 1 , with L samples, leads to:
[Equation]
where ξ l s are randomly drawn from S K − 1 , and ¯ z = 1 N (cid:80)
Nn =1 φ ( x An ; θ ∗ A ) .
These derivations give birth to our proposed algorithm shown in Algorithm 1.
Given that calculation of matrix Γ is not practically feasible (due to the large number of parameters of a deep neural network), we follow the work of (Kirkpatrick et al., 2017) and approximate Γ to be a diagonal matrix, which simpliﬁes to: (cid:2)
[Equation]
where ¯ φ ∗ A = 1 N (cid:80) Nn =
[Equation]
x An
[Equation]
θ ∗ A ) .
4. O NLINE EXTENSION OF THE ALGORITHM
To extend the framework into sequential learning of multiple tasks and to avoid memorizing taskspeciﬁc Γ s (or their diagonals), we follow the EWC++ framework proposed by Chaudhry et al.
(2018), which is, in essence, identical to the online-EWC proposed by Schwarz et al.
The
EWC++ (and online-EWC) methods ameliorate the need for predicting task identities and calculating task-speciﬁc FIMs and keep the memory requirement of the method constant (EWC requires linear growth of memory as a function of number of tasks).
In EWC++, given F ( t − 1) θ at task ( t − 1) , the accumulated FIM after learning task t is calculated as F ( t ) θ
=
αF θ ∗ t +(1 − α ) F ( t − 1) θ , where F θ ∗ t is the task speciﬁc FIM for task t , and α ∈ [0 , 1) is a hyperparameter that indicates the importance of preserving the most recent task over the older ones.
Similarly, we use,
[Equation]
(20) to obtain the sliced-Cram´er regularizer for task ( t + 1) .
Input: Data, X A
= { x An ∼ p AX } Nn =1 , and the optimized neural network, φ ( · ; θ ∗ A ) , for Task A .
Parameters: Number of random projections, L .
Output: Synaptic importance matrix Γ 1 Calculate the mean response of the network at the targeted layer: ¯ φ ∗ A = 1 N (cid:80)
Nn =1 φ ( x An ; θ ∗ A ) .
2 Initialize the synaptic importance matrix, [Γ] i,j = 0 .
3 for l ← 1 to L do
Sample ξ l from S K −
Slice the mean response: ρ = ξ l · ¯ φ ∗ A
Calculate ∇ θ ρ using auto-differentiation
Update
) T
[Figure]
N UMERICAL E XPERIMENTS 5. P ERMUTED MNIST
We ﬁrst test our proposed algorithm on the benchmark permuted MNIST task and compare the performance with online-EWC and our implementation of the online-MAS algorithm.
For this experiment, we used a single head model that learns ten tasks, where each task contains a permuted version of the original MNIST dataset.
We note that the reported results are a function of the architecture of the underlying network.
Meaning that, while all three methods perform well on this task for larger networks, the true competitiveness of the proposed method emerges for smaller networks where “over-estimation” of the synaptic importances signiﬁcantly hinders learning of the subsequent tasks and leads to intransigence.
For this experiment, we used a fully-connected network (i.e., a multi-layer perceptron) with the following architecture, 784 → 1024 → 512 → 256 → 10 neu- rons, and for all optimizations we used the ADAM optimizer with learning rate, lr = 1 e − 4 .
For our proposed method, SCP, we used L = 100 slices.
We repeated each experiment 10 times (with different permutations), and reported the average accuracy over all tasks in Figure 3 (a).
We can see that SCP is capable of utilizing the capacity of the network better, which points to the fact that regularizing the distribution as opposed to the samples provides a less restricted regularization for the network and still enables the network to freely move individual samples so long as the overall latent distribution of the data is consistent.
The regularization coefﬁcients for each algorithm was cross-validated on the following grid,
λ ∈ { 1 e + i | i ∈ [ − 3 , − 2 , ..., 9] } and the optimal value was used to report the results in Figure 3 (a).
Moreover, in Figure 3 (b) we show the histogram of the logarithm of the product of the regularization coefﬁcients with importance parameters for each method (i.e., λ times the diagonal values of
F θ , Ω , and Γ for EWC, MAS, and SCP, respectively).
One can see that the optimal values of the regularization coefﬁcients provide, more or less, scale-consistent synaptic importances for all methods and distinguishing factor between the methods is on the difference between these distributions.
An-
[Figure] [Figure]
other interesting observation is that the distribution of the synaptic importances are more similar for sample-based methods (i e., EWC and MAS) compared to the proposed distribution-based method.
5. S EQUENTIAL L EARNING OF A UTO -E NCODERS
Next we consider an experiment consisting of unsupervised/self-supervised sequential learning.
To that end we learn an auto-encoder on single digits of the MNIST dataset sequentially.
The model is chosen to be a fully connected auto-encoder, with the following encoder 728 → 1024 → 1024 → 1024 → 256 , a mirrored decoder 256 → 1024 → 1024 → 1024 → and Rectiﬁed Linear Unit (ReLU) activations.
For the loss function, we used cross-entropy plus the (cid:96) 1 -norm of the reconstruction error.
Similar to the previous experiment, we used the ADAM optimizer (Kingma &
Ba, 2014) for training the network with lr = 1 e − 4 .
We perform 50 epochs of learning on each digit, before switching to the next digit.
For consolidation, we used L = 100 slices for SCP.
Finally, we permute the order of the digits and run our experiments 10 times, and compare ‘No Consolidation,’ with MAS, and SCP.
Due to the unsupervised nature of the experiment, the EWC framework does not apply here.
Figure 4 demonstrate the results of this experiment.
Panel (a) shows the reconstruction of a sample digit from one of our runs with the input digit sequence of [2 , 7 , 4 , 0 , 9 , 1 , 6 , 5 , 8 , 3] .
It can be seen that both MAS and SCP can successfully retain the learned knowledge while acquiring new knowledge, while without consolidation the auto-encoder suffers from catastrophic forgetting.
Panel (b) shows the average (cid:96) 1 -norm of the reconstruction error for all seen tasks over the 10 runs.
We can see that SCP and MAS are qualitatively on par, and SCP provides a modest yet statistically signiﬁcant improvement over MAS for this task.
5. S EMANTIC S EGMENTATION OF SYNTHIA D ATASET
Lastly, we go beyond the benchmark yet less practical MNIST dataset and address catastrophic forgetting in a more interesting/critical application of autonomous vehicles.
We speciﬁcally consider the problem of learning semantic segmentation of road scenes in a sequential manner, where the input distribution of the data changes over time.
Semantic segmentation is the task of assigning a class label to every pixel of an input image.
To that end, we use two sequences of the SYNTHIA dataset (Ros et al., 2016), namely ‘SYNTHIA-SEQS-01-SUMMER’ and ‘SYNTHIA-SEQS-01-WINTER’ as Task 1 and Task 2, respectively.
There are 13 classes in the dataset namely: Miscellaneous, Sky,
Building, Road, Sidewalk, Fence, Vegetation, Pole, Car, Sign, Pedestrian, Cyclist, and Lane Marking.
We keep the last 100 frames of each sequence as the testing-set and train a deep convolutional
U-Net architecture (Ronneberger et al., 2015) on the tasks mentioned above.
For the loss function, we used (1 − Dice )
(Zou et al., 2004), and each task was learned over 100 epochs.
For the optimizer, we used the ADAM optimizer (Kingma & Ba, 2014) with learning rate, lr = 1 e − 4 .
For consolidation, we used L = 100 slices for SCP.
sequentially summer a qualitative comparison of online-EWC, MAS, and SCP on a sample test frame is in Figure 5.
The ﬁrst row shows the performance on T1 after learning T (base model), the second row shows performance on T2 after learning on T1 and then T (indicator of intransigence), and the third row shows performance on T1 after learning on T and then T2 (indicator of the catastrophic forgetting). We can see that catastrophic forgetting happens when no synaptic consolidation is leveraged. Moreover, compared to online-EWC and MAS, SCP suffers less from intransigence as fewer artifacts are present in the second row.
Lastly, SCP overcomes catastrophic-forgetting more successfully compared to EWC and MAS as it is apparent from the lack of artifacts in the last row.
Finally, we provide a quantitative comparison between the methods in Figure 6, where we report the Dice score (Zou et al., 2004), averaged over ten runs, for each task and each method during the sequential training.
As can be seen, SCP signiﬁcantly outperforms online-EWC and MAS on this task both in overcoming catastrophic forgetting (left plot) and overcoming intransigence (right plot).
C ONCLUSION
We introduced a new generic approach towards selective synaptic plasticity for preserving the distribution of the network’s output, at an arbitrary layer, for previously learned tasks.
We started from a memory-replay-based regularization that penalized the change in the distribution of the network’s output and showed that a second-order Taylor expansion of such regularization would lead to a selective synaptic plasticity approach that does not require memory of samples from previously seen tasks.
Furthermore, we proposed the sliced-Cram´er distance as a suitable metric for preserving these distributions, which leads to a straightforward algorithm for selective plasticity.
Also, using a similar approach, we reverse-engineered the Memory Aware Synapses (MAS) framework and provided a geometrically meaningful regularization that leads to this algorithm.
We then compared the online-EWC, MAS, and SCP methods on a variety of learning tasks, including supervised and unsupervised/self-supervised learning, and consistently showed competitive performance.
S UPPLEMENTARY M ATERIALS 7. T AYLOR E XPANSION OF THE KL-D IVERGENCE
For the sake of completion
[Equation]
here we derive the second-order Taylor expansion of D KL ,
[Equation]
around θ 0 where we can write θ = θ
+ δ θ .
The second-order Taylor expansion is:
[Equation]
Where for the ﬁrst-order term we have:
[Equation]
Therefore at θ = θ 0 we have:
[Equation]
and for the second-order term:
[Equation]
Therefore at θ = θ 0 we have:
[Equation]
which is the Fisher Information Matrix (FIM).
Finally, putting everything together we have:
[Equation]
which concludes the derivation.
7. T AYLOR E XPANSION OF THE MAS R EGULARIZER
Here we drive the second-order Taylor expansion of the MAS regularizer in equation 6,
[Equation]
around θ 0 .
The second-order Taylor expansion is:
[Equation]
Where for the ﬁrst-order term we have:
[Equation]
which for θ = θ 0 is zero.
For the second-order term we have:
[Equation]
where evaluated at θ = θ
the second term on the right-hand-side vanishes.
Finally
[Equation]
putting everything together we have:
[Equation]
which is the term reported in the paper
[Equation]
7. T AYLOR E XPANSION OF THE S LICED -C RAM ´ ER DISTANCE
Here we derive the second-order Taylor expansion of the squared 2-Sliced-Cram ´ er distance,
[Equation]
around θ 0 , as reported in equation 16.
The second-order Taylor expansion is:
[Equation]
Where for the ﬁrst-order term we have:
[Equation]
which for θ = θ 0 is equal to zero.
For the second-order term we have:
[Equation]
which evaluated at θ = θ 0 is:
[Equation]
Finally, putting everything together we have:
[Equation]
[Figure]
7. MNIST- TO -SVHN E XPERIMENT
Given the space constraint of the conference, we include our results on the MNIST-to-SVHN experiment in the supplementary material.
In this experiment, Task 1 is learning the MNIST digits, and
Task 2 is learning the SVHN digits.
We show sample images from these two datasets in Figure 7a.
In this experiment, we used a VGG-like architecture, for which we include the details in 8.
[Figure]
with
We performed incremental learning with no consolidation, Online-EWC, MAS, and SCP.
The importance parameters were cross-validated on a coarse grid (due to the computational and time restrictions) of { 1 e + i | i ∈ [2 , 3 , 4 , 5 , 6] } , where the optimal parameters for the methods where 1 e + 3 for SCP, 1 e +5 for MAS, and 1 e +4 for Online-EWC.
We chose 20 epochs per task.
Figure 7b shows the results of the experiment.
Note that the ﬁrst plot shows the testing performance of the methods on Task1 (i.e., MNIST dataset), where the blue shade indicates the ﬁrst 20 epochs (training on the
MNIST dataset), and the red shade indicates the second 20 epochs (training on the SVHN dataset).
As can be seen, all methods can address catastrophic forgetting, while Online-EWC and SCP outperform MAS (However, a one should perform a ﬁner grid search on the importance-parameters for a deﬁnitive evaluation).
The middle plot shows the testing performance of the model on the SVHN dataset (Task 2).
The MAS and SCP methods outperform online-EWC, and SCP performs slightly better than MAS (i.e., overcomes intransigence better).
The third plot shows the average testing performance of the methods on both tasks.
Similarly, we can see that SCP outperforms MAS and
Online-EWC.
We repeated the experiments ten times, and the plots show the average performance.
7. CIFAR100 E XPERIMENTS
The CIFAR100 (Krizhevsky et al., 2009) dataset contains images from 20 super-classes, where each super-class contains ﬁve sub-classes.
For this dataset, we considered an experiment in which the tasks are supervised classiﬁcation of the super-classes.
In short, we split the data into ﬁve sequential tasks, where each task contains a sub-class from all the 20 super-classes.
We used a Wide Residual
Network (Wide-ResNet) Zagoruyko & Komodakis (2016) network as our model and utilized the data augmentation suggested by (DeVries & Taylor, 2017).
Figure 9a shows sample super-classes together with data augmentation.
For each method, we used a coarse grid search for the importance parameters { 1 e + i | i ∈
Unfortunately, and due to the lack of time, we were able to only run the experiment for each method once.
We report the average classiﬁcation results in Figure 9b.
For the ﬁnal version of the paper, we will complete this experiment with 1) a ﬁner grid search over the importance parameters and 2) multiple runs of the experiments and reporting the average.