E NERGY -B ASED M ODELS FOR C ONTINUAL L EARNING
A BSTRACT
We motivate Energy-Based Models (EBMs) as a promising model class for continual learning problems.
Instead of tackling continual learning via the use of external memory, growing models or regularization, EBMs have a natural way to support a dynamically-growing number of tasks or classes that causes less interference with previously learned information.
Our proposed version of EBMs for continual learning is simple, efﬁcient and outperforms baseline methods by a large margin on several benchmarks.
Moreover, our proposed contrastive divergence based training objective can be applied to other continual learning methods, resulting in substantial boosts in their performance.
We also show that EBMs are adaptable to a more general continual learning setting where the data distribution changes without the notion of explicitly delineated tasks.
These observations point towards EBMs as a class of models naturally inclined towards the continual learning regime
I NTRODUCTION
Humans are able to rapidly learn new skills and continuously integrate them with prior knowledge.
The ﬁeld of Continual Learning (CL) seeks to build artiﬁcial agents with the same capabilities (Parisi et al., 2019).
In recent years, continual learning has seen increased attention, particularly in the context of classiﬁcation problems.
Continual learning requires models to remember prior skills as well as incrementally learn new skills, without necessarily having a notion of an explicit task identity.
Standard neural networks (He et al., 2016; Simonyan & Zisserman, 2014) experience the catastrophic forgetting problem and perform poorly in this setting.
Different approaches have been proposed to mitigate catastrophic forgetting, but many rely on the usage of external memory (Li & Hoiem, 2017), additional models (Shin et al., 2017), or auxiliary objectives and regularization (Maltoni & Lomonaco, 2019), which can constrain the wide applicability of these methods.
In this work, we propose a new approach towards continual learning on classiﬁcation tasks.
Most existing CL approaches tackle these tasks by utilizing normalized probability distribution (i.e., softmax output layer) and trained with a cross-entropy objective.
In this paper, we argue that by viewing classiﬁcation from the lens of training an un-normalized probability distribution, we can signiﬁcantly improve continual learning performance in classiﬁcation problems.
In particular, we interpret classiﬁcation as learning an Energy-Based Model (EBM) across classes.
Training becomes a wakesleep process, where the energy of an input data at its ground truth label is decreased while the energy of the input at (an)other selected class(es) is increased.
An important advantage is that this framework offers freedom to choose what classes to update in the continual learning process.
By contrast, the cross entropy objective reduces the likelihood of all negative classes when given a new input, creating updates that lead to catastrophic forgetting.
The energy function, which maps an input-label pair to a scalar energy, also provides a way for the model to select and ﬁlter portions of the input that are relevant towards the classiﬁcation on hand.
We show that this enables EBMs training updates for new data to interfere less with previous data.
In particular, our formulation of the energy function allows us to compute the energy of an input by learning a conditional gain based on the class label, which serves as an attention ﬁlter to select the most relevant information.
In the event of a new class, a new conditional gain can be learned.
These unique properties beneﬁt EBMs in addressing two important open challenges in continual learning.
First, we show that EBMs are promising for class-incremental learning, which is one of the most challenging settings for continual learning (van de Ven & Tolias, 2019).
Generally, successful existing methods for class-incremental learning either store data or use generative replay, which has disadvantages in terms of memory and/or computational efﬁciency.
We show that EBMs perform well in class-incremental learning without using replay and without relying on stored data.
The second open challenge that EBMs can address is continual learning without task boundaries.
Typically, a continual learning problem is set up as a sequence of distinct tasks with clear boundaries that are known to the model (the boundary-aware setting).
Most existing continual learning methods rely on these known boundaries for performing certain consolidation steps (e.g., calculating parameter importance, updating a stored copy of the model).
However, assuming such clear boundaries is not always realistic, and often a more natural scenario is the boundary-agnostic setting (Zeno et al., 2018; Rajasegaran et al., 2020), in which data distributions gradually change without a clear notion of task boundaries.
While many common CL methods cannot be used without clear task boundaries, we show that EBMs can be naturally applied to this more challenging setting.
Our work has three main contributions: First, we introduce energy-based models for classiﬁcation
CL problems.
We show that EBMs can naturally deal with challenging problems in CL, including the boundary-agnostic setting and class-incremental learning without using replay.
Secondly, we propose an energy-based training objective that is simple and broadly applicable to different types of models, with signiﬁcant boosts on their performance.
This contrastive divergence based training objective can naturally handle the dynamically growing number of classes and signiﬁcantly reduces catastrophic forgetting.
Lastly, we show that the proposed EBMs perform strongly on four standard
CL benchmarks.
These observations point towards EBMs as a class of models naturally inclined towards the CL regime and as an important new baseline upon which to build further developments.
The code are made public to facilitate further research 1 .
C ONTINUAL L EARNING WITH SOFTMAX - BASED CLASSIFIERS
The most common way to do classiﬁcation with deep neural networks is to use a softmax output layer in combination with a cross-entropy loss.
In CL, virtually all existing methods for classiﬁcation are based on the softmax-based classiﬁer (SBC) (Li & Hoiem, 2017; Zenke et al., 2017).
Given an input x ∈ R D and a discrete set Y = { 1 , . . .
, N } of N possible class labels, a traditional softmax-based classiﬁer deﬁnes the conditional probabilities of those labels as p θ ( y | x ) = exp ([ f θ ( x )]
y ) /
(cid:80) i ∈Y exp ([ f θ ( x )] i ) , for all y ∈ Y , where f θ ( x ) : R D → R N is a feed-forward neural network, parameterized by θ , that maps an input x to a N -dimensional vector of logits.
[ · ] i indicates the i th element of a vector.
A schema of SBC is shown in Figure 1 left.
Training.
A softmax-based classiﬁer is typically trained by optimizing the cross-entropy loss function.
For a given input x and corresponding ground truth label y
+ , the cross-entropy loss is
L CE ( θ ; x , y + )
= − log ( p θ ( y + | x )) .
Inference.
Given an input x , the class label predicted by the softmax-based classiﬁer is the class with the largest conditional probability ˆ y = arg max y ∈Y p θ ( y | x ) .
2. S OFTMAX - BASED CLASSIFIERS FOR CONTINUAL LEARNING
When used for continual learning, and in particular when used for class-incremental learning, softmax-based classiﬁers face several challenges.
One important issue is that softmax-based classiﬁers compute the cross-entropy loss over all classes (or sometimes over all classes that have been seen so far).
As a result, when training on a new task, the likelihood of the currently observed classes is increased, but the likelihood of old classes is too heavily suppressed since they are not encountered in the new task.
The softmax operation introduces competitive, winner-take-all dynamics that make the classiﬁer catastrophically forget past tasks.
We show such phenomenon of SBC in Section 4.1.3.
C ONTINUAL L EARNING WITH E NERGY -B ASED M ODELS
In this section, we propose a simple but efﬁcient energy-based training objective that can successfully mitigate the catastrophically forgetting in continual learning.
3. E NERGY - BASED MODELS
EBMs (LeCun et al., 2006) are a class of maximum likelihood models that deﬁne the likelihood of a data point x ∈ X ⊆ R D using the Boltzmann distribution:
[Equation]
where E θ ( x ) : R D → R , known as the energy function, maps each data point x to a scalar energy value, and Z ( θ ) is the partition function.
In deep learning applications, the energy function E θ is a neural network parameterized by θ .
EBMs are powerful models that have been applied to different domains, such as structured prediction Belanger & McCallum (2016); Gygli et al. ; Rooshenas et al.
Tu & Gimpel (2019), machine translation Tu et al. text generation Deng et al. (2020), reinforcement learning Haarnoja et al. (2017), image generation Salakhutdinov & Hinton (2009);
Du et al.
(2020a;b; 2019);
Xie et al.
; Nijkamp et al.
(2019), memory modeling Bartunov et al.
(2019), classiﬁcation Grathwohl et al.
(2019), and biologically-plausible training Scellier & Bengio (2017).
As far as we are aware, EBMs for CL has so far remained unexplored.
3. E NERGY - BASED MODELS FOR CLASSIFICATION
To solve the classiﬁcation tasks, we adapt the above general formulation of an EBM as follows.
Given inputs x ∈ R D and a discrete set Y of possible class labels, we propose to use the Boltzmann distribution to deﬁne the conditional likelihood of label y given x :
[Equation]
where E θ ( x , y ) : ( R D , N ) → R is the energy function that maps an input-label pair ( x , y ) to a scalar energy value, and Z ( θ ; x ) is the partition function for normalization.
Training.
We want the distribution deﬁned by E θ to model the data distribution p D , which we do by minimizing the negative log likelihood of the data
[Equation]
with the expanded form:
[Equation]
Equation 4 minimizes the energy of x at the ground truth label y and minimizes the overall partition function by increasing the energy of x at other labels y (cid:48) .
Inference.
Given an input x , the class label predicted by our EBMs is the class with the smallest energy at x : ˆ
y = arg
min y (cid:48) ∈Y E θ ( x , y (cid:48) ) .
3. E NERGY - BASED MODELS FOR CONTINUAL LEARNING 3.3. EBM T RAINING O BJECTIVE
We notice that in Equation 4, the energy over all class labels y (cid:48) given data x are maximized.
Directly maximizing energy across all labels raises the same problem as the softmax-based classiﬁer models that the old classes are suppressed when training a model on new classes and thus cause the catastrophic forgetting.
Inspired by (Hinton, 2002; Du & Mordatch, 2019;
Xie et al., 2016), we ﬁnd that the contrastive divergence approximation of Equation 4 can mitigate this problem and lead to a simpler equation.
To do so, we deﬁne the following contrastive divergence loss:
[Equation]
where y is the ground truth label of data x and y − is a negative class label randomly sampled from the set of class labels in the current training batch Y B .
[Figure]
N G & Different from the SBC, EBMs maximize likelihood not by normalizing over all classes but instead by contrastively increasing the energy difference between the ground truth label and another negative label for a given data point.
This operation causes less interference with previous classes and enables EBMs to suffer less from catastrophic forgetting.
Importantly, such a sampling strategy also allows our EBMs to be naturally applied, without any modiﬁcation, to different CL settings described in Section E.1.
Since we select the negative sample(s) from the current batch, our
EBMs do not require knowledge of the task on hand.
This allows application of EBMs in the boundary-agnostic setting, in which the task boundaries are not given.
We ﬁnd that the proposed EBM training objective is efﬁcient enough to achieve good performance on different CL datasets (Table 1 and Table 4).
We note however that it is possible to use other strategies for choosing the negative classes in the partition function in Equation 4.
In Table 3, we explore alternative strategies: 1) using all classes in the current training batch Y B as negative classes, and 2) using all classes seen so far as negative classes.
The usage of negative samples in the
EBM training objective provides freedom for models to choose which classes to train on which is important for preventing catastrophic forgetting in continual learning.
EBMs are not limited to modeling the conditional distribution between x and y as shown in Equation 2.
Another way to use the Boltzmann distribution is to deﬁne the joint likelihood of x and y .
In
Supplement Section A, we show that a different training objective can further improve the results.
3.3. E NERGY NETWORK
Another important difference from softmax-based classiﬁers is that the choice of model architectures becomes more ﬂexible in EBMs.
Traditional classiﬁcation models only feed in x as input.
In contrast, EBMs have many different ways to combine x and y in the energy function with the only requirement that E θ ( x , y ) : ( R D , N ) → R .
In EBMs, we can treat y as an attention ﬁlter or gate to select the most relevant information between x and y .
To compute the energy of any data x and class y pair, we use y to inﬂuence a conditional gain on x , which serves as an attention ﬁlter (Xu et al., 2015) to select the most relevant information between x and y .
In Figure 1 (right), we ﬁrst send x into a small network to generate the feature f ( x ) .
The label y is mapped into a same dimension feature g ( y ) using a small learned network or a random projection.
We use the gating block G to select the most relevant information between x and y : m ( x , y )
=
G ( f ( x ) , g ( y )) .
The output is ﬁnally sent to weight layers to generate the energy value
E θ ( x , y ) .
See Supplement Section D for more details about our model architectures.
Our EBMs allow any number of classes in new batches by simply training or deﬁning a new conditional gain g ( y ) for the new classes and generating its energy value with data point x .
3.3. I NFERENCE
[Table]
During inference, because we evaluate according to the class-incremental learning scenario (van de Ven & Tolias, 2019; Tao et al., 2020), the model must predict a class label by choosing from all classes seen so far.
Let x k be one data point from a batch B k with an associated discrete label
y ∈
Y k , where
Y k contains the class labels in B k .
Then there are Y =
(cid:83) Kk =1 Y k different class labels in total after seeing all the batches.
The MAP estimate is ˆ y = arg
min y E
θ K ( x k , y ) , where y ∈ (cid:83)
Y k , and E
θ K ( x k , y ) is the energy function with parameters θ K resulting from training on the batches {B 1 , · · · , B K } .
The energy function can compute an energy for any discrete class input, including unseen classes, which avoids needing to predeﬁne the number of classes in advance.
E XPERIMENTS
In this section, we want to answer the following questions: How do the proposed EBMs perform on different CL settings?
Can we apply the EBM training objective to other methods?
And can we qualitatively understand the differences between EBMs and baselines?
We report experiments on the boundary-aware setting in Section 4.1 and the boundary-agnostic setting in Section 4.2.
E XPERIMENTS ON B OUNDARY -A WARE S ETTING 4.1. D ATASETS AND EVALUTION PROTOCOLS
We evaluate the proposed EBMs on the split MNIST (Zenke et al., 2017), permuted MNIST (Kirkpatrick et al., 2017), CIFAR-10 (Krizhevsky et al., 2009), and CIFAR-100 (Krizhevsky et al., 2009) datasets. The split MNIST dataset is obtained by splitting the original MNIST dataset (LeCun et al., 1998) into 5 tasks with each task having 2 classes. This dataset has 60,000 training images and 10,000 test images. The permuted MNIST protocol has 10 tasks, each task with 10 classes.
For each task, the original images pixels are randomly permuted to generate 32 × 32 images.
We separate
CIFAR-10 into 5 tasks, each task with 2 classes. Similarly, CIFAR-100 is split into 10 tasks with each task having 10 classes. These last two datasets each have 50,000 training images and 10, test images. As noted by (van de Ven & Tolias, 2019), the above task protocols could be evaluated according either the task-incremental, domain-incremental, or class-incremental learning scenario.
Most CL approaches perform well on the ﬁrst two simpler scenarios, but fail when asked to do classincremental learning, which is considered as the most natural and also the hardest setting for CL (Tao et al., 2020; He et al., 2018). In this paper, we perform all experiments according to the classincremental learning scenario.
4.1. C OMPARISONS WITH EXISTING METHODS
[Table]
The most successful existing methods for Class-IL either rely on an external quota of memory (Rebufﬁ et al., 2017; Hayes et al., 2020) or on using generative replay (Shin et al., 2017; van de Ven et al., 2020).
One of their disadvantages is that they are relatively expensive in terms of memory and/or computation.
In this paper, we focus on CL without using replay and without storing data.
[Table]
Comparisons with baselines.
We compare our proposed EBM method with available baseline models that do not use replay, including the standard softmax-based classiﬁer (SBC), EWC (Kirkpatrick et al., 2017), Online EWC (Schwarz et al., 2018),
SI (Zenke et al., 2017), LwF (Li & Hoiem, 2017), MAS (Aljundi et al., 2019), and BGD (Zeno et al., 2018).
The Class-IL results on four datasets are shown in Table 1.
All the baselines and EBMs use similar model architectures with similar number of model parameters for fair comparison.
For split MNIST and permuted MNIST, we use several fully-connected layers.
For CIFAR-10 and CIFAR-100, we use a convolutional network (Supplement Section D for details).
For CIFAR-100, all compared models used convolutional layers that were pre-trained on
CIFAR-10. Similar training regimes were used for the EBMs and baselines. On split MNIST, permuted MNIST, and CIFAR-10, we trained for 2000 iterations per task. On CIFAR-100, we trained for 5000 iterations per task.
All experiments used the Adam optimizer with learning rate 1 e − 4 .
Each experiment was performed at least 10 times with different random seeds, with results reported as the mean ± SEM in Table 1.
EBMs have a signiﬁcant improvement over the baseline methods on all the datasets, showing that EBMs forget less when updating models for new tasks.
EBM training objective on existing approaches.
Our proposed energy-based training objective is simple and can also be directly applied to existing CL approaches.
We test this by modifying
[Figure]
Label Distributation (Class-IL)
[Figure]
Label Distributation (Class-IL)
[Figure]
Label Distributation (Class-IL)
[Figure]
Label Distributation (Class-IL)
[Figure]
Label Distributation (Class-IL)
[Figure]
5  9  1 2 31 2 34512 3 45 6 7123456 7 8 910 EBM
Label Distributation (Class-IL)
splitMNIST
CLS
EBM    5  9  1 2 31 2 34512 3 45 6 7123456 7
Label Distributation (Class-IL)
splitMNIST
CLS
EBM    5  9  1 2 31 2 34512 3 45 6 7123456 7
L a b e l D i s t
r
i bu t a t i o n
) s p lit M N I S T
Label Distributation (Class-IL)
[Figure]
CLS
EBM
[Figure]
[Figure]
[Figure]
the training objective of baseline models to that of our proposed energy objective, which computes the softmax normalization only over class labels in the current training batch.
In Table 2, we ﬁnd that our proposed objective signiﬁcantly improves the performance of different CL methods.
This is because the new training objective does not suppress the probability of old classes when improving the probability of new classes.
Our training objective provides an orthogonal direction to tackle the
CL problem and is simple to implement on existing CL approaches.
Effect of energy training objective.
We conduct an experiment on the CIFAR-100 dataset to investigate how different EBM training objectives inﬂuence the CL results.
We compare three strategies for selecting the negative samples as described in Section 3.3.1.
The ﬁrst strategy uses all seen classes so far as negative labels ( All Neg Seen ).
The second one takes all the classes in the current batch as negative labels ( All Neg Batch ).
The last one randomly selects one class from the current batch as the negative as described in Equation 5 ( 1 Neg Batch ).
In Table 3, we ﬁnd using only one negative sample generates the best result, and using negatives sampled from classes in the current batch is better than from all seen classes.
Since our EBM training objective aims at improving the energy of negative samples while decreasing the energy of positive ones, sampling negatives from the current batch has less interference with previous classes than sampling from all seen classes.
4.1. Q UALITATIVE ANALYSIS
Most existing CL methods are based on the softmax-based classiﬁers (Pellegrini et al., 2019; Zenke et al., 2017).
To better understand why EBMs suffer less from catastrophic forgetting, we qualitatively compare our EBMs and the standard SBC model shown in Figure 1 in this part.
Energy landscape.
In Figure 2, we show the energy landscapes after training on task 9 and task 10 of the permuted MNIST dataset.
For SBC, the energy is given by the negative of the predicted probability.
Each datapoint has 100 energy values (EBM) or probabilities (SBC) corresponding to the 100 labels in the dataset.
Dark elements on the diagonal indicate correct predictions.
After training on task T 9 , SBC assigns high probabilities to classes from T 9 (80-90) for almost all the data from T 1 to T 9 .
After learning T 10 , the highest probabilities shift to classes from T 10 (90-100).
SBC tends to assign high probabilities to new classes for both old and new data, indicating forgetting.
In contrast, EBM has low energies across the diagonal, which means that after training on new tasks,
EBM still assigns low energies to the true labels of data from previous tasks.
This shows that EBM is better than SBC at learning new tasks without catastrophically forgetting of old tasks.
Predicted class distribution.
In Figure 3, for the split MNIST dataset, we plot the proportional distribution of predicted classes.
Only data from the tasks seen so far was used for this ﬁgure.
Taking the second panel in the ﬁrst row as an example, it shows the distribution of predicted labels on test data from the ﬁrst two tasks after ﬁnishing training on the second task.
Since the number of test images from each class are similar, the ground truth proportional distribution should be uniform over those four classes.
After training on the ﬁrst task, the predictions of SBC are roughly uniformly distributed over the ﬁrst two classes (ﬁrst panel).
However, after learning new tasks, SBC only predicts classes from the most recent task and fails to correctly memorize previous classes.
In contrast, the predictions of EBM are substantially more uniformly distributed over all seen classes.
we provide further comparisons between SBC and EBMs in Supplement Section B and Section C.
4. E XPERIMENTS ON B OUNDARY -A GNOSTIC S ETTING
When applying continual learning in real life, boundaries are not usually well deﬁned between different tasks.
However, most existing CL methods rely on the presence of sharp boundaries between tasks to determine when to consolidate the knowledge.
We show that EBMs are able to ﬂexibly perform CL across different setups, and perform well on the boundary-agnostic setting as well.
4.2. D ATASETS AND EVALUTION PROTOCOLS
For the boundary-agnostic setting, we use the same datasets as the boundary-aware setting in Section 4.1.1.
We use the code of “continuous task-agnostic learning” proposed by (Zeno et al., 2018) to generate a continually changing data stream.
See supplement Section E.1 for details.
All experiments are performed according to the Class-IL scenario.
4.2. C OMPARISON WITH EXISTING METHODS
[Table]
We focus on the investigation of the performance of different model architectures with similar model size and memory footprint, and thus do no compare with replay-based methods.
Since there is no knowledge on the number of tasks, many methods of continual learning that rely on task boundaries are generally inapplicable.
One trivial adaptation is to take the core action after every batch step instead of every task.
However, doing such adaptation is impractical for most algorithms, such as EWC, because of the large computational complexity.
However, we managed to run the Online EWC, SI, and BGD baselines in this way.
All compared methods used similar model architectures as in the boundary-aware setting.
Each experiment was performed 5 times with different random seeds, the results are reported as the mean ± SEM over these runs.
The results are shown in Table 4.
We observe that EBMs have a signiﬁcant improvement on all the datasets.
The experiments show that EBMs have good generalization ability for different continual learning problems as EBMs can naturally handle data streams with and without task boundaries.
C ONCLUSION
In this paper, we show that energy-based models are a promising class of models in a variety of different continual learning settings.
We demonstrate that EBMs exhibit many desirable characteristics to prevent catastrophic forgetting in continual learning, and we experimentally show that
EBMs obtain strong performance on the challenging class-incremental learning scenario on multiple benchmarks, both on the boundary-aware and boundary-agnostic settings.
One drawback of the current EBM method is that it computes the energy of a data point and all class labels during inference.
One way to speed up is to make hierarchical decisions, e.g., ﬁrst decide whether it is a cat or a dog, then decide the speciﬁc breed.
Study this case could be a good direction for future research.
Appendices
In Section A, we provide more details about the alternative EBM training objective mentioned in the main paper Section 3.3.1.
Section B and Section C show more comparisons and analysis of EBMs and baselines.
In Section D, we provide the model architecture details of the proposed EBMs and baselines on difference continual learning datasets.
Section E show some related works of different types of continual learning settings and approaches.
A A LTERNATIVE EBM T RAINING O BJECTIVE A. E NERGY - BASED MODELS FOR CLASSIFICATION
In the main paper Section 3.3.1, we mentioned an alternative EBM training objective can further improve the continual learning results.
Here we provide more details about this training objective.
We propose to use the Boltzmann distribution to deﬁne the joint likelihood of image x and label y :
[Equation]
where E θ ( x , y ) : ( R D , N ) → R is the energy function that maps an input-label pair ( x , y ) to a scalar energy value, and Z ( θ ) is the partition function for normalization.
Training.
We want the distribution deﬁned by E θ to model the joint data distribution p D , which we do by minimizing the negative log likelihood of the data
[Equation]
with the expanded form:
[Equation]
Equation 8 minimizes the energy of x at the ground truth label y and minimizes the overall partition function by increasing the energy of any other randomly paired x (cid:48) and y (cid:48) .
Inference.
Given an input x , the class label predicted by our EBMs is the class with the smallest energy at x :
[Equation]
A. E NERGY - BASED MODELS FOR CONTINUAL LEARNING
As described in the main paper Section 3.3.1, directly maximizing energy across all labels of a data point x raises the same problem as the softmax-based classiﬁer models that the old classes are suppressed when training a model on new classes and thus cause catastrophic forgetting.
Inspired by Hinton (2002)
; Du & Mordatch (2019);
Xie et al.
, we ﬁnd that the contrastive divergence approximation of Equation 8 can mitigate this problem and lead to a simpler equation.
We approximate Equation 8 by sampling a random pair of image x (cid:48) and label y (cid:48) from the current training batch to approximate the partition function.
Our training objective is given by:
[Equation]
where y is the ground truth label of data x .
This training objective is reminiscent of the contrastive divergence training objective used to train
EBMs in the main paper Equation 5.
The major difference is that we utilize both images and labels from the current batch as our contrastive samples instead of just labels used in the main paper
Equation 5.
We show in the experiments that using the proposed contrastive training objective in this supplement Equation 10 can further improve the continual learning performance.
[Table]
A. I NFERENCE
We use the same inference methods as described in the main paper Section 3.3.3 to perform the
Class-IL evaluation on the continual learning datasets.
The model predicts the class label
ˆ y of a data point x k from all class labels, where x k is one data point from a batch B k with an associated discrete label
y ∈
Y k and Y k contains the class labels in B k .
Then there are Y =
(cid:83) Kk =1 Y k different class labels in total after seeing all the batches.
The MAP estimate is
[Equation]
(11) where E θ K ( x k , y ) is the energy function with parameters θ K resulting from training on the batches {B 1 , · · · , B K } .
A.
C OMPARISONS WITH EXISTING METHODS
We follow the experiments performed in the main paper Section 4.1.2 and evaluate the Class-IL on the split MNIST (Zenke et al., 2017) and permuted MNIST (Kirkpatrick et al., 2017) datasets on the
Boundary-Aware setting.
We compare EBMs using different training objectives and the baseline approaches in this supplement Table 5.
All the baselines and EBMs use similar model architectures with similar number of model parameters for fair comparison.
“EBM” means the results of the training objective used in the main paper Equation 5.
“EBM Alt CD” represents the alternative training objective described in this supplement Equation 10.
EBMs have a signiﬁcant improvement over the baseline methods on all the datasets, showing that EBMs forget less when updating models for new tasks.
“EBM Alt
CD” can further improve the continual learning performance.
B A DDITIONAL EXPERIMENTS B. I S THE STRONG PERFORMANCE OF EBM S DUE TO THE E NERGY T RAINING O BJECTIVE OR DUE TO THE L ABEL C ONDITIONING ?
In the main paper Section 4.1.2, we investigate the effect of energy training objective .
Here we conduct more experiments to investigate the effect of the label conditioning architecture in EBMs.
Effect of label conditioning.
We test whether the label conditioning in our EBMs is important for their performance in the main paper Table 2.
As mentioned in Section 4.1.2, we modify baseline models using our training objective.
EBMs still outperform the modiﬁed baselines, implying that the label conditioning architecture also contributes to why EBMs suffer less from catastrophic forgetting.
We further show the testing accuracy of each task as the training progresses in Figure 4.
We compare the standard classiﬁer (SBC), classiﬁer using our training objective (SBC*), and our EBMs on two different datasets.
The left ﬁgures show results on the split MNIST dataset while the right ﬁgures show results on the permuted MNIST dataset.
We ﬁnd that the accuracy of old tasks in SBC drop sharply when learning new tasks, while the EBM training objective used in SBC* is better.
The curve on EBMs drops even slower than SBC*, implying our EBMs can mitigate the forgetting problem. [Figure] EBM T SBC*
EBM T T T4 T A cc u r ac y 1.0 0.
SBC
T1 T
T
T
T5 T1 T
T
T
T5 T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T1 T2 T3 T4 T5 T6 T7 T8 T9 T10 T1 T2 T3 T4 T5 T6 T7 T8 T9 T
[Table]
To summarize
[Table]
we show that the strong performance of our EBMs is due to both the EBM training objective and the label conditioning architecture.
Moreover, these results indicate that surprisingly, and counterintuitively, directly optimizing the cross-entropy loss used by existing approaches may not be the best way to approach continual learning [Table]
B. C OMPARISON OF DIFFERENT ARCHITECTURES
EBMs allow ﬂexibility in integrating data information and label information in the energy function.
To investigate where and how to combine the information from data x and label y , we conduct a series of experiments on CIFAR-10.
Table 6 shows four model architectures (V1-V4) that combine x and y in the early, middle, and late stages, respectively (see Supplement Section D for more details).
We ﬁnd combining x and y in the late stage (V4) performs the best.
We note that instead of learning a feature embedding of label y , we can use a ﬁxed projection matrix which is randomly sampled from the uniform distribution U (0 , 1) .
Even using this ﬁxed random projection can already generates better results than most baselines in Table 1.
Note further that the number of trainable parameters in the “Fix” setting is much lower than that of the baselines.
Using a learned feature embedding of y can further improve the result.
We may also apply different normalization methods over the feature channel of y .
We ﬁnd that Softmax ( End Softmax (V4) ) is better than the L normalization ( End Norm2 (V4) ) and no normalization ( End (V4) ).
B. E FFECT OF DIFFERENT NUMBERS OF TASKS
To test the generality of our proposed EBMs, in Table 7 we repeat the boundary-aware experiments on CIFAR-100 for different number of classes per task.
In Table 1, the CIFAR-100 dataset was split up into 10 tasks, resulting in 10 classes per task.
Here we additionally split CIFAR-100 up intotasks (i.e., 20 classes per task), 20 tasks (i.e., 5 classes per task) and 50 tasks (i.e., 2 classes per task).
Our EBM substantially outperforms the baselines on all settings.
B. C AN EBM S USE R EPLAY ?
Although EBMs already achieve good performance on Class-IL without using any replay, we found that EBMs are ﬂexible enough to be combined with replay-based approaches to further improve the [Table] [Table] performance.
In Table 8, we show the results of SBC and EBM using exact replay.
When training a new task, we mix the new data with data sampled from a memory buffer that stores examples of previously learned tasks to train the models.
The examples stored in the buffer are randomly selected from the classes encountered so far, and the available memory budget k is equally divided over all classes encountered so far.
On the split MNIST, permuted MNIST, and CIFAR-10 datasets, we use a memory budget of k = 1000 .
On the CIFAR-100 datasets, we use a memory budget of k = 2000 .
Taking the split MNIST dataset as an example, after we ﬁnishing training the ﬁrst task, we randomly select 1000 data-label pairs from the ﬁrst task and save them in the replay buffer.
Then we train the model on the second task.
In each training batch, we randomly sample a set of data-label pairs from the second task, and randomly sample a set of data-label pairs from the replay buffer.
We compute the ﬁnal loss by adding the loss of data from the current task L CD current ( θ ; x , y ) and the loss of data from the replay buffer L CD replay ( θ ; x , y ) using the following equation:
[Equation]
After we ﬁnishing training the second task, we randomly select 500 data-label pairs from the replay buffer and randomly sample 500 data-label pairs from the second task and update the replay buffer using the new sampled data-label pairs.
Note we always keep the number of data-label pairs in the replay buffer at 1000.
Then we train the model on the third task.
In each training batch, we randomly sample a set of data-label pairs from the third task, and randomly sample a set of data-label pairs from the replay buffer.
We compute the ﬁnal loss using Equation 12.
We use such a strategy to train the models until the ﬁnal task.
The baseline model, i.e . “SBC ER”, in Table 8 uses replay in the same way.
The only difference from our EBM is that “SBC ER” uses the cross-entropy loss as described in the main paper Sectionbut “EBM ER” uses the proposed EBM training objective.
Each experiment was performed 5 times with different random seeds, with results reported as the mean ± SEM.
We use the similar training regimes for EBMs and SBC.
On split MNIST, permuted MNIST, and CIFAR-10, we trained for 2000 iterations per task.
On CIFAR-100, we trained for 5000 iterations per task.
In each training batch, we sampled 128 data-label pairs from the current task and 128 data-label pairs from the replay buffer (start from the second task) to train the model.
All experiments used the Adam optimizer with learning rate 1 e − 4 .
We report the results on 4 datasets.
Note these numbers might be slightly different from results reported in some existing works because of the usage of different model architectures, different memory sizes, and different ways to split the datasets.
In our experiments, we control the baselines and EBMs to have similar model architectures with similar number of model parameters and the same buffer sizes on each dataset.
After using extra memory, both SBC and EBM have improvements.
EBMs still outperform SBC, especially on more challenging datasets, e.g., CIFARand CIFAR-100.
Interestingly, on CIFAR-100, we ﬁnd that EBMs without using replay (30.28%;
Table 1) perform better than SBC with using replay (28.57%; Table 8).
Again our focus is try to address the CL problems without using replay or stored data.
Our results show that the proposed EBM formulation provides an orthogonal direction to tackle the CL problems.
This approach can be further combined with existing CL methods.
B. C LASS CONFUSION MATRIX AT THE END OF LEARNING
We show confusion matrices for EBM and SBC.
A confusion matrix illustrates the relationship between the ground truth labels and the predicted labels.
Figure 5 in this supplement shows the confusion matrices after training on all the tasks on the split MNIST dataset and permuted MNIST dataset.
The standard classiﬁer tends to only predict the classes from the last task (class 8, 9 for split
MNIST and classes 90-100 for permuted MNIST).
The EBMs on the other hand have high values
[Figure]
[Figure]
[Equation] [Figure] [Equation] along the diagonal, which indicates that the predicted results match the ground truth labels for all the sequentially learned tasks.
C A DDITIONAL A NALYSES
Extending the results presented in the main paper Section 4.1, here we further compare EBMs with the baseline models by providing additional quantitative analyses of their performance.
We show the model capacity comparisons in Section C.1 and parameter importance measurement in Section C.2.
C. M ODEL CAPACITY
Another hypothesized reason for why EBMs suffer less from catastrophic forgetting than standard classiﬁers is potentially their larger effective capacity.
To analyze effective capacity of our models, we test the model capacity of the standard classiﬁer and EBMs on both the generated images and natural images.
Model capacity on generated images.
We generate a large randomized dataset of 32 × 32 images with each pixel value uniformly sampled from -1 to 1.
Each image is then assigned a random class label between 0 and 10.
We measure the model capacity by evaluating to what extent the model can ﬁt a such dataset.
For both the standard classiﬁer and the EBM, we evaluate three different sizes of models (small, medium, and large).
For a fair comparison, we control the EBM and classiﬁer have similar number of parameters.
The Small EBM and SBC have 2 , 348 , 545 and 2 , 349 , 032 parameters respectively.
The medium models have 5 , 221 , 377 (EBM) and 5 , 221 , 352 (SBC) parameters while the large models have 33 , 468 , 417 (EBM) and 33 , 465 , 320 (SBC) parameters.
We use the model architectures in this supplement Table 9 for EBMs and classiﬁers.
The resulting training accuracies are shown in this supplement Figure 6 with the number of data ranges from one to ﬁve millions.
Given any number of datapoints, EBM obtains higher accuracy than the classiﬁer, demonstrating that indeed EBM has larger capacity to memorize data given a similar number of parameters.
The gap between EBM and SBC increases when the models become larger.
The larger capacity of EBM potentially enables it to memorize more data and mitigate the forgetting problem.
Model capacity on natural images.
We also compare classiﬁers and EBMs on natural images from CIFAR-10.
Each image is assigned a random class label between 0 and 10.
We use the same network architecture as in Table 9 but with a hidden unit size of h = 256 .
Since there are only 50 , 000 images on CIFAR-10, we use a small classiﬁer and EBM and train them on the full dataset. [Figure]
After training 100000 iterations, the EBM obtains a top-1 prediction accuracy of 82 . 81 , while the classiﬁer is 42 .
We obtain the same conclusion that EBM has larger capacity to memorize data given a similar number of parameters.
C. P ARAMETER IMPORTANCE
To further understand why EBMs suffer less from catastrophic forgetting than standard classiﬁers, we design an experiment to test the importance of model parameters on past data.
Inspired by the elastic weight consolidation (EWC) (Kirkpatrick et al., 2017), we estimate the importance of parameters for each tasks using the diagonal elements of the Fisher information matrix (FIM) F .
Let θ i be the model parameters after training on task T i .
Given one of previous tasks T j , j <
i , we evaluate how important each parameter is for tasks T j .
The k th diagonal of F is deﬁned as the gradient on the EBM loss
[Equation]
(13) where x is sampled from tasks T j and E ( x , y ) is the energy value of the input data x and ground truth label y .
The class label y − ∈
Y j are randomly selected from the current batch.
Here we use a single negative class.
The above equation assigns high values to parameters crucial to task
T
j as their gradients with respect to the loss are larger.
Since the diagonal elements of the ﬁsher information matrix measure the importance of each parameter to a given task, the density of diagonal elements represents the proportion of important parameters over all parameters.
More density means more parameters are important for the given task and less parameters can be recruited for new tasks.
Ideally, we expect these values to be sparse.
In this supplement Figure 7, we show the diagonal elements of the standard classiﬁer (SBC), classiﬁer using our training objective (SBC*), and our EBMs on the split MNIST dataset.
For SBC and SBC*, we follow (Kirkpatrick et al., 2017) to compute their ﬁsher information matrices.
For comparisons across multiple models, we normalize the FIM diagonal elements of each method to be between 0 and 1 and report the normalized results in Figure 7.
For example, “Fisher 5 on data 1” shows the diagonal elements of the Fisher information matrix obtained by Equation 13 using the model parameters θ 5 (after training on task T 5 ) and data x , y, y − from task T 1 .
The distribution of
EBMs is sparser than SBC and SBC* indicating that EBMs have fewer important parameters for previous data.
Updating parameters for the new task will have less negative impact on old tasks.
In addition, more parameters can be used for learning new tasks if the distribution is sparse.
This may provide another explanation for why EBMs can mitigate catastrophic forgetting.
D M ODEL ARCHITECTURES
In this section, we provide details of the model architectures used on the different datasets.
Images from the split MNIST and permuted MNIST datasets are grey-scale images.
The baseline models for these datasets, similar as in (van de Ven & Tolias, 2019), consist of several fully- [Equation] [Equation] connected layers.
For the EBMs we use similar number of parameters.
The model architectures of
EBMs on the split MNIST dataset and permuted MNIST dataset are the same, but have different input and output dimensions and hidden sizes.
The model architectures of EBMs and baseline models on the split MNIST dataset are shown in this supplement Table 10.
The model architectures of EBMs and baseline models on the permuted MNIST dataset are shown in Table 11.
Images from the CIFAR-10 and CIFAR-100 datasets are RGB images.
For CIFAR-10, we use a small convolutional network for both the baseline models and the EBMs.
The model architectures of EBMs and baseline models on the CIFAR-10 dataset are shown in Table 12.
We investigate different architectures to search for the effective label conditioning on EBMs training as described in the main paper Section 4.1.2.
The model architectures used on the CIFAR-100 dataset are detailed in Table 13.
E R ELATED WORK E. C ONTINUAL LEARNING SETTINGS
Boundary-aware versus boundary-agnostic .
In most existing CL studies, models are trained in a “boundary-aware” setting, in which a sequence of distinct tasks with clear task boundaries is given (e.g., Kirkpatrick et al., 2017; Zenke et al., 2017; Shin et al., 2017). There are no overlaps between [Table]
[Table] [Table] any two tasks; for example task 1 has data class labels “1,2” and task 2 has data with class labels “3,4”.
Models are ﬁrst trained on the ﬁrst task and then move to the second one.
Moreover, models are typically told when there is a transition from one task to the next.
However, it could be argued that it is more realistic for tasks to change gradually and for models to not be explicitly informed about the task boundaries.
Such a boundary-agnostic setting has been explored in (Zeno et al., 2018;
Rajasegaran et al., 2020; Aljundi et al., 2019).
In this setting, models learn in a streaming fashion and the data distributions gradually change over time.
In (Zeno et al., 2018), the percentage of “1s” gradually decrease while the percentage of “2s” increases during training.
Importantly, most existing
CL approaches are not applicable to this setting as they require the task boundaries to decide when to perform certain consolidation steps.
In this paper, we will show that our proposed approach can naturally handle both the boundary-aware and boundary-agnostic settings.
Task-incremental versus class-incremental learning .
Another important distinction in CL is between task-incremental learning ( Task-IL ) and class-incremental learning ( Class-IL ) (van de Ven &
Tolias, 2019; Prabhu et al., 2020).
In Task-IL , also referred to as the multi-head setting (Farquhar & Gal, 2018), models predict the label of an input data by choosing only from the labels in the task where the data come from.
In Class-IL , also referred to as the single-head setting, models chose between the classes from all tasks so far when asked to predict the label of an input data.
Class-IL is more challenging than Task-IL as it requires models to select the correct labels from the mixture of new and old classes.
Generally, to perform well on Class-IL , existing methods need to store data, use replay, or pretrain models from another large dataset (Rebufﬁ et al., 2017; Rajasegaran et al., 2019; Belouadah et al., 2020; Maltoni & Lomonaco, 2019; Hayes & Kanan, 2020).
E. C ONTINUAL LEARNING APPROACHES
Numerous methods have been proposed for continual learning.
Here we broadly partition them into three categories: task-speciﬁc, regularization, and replay-based approaches.
Task-speciﬁc methods.
One way to reduce interference between tasks is by using different parts of a neural network for different tasks.
For a ﬁxed-size network, such specialization could be achieved by learning a separate mask for each task (Fernando et al., 2017; Serra et al., 2018), by a priori deﬁning a different, random mask for every task (Masse et al., 2018), or by using a different set of parameters for each task (Zeng et al., 2019;
Hu et al., 2019).
Other methods let models grow or recruit new resources when learning new tasks, such as progressive neural networks (Rusu et al., 2016) and dynamically expandable networks (Yoon et al., 2017).
Although these methods are generally successful in reducing catastrophic forgetting, a key disadvantage is that they require knowledge of task identities during training and testing.
They are therefore not suitable for Class-IL .
Regularization-based methods.
Regularization is used in CL to encourage the stability of those aspects of the network that are important for previous tasks.
A popular strategy is to add a regularization loss to penalise changes of important parameters.
EWC (Kirkpatrick et al., 2017)) and online
EWC (Schwarz et al., 2018) evaluate the importance of parameters using the diagonal elements in the ﬁsher information matrices, while SI (Zenke et al., 2017) estimates the parameters’ importance online. LwF (Li & Hoiem, 2017) regularizes the network at the functional level.
Although regularization-based methods can be computationally efﬁcient, a disadvantage is that typically they gradually reduce the model’s capacity for learning new tasks. Moreover, while in theory these methods can be used for Class-IL , in practice they have been shown to fail on such problems (Farquhar & Gal, 2018; van de Ven & Tolias, 2019). Replay methods. To preserve knowledge, replay methods periodically rehearse previous information during training (Robins, 1995).
Exact or experience replay based methods store data from
[Table] x = Conv2d(3 × 3, 3, 32) (x) x = ReLU (x) x = Conv2d(3 × 3, 32, 32) (x) x = ReLU (x) x
= Maxpool(2, 2) (x) x = Conv2d(3 × 3, 32, 64) (x) x = ReLU (x) x = Conv2d(3 × 3, 64, 64) (x) x = ReLU (x) y
=
Embedding(N, 64) (y)
y = Softmax(dim=-1) (y) y
=
y * y.shape[-1] x = x * y x = Maxpool(2, 2) (x) x = FC(2304, 1024) (x) x = ReLU(x) out = FC(1024, 1) (x) (d) EBM:
End Fix Softmax (V4)
Input: x, y x = Conv2d(3 × 3, 3, 32) (x) x = ReLU (x) x = Conv2d(3 × 3, 32, 32) (x) x = ReLU (x) x = Maxpool(2, 2) (x) x = Conv2d(3 × 3, 32, 64) (x) x = ReLU (x) x = Conv2d(3 × 3, 64, 64) (x) x = ReLU (x) x = Maxpool(2, 2) (x) x = FC(2304, 1024) (x) y
=
Random Projection (y) y
= Softmax(dim=-1) (y) y
=
y * y.shape[-1] x = x * y out = FC(1024, 1) (x) (e) EBM:
End [Table] (V4)
Input: x, y x = Conv2d(3 × 3, 3, 32) (x) x = ReLU (x) x = Conv2d(3 × 3, 32, 32) (x) x = ReLU (x) x = Maxpool(2, 2) (x) x = Conv2d(3 × 3, 32, 64) (x) x = ReLU (x) x = Conv2d(3 × 3, 64, 64) (x) x = ReLU (x) x = Maxpool(2, 2) (x) x = FC(2304, 1024) (x) y =
Embedding(N, 1024) (y)
y = Softmax(dim=-1) (y) y
=
y * y.shape[-1] x = x * y out = FC(1024, 1) (x) (f) Baseline models
Input: x x = Conv2d(3 × 3, 3, 32) (x) x = ReLU (x) x = Conv2d(3 × 3, 32, 32) (x) x = ReLU (x) x = Maxpool(2, 2) (x) x = Conv2d(3 × 3, 32, 64) (x) x = ReLU (x) x = Conv2d(3 × 3, 64, 64) (x) x = ReLU (x) x = Maxpool(2, 2) (x) x = FC(2304, 1024) (x) x = ReLU (x) x = FC(1024, 1024) (x) x = ReLU (x) out = FC(1024, 10) (x) previous tasks and revisit them when training on new tasks.
Although straightforward, such methods face critical non-trivial questions, such as how to select the data to be stored and how to use them (Lopez-Paz & Ranzato, 2017; Hou et al., 2019; Wu et al., 2019; Mundt et al., 2020).
An alternative
[Table]
= Conv2d(3 × 3, 3, 16) (x) x = BatchNorm (x) x = ReLU (x) x
= Conv2d(3 × 3, 16, 32) (x) x = BatchNorm (x) x = ReLU (x) x = Conv2d(3 × 3, 32, 64) (x) x = BatchNorm (x) x = ReLU (x) x = Conv2d(3 × 3, 64, 128) (x) x = BatchNorm (x) x = ReLU (x) x = Conv2d(3 × 3, 128, 256)
(x) x = FC(1024, 2000)
(x) x = ReLU (x) x = BinaryMask
(x, y) x = FC(2000, 2000) (x) x = ReLU (x) x =
BinaryMask (x, y) out = FC(2000, 1) (x)
[Table]
Conv2d(3 ×
[Table]
x = BatchNorm (x) x = ReLU (x) x = Conv2d(3 × 3, 16, 32) (x) x = BatchNorm (x) x = ReLU (x) x = Conv2d(3 × 3, 32, 64) (x) x = BatchNorm (x) x = ReLU (x) x = Conv2d(3 × 3, 64, 128) (x) x = BatchNorm (x) x =
[Table]
is to generate the replayed data.
In (Shin et al., 2017), a generative model is sequentially trained to generate samples of previous tasks.
While both types of replay can mitigate forgetting, an important disadvantage is that they are computationally relatively expensive.
Additionally, storing data might not always be possible while incrementally training a generative model is a challenging problem in itself (Lesort et al., 2019; van de Ven et al., 2020).
In contrast, we propose EBMs for continual learning that reduce catastrophic forgetting without requiring knowledge of task-identity, without gradually restricting the model’s learning capabilities, and without using stored data.