E NERGY -B ASED M ODELS FOR C ONTINUAL L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
We motivate Energy-Based Models (EBMs) as a promising model class for continual learning problems.
Instead of tackling continual learning via the use of external memory, growing models, or regularization, EBMs have a natural way to support a dynamically-growing number of tasks and classes and less interference with old tasks.
We show that EBMs are adaptable to a more general continual learning setting where the data distribution changes without the notion of explicitly delineated tasks.
We also ﬁnd that EBMs outperform the baseline methods by a large margin on several continual learning benchmarks.
These observations point towards EBMs as a class of models naturally inclined towards the continual learning regime.
I NTRODUCTION
Humans are able to rapidly learn new skills and continuously integrate them with prior knowledge.
The ﬁeld of Continual Learning (CL) seeks to build artiﬁcial agents with the same capabilities (Parisi et al., 2019).
In recent years, CL has seen increased attention, particularly in the context of classiﬁcation problems.
A crucial characteristic of continual learning is the ability to learn new data without forgetting prior data.
Models must also be able to incrementally learn new skills, without necessarily having a notion of an explicit task identity.
However, standard neural networks (He et al., 2016; Simonyan & Zisserman, 2014; Szegedy et al., 2015) experience the catastrophic forgetting problem and perform poorly in this setting.
Different approaches have been proposed to mitigate catastrophic forgetting, but many rely on the usage of external memory (Lopez-Paz &
Ranzato, 2017; Li & Hoiem, 2017), additional models (Shin et al., 2017), or auxiliary objectives and regularization (Kirkpatrick et al., 2017; Schwarz et al., 2018; Zenke et al., 2017; Maltoni &
Lomonaco, 2019), which can constrain the wide applicability of these methods.
In this work, we focus on classiﬁcation tasks.
These tasks are usually tackled by utilizing normalized probability distribution (i.e., softmax output layer) and trained with a cross-entropy objective.
In this paper, we argue that by viewing classiﬁcation from the lens of training an un-normalized probability distribution, we can signiﬁcantly improve continual learning performance in classiﬁcation problems.
In particular, we interpret classiﬁcation as learning an Energy-Based Model (EBM) across seperate classes (Grathwohl et al., 2019).
Training becomes a wake-sleep process, where the energy of an input data and its ground truth label is decreased while the energy of the input and another selected class is increased.
This offers freedom to choose what classes to update in the CL process.
By contrast, the cross entropy objective reduces the likelihood of all negative classes when given a new input, creating updates that lead to forgetting.
The energy function, which maps a data and class pair to a scalar energy, also provides a way for the model to select and ﬁlter portions of the input that are relevant towards the classiﬁcation on hand.
We show that this enables EBMs training updates for new data to interfere less with previous data.
In particular, our formulation of the energy function allows us to compute the energy of a data by learning a conditional gain based on the input label which serves as an attention ﬁlter to select the most relevant information.
In the event of a new class, a new conditional gain can be learned.
These unique beneﬁts are applicable across a range of continual learning tasks.
Most existing works (Kirkpatrick et al., 2017; Zhao et al., 2020) toward continual learning typically learn a sequence of distinct tasks with clear task boundaries ( Boundary-Aware ).
Many of these methods depend on knowing the task boundaries that can provide proper moments to consolidate knowledge.
However, this scenario is not very common in the real world, and a more natural scenario is the Boundary-
Agnostic setting (Zeno et al., 2018; Rajasegaran et al., 2020), in which data gradually changes without a clear notion of task boundaries.
This setting has also been used as a standard evaluation in the continual reinforcement learning (Al-Shedivat et al., 2017; Nagabandi et al., 2018).
Many common
CL methods are not applicable to the Boundary-Agnostic scenario as the task boundaries are unknown or undeﬁned.
In contrast, EBMs are readily applied to this setting without any modiﬁcation and are able to support both Boundary-Aware and Boundary-Agnostic settings.
There are four primary contributions of our work.
First, we introduce energy-based models for classiﬁcation CL problems in both boundary-aware and boundary-agnostic regimes.
Secondly, we use the standard contrastive divergence training procedure and show that it signiﬁcantly reduces catastrophic forgetting.
Thirdly, we propose to learn new conditional gains during the training process which makes EBMs parameter updates cause less interference with old data.
Lastly, we show that in practice EBMs bring a signiﬁcant improvement on four standard CL benchmarks, split MNIST, permuted MNIST, CIFAR-10, and CIFAR-100.
These observations towards EBMs as a class of models naturally inclined towards the CL regime.
R ELATED WORK 2. C ONTINUAL LEARNING SETTINGS
Boundary-aware versus boundary-agnostic .
In most existing continual learning studies, models are trained in a “boundary-aware” setting, in which a sequence of distinct tasks with clear task boundaries is given (e.g., Kirkpatrick et al., 2017; Zenke et al., 2017; Shin et al., 2017).
Typically there are no overlapping classes between any two tasks; for example task 1 has data with ground truth class labels “1,2” and task 2 has data with ground truth class labels “3,4”.
In this setting, models are ﬁrst trained on the entire ﬁrst task and then move to the second one.
Moreover, models are typically told when there is a transition from one task to the next.
However, it could be argued that it is more realistic for tasks to change gradually and for models to not be explicitly informed about the task boundaries.
Such a boundary-agnostic setting has been explored in (Zeno et al., 2018; Rajasegaran et al., 2020; Aljundi et al., 2019).
In this setting, models learn in a streaming fashion and the data distributions gradually change over time.
For example, the percentage of “1s” presented to the model might gradually decrease while the percentage of “2s” increases.
Importantly, most existing continual learning approaches are not applicable to the boundary-agnostic setting as they require the task boundaries to decide when to consolidate the knowledge (Zeno et al., 2018).
In this paper, we will show that our proposed approach can also be applied to the boundary-agnostic setting.
Task-incremental versus class-incremental learning .
Another important distinction in continual learning is between task-incremental learning and class-incremental learning (van de Ven & Tolias, 2019;
Prabhu et al., 2020).
In task-incremental learning, also referred to as the multi-head setting (Farquhar & Gal, 2018), models have to predict the label of an input data by choosing only from the labels in the task where the data come from.
On the other hand, in class-incremental learning, also referred to as the single-head setting, models have to chose between the classes from all tasks so far when asked to predict the label of an input data.
Class-incremental learning is substantially more challenging than task-incremental learning, as it requires models to select the correct labels from the mixture of new and old classes.
So far, only methods that store data or use replay have been shown to perform well in the class-incremental learning scenario (Rebufﬁ et al., 2017; Rajasegaran et al., 2019).
In this paper, we try to tackle class-incremental learning without storing data and replay.
2. C ONTINUAL LEARNING APPROACHES
In recent years, numerous methods have been proposed for CL.
Here we broadly partition these methods into three categories: task-speciﬁc, regularization, and replay-based approaches.
Task-speciﬁc methods.
One way to reduce interference between tasks is by using different parts of a neural network for different problems.
For a ﬁxed-size network, such specialization could be achieved by learning a separate mask for each task (Fernando et al., 2017; Serra et al., 2018), by a priori deﬁning a different, random mask for every task to be learned (Masse et al., 2018) or by using a different set of parameters for each task (Zeng et al., 2019;
Hu et al., 2019).
Other methods let a neural network grow or recruit new resources when it encounters new tasks, examples of which are progressive neural networks (Rusu et al., 2016) and dynamically expandable networks (Yoon et al., 2017).
Although these task-speciﬁc approaches are generally successful in reducing catastrophic forgetting, an important disadvantage is that they require knowledge of the task identity at both training and test time.
These methods are therefore not suitable for class-incremental learning.
Regularization-based methods.
Regularization is used in continual learning to encourage stability of those aspects of the network that are important for previously learned tasks.
A popular strategy is to add a regularization loss to penalise changes to model parameters that are important for previous tasks.
EWC (Kirkpatrick et al., 2017)) and online EWC (Schwarz et al., 2018) evaluate the importance of each parameter using the diagonal elements in the ﬁsher information matrices, while SI (Zenke et al., 2017) tracks the past and current parameters and estimates their importance online.
An alternative strategy is to regularize the network at the functional level.
Learning without
Forgetting (Li & Hoiem, 2017) uses knowledge distillation to encourage stability of the network’s learned input-output mapping.
However, these regularization-based approaches gradually reduce the model’s capacity for learning new tasks and they have been shown to consistently fail in the class-incremental learning (Farquhar & Gal, 2018; van de Ven & Tolias, 2019).
Replay methods.
To preserve knowledge, replay methods periodically rehearse previously acquired information during training (Robins, 1995).
One way to do this, referred to as exact or experience replay, is to store data from previous tasks and revisit them when training on a new task.
Although this might seem straightforward, critical non-trivial questions are how to select the data to be stored as well as exactly how to use them (Rebufﬁ et al., 2017; Lopez-Paz & Ranzato, 2017; Rajasegaran et al., 2019;
Hou et al., 2019; Wu et al., 2019; Mundt et al., 2020).
An alternative to storing data is to generate the data to be replayed.
In such generative replay (Shin et al., 2017), a generative model is sequentially trained to generate input samples representative of those from previously seen tasks.
While both types of replay can prevent catastrophic forgetting in both task- and class-incremental learning settings, an important disadvantage is that these methods are computationally relatively expensive as each replay event requires at least one forward and one backward pass through the model.
In addition, storing data might not always be possible while incrementally training a generative model is a challenging problem in itself (Lesort et al., 2019; van de Ven et al., 2020).
In contrast, our EBMs reduce catastrophic forgetting without requiring knowledge of task-identity, without gradually restricting the model’s learning capabilities and without using stored data.
B ACKGROUND
In this section, we will introduce traditional continual learning methods, which are typically modiﬁed based on the feed-forward classiﬁer.
We will show the limitation of using such structures and how EBMs can be applied to these problems in the next section.
3. C ONTINUAL L EARNING WITH FEED - FORWARD CLASSIFIER
Deﬁne T as a classiﬁcation task with inputs x and associated discrete classiﬁcation labels y ∈ Y ,
where Y = { 1 , . . .
, N } contains N classes.
Traditional classiﬁer approaches to continual learning typically predict the probabilities of all the N classes using a feed-forward network
[Equation]
The input is a data x ∈ R D and the output is probabilities of all classes p θ ∈ R N .
f
( x ) : R D
→ R N is the feed-forward function that maps data x into a N -dimensional vector.
The ﬁnal layer in such classiﬁer networks is predeﬁned to predict up to N different classes.
Existing continual learning methods (Kirkpatrick et al., 2017; Masse et al., 2018) tackle the continual learning problem typically using a feed-forward classiﬁer.
However, such classiﬁer architectures do not perform well in the Class-IL setting, where models have to predict the label of a given data from all classes (Tao et al., 2020a;
He et al., 2018).
The standard feed-forward classiﬁer computes the
Softmax and cross-entropy loss over all seen classes.
When training on a new task, they improve the likelihood of ground truth classes but suppress the likelihood of old classes since they are all negative classes for data in the current task.
This operation introduces competitive, winner-take-all dynamics that impact the strength of old classes and thus make the classiﬁer forget past tasks.
3. E NERGY - BASED MODELS
Energy based models (LeCun et al., 2006) are a class of maximum likelihood models that deﬁne the likelihood of a data point x ∈ R D using the Boltzmann distribution
[Equation]
where E θ ( x ) : R D → R , known as the energy function, maps each input data to a scalar, and Z ( x ) is the partition function over all data points.
EBMs are a powerful generative models that have been applied to different tasks, such as structured prediction (Belanger & McCallum, 2016; Gygli et al., 2017; Rooshenas et al., 2019; Tu &
Gimpel, 2019), machine translation (Tu et al., 2020), and image generation (Xie et al., 2016; 2018;
Grathwohl et al., 2019).
(Belanger & McCallum, 2016) introduce energy networks for structured prediction.
They propose energy networks can be naturally posed as structured prediction, since the labels exhibit rich interaction structure.
Tu & Gimpel (2019) propose a structured energy function for sequence labeling tasks.
Rooshenas et al.
(2019) introduce a search-guided approach to obtain training pairs using a combination of the energy function and reward function that allows the lightly-supervised training of structured prediction energy networks.
Tu et al.
(2020) treat the nonautoregressive translation system as an inference network that ﬁnds the translation by minimizing the energy of a pretrained autoregressive model.
Xie et al.
(2016) is the ﬁrst paper to use ConvNetparameterized EBMs with Langevin dynamics for image generation.
They show that a generative random ﬁeld model can be derived from the commonly used discriminative ConvNet.
(Xie et al., 2018) cooperatively train the descriptor and generator using MCMC sampling.
C ONTINUAL L EARNING WITH E NERGY -B ASED M ODELS
In this section, we describe how EBMs can be used for classiﬁcation and we discuss how they can overcome the above mentioned problems in the continual learning setting.
Note our analysis is based on the Class-IL setting which is one of the most natural and also the hardest settings used in
CL (van de Ven & Tolias, 2019; He et al., 2018; Tao et al., 2020b).
4. E NERGY
Different from existing works using EBM for structured prediction (Belanger & McCallum, 2016;
Gygli et al., 2017; Rooshenas et al., 2019), such as multi-label classiﬁcation tasks, CL does not access previous data while training on new data and thus the biggest challenge is to prevent catastrophic forgetting.
The main focus of this paper is to mitigate the catastrophic forgetting in CL without using replay buffers to store past data or models.
To make the EBM suitable for classiﬁcation, we use the Boltzmann distribution to deﬁne the likelihood of y conditioned on x :
[Equation]
where E θ ( x , y ) : ( R D , N ) → R is the energy function that maps x and y to a scalar energy value, and Z ( y | x ) is the partition function that ensures (cid:80)
y p θ ( y | x ) = 1 for any given x .
To minimize the negative log likelihood of the ground truth label y + , inspired by the contrastive divergence proposed by (Hinton, 2002), we minimize the energy of x at the ground truth label y
+ while increasing the energy of x at other labels.
We use the following loss and its corresponding gradient (see Appendix B for the derivation):
[Equation]
whereby N is the set of ‘negative classes’.
Following common practice in contrastive divergence, we sample the set of negative classes N from the set of class labels in the current batch Y B , using only a single negative class per training sample.
We note however that it is possible to use different strategies for choosing the negative classes, and in the experiments reported in the top half of Tablewe explore alternative strategies.
If there are batches that contain only a single class label, sampling a negative class from the current batch does not work, but one could instead select whichever other class was seen most recently as a negative class, or one could sample from all other classes seen so far.
Another solution might be to always use a fake label representing “not exist” as negative sample.
As opposed to the classiﬁer architectures that output a ﬁxed N -dimensional probability vector of all classes, this formulation thus provides freedom in the choice of which class labels to train on.
This is important because an important issue with standard classiﬁer models is that they suppress the strength of old classes and thus make the classiﬁer forget past tasks as mentioned in Section 3.1.
In contrast, EBMs deﬁne the probability without normalizing over all classes but instead sample a limited number of negative classes.
This operation cause less interference with old classes without using any extra information which contributes to why EBMs suffer less from catastrophic forgetting.
[Figure]
Another advantage of our EBMs is that the choice of model architectures also becomes more ﬂexible.
The traditional classiﬁcation models can only feed in x as input and the classiﬁcation results only depend on the neural network trained on the current data x .
Thus the models usually fail on the old data after updating network parameters trained on the new data.
In contrast, EBMs have many different ways to combine x and y in the energy function E θ ( x , y ) with the only requirement that E θ ( x , y ) : ( R D , N ) → R .
In EBMs, we can treat y as an attention ﬁlter or gate to select the most relevant information between x and y as described in Section 4.2.
Since the negative sample can be sampled from the current batch,
EBMs do not require knowledge of the task on hand, allowing applications across different continual learning scenarios, including those when the underlying task is unclear.
More broadly speaking,
EBMs can go out of the scope of classiﬁcation problems, and similar ideas can be applied to regression (Gustafsson et al., 2020), generation (Du & Mordatch, 2019;
Du et al., 2020), and reinforcement learning tasks (Parshakova et al., 2019), although we do not explore them in this work.
4. E NERGY NETWORK .
To compute the energy of any data x and class label y pair, we use y to inﬂuence a conditional gain on x , which serves as an attention ﬁlter (Xu et al., 2015) to select the most relevant information between x and y .
In Figure 1, we ﬁrst send x into a small network to generate the feature f ( x ) .
The label y is mapped into a same dimension feature space g ( y ) using a FC layer or a random projection.
We use the Softmax function over each channel of g ( y ) and combine it with x :
[Equation]
where f c and g c are the c th channel of f ( x ) and g ( y ) respectively.
The output is ﬁnally sent to a fully connected layer to generate the energy value E θ ( x , y ) .
For more details on our model architectures, see Appendix C.
Our EBMs allow any number of classes in new batches by simply training or deﬁning a new conditional gain g ( y ) for the new classes and generating its energy value with data point x .
This formulation gives us freedom to learn new classes without pre-deﬁning their number in advance.
We note that also in a standard classiﬁer new classes can be dynamically added, but there new class heads need to be added to the softmax output layer.
Class inference.
During inference, the model must predict a class label from all classes seen so far.
let x k be one data point from a batch B k with an associated discrete label
y ∈
Y k , where Y k contains classes in B k .
There are Y =
(cid:83) Kk =1 Y k different classes in total after seeing all the batches.
The MAP estimate is
[Equation]
(7) where E θ K ( x k , y ) is the energy function with parameters θ K resulting from training on the batches { B 1 , · · · , B K } .
The energy function can compute an energy for any discrete class input, including unseen classes.
This avoids needing to predeﬁne the number of classes in advance, as is necessary with traditional CL models.
E XPERIMENTS
In this section, we want to investigate several questions, including how does the proposed EBMs perform on different CL settings?
Can we qualitatively understand the differences between EBMs and baselines?
Is the proposed EBM training objective better than other objectives?
Can we apply the
EBM training objective to other baseline methods?
And ﬁnally what is the the best architecture for
[Table]
[Figure]
label conditioning that causes the minimum interference with old data?
To answer these questions, we ﬁrst introduce experiments on the Boundary-Aware setting which typically learn a sequence of distinct tasks with clear task boundaries in Section 5.1.
We then show that EBMs can be naturally applied to the Boundary-Agnostic setting where the data gradually changing in a steaming fashion without the notion of task boundaries in Section 5.2.
5. E XPERIMENTS ON B OUNDARY -A WARE S ETTING 5.1. D ATASETS AND EVALUTION PROTOCOLS
Datasets.
We evaluate the proposed EBMs on the split MNIST (Zenke et al., 2017), permuted
MNIST (Kirkpatrick et al., 2017), CIFAR-10 (Krizhevsky et al., 2009), and CIFAR-100 (Krizhevsky et al., 2009) datasets.
We follow existing approaches and separate the datasets into several splits.
The split MNIST dataset is obtained by splitting the original MNIST dataset (LeCun et al., 1998) into 5 tasks with each task has 2 classes.
It has 60,000 training images and 10,000 test images.
The permuted MNIST has 10 tasks, each task with 10 classes.
For each task, the original images pixels are randomly permuted to generate 32 × 32 images.
We separate CIFAR-10 into 5 tasks, each task with 2 classes.
There are 50,000 training images and 10,000 test images.
Similarly, CIFAR-100 is split into ten tasks with each task has 10 classes.
Evaluation protocols.
Task-incremental Learning ( Task-IL ), Domain-incremental Learning ( Domain-IL ), and Class-incremental Learning ( Class-IL ) are three common evaluation metrics used by existing works (van de Ven & Tolias, 2019; Prabhu et al., 2020).
Most approaches perform fairly good on the ﬁrst two simpler settings, but fail on the Class-IL setting, which is considered as the most natural and also the hardest setting for continual learning (Tao et al., 2020a;
He et al., 2018;
Tao et al., 2020b).
Class-IL predicts the label of a given data from all seen classes.
In this paper, we consider Class-IL as our evaluation metric.
5.1. C OMPARISONS WITH EXISTING METHODS
Due to the difﬁculty on the class incremental setting, most approaches rely an external quota of memory.
Such replay based methods (Shin et al., 2017;
Rebufﬁ et al., 2017) use extra memory, larger or extra models, and are computationally relatively expensive.
In this paper, we focus on continual learning without using replay and without storing data.
We compare the proposed method with available baseline models that do not use replay or stored data, including a standard classiﬁcation model (CLS), EWC (Kirkpatrick et al., 2017), Online EWC (Schwarz et al., 2018), SI (Zenke et al., 2017), LwF (Li & Hoiem, 2017), MAS (Aljundi et al., 2019), and BGD (Zeno et al., 2018).
The continual learning results on four datasets are shown in
Table 1.
All the baselines and EBMs are based on the same model architecture.
For split MNIST and permuted MNIST, we use several fully-connected layers as in (van de Ven & Tolias, 2019).
For CIFAR-10 and CIFAR-100, we use the convolutional network (see Appendix C).
For all the baselines and EBMs on CIFAR-100, we pre-train the convolutional layers on CIFAR-10 and only ﬁne-tune the fully-connected layers.
Similar training regimes are used for the EBMs and baselines.
On the split MNIST, permuted
MNIST, and CIFAR-10 datasets, we train for 2000 iterations per task.
On the CIFAR-100 dataset we train for 5000 iterations per task.
For all experiments, we use the adam optimizer with learning rate 1 e − 4 .
Each experiment in Table 1 is run 20 times with different random seeds, with results reported as the mean ± SEM.
EBMs have a signiﬁcant improvement on all the datasets, showing that EBMs forget less when updating models for new tasks.
5.1. Q UALITATIVE ANALYSIS
Energy landscape.
To better understand why EBMs suffer less from catastrophically forgetting, we qualitatively compare the change in energy landscapes of EBMs and baseline methods as the learning progresses.
We show the energy landscapes after training on task 9 and task 10 on the permuted MNIST dataset in Figure 2.
For the classiﬁer, we show its negative probabilities on all classes.
Each datapoint has 100 energy values (EBM) or probabilities (CLS) corresponding to the 100 labels in the dataset and the values are normalized over 100 classes for each datapoint.
The dark diagonal elements indicate the model predictions are correct for both old and new data.
After training on task T 9 , CLS assigns high probabilities to classes from task T 9 (80-90) for almost all the data from task T 1 to T 9 .
However, the highest probabilities shift to classes from task T 10 (90-100) after learning task T 10 .
This means classiﬁer tends to assign high probabilities to new classes for both old and new data, indicating forgetting.
EBM on the other hand, tends to have low energies across the diagonal, which means that after training on new tasks, EBM still assigns low energies to true labels of previous data.
This indicates that EBM is better at learning new tasks without catastrophically forgetting of old tasks.
Label Distributation (Class-IL)
splitMNIST
CLS
EBM    5  9  131 2 34512 3 45 6 712 3 47
Label Distributation (Class-IL)
splitMNIST
CLS
EBM    5  9  131 2 34512 3 45 6 712 3 47
Label Distributation (Class-IL)
splitMNIST
CLS
EBM    5  9  131 2 34512 3 45 6 712 3 47
Label Distributation (Class-IL)
splitMNIST
CLS
EBM    5  9  131 2 34512 3 45 6 712 3 47
Label Distributation (Class-IL)
splitMNIST
CLS
EBM    5  9  131 2 34512 3 45 6 712 347 8 910 EBM
Label Distributation (Class-IL)
splitMNIST
CLS
EBM    5  9  131 2 34512 3 45 6 712 3 47
Label Distributation (Class-IL)
splitMNIST
CLS
EBM    5  9  131 2 34512 3 45 6 712 3 47
L a b e l
D i s t
r
i bu t a t i o n (
C l a ss - I L ) s p
lit M
N I S T
C L S E B M 13  791 2 1  12 3 45 6 712 3 4 56 7 8 splitMNIST
CLS
EBM    5      131 2 12 3 45712 3 47 8    9 10 5 6 CLS
Predicted class distribution.
In Figure 3, we plot the proportional distribution of all seen classes so far on the splitMNIST dataset.
Taking the second ﬁgure in the ﬁrst row as an example, it shows the distribution of predicted labels after training on the second task which means there are four seen classes { 1 , 2 , 3 , 4 } .
For each image from the ﬁrst and second task, we compute its top-1 class prediction and accumulate the prediction on each class to obtain the proportional distribution graph over all seen classes.
Since the number of images in each class are almost the same, the ground truth proportional distribution should be uniform over all the classes.
CLS can predict the ﬁrst two classes uniformly after training on task(ﬁrst column).
However, after learning new tasks, classes from the new task have larger proportions than old classes which means the model tends to predict new classes and forget the old classes.
The last ﬁgure in row one gives very high proportions for classes from the last task but very low proportions for previously classes.
CLS almost forgets all previous tasks and only predict classes from the last task even the input images are from old tasks.
EBM covers all seen classes, indicating memorizing the old task and preventing forgetting.
We provide more analysis about EBMs and baselines from different perspectives in the Appendix, including the confusion matrix in Appendix A.2, the model capacity in Appendix A.3, and parameter importance in Appendix A.
. 5.1. I S THE STRONG PERFORMANCE OF EBM S DUE TO THE E NERGY T RAINING O BJECTIVE OR DUE TO THE L ABEL C ONDITIONING ?
Effect of energy training objective.
We conduct an experiment on the CIFAR-100 dataset to investigate how different training objectives inﬂuence the CL results.
In Equation 4, the negative labels y
i ares sampled from Y B .
We test three different sampling strategies in Table 2.
The ﬁrst one uses all seen classes so far as negative labels which is similar to the traditional feed-forward classiﬁers ( All
Neg Seen (V4) ).
The second one takes all the classes in the current batch as negative labels ( All Neg
Batch (V4) ).
The last one randomly selects one class from the current batch as the negative as we reported in Section 4.1 ( 1 Neg Batch (V4) ).
Note that the negative labels do not include the ground truth class in all these three strategies.
We found using only one negative sample generates the best result and using negatives sampled from classes in the current batch is better than from all seen classes.
This conclusion is consistent with our analysis in Section 4.1.
Since the loss in Equationaims at improving the energy of negative samples while decreasing the energy of positive ones, sampling negatives from the current batch has less interference with previous classes than sampling from all seen classes.
Using a single negative causes the minimum suppression on negative samples and thus has the best result.
Overall, our results indicate that surprisingly, and counterintuitively, directly optimizing the cross-entropy loss used by existing approaches may not be the best way for continual learning.
Effect of label conditioning.
Next
[Table]
we test whether the label conditioning in our EBMs is important for their performance.
We test this by modifying the training objective of a standard classiﬁer to that of our proposed energy objective, which is the same as running our EBMs without the label conditioning.
The results of this comparison are listed in Table 3.
The new training objective does not suppress the probability of old classes when improving the probability of new classes and thus achieves better results.
However, EBMs still outperform the baselines, implying that the label conditioning architecture also contributes to why EBMs suffer less from catastrophic forgetting.
We also show the testing accuracy curve in Appendix A.
[Table]
To summarize
[Table]
we showed that the strong performance of our EBMs is due to both the energy training objective and the label conditioning
. 5.1. C OMPARISON OF DIFFERENT ENERGY NETWORK ARCHITECTURES
[Table]
EBMs allow ﬂexibility in integrating data information and label information in the energy function.
To investigate where and how to combine the information from x and y , we conduct a series of experiments on the CIFARdataset.
Table 2 shows four model architectures (V1-V4) that combine x and y in the early, middle, and late stages, respectively (see Appendix C for the architecture details).
We ﬁnd combining x and y in the late stage (V4) performs the best.
We note that instead of learning a feature embedding of label y , we can use a ﬁxed projection matrix which is randomly sampled from the uniform distribution U (0 , 1) .
Even using this ﬁxed random projection can already generates better results than most baselines in Table 1.
Note further that the number of trainable parameters in the “Fix” setting is much lower than that of the baselines.
Using a learned feature embedding of y can further improve the performance.
We may also apply different normalization methods over the feature channel of y .
We ﬁnd that Softmax ( End Softmax (V4) ) is better than the
L2 normalization ( End Norm2 (V4) ) and no normalization ( End (V4) ).
5.1. I NTERFERENCE WITH PAST DATA
Here we test the effects of label conditioning and the energy training objective in a different way, by evaluating their ability to prevent interference with past data when learning classiﬁcation on new classes.
Formally, let x i and x j be two data points and θ i be the model parameters after training on x i .
The model parameters change (cid:52) θ ( x j ) after learning new data x j .
We test the difference between the losses L ML ( x i | θ i ) and L ML ( x i | θ i + (cid:52) θ ( x j )) .
Ideally, we expect L ML ( x i | θ i + (cid:52) θ ( x j )) ≤
L ML ( x i | θ i ) for x
i (cid:54) =
x j , which means the update on new data has no inﬂuence or positive inﬂuence on the old data.
The ﬁrst-order expansion gives L ML ( x i | θ i + (cid:52) θ ( x j ) )
≈ L ML ( x i | θ i ) + ∇ θ i L ML ( x i | θ i ) T (cid:52) θ ( x j ) .
To make our desired equality hold, the gradient at x i should be ∇ θ i L ML ( x i | θ i ) T (cid:52) θ ( x j ) ≤ 0 .
φ = ∇ θ i L ML ( x i | θ i ) T (cid:52) θ
( x j ) can be used to measure the inﬂuence of updating models on new data on the performance of old data.
The smaller the value is, the less negative inﬂuence on old data, and thus the better the model is in preventing forgetting.
We compare EBMs with the standard classiﬁer (CLS) and CLS using our EBM training objective (CLS*) (see Appendix A.5 for the implementation details).
For results on split MNIST, the average value of φ over all tasks of CLS, CLS*, and EBMs are φ CLS = 0 .
168 , φ CLS* = 0 .
004 , and φ EBM = − 3 .
743 e − 5 , respectively.
For CIFAR-10, the results are φ CLS = 1 . 104 , φ CLS* = 0 . 043 , and φ EBM = 4 .
414 e − 5 , respectively.
We found EBMs have smaller φ than CLS and CLS*, which is consistent with our analysis that the conditional gain g ( y ) serves as an attention ﬁlter that could select the
[Table] [Table]
relevant information between x and y and make parameter updates cause less interference with old data, which contributes to why EBMs suffer less from catastrophic forgetting.
5. E XPERIMENTS ON B OUNDARY -A GNOSTIC S ETTING
When applying continual learning in real life, boundaries are not usually well deﬁned between different tasks.
Most existing continual learning methods are often tailored to the problem setup where a ﬁxed boundary between tasks is deﬁned, and must be informed when the tasks are switched during training.
We show that EBMs are able to ﬂexibly perform continual learning across different problem setups, and perform well on the Boundary-Agnostic setting as well.
5.2. D ATASETS AND EVALUTION PROTOCOLS
We evaluate our proposed continual learning method on the Boundary-Agnostic setting on four datasets, including split MNIST, permuted MNIST, CIFAR-10, and CIFAR-100.
We use the code of “continuous task-agnostic learning” proposed by (Zeno et al., 2018) to generate the training/testing data.
The transition between tasks goes slowly over time using a sampling function based on the probability of each task.
There is a mixture of data from two different tasks during the transition and the proportion of new data gradually increases.
Similar to the Boundary-Aware setting, We use
Class-IL as our evaluation metric for the Boundary-Agnostic continual learning.
5.2. C OMPARISON WITH EXISTING METHODS
We focus on the investigation on the performance of different model architectures with similar model size and memory footprint.
We do not directly compare with replay-based methods as they use extra memory, larger or extra models.
Since there is no knowledge on the number of tasks, previous methods of continual learning that rely on task boundaries are generally inapplicable.
One trivial adaptation is to take the core action after every batch step instead of every task.
However, doing such adaptation is impractical for most algorithms, such as EWC, because of the large computational complexity.
We successfully get the result of CLS, Online EWC, SI, and BGD baselines.
Overall classiﬁcation results on four datasets are shown in Table 4.
All the baselines and EBM are based on the same model architecture as in the Boundary-Aware setting.
Each experiment was performed 5 times with different random seeds, the results are reported as the mean ± SEM over these runs.
We observe that EBMs have a signiﬁcant improvement on all the datasets.
EBMs can be naturally applied to both the Boundary-Aware and BoundaryAgnostic settings without any modiﬁcation, as EBMs have a very ﬂexible training objective.
The experiments demonstrate EBMs have good generalization ability for different continual learning problems.
EBMs can naturally handle different number of new classes, new tasks, data streams with and without clear task boundaries.
C ONCLUSION
In this paper, we found that energy-based models are a promising class of models towards a variety of different continual learning settings.
We demonstrated that EBMs exhibit many desirable characteristics to prevent catastrophic forgetting in CL, and we experimentally showed that EBMs obtain the state-of-the-art performance on the Class-IL and Boundary-Agnostic settings on multiple benchmarks.
A A DDITIONAL A NALYSES
Extending the results presented in Section 5.1, here we further compare EBMs with the baseline models by providing additional quantitative analyses of their performance.
We show testing accuracy curves in Section A.1, confusion matrices between the ground truth labels and model predictions in Section A.2, model capacity comparisons in Section A.3, and parameter importance measurement in Section A.4.
In Section A.5, we provide more details of the label conditioning analysis mentioned in Section 5.1.5.
Finally, in Section A.6 we perform the split CIFAR-100 protocol with several different number of tasks to test the generality of the proposed EBMs.
A. T ESTING ACCURACY CURVE
In Section 5.1.4, we show that the proposed EBM training objective is also applicable to baseline approaches and improves their performance signiﬁcantly on the Class-IL setting.
Figure 4 shows the testing accuracy of each task as the training progresses.
We compare the standard classiﬁer (CLS), classiﬁer using our training objective (CLS*), and our EBMs.
The left ﬁgures show results on the split MNIST dataset while the right ﬁgures show results on the permuted MNIST dataset.
We observe that the accuracy of old tasks in CLS drop sharply when learning new tasks, while the
EBM training objective can mitigate the forgetting problem.
The curve on EBMs drops even slower than CLS and CLS*, implying the proposed energy objective can mitigate the catastrophic forgetting problem.
[Figure]
A. C LASS CONFUSION MATRIX AT THE END OF LEARNING
In Section 5.1.3, we show the qualitative analysis of EBMs and baselines.
Except the energy landscape and predicted class distribution, we also generate the confusion matrices of EBMs and baseline approaches.
Confusion matrix shows the relationship between ground truth labels and predicted labels.
In Figure 5, we draw the confusion matrices after training on all the tasks on the split MNIST dataset and permuted MNIST dataset.
The classiﬁer tends to predict the classes from the last task (class 8, 9 for split MNIST and 90-100 for permuted MNIST).
EBMs have high values along the diagonal that means the predicted results matching the ground truth labels for all the sequentially learned tasks.
This demonstrates that EBMs are better at learning new tasks without catastrophically forgetting of old tasks.
[Figure]
A. M ODEL CAPACITY
Another hypothesized reason for why EBMs suffer less from catastrophic forgetting than standard classiﬁers is potentially their larger effective capacity.
To analyze effective capacity of our models, we test the model capacity of the standard classiﬁer and EBMs on both the generated images and natural images.
Model capacity on generated images.
We generate a large randomized dataset of 32 × 32 images with each pixel value uniformly sampled from -1 to 1.
Each image is then assigned a random class label between 0 and 10.
We measure the model capacity by evaluating to what extent the model can ﬁt a such dataset.
For both the standard classiﬁer and the EBM, we evaluate three different sizes of models (small, medium, and large).
For a fair comparison, we control the EBM and classiﬁer have similar number of parameters.
The Small EBM and CLS have 2 , 348 , 545 and 2 , 349 , 032 parameters respectively.
The medium models have 5 , 221 , 377 (EBM) and 5 , 221 , 352 (CLS) parameters while the large models have 33 , 468 , 417 (EBM) and 33 , 465 , 320 (CLS) parameters.
We use the model architectures in Table 5a and Table 5b for EBMs and classiﬁers.
The resulting training accuracies are shown in Figure 6 with the number of data ranges from one to ﬁve millions.
Given any number of datapoints, EBM obtains higher accuracy than the classiﬁer, demonstrating that indeed EBM has larger capacity to memorize data given a similar number of parameters.
The gap between EBM and CLS increases when the models become larger.
The larger capacity of EBM potentially enables it to memorize more data and mitigate the forgetting problem.
Model capacity on natural images.
We also compare classiﬁers and EBMs on natural images from
CIFAR-10.
Each image is assigned a random class label between 0 and 10.
We use the same network architecture as in Table 5a and Table 5b, but with a hidden unit size of h = 256 .
Since there are only 50 , 000 images on CIFAR-10, we use a small classiﬁer and EBM and train them on the full dataset.
After training 100000 iterations, the EBM obtains a top-1 prediction accuracy of 82 . 81 , while the classiﬁer is 42 .
We obtain the same conclusion that EBM has larger capacity to memorize data given a similar number of parameters.
[Table]
[Equation]
[Table]
[Equation]
A. P ARAMETER IMPORTANCE
To further understand why EBMs suffer less from catastrophic forgetting, we design an experiment to test the importance of model parameters on past data.
Inspired by the elastic weight consolidation (EWC) (Kirkpatrick et al., 2017), we estimate the importance of parameters for each tasks using the diagonal elements of the Fisher information matrix (FIM) F .
Let θ i be the model parameters after training on task T i .
Given one of previous tasks T j , j <
i , we evaluate how important each parameter is for tasks T j .
The k th diagonal of F is deﬁned as the gradient on the EBM loss
[Equation]
i
[Equation]
where x is sampled from tasks T j and E ( x , y + ) is the energy value of the input data x and ground truth label y + .
y (cid:48)
∈
Y j are classes randomly selected from the current batch.
Here we use a single negative class.
The above equation assigns high values to parameters crucial to task
T
j as their gradients with respect to the loss are larger.
Since the diagonal elements of the ﬁsher information matrix measure the importance of each parameter to a given task, the density of diagonal elements represents the proportion of important parameters over all parameters.
More density means more parameters are important for the given task and less parameters can be recruited for new tasks.
Ideally, we expect these values to be sparse.
In Figure 7, we show the diagonal elements of the standard classiﬁer (CLS), classiﬁer using our training objective (CLS*), and our EBMs on the split MNIST dataset.
For CLS and CLS*, we follow (Kirkpatrick et al., 2017) to compute their ﬁsher information matrices.
For comparisons across multiple models, we normalize the FIM diagonal elements of each method to be between 0 and 1 and report the normalized results in Figure 7.
For example, “Fisher 5 on data 1” shows the diagonal elements of the Fisher information matrix obtained by Equation 8 using the model parameters θ 5 (after training on task T 5 ) and data x , y + , y (cid:48) from task T 1 .
The distribution of EBMs is sparser than CLS and CLS* indicating that EBMs have fewer important parameters for previous data.
Updating parameters for the new task will have less negative impact on old tasks.
In addition, more parameters can be used for learning new tasks if the distribution is sparse.
This may provide another explanation for why EBMs can mitigate catastrophic forgetting.
A. M ORE DETAILS OF THE I NTERFERENCE WITH PAST DATA ANALYSIS
In Section 5.1.5, we test the interference of models trained on new data with past data.
We compare the proposed EBMs with a standard classiﬁer (CLS) and CLS using our EBMs training objective (CLS*).
For CLS and CLS*, we use the cross-entropy loss to get their φ , i.e., the cross-entropy losses on the old data before and after training on a new data.
We obtain θ i by training on 200 randomly selected data from task1 and compute its gradient after training.
Then we perform a single step optimization on 200 data that are randomly selected from task2 and average their parameters to get θ j .
We compute φ using the inner product of (cid:52) θ
( x j )
=
θ j − θ
i and the gradient ∇ θ i L ML ( x i | θ i ) .
For comparisons among multiple approaches, we normalize the φ of each method by dividing its maximum element value of θ i and scale φ based on the length of θ , as CLS and EBMs have different number of parameters.
A. S PLIT CIFAR-100 WITH DIFFERENT NUMBERS OF TASKS
To test the generality of our proposed EBMs, in Table 6 we repeat the boundary-aware experiments on spit CIFAR-100 (see Table 1 in the main text) for different number of classes per task.
In Tablethe CIFAR-100 dataset was split up into 10 tasks, resulting in 10 classes per task.
Here we additionally perform the split CIFAR-100 protocol with 5 different tasks (i.e., 20 classes per task), withdifferent tasks (i.e., 5 classes per task) and with 50 different tasks (i.e., 2 classes per task).
For all settings, we ﬁnd that our EBM substantially outperforms the baselines.
[Table]
B D ERIVATION OF LOSS GRADIENT
The derivation of the loss gradient in Equation 5 is ∂ L ML
[Equation]
)) is a constant and can be moved inside the summation
[Equation]
C M ODEL ARCHITECTURES
Here we provide details of the model architectures used on the different datasets.
Images from the split MNIST and permuted MNIST datasets are grey-scale images.
The baseline models for these datasets, similar as in (van de Ven & Tolias, 2019), consist of several fullyconnected layers.
For the EBMs we use similar number of parameters.
The model architectures of
EBMs on the split MNIST dataset and permuted MNIST dataset are the same, but have different input and output dimensions and hidden sizes.
The model architectures of EBMs and baseline models on the split MNIST dataset are shown in Table 7a and Table 7b respectively.
The model architectures of EBMs and baseline models on the permuted MNIST dataset are shown in Table 8a and Table 8b.
Images from the CIFAR-10 and CIFAR-100 datasets are RGB images.
For CIFAR-10, we use a small convolutional network for both the baseline models and the EBMs.
The model architectures of EBMs on the CIFAR-10 dataset are shown in Table 9a, Table 9b, Table 9c, Table 9d, and Table 9e.
We investigate different architectures to search for the effective label conditioning on EBMs training as described in Section 5.1.5.
The baseline models on CIFAR-10 are given in Table 9f.
The model architectures used on the CIFAR-100 dataset are detailed in Table 10.
[Table]
[Equation]
[Table]
[Equation]
[Equation] [Equation]
[Table]
, y x = Conv2d(3 × 3, 3, 32) (x) x = ReLU (x) x = Conv2d(3 × 3, 32, 32) (x) x = ReLU (x) x
= Maxpool(2, 2) (x) x = Conv2d(3 × 3, 32, 64) (x) x = ReLU (x) x = Conv2d(3 × 3, 64, 64) (x) x = ReLU (x) y
=
Embedding(N, 64) (y)
y = Softmax(dim=-1) (y) y
=
y * y.shape[-1] x = x * y x = Maxpool(2, 2) (x) x = FC(2304, 1024) (x) x = ReLU(x) out = FC(1024, 1) (x) (d) EBM:
End Fix Softmax (V4)
Input: x, y x = Conv2d(3 × 3, 3, 32) (x) x = ReLU (x) x = Conv2d(3 × 3, 32, 32) (x) x = ReLU (x) x = Maxpool(2, 2) (x) x = Conv2d(3 × 3, 32, 64) (x) x = ReLU (x) x = Conv2d(3 × 3, 64, 64) (x) x = ReLU (x) x = Maxpool(2, 2) (x) x = FC(2304, 1024) (x) y
=
Random Projection (y) y
= Softmax(dim=-1) (y) y
=
y * y.shape[-1] x = x * y out = FC(1024, 1) (x) (e) EBM:
End Softmax (V4)
Input: x, y x = Conv2d(3 × 3, 3, 32) (x) x = ReLU (x) x = Conv2d(3 × 3, 32, 32) (x) x = ReLU (x) x = Maxpool(2, 2) (x) x = Conv2d(3 × 3, 32, 64) (x) x = ReLU (x) x = Conv2d(3 × 3, 64, 64) (x) x = ReLU (x) x = Maxpool(2, 2) (x) x = FC(2304, 1024) (x) y =
Embedding(N, 1024) (y)
y = Softmax(dim=-1) (y) y
=
y * y.shape[-1] x = x * y out = FC(1024, 1) (x) (
[Table]
models
Input: x x = Conv2d(3 × 3, 3, 32) (x) x = ReLU (x) x = Conv2d(3 × 3, 32, 32) (x) x = ReLU (x) x = Maxpool(2, 2) (x) x = Conv2d(3 × 3, 32, 64) (x) x = ReLU (x) x = Conv2d(3 × 3, 64, 64) (x) x = ReLU (x) x = Maxpool(2, 2) (x) x = FC(2304, 1024) (x) x = ReLU (x) x = FC(1024, 1024) (x) x = ReLU (x) out = FC(1024, 10) (x)
Input: x, y x = Conv2d(3 × 3, 3, 16) (x) x = BatchNorm (x) x = ReLU (x) x = Conv2d(3 × 3, 16, 32) (x) x = BatchNorm (x) x = ReLU (x) x = Conv2d(3 × 3, 32, 64) (x) x = BatchNorm (x) x = ReLU (x) x = Conv2d(3 × 3, 64, 128) (x) x = BatchNorm (x) x = ReLU (x) x = Conv2d(3 × 3, 128, 256) (x) x = FC(1024, 2000) (x) x = ReLU (x) x = BinaryMask
(x, y) x = FC(2000, 2000) (x) x = ReLU (x) x =
BinaryMask (x, y) out = FC(2000, 1) (x)
[Table]
x = Conv2d(3 × 3, 3, 16) (x) x = BatchNorm (x) x = ReLU (x) x
= Conv2d(3 × 3, 16, 32) (x) x = BatchNorm (x) x = ReLU (x) x = Conv2d(3 × 3, 32, 64) (x) x = BatchNorm (x) x = ReLU (x) x = Conv2d(3 × 3, 64, 128) (x) x = BatchNorm (x) x
[Equation]
ReLU (x) x
[Equation]
Conv2d(
[Equation] [Equation]
[Equation]
x) x
[Equation]
[Equation]
x) x
[Equation]
(x) x
[Equation]
(x) x
[Equation]
(x) out = FC(2000, 100) (x)