Deep Class-Conditional Gaussians for Continual Learning
Abstract
The current state of the art for continual learning with frozen, pre-trained embedding networks are simple probabilistic models defined over the embedding space, for example class conditional Gaussians.
As yet, in the task-incremental online setting, it has been an open question how to extend these methods to when the embedding function has to be learned from scratch.
In this paper, we propose DeepCCG, an empirical Bayesian method which learns online both a class conditional
Gaussian model and an embedding function.
The learning process can be interpreted as using a variant of experience replay, known to be effective in continual learning.
As part of our framework, we decide which examples to store by selecting the subset that minimises the KL divergence between the true posterior and the posterior induced by the subset.
We demonstrate performance task-incremental online settings, including those with overlapping tasks.
Our method outperforms all other methods, including several other replay-based methods.
Introduction
Real world use of deep learning methods can often necessitate dynamic updating of solutions on non-stationary data streams [10, 3].
This is one of the main problems studied in continual learning and as a result, continual learning is of increasing interest to the machine learning community, with many proposed approaches [28] and settings [17, 3, 7].
The setting considered here is task-incremental online learning where a data stream is split into a sequential set of tasks and methods are given information about what the current task is [35, 30].
Each task is encapsulated by a representative dataset (considered to be sampled i.i d. from a task distribution) which is given to a method batch by batch.
Different tasks will generally be associated with different distributions, as well as different target problems, which are summarised in a task objective function.
We focus on classification [17], where the classes being considered vary between tasks.
The overall objective is, after seeing all of the tasks, to perform well on all of them (given memory constraints).
Currently one of the best ways to approach continual learning is to use a frozen pretrained embedding function and define a simple probabilistic model on top to classify the data [27, 14].
However, in some real-world settings it is necessary to learn the embedding function online, for example it might be impossible to pretrain the embedding function due to a lack of data or due to distribution shifts a frozen pretrained encoder may become outdated and so perform badly [27].
Therefore, it is an interesting question to see how these methods can be adapted to settings when the embedding function must be continually learnt.
We explore this question, presenting the method DeepCCG, where we use a class conditional Gaussian model on top of a neural network embedding function.
A key part of our method is that the class conditional Gaussian model can be re-estimated on a memory buffer as the embedding function changes, reducing forgetting.
Additionally, another key part of DeepCCG is a method for selecting what samples to store in memory, where we select examples that preserve the means of the per-class clusters formed by the embedded data.
In our experiments we look at two specific settings, the commonly used disjoint tasks setting, where each task contains different classes to any other [7] and an underexplored shifting window setting, where there is an overlap between what classes are in each task.
The reason we look at a setting with an overlap between tasks is to explore a methods ability for positive transfer, which is more rewarded in this setting as there is greater shared information [4].
The results of our experiments show that DeepCCG performed the best out of all methods tested, gaining an average performance improvement of 2 .
145% in terms of mean average accuracy, showing the potential of our approach.
Task-incremental setting
[Figure]
At time j the learner is given a sample of data D j and has a memory set of stored datapoints M j .
The memory is split into the replay data R j and the rest
M j /R j .
Learning happens by taking a gradient step on the parameters of the embedding function ϕ
using a conditional marginal likelihood function over D j and R j , where M j /R j is used to induce a posterior over the means of the per-class Gaussians and so define the conditional marginal likelihood function used.
Task-incremental online continual learning is a setting where the learner is given a sequence of batches of data one by one, along with a task identifier for each batch.
Different batches may be drawn from different tasks according to some unknown non-stationary process.
The goal is for the learner to be able to perform well on all tasks at the end of training [17, 30].
Specifically, we consider classification tasks.
Let X denote a shared data space, and C denotes the set of all classes being considered.
Let t ∈ T denote a particular task, which generates data
S t = { ( x i , y i ) | i = 1 , 2 , . . .
} , containing data instances x i ∈ X and labels y
i ∈
C t , where C t ⊂ C are the subset of classes relevant for this task.
The learner is provided with a temporal sequence (of length N ) of data batches and task identifiers (( D j , t D j ) | j = 1 , 2 , . . .
, N ) , where j indexes batches and t D j denotes the task identifier for batch D j .
At each time step j , the batch
D j is a sample from a single task and the task identifier t D j determines this task, indicating the classes which are relevant for that task.
During training, the learner receives each batch in sequential order; after training on a batch the learner must discard all the data it does not store in its fixed-size memory buffer.
At test time, the learnt model is given a previously-unseen test set from all the classes seen during training.
This setting is commonly used in the continual learning literature [5, 30].
Deep class conditional Gaussians
We propose an empirical Bayesian method to contiually learn both the embedding function and a class conditional Gaussian model, which we name DeepCCG (we also provide a more general method in
Appendix B).
We use a neural network for the embedding function f ϕ and have a fixed size memory buffer storing previously seen examples and their task identifiers.
We define a class conditional
Gaussian model in the embedding space Z and use the inverse model of that class
conditional
Gaussian to define p ( y | z, t, θ ) , where the parameters θ = { µ y | y ∈ C } , that is the parameters of this model are the means of the per-class Gaussians.
The reason we chose to use a class conditional
Gaussian model is due to it having state-of-the art performance when using a frozen pretrained embedding function [27, 14] and that by using its associated conjugate prior, the posterior distribution for θ is tractable and easy to compute.
Likewise, the marginal likelihood given by the inverse model is fully tractable.
More formally, for each task t we define
[Equation]
In this paper, we choose V
= a I , where in practice we take a → ∞ , and Cat( C t , (1 / | C t | ) 1 ) is the uniform categorical distribution over the classes of task t .
Learning the embedding To update the embedding function f ϕ , in response to the arrival of batch
D j , with task identifier t D j , and having received the memory buffer M j , we take the following steps (shown in Figure 1).
First, we select a set R j ⊂ M j , of size r , from the memory buffer.
Then we condition on the rest of the examples, M j /R j , to obtain the posterior predictive distributions p ( y | f ϕ ( x ) , t ( x,y ) , M j /R j ) for each sample (
x, y ) ∈
D j ∪ R j , where t ( x,y ) is the task identifier for the sample.
The posterior predictive distributions can be shown to be (see Appendix D for the derivation):
[Equation]
where M Zj,y = { f ϕ ( x ) | ( x, y ′ )
∈
M j ∧ y ′
=
y } are the embeddings of samples in the memory buffer with class y .
R Zj is defined likewise and we use the notation S to denote the mean of the elements of a set S .
As in meta-learning [12, 16], the posterior predictive distributions are used as a fixed likelihood term for each individual unseen data item, leading to a log-marginal likelihood as the objective in the gradient decent update used:
[Equation]
where t ( x,y ) is the task identifier associated with sample ( x, y ) .
This update method can be seen as a variant experience repay, a very effective continual learning approach [35, 39, 25].
Sample selection
The second key component to DeepCCG is the sample selection for the memory buffer.
Because all the information available to inform the value of the parameters so far is encapsulated in the full posterior over the parameters, we target choosing a set of memory samples that best recreate that posterior, preventing as much parameter information loss as possible.
Hence, to perform sample selection we minimise the KL divergence between two posterior distributions: the posterior over parameters induced by the new memory being optimized, and the posterior induced by the current batch and the old memory.
We keep the number of samples in memory for each class balanced, therefore minimising the KL divergence is equivalent to minimising the per-class KL divergence which reduces to the squared Euclidean distance between the means of a class’s embedded data.
Formally,
[Equation]
where M j +1 ,y is the new memory to be selected for class y , M j,y is the is the set of samples of class y in memory when receiving the current batch of data and D j,y is the set of samples of class y in the current batch.
Performing the minimization in Eq.
4 is computationally hard, so we utilise a relaxation of the problem using lasso [13], whereby our method selects the new memory by assigning to each embedded input
z i a zero-to-one weight β i and performing gradient decent on the loss
[Equation]
Then, after termination, our method selects the m samples with the largest weights to be the samples stored in memory, where they are stored along with their task identifiers.
Experiments
Benchmarks In our experiments we look at two different settings: disjoint tasks and a shifting window setting.
Furthermore, in these settings we utilize three datasets CIFAR-100 [21], MiniImageNet [36] and CIFAR-10 [21].
For disjoint tasks, we split the datasets using the classes of the samples, splitting evenly the classes across a certain number of tasks and assigning all samples with that class to that task [7, 5].
We split CIFAR-10 into 5 tasks where there are 2 classes per task and for
CIFAR-100 and MiniImageNet we split the dataset into 20 tasks with 5 classes per task.
While in the shifting window setting we split the datasets up into tasks by fixing an ordering of the classes c 1 , . . .
, c k and construct the t th task by selecting a set of samples from classes c t , . .
c t + l , where l is the window length, there is no overlap between any two of the task datasets and each task has the same number of samples per-class.
For CIFAR-10 we use a window length of 2 and for CIFARand
MiniImageNet we use a window length of 5 .
Additionally, for all experiments we train withsamples per-class.
We also perform an ablation of our method in Appendix F.
Methods We compare DeepCCG to representative methods of the main paradigms of continual learning: EWC [18, 20], PackNet [24], ER-Reservoir [1], EntropySS [38] and GSS [2].
[Table]
We also compare against two baselines: SGD which is when learning is performed using SGD with no modification and Multi-Task which is a rough upper bound and is when we learn the base neural network offline, training on the same number of samples per batch as the replay methods tested, which is 20 ( 10 new samples and 10 replayed), with the same number of batches, to more fairly upper bound their performance.
All methods use the same underlying embedding network for all experiments which is a modified ResNet18 with six times fewer filters across all layers and Instance Normalisation [34] instead of Batch Normalisation layers [19], which is similar to other work [25, 9].
Additionally, all methods use a batch size of 10 , with all replay methods having the same sample replay size of 10 and a memory size of 10 samples per-class for all experiments.
[Table]
Results We see from Table 2 and Table 1 that DeepCCG performs the best out of all the methods compared.
For the shifting window setting, on CIFAR-100, MiniImageNet and CIFAR-10, DeepCCG gets mean average accuracies of 56 .
18% and 74 .
13% , respectively, which is 2 .
14% and 3 .
1% better than any other method.
For the disjoint tasks setting DeepCCG also performs well, where for CIFAR-100, MiniImageNet and CIFAR-10, DeepCCG achieves a mean average accuracy of 60 .
04% and 69 .
29% which is 2 .
15% , 1 . 92% , 1 . 99% better than any other method, respectively.
It is important to note however that in the shifting window setting DeepCCG is still far below the multi-task performance of 90 .
81% and 96 .
20% for CIFAR-100, MiniImageNet and CIFAR-10, respectively.
This shows that in settings with more overlap between tasks there is still much work to be done on continual learning methods.
Conclusions
In this work we have demonstrated that using a replay based empirical Bayesian procedure for continual learning is a promising direction and can expand the use of simple probabilistic models from when the embedding function is frozen to when it is learnt online.
We have proposed DeepCCG, a method to learn continually both a class conditional Gaussian model and the embedding function it is defined on top of.
As part of the method we proposed a way of selecting samples to store in memory by approximately selecting the best subset that minimises the KL divergence between the posterior induced by all the currently accessible data and the posterior induced by the subset.
The results show that DeepCCG performs well for task-incremental online learning, outperforming all other methods in our experiments.
For future work, potential directions would be to look at adapting this approach to class-incremental learning and in settings with imbalanced and/or noisy data [33, 40, 2].
A Related work
When using frozen pretrained embedding functions in continual learning simple metric-based probabilistic models have been shown to have state-of-the-art performance [27, 14, 29].
For example, [14] show that online linear discriminant analysis (LDA) on top of a frozen pretrained backbone is the best performing method in their experiments and [27] show that class-conditional Gaussian models perform the best in certain settings.
These probabilistic models have the advantage that it is possible to learn the same estimate of parameters given any ordering of the data, including i.i.d orderings.
However, in many cases frozen pretrained embedding functions cannot be used, which is the case we look at in this work, due to a lack of relevant data for pretraining or due to distribution shift a pretrained embedding function will perform badly [27].
When the embedding function has to be learnt, there are three main paradigms for solving continual learning problems: regularisation , parameter-isolation and replay [7].
Regularisation methods aim to prevent catastrophic forgetting by adding terms to the loss which try to prevent the network from erasing the information of previous tasks [20, 18].
Parameter-Isolation methods look at controlling what parameters of a neural network are used for what tasks [24], which can often be seen as a hard version of regularisation methods.
Finally, Replay methods aim to solve continual learning problems by storing a subset of previously seen samples, which are then trained on alongside new incoming data.
This is the approach our method is part of.
Replay methods have been shown to have competitive if not the best performance across many settings [35, 39, 25].
The standard replay method is experience replay (ER) [6, 1], which in the setting explored in this work where we can not store all the current task’s data, selects samples to store using reservoir sampling [37], we call this variant ER-reservoir.
One of the main questions to be answered by a replay-based approach is how to select what samples to store in memory.
While reservoir sampling has been shown to be very effective [38] there have been other methods proposed for sample selection, for example ones which use information-theoretic criteria [38] and others maximising the diversity of the gradients of stored examples [2].
There has also been a Bayesian method proposed to select samples called InfoGS [33], which is somewhat similar to our method but only uses the probability model to select samples, not for prediction or training, and unlike our work looks at class-incremental learning , where methods do not have access to task identifiers.
There has been considerable work on using Bayesian methods in continual learning [26, 8, 22], perhaps inspired by the fact that true online Bayesian inference cannot suffer from catastrophic forgetting [26].
Bayesian perspectives have mainly been used for regularisation based methods, where a popular approach is to use variational inference [26, 11, 41].
These variational inference methods focus on the offline continual learning setting where a learner has access to all of the data of a task at the same time and, in previous work, are often limited to being used in conjunction with small neural networks, mainly due to the need to sample multiple networks when calculating the loss [15, 26].
Therefore, these methods are not suited to the settings we consider in this paper.
Bayesian methods have also been used in generative replay based approaches, where instead of storing and replaying real samples they use generated pseudo-samples [31].
However, when it comes to replay, with real examples, there has been relatively little work on using Bayesian methods, specifically for empirical Bayesian methods where there is a requirement to prevent forgetting when fitting the embedding function, which we aim to help to fill in by proposing our method.
In this work we look at two types of settings, one where the sets of class labels are disjoint between tasks and another where there is an overlap been the classes seen in each task.
Disjoint tasks is a commonly explored setting [7] while having some class overlap between tasks is not often studied, but arguably is more realistic.
Slightly tangential to the settings considered here, there has been some work looking at class overlap settings where blurry tasks are considered [30, 4, 2], that is when a disjoint task setting is modified by adding to each task a small number of samples sampled from any class, often around 10% of the original task’s size.
B Continual learning of embedding functions and probabilistic models
To solve the problem of exploiting simple probabilistic models, like LDA, when the embedding function has to be learnt online, we propose an empirical Bayesian method (shown in Algorithm 1).
Let Z denote an embedding space.
We assume that we have some embedding function f ϕ : X → Z and a fixed memory of size m , which counts the number of items (each an element of X × C × T ) that can fit in a memory buffer.
The memory buffer stores as one element a sample ( x, y ) and its associated task index t ( x,y ) —in common with previous work [5].
We propose to use a Bayesian probabilistic model with parameters θ , such that
[Equation]
and assume the labels are conditionally independent given the task, θ and the data instances (each an element of X ).
Therefore if f ϕ is fixed, we can update our beliefs about the parameters on seeing batch D j and its task identifier
t D j by simply using Bayes rule, calculating the posterior p ( θ | D j , D <j ) .
However, in our setting, we need to update f ϕ continually, which would change the value of p ( θ | D j , D <j ) .
We cannot recompute p ( θ | D j , D <j ) as we are unable to store all the previous samples in memory.
Therefore, we propose to store in the buffer a set of representative samples with which we can compute an approximate posterior after a change in f ϕ .
To achieve this goal, after every batch, we select a subset of data points, from the previous memory and current batch, to store in memory.
We choose the subset which minimises the KL divergence between the posterior distribution induced by the subset and the posterior distribution induced by the whole data,
[Equation]
where M j is the memory which was carried forward when receiving the j th batch of data.
To update the embedding function, for each arriving batch D j , task identifier t D j and corresponding memory M j , the method randomly subsamples R j ⊂ M j of given size r .
Then we update f ϕ using
[Equation]
where S tasks =
{ t ( x,y ) | ( x, y ) ∈ R j } , η is a learning rate, D Zj = { f ϕ ( x ) |
( x, y ) ∈
D j } , D Yj =
{ y | ( x
, y ) ∈
D j } and R Zj and R Yj are defined likewise.
This is a gradient step on a conditional marginal log-likelihood (i.e. we compute the marginal log-likelihood on some of the data having conditioned on the rest).
Conditioning on part of the data helps to reduce the sensitivity of learning in respect to a potentially uninformative prior and has been shown in previous work to increase generalisation performance [23].
[Table] [Table]
One way to view of our update procedure for the embedding function f ϕ is that it is a variant of experience replay, with a different loss function.
Experience replay is a highly performing method to prevent forgetting [35, 39, 25] [Table] so therefore our method to learn the embedding function should be resistant to forgetting.
Additionally, we note that their exists methods to select what examples to replay [1, 32] which are complementary/orthogonal to this work [Table] and could be combined with our approach.
We further note that while our general formulation requires the continual learning setting to provide task identifiers, if p ( y | x, θ, t ( x,y ) )
=
p ( y | x, θ ) [Table] where the distribution of x can still shift between tasks, our general approach can be used in task-agnostic settings as well, which we leave to future work [Table]
C Algorithm for DeepCCG
[Table]
D Details of learning the embedding function
For our deep class conditional Gaussian method (DeepCCG), to compute the update to the embedding function f ϕ , when given a batch D j and a corresponding memory M j , we proceed using the following steps.
Firstly, we randomly sample R j ⊂ M j of size r from memory.
Then using only the other samples in memory, M j /R j , we compute the posterior density for each class mean— µ c , with c ∈ C :
[Equation]
where M zj,c = { f ϕ ( x ) | ( x, y ) ∈
M j ∧ y = c } are the embeddings of points in the memory buffer with class y .
R z
j is defined likewise and we use the notation S to denote the mean of the elements of a set uninformative prior will often result in a flat marginal likelihood which is not informative of what embedding function fits the data well.
Whereas, conditioning the marginal likelihood on M j /R j , the posterior we average over is more informative, having some belief of where the position of embedded data instances should be, and so provides a better signal to fit the embedding function.
S .
Then we compute the posterior distribution of the embedded inputs z ∈ D
zj ∪ R zj for each class c ∈ C utilizing
[Equation]
Next, we compute the posterior predictive for each sample (
x, y ) ∈
D j ∪ R j with a task identifier t ( x,y ) , which is known for samples in the current batch and is stored by our method for samples stored in memory, and where z = f ϕ ( x ) using
[Equation]
Finally [Equation] we update the embedding function by performing a gradient step using the formula
[Equation]
which can be seen as a per-sample conditional marginal log-likelihood.
By using Eq. 13 and 15, the closed form of the posterior predictive distribution for a sample ( x, y ) with task identifier t ( x,y ) and where z = f ϕ ( x ) is
[Equation]
E Additional Experimental details
In addition to the experimental details mentioned in the main body of the paper, there are a few more details to mention.
Firstly, for methods with hyperparameters we performed a grid search using the same experimental set up as the real experiments and 10% of the training data as validation data.
This led to the selection of the hyperparameters of EWC and PackNet, the two methods with free hyperparameters, shown in Table 3.
Secondly, we use the same learning rate of 0 .
1 and the standard gradient decent optimiser for all methods.
These were chosen by looking at the commonly selected values in previous work [25] and were shown to be performative for all methods tested.
Finally, all experiments were run on a laptop with a single NVIDIA GeForce GTX 1050 GPU.
[Table] [Table]
[Table]
F Ablation Study
One way to view DeepCCG is as a relative of ER-reservoir, where the methods differ in the sample section mechanism used and the probabilistic model defined on top of the embedding function.
Therefore, to analyse where the performance of DeepCCG comes from, we ablate our method by separately changing these two components to what they are in ER-reservoir.
The results are presented in Table 4 and show that both components on their own do not improve performance over ER-reservoir but together they do.
We go into detail of the ablation of both components, the sample selection mechanism and probabilistic model, in the next two paragraphs and then present an ablation on the memory size used for the replay methods tested, where we show that DeepCCG performs the best over all the memory sizes tested on.
Sample selection To explore the performance of our memory selection mechanism, we compare our full method to when we select samples using reservoir sampling (DeepCCG-reservoir), showing that the full method performs the best in the setting tested on.
DeepCCG achieves a mean average accuracy 56 .
62% whereas DeepCCG-reservoir obtains a lower mean average accuracy of 49 .
95% (see Table 4).
This shows that our memory selection mechanism is necessary for DeepCCG to obtain good performance.
Additionally, we predict that our memory selection mechanism will also perform well in settings with noisy, OOD and/or imbalanced data, unlike reservoir sampling [33, 40, 2], which we leave to future work to validate.
Class conditional Gaussian model To understand the performance of using a class conditional
Gaussian model instead of the standard discriminative softmax model, we look at the performance of
DeepCCG without using the class conditional Gaussian model.
That is, we evaluate the performance of DeepCCG when we replace the class conditional Gaussian model with a standard output head (a fully connected layer, then a softmax), while still using our sample selection method—calling it
DeepCCG-standardHead.
The results show that DeepCCG-standardHead performs poorly, achieving an average accuracy of 44 .
05% (see Table 4), which is significantly worse than when using reservoir sampling, as in ER-reservoir, which gets a mean average accuracy of 54 .
This is to be expected as our sample selection strategy is designed with the assumption that our embedding function will cluster the data into per-class clusters which may not be true when using a standard multi-head output layer, while when using a class conditional Gaussian model the embedding function will be encouraged to create per-class clusters.
Memory size One additional useful experiment is looking at the relationship between performance and the size of memory used for DeepCCG.
Therefore, in Table 5 we show the performance of
DeepCCG and the other replay methods compared against with varying memory size.
Table 5 shows that when the memory size is increased to m = 1250 , DeepCCG has an improvement in average accuracy relative to other methods, as it achieves 2 .
76% better than any other method for m = 1250 , while for m = 1000 it achieves 2 .
57% better than any other method, a 0 .
19% improvement.
When the memory size is decreased to m = 750 , we see that DeepCCG’s performance drops more than other methods as it is only 0 .
25% better than other methods in this case.
Therefore, our experiments show that compared to other replay methods DeepCCG’s performance increases the most when m increases and that for small buffer sizes DeepCCG performs less well, potentially due to the fact that the examples in the memory buffer are used to infer the posterior over the means of the per-class
Gaussians and so the method needs a given amount of examples to specify the means well.
We also note that in our experiments the performance ranking of the methods does not change with m ..