A N EURAL D IRICHLET P ROCESS M IXTURE M ODEL FOR T ASK -F REE C ONTINUAL L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite the growing interest in continual learning, most of its contemporary works have been studied in a rather restricted setting where tasks are clearly distinguishable and task boundaries are known during training.
However, if our goal is to develop an algorithm that learns as humans do, this setting is far from realistic and it is essential to develop a methodology that works in a task-free manner.
Meanwhile, among several branches of continual learning, expansion-based methods have the advantage of eliminating catastrophic forgetting by allocating new resource to learn new data.
In this work, we propose an expansion-based approach for task-free continual learning for the ﬁrst time.
Our model, named Continual
Neural Dirichlet Process Mixture (CN-DPM), consists of a set of neural network experts that are in charge of a subset of the data.
CN-DPM expands the number of experts in a principled way under the Bayesian nonparametric framework.
With extensive experiments, we show that our model successfully performs taskfree continual learning for both discriminative and generative tasks such as image classiﬁcation and image generation.
I NTRODUCTION
Humans consistently encounter new information throughout their lifetime.
The way the information is provided, however, is vastly different from that of conventional machine learning where each minibatch is iid-sampled from the whole dataset.
Data points adjacent in time can be highly correlated and the overall distribution of the data can shift drastically as the training progresses.
Continual learning (CL) aims at imitating incredible human’s ability of learning from a non-iid stream of data without catastrophically forgetting the previously learned knowledge.
Most CL approaches (Aljundi et al., 2018; 2017; Lopez-Paz & Ranzato, 2017; Kirkpatrick et al., 2017; Rusu et al., 2016; Shin et al., 2017; Yoon et al., 2018) assume that the data stream is explicitly divided into a sequence of tasks that are known at training time.
Since this assumption is far from realistic, task-free CL is more practical and demanding but has been largely understudied with only few exceptions of (Aljundi et al., 2019a;b).
In this general CL, not only is explicit task deﬁnition unavailable but also the data distribution gradually shifts without clear task boundary.
Meanwhile, existing CL methods can be classiﬁed into three different categories (Parisi et al., 2019): regularization, replay, and expansion methods.
Regularization and replay approaches address the catastrophic forgetting by regularizing the update of a speciﬁc set of weights or replaying the previously seen data, respectively.
On the other hand, the expansion methods are different from the two approaches in that it can expand the model architecture to accommodate new data instead of ﬁxing it beforehand.
Therefore, the expansion methods can bypass catastrophic forgetting by preventing pre-existing components from being overwritten by the new information.
The key limitation of prior expansion methods, however, is that the decisions of when to expand and which resource to use heavily rely on explicitly given task deﬁnition and heuristics.
In this work, our goal is to propose a novel expansion-based approach for task-free CL which, to the best of our knowledge, has not been discussed yet.
Inspired by Mixture of Experts (MoE) (Jacobs et al., 1991), our model consists of a set of experts , each of which is in charge of a subset of the data in a stream.
The model expansion ( i.e . adding more experts) is governed by the Bayesian nonparametric framework, which determines the model complexity by the data, as opposed to the parametric methods that ﬁx the model complexity before training.
We formulate the task-free CL as an online variational inference of Dirichlet process mixture models consisting of a set of neural experts; thus we name our approach as the Continual Neural Dirichlet Process Mixture (CN-DPM) model.
We highlight the key contributions of this work as follows.
• We are the ﬁrst to propose an expansion-based approach for task-free CL.
Hence, our model not only prevents catastrophic forgetting but also is applicable to the setting where no task deﬁnition and boundaries are given at both training and test time.
Our model named CNDPM consists of a set of neural network experts, which are expanded in a principled way built up on the Bayesian nonparametrics that have not been adopted in general CL research.
• Our model can deal with both generative and discriminative tasks of CL.
With several benchmark experiments of CL literature on MNIST, SVHN and CIFAR 10/100, we show that our model successfully performs multiple types of CL tasks including image classiﬁcation and generation.
B ACKGROUND AND R ELATED W ORK 2. C ONTINUAL L EARNING
Parisi et al.
(2019) classify CL approaches into three branches: regularization (Kirkpatrick et al., 2017; Aljundi et al., 2018), replay (Shin et al., 2017) and expansion (Aljundi et al., 2017; Rusu et al., 2016; Yoon et al., 2018) methods.
Regularization and replay approaches ﬁx the model architecture before training, and prevent catastrophic forgetting by regularizing the change of a speciﬁc set of weights or replaying previously learned data.
Hybrids of replay and regularization also exist such as Gradient Episodic Memory (GEM) (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2019a).
On the other hand, methods based on expansion add new network components to learn new data.
Conceptually, such direction has the following advantages compared to the ﬁrst two: (i) catastrophic forgetting can be eliminated since new information is not overwritten on pre-existing components and (ii) the model capacity is determined adaptively depending on the data.
Task-Free Continual Learning .
All the works mentioned above heavily rely on explicit task deﬁnition.
However, in real-world scenarios, task deﬁnition is rarely given at training time.
Moreover, the data domain may gradually shift without any clear task boundary.
Despite its importance, task-free
CL has been largely understudied; to the best of our knowledge, there are only two works (Aljundi et al., 2019a;b), each based on regularization and replay.
Speciﬁcally, Aljundi et al. (2019a) extend
MAS (Aljundi et al., 2018) by adding heuristics to determine when to update the importance weights with no task deﬁnition.
In their following work (Aljundi et al., 2019b), they improve the memory management algorithm of GEM (Lopez-Paz & Ranzato, 2017) such that the memory elements are carefully selected to minimize catastrophic forgetting.
Compared to the previous research, our work is the ﬁrst to propose an expansion-based task-free CL method.
2. D IRICHLET P ROCESS M IXTURE M ODELS
We brieﬂy review the Dirichlet process mixture (DPM) model (Antoniak, 1974; Ferguson, 1983) and a variational method to approximate the posterior of DPM models in an online setting: Sequential
Variational Approximation (SVA) (Lin, 2013).
For a more detailed review, refer to Appendix A.
Dirichlet Process Mixture (DPM) .
The DPM model is often applied to clustering problems where the number of clusters is not known in advance.
The generative process of a DPM model is
[Equation]
where x n is the n -th data, and θ n is the n -th latent variable sampled from G , which itself is a distribution sampled from a Dirichlet process (DP).
The DP is parameterized by a concentration parameter α and a base distribution G 0 .
Since G is discrete with probability 1 (Teh, 2010), same values can be sampled multiple times for θ .
If θ n = θ m , the two data points x n and x m belong to the same cluster.
An alternative formulation uses the variable z n that indicates to which cluster the n -th data belongs such that θ n = φ z n where φ k is the parameter of k -th cluster.
In the context of this paper, φ k refers to the parameters of k -th expert.
Approximation of Posterior of DPM Models .
Since the exact inference of the posterior of DPM models is infeasible, approximate inference methods are applied.
Among many approximation methods, we adopt the Sequential Variational Approximation (SVA)
(Lin, 2013).
While the data is given one by one, SVA sequentially determines ρ n and ν k which are the variational approximation for the distribution of z n and φ k respectively.
Since ρ n satisﬁes (cid:80) k ρ n,k = 1 and ρ n,k > = 0 , ρ n,k can be interpreted as the probability of n -th data belonging to k -th cluster and is often called responsibility.
ρ n +1 and ν ( n +1) at step n + 1 are computed as:
[Equation]
In practice, SVA adds a new component only when ρ K +1 is greater than a certain threshold (cid:15) .
If
G 0 and p ( x i | φ ) are not a conjugate pair, stochastic gradient descent (SGD) is used to ﬁnd the MAP estimation ˆ φ with a learning rate of λ instead of calculating the whole distribution ν k :
[Equation]
DPM for Discriminative Tasks .
DPM can be extended to discriminative tasks where each data point is an input-output pair ( x, y ) and the goal is to learn the conditional distribution p ( y | x ) .
To use DPM, which is a generative model, for discriminative tasks, we ﬁrst learn the joint distribution p ( x, y ) and induce the conditional distribution from it: p ( y | x )
= p ( x, y ) / (cid:82) y p ( x, y ) .
The joint distribution modeled by each component can be decomposed as p ( x, y | z )
= p ( y | x, z ) p ( x | z ) (Rasmussen & Ghahramani, 2002; Shahbaba & Neal, 2009).
DPM in Meta-Learning .
Recent works (Jerfel et al., 2018; Nagabandi et al., 2019) in online metalearning use DPM models to add new components without supervision.
Their approach, however, lacks a generative component which is a crucial element to complete the DPM formulation.
As a consequence, it is hard to extend their methods beyond meta-learning.
In contrast, our method implements a sound DPM model and is applicable to general CL.
A PPROACH
We aim at general task-free CL where the number of tasks and task description are not available at both training and test time.
We even consider the case where the data stream cannot be split into separate tasks in Appendix F. All of the existing expansion methods are not task-free since they require task deﬁnition at training (Aljundi et al., 2017) or even at test time (Rusu et al., 2016; Xu & Zhu, 2018; Li et al., 2019).
We propose a novel expansion method that automatically determines when to expand and which component to use.
We ﬁrst deal with generative tasks and generalize into discriminative ones.
3. C ONTINUAL L EARNING AS M ODELING OF THE M IXTURE D ISTRIBUTION
We can formulate a CL scenario as a stream of data involving different tasks D 1 , D 2 , ... where each task D k is a set of data sampled from a (possibly) distinct distribution p ( x | z = k ) .
If K tasks are given so far, the overall distribution is expressed as the mixture distribution:
[Equation]
where p ( z = k ) can be approximated by N k /N where N k = |D
k | and N = (cid:80)
k N k .
The goal of CL is to learn the mixture distribution in an online manner.
Regularization and replay methods directly model the approximate distribution p ( x ; φ ) parameterized by a single component φ and update it to ﬁt the overall distribution p ( x ) .
When updating φ , however, they do not have full access to all the previous data, and thus the information of previous tasks is at risk of being lost as more tasks are learned.
Another way to solve CL is to use a mixture model: approximating each p ( x | z = k ) with p ( x ; φ k ) .
If we learn a new task distribution p ( x |
z
= K + 1) with new parameter φ K + and leave the existing parameters intact, we can preserve the knowledge of the previous tasks.
The expansion-based CL methods follow this idea.
Similarly, in the discriminative task, the goal of CL is to model the overall conditional distribution which is a mixture of task-wise conditional distribution p ( y | x, z = k ) :
[Equation]
Prior expansion methods use expert networks each of which models a task-wise conditional distribution p ( y | x ; φ k ) 1 .
However, a new problem arises in expansion methods: choosing the right expert given x , i.e . p ( z | x ) in Eq.(6).
Existing methods simply assume that explicit task descriptor z is given, which is generally not true in human-like learning scenarios.
That is, we need a gating mechanism that can infer p ( z | x ) only from x ( i.e . which expert should process x ).
With the gating, the model prediction naturally reduces to the sum of expert outputs weighted by the gate values, which is the mixture of experts (MoE) (Jacobs et al., 1991) formulation: p ( y | x ) ≈ (cid:80) k p
( y | x ; φ k ) p ( z = k | x ) .
However, it is not possible to use a single gate network as in Shazeer et al.
(2017) to model p ( z | x ) in
CL; since the gate network is a classiﬁer that ﬁnds the correct expert for a given data, training it in an online setting causes catastrophic forgetting.
Thus, one possible solution to replace a gating network is to couple each expert k with a generative model that represents p ( x | z = k ) as in Rasmussen &
Ghahramani (2002) and Shahbaba & Neal (2009).
As a result, we can build a gating mechanism without catastrophic forgetting as
[Equation]
where p ( z = k ) ≈ N k /N .
We also differentiate the notation for the parameters of discriminative models for classiﬁcation and generative models for gating by the superscript D and G .
If we know the true assignment of z , which is the case of task-based CL, we can independently train a discriminative model ( i.e . p ( y | x ; φ Dk ) ) and a generative model ( i.e . p ( x ; φ Gk ) ) for each task k .
In task-free CL, however, z is unknown so the model needs to infer the posterior p ( z | x, y ) .
Even worse, the total number of experts is unknown beforehand.
Therefore, we propose to employ a
Bayesian nonparametric framework, speciﬁcally the Dirichlet process mixture (DPM) model, which can ﬁt a mixture distribution with no preﬁxed number of components.
We use SVA described in section 2.2 to approximate the posterior in an online setting.
Although SVA is originally designed for the generative tasks, it is easily applicable to discriminative tasks by making each component k to model p ( x, y | z )
= p ( y | x, z ) p ( x | z ) .
3. T HE C ONTINUAL N EURAL D IRICHLET P ROCESS M IXTURE (CN-DPM) M ODEL
The proposed approach for task-free CL, named Continual Neural Dirichlet Process Mixture (CNDPM) model, consists of a set of experts , each of which is associated with a discriminative model (classiﬁer) and a generative model (density estimator).
More speciﬁcally, the classiﬁer models p ( y | x, z = k ) , for which we can adopt any classiﬁer or regressor using deep neural networks, and the density estimator describes the marginal likelihood p ( x | z = k ) , for which we can use any explicit density model such as VAEs (Kingma & Welling, 2014) and PixelRNN (Oord et al., 2016).
We respectively denote the classiﬁer and the density estimator of expert k as p ( y | x ; φ Dk ) and p ( x ; φ Gk ) , where φ Dk and φ Gk are the parameters of the models.
Finally, the prediction p ( y | x ) can be obtained from Eq.(7) by plugging in the output of the classiﬁer and the density estimator.
Note that the number of experts is not pre-ﬁxed but expanded via the DPM framework.
Figure 1 illustrates the overall training and inference process of our model.
Training .
We assume that a sequence of samples arrive one at a time during training.
For a new sample, we ﬁrst decide whether the sample should be assigned to an existing expert or a new expert should be created for it.
Suppose that samples up to ( x n , y n ) are processed and K
[Figure]
Training (b) Inference 𝑥
[Figure]
𝑦
[Figure]
𝜙
[Figure]
𝜌
[Figure]
𝜙 0𝐷
[Figure]
𝜙 0𝐺 𝜙 0𝐷 𝜙
[Figure]
𝛼
STM STM
Train a new expert
Sleep if
STM is full
Wake 𝑁 2 𝑁 𝑥 𝑁 𝐾 𝑁 𝑝 𝑥, 𝑦 𝑁 1 𝑁 𝜙 𝐾+
𝐷 𝜙 𝐾+1𝐺 𝑁 𝐾+ 𝑝 𝑧
= 𝑘 = 𝑁 𝑘 𝑁 𝑝 𝑥, 𝑦;𝜙𝑝 𝑥, 𝑦;𝜙𝑝 𝑥, 𝑦;𝜙 𝐾 𝑝 𝑦|𝑥
we ﬁrst compute experts are already created when a new sample ( x n +1 , y n +1 ) arrives.
We compute the responsibility ρ n +1 ,k as follows:
[Equation]
if
[Equation]
where G 0 is a distribution corresponding to the weight initialization.
If arg max k ρ n +1 ,k (cid:54) =
K + 1 , the sample is assigned to the existing experts proportional to ρ n +1 ,k , and the parameters of the experts are updated with the new sample by Eq.(4) such that ˆ φ k is the MAP approximation given the data assigned up to the current time step.
Otherwise, we create a new expert.
Short-Term Memory .
However, it is not a good idea to create a new expert immediately and initialize it to be the MAP estimation given x n +1 .
Since both classiﬁer and density estimator of an expert are neural networks, training the new expert with only a single example leads to severe overﬁtting.
To mitigate this issue, we employ short-term memory (STM) to collect sufﬁcient data before creating a new expert.
When a data point is classiﬁed as new, we store it to the STM.
Once the STM reaches its maximum capacity M , we stop the data inﬂow for a while and train a new expert with the data in the STM for multiple epochs until convergence.
We call this procedure sleep phase .
After sleep, the STM is emptied and the newly trained expert is added to the expert pool.
During the subsequent wake phase, the expert is learned from the data assigned to it.
This STM trick assumes that the data in the STM belong to the same expert.
We empirically ﬁnd that this assumption is acceptable in many CL settings where adjacent data are highly correlated.
The overall training procedure is described in Algorithm 1.
Note that we use ρ n, 0 instead of ρ n,K +1 in the algorithm for brevity.
Inference .
At test time, we infer p ( y | x ) from the collaboration of the learned experts as in Eq.(7).
Techniques for Practicality .
Naively adding a new expert has two major problems: (i) the number of parameters grows unnecessarily large as the experts redundantly learn common features and (ii) there is no positive transfer of knowledge between experts.
Therefore, we propose a simple method to share parameters between experts.
When creating a new expert, we add lateral connections to the features of the previous experts similar to Rusu et al.
To prevent catastrophic forgetting in the existing experts, we block the gradient from the new expert.
In this way, we can greatly reduce the number of parameters while allowing positive knowledge transfer.
More techniques such as sparse regularization in Yoon et al.
(2018) can be employed to further reduce redundant parameters.
As they are orthogonal to our approach, we do not use such techniques in our experiments.
Another effective technique that we use in the classiﬁcation experiments is adding a temperature parameter to the classiﬁer.
Since the range of log p ( x | z ) is far broader than log p ( y | x, z ) , the classiﬁer has almost no effect without proper scaling.
Thus, we can increase overall accuracy by adjusting the relative importances of images and labels.
We also introduce an algorithm to prune redundant experts in
Appendix D, and discuss further practical issues of CN-DPM in Appendix B.
Require: Data ( x 1 , y 1 ) , ..., ( x N , y N ) , concentration α , base measure G 0 , short-term memory capacity M , learning rate λ 1: M ← ∅ { Short-term memory } 2: K ← 0 { Number of experts } 3: N 0 ← α ; ˆ φ 0 ← Sample( G 0 ) 4: for n = 1 to N do 5: for k = 0 to K do 6: l k ← p ( y n | x n ; ˆ φ Dk ) p ( x n ; ˆ φ Gk ) 7: ρ n,k ← N k l k 8: end for 9: ρ n, 0: K ← ρ n, 0: K / (cid:80)
Kk =
ρ n,k 10: if arg max k ρ n,k = 0 then 11: { Save x n to short-term memory } 12:
M ← { x n } ∪ M 13: if |M| ≥ M then { Add new expert } 14: ˆ φ K +1 ← FindMAP( M , G 0 ) 15:
N K +1 ← |M| ; M ← ∅ 16:
K ← K +17: end if 18: else { Update existing experts } 19: ρ n, 1: K ← ρ n, 1: K / (cid:80)
Kk =1 ρ n,k 20: for k = 1 to K do 21:
N k ← N k + ρ n,k 22:
ˆ φ k ← ˆ φ k + ρ n,k λ ∇ ˆ φ k log l k 23: end for 24: end if 25: end for
E XPERIMENTS
We evaluate the proposed CN-DPM model in task-free CL with four benchmark datasets.
Appendices include more detailed model architecture, additional experiments and analyses.
4. C ONTINUAL L EARNING S CENARIOS
A CL scenario deﬁnes a sequence of tasks where the data distribution for each task is assumed to be different from others.
Below we describe the task-free CL scenarios used in the experiments.
At both train and test time, the model cannot access the task information.
Unless stated otherwise, each task is presented for a single epoch ( i.e . a complete online setting) with a batch size of 10.
Split-MNIST (Zenke et al., 2017).
The MNIST dataset (LeCun et al., 1998) is split into ﬁve tasks, each containing approximately 12K images of two classes, namely (0/1, 2/3, 4/5, 6/7, 8/9).
We conduct both classiﬁcation and generation experiments in this scenario.
MNIST-SVHN (Shin et al., 2017).
It is a two-stage scenario where the ﬁrst consists of MNIST and the second contains SVHN (Netzer et al., 2011).
This scenario is different from Split-MNIST; in
Split-MNIST, new classes are introduced when transitioning into a new task, whereas the two stages in MNIST-SVHN share the same set of class labels and have different input domains.
Split-CIFAR10 and Split-CIFAR100 .
In Split-CIFAR10, we split CIFAR10 (Krizhevsky & Hinton, 2009) into ﬁve tasks in the same manner as Split-MNIST.
For Split-CIFAR100, we buildtasks each containing 5 classes according to the pre-deﬁned superclasses in CIFAR100.
The training sets of CIFAR10 and CIFAR100 consist of 50K examples each.
To the best of our knowledge, we are ﬁrst to report Split-CIFAR100 performance without using task information at test time.
In
Split-CIFAR100 experiments of all previous works (Rebufﬁ et al., 2017; Zenke et al., 2017; LopezPaz & Ranzato, 2017; Aljundi et al., 2019c; Chaudhry et al., 2019a)
a distinct output head is used for each task, and the task information to select the corresponding output head is given at both training and test time.
Knowing the right output head, however, the task reduces to 5-way classiﬁcation.
Therefore, our setting is far more difﬁcult than the prior works since the model has to perform 100-way classiﬁcation only from the given input.
4. C OMPARED M ETHODS
All the following baselines use the same base network that will be discussed in section 4.3.
[Table]
[Table] [Table]
[Figure] [Figure]
iid-ofﬂine and iid-online .
iid-ofﬂine shows the maximum performance achieved by combining standard training techniques such as data augmentation, learning rate decay, multiple iteration (up to 100 epochs), and larger batch size.
iid-online is the model trained with the same number of epoch and batch size with other CL baselines.
Fine-tune .
As a popular baseline in the previous works, the base model is naively trained as data enters.
Reservoir .
As Chaudhry et al.
(2019b) show that simple experience replay (ER) can outperform most CL methods, we test ER with reservoir sampling as a strong baseline.
Reservoir sampling randomly chooses a ﬁxed amount of samples with a uniform probability from an indeﬁnitely long stream of data, and thus it is suitable for managing the replay memory in task-free CL.
At each training step, the model is trained using a mini-batch from the data stream and another one of the same size from the memory.
Gradient-Based Sample Selection (GSS) .
Aljundi et al.
(2019b) propose a sampling method called
GSS that diversiﬁes the gradients of the samples in the replay memory.
Since it is designed to work in task-free settings, we report the scores in their paper for comparison.
4. M ODEL A RCHITECTURE
Split-MNIST .
Following Hsu et al.
(2018), we use a simple two-hidden-layer MLP classiﬁer with
ReLU activation as the base model for classiﬁcation.
The dimension of each layer is 400.
For generation experiments, we use VAE whose encoder and decoder have the same hidden layer conﬁguration with the classiﬁer.
Each expert in CN-DPM has the similar classiﬁer and VAE with smaller hidden dimensions.
The ﬁrst expert starts with 64 hidden units per layer, and add 16 units when a new expert is added.
For classiﬁcation, we adjust hyperparameter α such that 5 experts are created.
For generation, we set α to produce 12 experts since more experts produce better score.
We set the memory size in both Reservoir and CN-DPM to 500 for classiﬁcation and 1000 for generation.
MNIST-SVHN and Split-CIFAR10/100 .
We use ResNet-18 (He et al., 2016) as the base model.
In
CN-DPM, we use a 10-layer ResNet for the classiﬁer and a CNN-based VAE.
The encoder and the decoder of VAE have two CONV layers and two FC layers.
We set α such that 2, 5, and 20 experts are created for each scenario.
The memory sizes in Reservoir, GSS and CN-DPM are set to 500 for
MNIST-SVHN and 1000 for Split-CIFAR10/100.
More details can be found in Appendix C.
4. R ESULTS OF T ASK -F REE C ONTINUAL L EARNING
All reported numbers in our experiments are the average of 10 runs.
Table 1 and 2 show our main experimental results.
In every setting, CN-DPM outperforms the baselines by signiﬁcant margins with reasonable parameter usage.
Table 2 and Figure 2 shows the results of Split-CIFAR10 experiments.
Since Aljundi et al.
(2019b) test GSS using only 10K examples of CIFAR10, which is 1/ of the whole train set, we follow their setting (denoted by 0.2 Epoch ) for fair comparison.
We also test a Split-CIFAR10 variant where each task is presented for 10 epochs.
The accuracy and the training graph of GSS are excerpted from the original paper where the accuracy is the average of three runs and the graph is from one of the runs.
In Figure 2, the bold line represents the average of 10 runs (except GSS which is a single run), and the faint lines are the individual runs.
Surprisingly,
Reservoir even surpasses the accuracy of GSS and proves to be a simple but powerful CL method.
One interesting observation in Table 2 is that the performance of Reservoir degrades as each task is extended up to 10 epochs.
This is due to the nature of replay methods; since the same samples are replayed repeatedly as representatives of the previous tasks, the model tends to be overﬁtted to the replay memory as training continues.
This degradation is more severe when the memory size is small as presented in Appendix I.
Our CN-DPM, on the other hand, uses the memory to temporarily buffer recent examples, so there is no such overﬁtting problem.
This is also conﬁrmed by the CN-DPM’s accuracy consistently increasing as learning progresses.
In addition, CN-DPM is particularly strong compared to other baselines when the number of tasks increases.
For example, Reservoir, which performs reasonably well in other tasks, scores poorly in Split-CIFAR100, which involves 20 tasks and 100 classes.
Even with the large replay memory of size 1000, the Reservoir suffers from the shortage of memory ( e.g . only 50 slots per task).
In contrast, CN-DPM’s accuracy is more than double of Reservoir and comparable to that of iid-online.
Table 3 analyzes the accuracy of CN-DPM in Split-CIFAR10/100.
We assess the performance and the forgetting of individual components.
At the end of each task, we measure the test accuracy of the responsible classiﬁer and report the average of such task-wise classiﬁer accuracies as Classiﬁer (init) .
We report the average of the task-wise accuracies after learning all tasks as Classiﬁer (ﬁnal) .
With little difference between the two scores, we conﬁrm that forgetting barely occurs in the classiﬁers.
In addition, we report the gating accuracy measured after training as Gating (VAEs) , which is the accuracy of the task identiﬁcation performed jointly by the VAEs.
The relatively low gating accuracy suggests that CN-DPM has much room for improvement through better density estimates.
Overall, CN-DPM does not suffer from catastrophic forgetting which is a major problem in regularization and replay methods.
As a trade-off, however, choosing the right expert arises as another problem in CN-DPM.
Nonetheless, the results show that this new direction is especially promising when the number of tasks is very large.
C ONCLUSION
In this work, we formulated expansion-based task-free CL as learning of a Dirichlet process mixture model with neural experts.
We demonstrated that the proposed CN-DPM model achieves great performance in multiple task-free settings, better than the existing methods.
We believe there are several interesting research directions beyond this work: (i) improving the accuracy of expert selection, which is the main bottleneck of our method, and (ii) applying our method to different domains such as natural language processing and reinforcement learning.
A R EVIEW OF D IRICHLET P ROCESS M IXTURE M ODEL
We review the Dirichlet process mixture (DPM) model and a variational method to approximate the posterior of DPM models in an online setting: Sequential Variational Approximation (SVA) (Lin, 2013).
Dirichlet Process .
Dirichlet process (DP) is a distribution over distributions that is deﬁned over inﬁnitely many dimensions.
DP is parameterized by a concentration parameter α ∈ R + and a base distribution G 0 .
For a distribution G sampled from DP( α, G 0 ) , the following holds for any ﬁnite measurable partition { A 1 , A 2 , ..., A K } of probability space Θ (Teh, 2010):
[Equation]
The stick-breaking process is often used as a more intuitive construction of DP:
[Equation]
Initially, we start with a stick of length one, which represents the total probability.
At each step k , we cut a proportion v k off from the remaining stick (probability) and assign it to the atom φ k sampled from the base distribution G 0 .
This formulation shows DP is discrete with probability 1 (Teh, 2010).
In our problem setting, G is a distribution over expert’s parameter space and has positive probability only at the countably many φ k which are independently sampled from the base distribution.
Dirichlet Process Mixture (DPM) Model .
The DPM model is often applied to clustering problems where the number of clusters is not known in advance.
The generative process of DPM model is
[Equation]
where x n is the n -th data, and θ n is the n -th latent variable sampled from G , which itself is a distribution sampled from a Dirichlet process (DP).
Since G is discrete with probability 1, same values can be sampled multiple times for θ .
If θ n = θ m , the two data points x n and x m belong to the same cluster.
An alternative formulation uses the indicator variable z n that indicates to which cluster the n -th data belongs such that θ n = φ z n where φ k is the parameter of k -th cluster.
The data x n is sampled from a distribution parameterized by θ n .
For a DP Gaussian mixture model as an example, each θ = { µ, σ 2 } parameterizes a Gaussian distribution.
The Posterior of DPM Models .
The posterior of a DPM model for given θ 1 , ..., θ n is also a DP (Teh, 2010):
[Equation]
The base distribution of the posterior, which is a weighted average of G 0 and the empirical distribution 1 n (cid:80) ni =
δ
( θ i ) , is in fact the predictive distribution of θ n +1 given θ 1: n (Teh, 2010):
[Equation] [Equation]
If we additionally condition x n and reﬂect the likelihood, we obtain (Neal, 2000):
[Equation]
(cid:33) , (14) where Z is the normalizing constant.
Note that θ n +1 is independent from x 1: n given θ 1: n .
Approximation of the Posterior of DPM Models .
Since the exact inference of the posterior of
DPM models is infeasible, approximate inference methods are adopted such as Markov chain Monte
Carlo (MCMC) (Maceachern, 1994; Escobar & West, 1995; Neal, 2000) or variational inference (Blei & Jordan, 2006; Wang & Dunson, 2011; Lin, 2013).
Among many variational methods, the
Sequential Variational Approximation (SVA) (Lin, 2013) approximates the posterior as
[Equation] [Equation]
where p ( z 1: n | x 1: n ) is represented by the product of individual variational probabilities
ρ z i for z
i , which greatly simpliﬁes the distribution.
Moreover, p ( G | x 1: n , z 1: n ) is approximated by a stochastic process q ( z ) ν
( G | z 1: n ) .
Sampling from q ( z ) ν
( G | z 1: n ) is equivalent to constructing a distribution as
[Equation]
[Equation]
where { C ( z ) 1 , C ( z ) 2 , ..., C ( z ) K } is the partition of x 1: n characterized by z .
The approximation yields the following tractable predictive distribution:
[Equation]
SVA uses this predictive distribution for sequential approximation of the posterior of z and φ .
[Equation]
While the data is given one by one, SVA sequentially updates the variational parameters; the following ρ n +1 and ν ( n +1) at step n + 1 minimizes the KL divergence between q ( z n +1 , φ ( n +1) | ρ 1: n +1 , ν ( n +1) ) and the posterior:
[Equation]
In practice, SVA adds a new component only when ρ n +1 ,K +1 is greater than a threshold (cid:15) .
It uses stochastic gradient descent to ﬁnd and maintain the MAP estimation of parameters instead of calculating the whole distribution ν k :
[Equation]
(22) where λ ( n ) k is a learning rate of component k at step n , which decreases as in the Robbins-Monro algorithm.
B P RACTICAL I SSUES OF CN-DPM
CN-DPN is designed based on strong theoretical foundations including the nonparametric Bayesian framework.
In this section, we further discuss some practical issues of CN-DPM with intuitive explanations.
Bounded expansion of CN-DPM .
The number of components in DPM model is determined by the data distribution and the concentration parameter.
If the true distribution consists of K clusters, the number of effective components converges to K under appropriate concentration parameter α .
Typically, the number of components is bounded by O ( α log N ) (Teh, 2010).
Experiments in Appendix
H empirically show that CN-DPM does not blindly increase the number of experts.
Continued increase of model capacity .
Our model capacity keeps increasing as it learns more tasks.
However, we believe this is one of the strengths of our method, since it may not make sense to use a ﬁxed-capacity neural network to learn an indeﬁnitely long sequence of tasks.
The underlying assumption of using a ﬁxed-capacity model is that the pre-set model capacity is adequate (at least not insufﬁcient) to learn the incoming tasks.
On the other hand, CN-DPM approaches the problem in a different direction: start small and add more as needed .
This property is essential in task-free settings where the total number of tasks are not known.
If there are too many tasks than expected, a ﬁxed-capacity model would not be able to successfully learn them.
Conversely, if there are less tasks than expected, resources would be wasted.
We argue that the expansion is a promising direction since it does not need to ﬁx the model capacity beforehand.
Moreover, we also introduce an algorithm to prune redundant experts in Appendix D,
Generality of the concentration parameter .
The concentration parameter controls how sensitive the model is to new data.
In other words, it determines the level of discrepancy between tasks that makes them modeled by distinct components.
As an example, suppose that we are designing a handwritten alphabet classiﬁer that continually learns in the real world.
In the development, we only have the character images for a half of alphabets, i.e .
from ‘a’ to ‘m’.
If we can ﬁnd a good concentration parameter α for the data from ‘a’ to ‘m’, the same α can work well with novel alphabets ( i.e . from ‘n’ to ‘z’) because the alphabets would have similar level of discrepancies between tasks.
Therefore, we do not need to access the whole data to determine α if the discrepancy between tasks is steady.
C M ODEL A RCHITECTURES AND E XPERIMENTAL D ETAILS C. B ASE M ODELS C.1. S PLIT -MNIST
Following Hsu et al.
(2018), we use two-hidden-layer MLP classiﬁer with 400 hidden units per layer.
For generation tasks, we use a simple VAE with the two-hidden-layer MLP encoder and decoder where each layer contains 400 units.
The dimension of latent code is set to 32.
We use
ReLU for all intermediate activation functions.
C.1. MNIST-SVHN AND S PLIT -CIFAR10/
We use ResNet-18 (He et al., 2016).
The input images are transformed to 32 × 32 RGB images.
C. CN-DPM C.2. S PLIT -MNIST
For the classiﬁers in experts we use a smaller version of the base MLP classiﬁer.
In the ﬁrst expert, we set the number of hidden units per layer to 64.
In the second or later experts, we introducenew units per layer which are connected to the lower layers of the existing experts.
For the encoder and decoder of VAEs, we use a two-layer MLP.
The encoder is expanded in the same manner as the classiﬁer.
However, we do not share the parameters beyond the encoders; with a latent code of dimension 16, we use the two-hidden-layer MLP decoder as done in the classiﬁer.
For generation tasks, we double the size; for example, we set the size of initial and additional hiden units toand 32, respectively.
C.2. S PLIT -CIFAR10/
The ResNet-18 base network has 8 residual blocks.
After passing through 2 residual blocks, the width and height of the feature are halved and the number of channels is doubled.
The initial number of channels is set to 64.
For the classiﬁers in CN-DPM, we use a smaller version of ResNet that has only four residual blocks and resizes the feature every block.
The initial number of channels is set to 20 in the ﬁrst expert, and 4 initial channels are added with a new expert.
Thus, 4, 8, 16, and 32 channels are added for the four blocks.
The ﬁrst layer of each block is connected to the last layer of the previous block of prior experts.
For the VAEs, we use a simple CNN-based VAEs.
The encoder has two 3 × 3 convolutions followed by two fully connected layers.
Each convolution is followed by 2 × 2 max-pool and ReLU activation.
The number of channels and hidden units are doubled after each layer.
In the ﬁrst expert, the ﬁrst convolution outputs 32 channels, while four new channels are added with each new expert.
As done for the VAE in Split-MNIST, each expert’s VAE has an unshared decoder with a 64dimensional latent code.
The decoder is the mirrored encoder where 3 × 3 convolution is replaced by 4 × 4 transposed convolution with a stride of 2.
C.2. MNIST-SVHN
For the classiﬁer, we use ResNet-18 with 32 channels for the ﬁrst expert and additional 32 channels for each new expert.
We use the same VAE as in Split-CIFAR10.
C. E XPERIMENTAL D ETAILS
We use the classiﬁer temperature parameter of 0.01 for Split-MNIST, Split-CIFAR10/100, and no temperature parameter on MNIST-SVHN.
Weight decay 0.00001 has been used for every model in the paper.
Gradients are clipped by value with threshold 0.5.
All the CN-DPM models are trained by the Adam optimizer.
During the sleep phase, we train the new expert for multiple epochs with a batch size of 50.
In classiﬁcation tasks, we improve the density estimation of VAEs by samplinglatent codes and averaging the ELBOs, following Burda et al. (2015).
C.3. S PLIT -MNIST
Learning rate of 0.0001 and 0.0004 has been used for the classiﬁer and VAE of each expert in the classiﬁcation task.
We use learning rate 0.003 for the VAE of each expert in generation task.
In the generation task, we decay the learning rate of the expert by 0.003 before it enters the wake phase.
Following the existing works in VAE literature, we use binarized MNIST for the generation experiments.
VAEs are trained to maximize Bernoulli log-likelihood in the generation task while
Gaussian log-likelihood is used for the classiﬁcation task.
C.3. S PLIT -CIFAR
Learning rate of 0.005 and 0.0002 has been used for the classiﬁer and VAE of each expert in CIFAR10.
We decay the learning rate of the expert by 0.1 before it enters the wake phase.
VAEs are trained to maximize Gaussian log-likelihood.
C.3. S PLIT -CIFAR
Learning rate of 0.0002 and 0.0001 has been used for the classiﬁer and VAE of each expert in
CIFAR10.
We decay the learning rate of the expert by 0.2 before it enters the wake phase.
VAEs are trained to maximize Gaussian log-likelihood.
C.3. MNIST-SVHN
Learning rate of 0.0001 and 0.0003 has been used for the classiﬁer and VAE of each expert in
CIFAR10.
We decay the learning rates of classiﬁer and VAE of each expert by 0.5 and 0.1 before it enters the wake phase.
VAEs are trained to maximize Gaussian log-likelihood.
D P RUNING R EDUNDANT E XPERTS
Lin (2013) propose a simple algorithm to prune and merge redundant components in DPM models.
Following the basic principle of the algorithm, we provide a pruning algorithm for CN-DPM.
First, we need to measure the similarities between experts to choose which expert to prune.
We compute the log-likelihood l nk = p ( x n , y n | ˆ φ k ) of each expert k for data ( x 1: N , y 1: N ) .
As a result, we can obtain K vectors with N dimensions.
We deﬁne the similarity s ( k, k (cid:48) ) between two experts k and k (cid:48)
as the cosine similarity between the two corresponding vectors l · k and l · k (cid:48) , i.e . s ( k, k (cid:48) )
=
l · k · l · k (cid:48) | l · k || l · k (cid:48) | .
If the similarity is greater than a certain threshold (cid:15) , we remove one of the experts with smaller
N k = (cid:80) n ρ n,k .
The N k data of the removed expert are added to the remaining experts.
Figure 4 shows an example of expert pruning.
We test CN-DPM on Split-MNIST with an α higher than the optimal value such that more than 5 experts are created.
In this case, seven experts are created.
If we build a similarity matrix as shown in Figure 4b, we can see which pair of experts are similar.
We then threshold the matrix at 0.9 in Figure 4c and choose expert pairs (2/3) and (5/6) for pruning.
Comparing N k within each pair, we can ﬁnally choose to prune expert 3 and 6.
After pruning, the test accuracy marginally drops from 87.07% to 86.01%.
[Figure] [Figure]
[Figure]
E C OMPARISON WITH T ASK -B ASED M ETHODS ON S PLIT -MNIST
Table 4 compares our method with task-based methods for Split-MNIST classiﬁcation.
All the numbers except for our CN-DPM are excerpted from Hsu et al.
(2018), in which all methods are trained for 4 epochs per task with a batch size of 128.
Our method is trained for 4 epochs per task with a batch size of 10.
The model architecture used in compared methods is the same with our baselines: a two-hidden-layer MLP with 400 hidden units per layer.
All compared methods use single output head, and the task information is given at training time but not at test time.
For CNDPM, we test two training settings where the ﬁrst one uses task information to select experts, while the second one infer the responsible expert by the DPM principle. Task information is not given at test time in both cases.
Notice that regularization methods often suffer from catastrophic forgetting while replay methods yield decent accuracies.
Even though the task-free condition is a far more difﬁcult setting, the performance of our method is signiﬁcantly better than regularization and replay methods which exploit the task description.
If task information is available at train time, we can utilize it to improve the performance even more.
[Table]
F F UZZY S PLIT -MNIST
In addition
[Table]
we experiment the case where the task boundaries are not clearly deﬁned
[Table]
which we call Fuzzy-Split-MNIST .
Instead of discrete task boundaries, we have transition stages between tasks where the data of existing and new tasks are mixed but the proportion of the new task linearly increases.
This condition adds another level of difﬁculty since it makes the methods unable to rely
Method
Acc.
[Table]
Param.
Fine-
[Figure]
[Table] 0 .478 K
Reservoir 88 .
64 ± 0 .
[Figure]
478 K
[Figure]
DPM 93.22 ± 0 .524 K 0K 12K 24K 36K 48K 60K
Learned examples
C u m u l a t
i v e t
a s k p r o p o r t i o n
T a s k
T a s k T a s k
T a s k T a s k
[Figure]
on clear task boundaries.
The scenario is visualized in Figure 5.
As shown in Table 5, CN-DPM can perform continual learning without task boundaries.
G G ENERATION OF S AMPLES
Even in discriminative tasks where the goal is to model p ( y | x ) , CN-DPM learns the joint distribution p ( x, y ) .
Since CN-DPM is a complete generative model, it can generate ( x, y ) pairs.
To generate a sample, we ﬁrst sample z from p ( z ) which is modeled by the categorical distribution
Cat( N 1 N , N 2 N , ..., N K N ) , i.e . choose an expert.
Given z
=
k , we ﬁrst sample x from the generator p ( x ; φ Gk ) , and then sample y from the discriminator p ( y | x ; φ Dk ) .
Figure 6 presents 50 sample examples generated from a CN-DPM trained on Split-MNIST for a single epoch.
We observe that
CN-DPM successfully generates examples of all tasks with no catastrophic forgetting.
H E XPERIMENTS WITH L ONGER C ONTINUAL L EARNING S CENARIOS
We present experiments with much longer continual learning scenarios on Split-MNIST, SplitCIFAR10 and Split-CIFAR100 in Table 6, 7 and 8, respectively.
We report the average of 10 runs with ± standard error of the mean.
To compare with the default 1-epoch scenario, we carry out experiments that repeat each task 10 times, which are denoted 10 Epochs .
In addition, we also present the results of repeating the whole scenario 10 times which are denoted 1 Epoch × 10 .
For example, in Split-MNIST, the 10 Epochs scenario consists of 10-epoch 0/1, 10-epoch 2/3, ..., 10-epoch 8/ tasks.
On the other hand, the 1 Epoch × 10 scenario revisits each task multiple times, i.e . 1-epoch 0/1, 1-epoch 2/3, ..., 1-epoch 8/9, 1-epoch 0/1, ..., 1-epoch 8/9.
We use the same hyperparameters tuned for the 1-epoch scenario.
We ﬁnd that the accuracy of Reservoir drops as the length of each task increases.
As mentioned in the main text, this phenomenon seems to be caused by the overﬁtting on the samples in the replay memory.
Since only a small amount of examples in the memory represents each task, replaying them for a long period degrades the performance.
On the other hand, the performance of our CN-DPM improves as the learning process is extended.
In the 1 Epoch × 10 setting, CN-DPM shows similar performance with 10 Epoch since the model sees each data point 10 times in both scenarios.
On the other hand, Reservoir’s scores in the 1 Epoch × 10 largely increase compared to both 1 Epoch and 10 Epoch
This difference can be explained by how the replay memory changes as training progresses.
In the 10 Epoch setting, if a task is ﬁnished, it is not visited again.
Therefore, the examples of the task in the replay memory monotonically decreases, and the remaining examples are replayed repeatedly.
As the training progresses, the model is overﬁtted to the old examples in the memory and fails to generalize in the old tasks.
In contrast, in 1 Epoch × 10 setting, each task is revisited multiple times, and each time a task is revisited, the replay memory is also updated with the new examples of the task.
Therefore, the overﬁtting problem in the old tasks is greatly relieved.
Another important remark is that CN-DPM does not blindly increase the number of experts.
If we add a new expert at every constant steps, we would have 10 times more experts in the longer scenarios.
However, this is not the case.
CN-DPM determines whether it needs a new expert on a data-by-data basis such that the number of experts is determined by the task distribution, not by the length of training.
[Table]
[Table] [Table]
I E XPERIMENTS WITH D IFFERENT M EMORY S IZES
In Split-CIFAR10/100 experiments in the main text, we set the memory size of Reservoir and CNDPM to 1000, following Aljundi et al. (2019b).
Table 9 compares the experimental results with different memory sizes of 500 and 1000 on Split-CIFAR10/100.
Compared to Reservoir, whose performance drops signiﬁcantly with a smaller memory, CN-DPM’s accuracy drop is relatively marginal.
[Table]
J T HE E FFECT OF C ONCENTRATION P ARAMETER
Table 10 shows the results of CN-DPM on Split-MNIST classiﬁcation according to the concentration parameter α , which deﬁnes the prior of how sensitive CN-DPM is to new data.
With a higher α , an expert tends to be created more easily.
In the experiment reported in the prior sections, we set log α = − 400 .
At log α = − 600 , too few experts are created and the accuracy is rather low.
As α increases, the number of experts grows along with the accuracy.
Although the CN-DPM model is task-free and automatically decides the task assignments to experts, we still need to tune the concentration parameter to ﬁnd the best balance point between performance and model capacity, as all Bayesian nonparametric models require.
effects of concentration parameter α . log α Acc. Experts
Param. −54 . 04 ± 2 .3 . 20 ± 0 .362 K −93 . 23 ± 0 .5 . 00 ± 0 .
[Figure]
K 93 .
54 ± 0 .
[Figure]
±
[Figure]
K T HE E FFECT OF P ARAMETER S HARING
Table 11 compares when the parameters are shared between experts and when they are not shared.
By sharing the parameters, we could reduce the number of parameters approximately 38% without sacriﬁcing the accuracy.
[Table]
L T RAINING G RAPHS
Figure 8 shows the training graphs of our experiments.
In addition to the performance metrics, we present the number of experts in CN-DPM and compare the total number of parameters with the baselines.
The bold lines represent the average of the 10 runs while the faint lines represent individual runs.
Figure 9 and Figure 10 show how the accuracies of each task change during training.
We also present the average accuracy of learned tasks at the bottom right.
[Figure]
[Figure]
[Figure]