M IXTURE - OF -V ARIATIONAL -E XPERTS FOR C ONTINUAL L EARNING
A BSTRACT
One weakness of machine learning algorithms is the poor ability of models to solve new problems without forgetting previously acquired knowledge.
The Continual Learning (CL) paradigm has emerged as a protocol to systematically investigate settings where the model sequentially observes samples generated by a series of tasks.
In this work, we take a task-agnostic view of continual learning and develop a hierarchical information-theoretic optimality principle that facilitates a trade-off between learning and forgetting.
We discuss this principle from a Bayesian perspective and show its connections to previous approaches to CL.
Based on this principle, we propose a neural network layer, called the Mixture-ofVariational-Experts layer, that alleviates forgetting by creating a set of information processing paths through the network which is governed by a gating policy.
Due to the general formulation based on generic utility functions, we can apply this optimality principle to a large variety of learning problems, including supervised learning, reinforcement learning, and generative modeling.
We demonstrate the competitive performance of our method in continual supervised learning and in continual reinforcement learning.
I NTRODUCTION
Acquiring new skills without forgetting previously acquired knowledge is a hallmark of human and animal intelligence.
Biological learning systems leverage task-relevant knowledge from preceding learning episodes to guide subsequent learning of new tasks to accomplish this.
Artiﬁcial learning systems, such as neural networks, usually lack this crucial property and experience a problem coined ”catastrophic forgetting” (McCloskey & Cohen, 1989).
Catastrophic forgetting occurs when we naively apply machine learning algorithms to solve a sequence of tasks T 1: t , where adaptation to task T t prompts overwriting of parameters learned for tasks T 1: t − 1 .
The Continual Learning (CL) paradigm (Thrun, 1998) has emerged as a way to investigate such problems systematically.
We can divide CL approaches into three broad categories: rehearsal and memory consolidation, regularization and weight consolidation, and architecture and expansion methods.
Rehearsal methods train a generative model to learn the data-generating distribution to reproduce data of old tasks (Shin et al., 2017; Rebufﬁ et al., 2017).
In contrast, regularization methods (e.g., Kirkpatrick et al., 2017; Ahn et al., 2019; Han & Guo, 2021a) introduce an additional constraint to the learning objective to prevent changes in task-relevant parameters.
Finally, CL can be achieved by modifying the design of a model during learning (e.g., Lin et al., 2019; Rusu et al., 2016; Golkar et al., 2019).
Despite recent progress in CL, there are still open questions (Parisi et al., 2019).
For example, most existing algorithms share a signiﬁcant drawback in that they require task-speciﬁc knowledge, such as the number of tasks and which task is currently at hand.
Approaches sharing this drawback are multi-head methods (e.g., El Khatib & Karray, 2019;
Nguyen et al., 2017; Ahn et al., 2019) and methods that compute a per-task loss, which requires storing old weights and task information (e.g.,
Kirkpatrick et al., 2017; Zenke et al., 2017; Sokar et al., 2021; Yoon et al., 2018; Chaudhry et al., 2021; Han & Guo, 2021b).
Extracting relevant task information is a difﬁcult problem, in particular when distinguishing tasks without contextual input (Hihn & Braun, 2020b).
Thus, providing the model with such task-relevant information yields overly optimistic results (Chaudhry et al., 2018a).
To deal with more realistic CL scenarios, therefore, models must learn to compensate for the lack of auxiliary information.
The approach we propose here tackles this problem by formulating a hierarchical learning system, that allows us to learn a set of sub-modules specialized in solving particular tasks.
To this end, we introduce hierarchical variational continual learning (HCVL) and devise the mixture-of-variational-experts layer (MoVE layers) as an instantiation of HVCL.
MoVE layers consist of M experts governed by a gating policy, where each expert maintains a posterior distribution over its parameters alongside a corresponding prior.
A sparse selection reduces computation as only a small subset of the parameters must be updated during the back-propagation of the loss (Shazeer et al., 2017).
To mitigate catastrophic forgetting we condition the prior distributions on previously observed tasks and add a penalty term on the Kullback-Leibler-Divergence between the expert posterior and its prior.
In ensemble methods two main questions arise.
The ﬁrst one concerns the question of optimally selecting ensemble members using appropriate selection and fusion strategies (Kuncheva, 2004).
The second one, is the question of how to ensure expert diversity (Kuncheva & Whitaker, 2003; Bian &
Chen, 2021).
We argue that ensemble diversity beneﬁts continual learning and investigate two complementary diversity objectives: the entropy of the expert selection process and a similarity measure between different experts based on Wasserstein exponential kernels in the context of determinantal point processes (Kulesza et al., 2012).
To summarize, our contributions are the following: ( i ) we extend VCL to a hierarchical multiprior setting, ( ii ) we derive a computationally efﬁcient method for task-agnostic continual learning from this general formulation, ( iii ) to improve expert specialization and diversity, we introduce and evaluate novel diversity measures.
This paper is structured as follows: we introduce our method in Section 2, we design, perform, and evaluate the main experiments in Section 3, in Section 4, we discuss novel aspects of the current work in the context of previous literature and conclude with a ﬁnal summary in Section 5.
H IERARCHICAL V ARIATIONAL C ONTINUAL L EARNING
In this section we ﬁrst extend the variational continual learning (VCL) setting introduced by Nguyen et al.
to a hierarchical multi-prior setting and then introduce a neural network implementation as a generalized application of this paradigm in Section 2.1.
VCL describes a general learning paradigm wherein an agent stays close to an old strategy (”prior”) it has learned on a previous task t − 1 while learning to solve a new task t (”posterior”).
Given datasets of input-output pairs D t = { x it , y it } N t
i
=0 of tasks t ∈ { 1 , ..., T } , the main learning objective of minimizing the log-likelihood log p θ ( y it | x it ) for task t is augmented with an additional loss term in the following way:
[Equation]
where p ( θ ) is a distribution over the models parameter θ and N t is the number of samples for task t .
The constraint encourages the agent to ﬁnd an optimal trade-off between solving a new task and retaining knowledge about old tasks.
Over the course of T datasets, Bayes’ rule then recovers the posterior
[Equation]
which forms a recursion: the posterior after seeing T datasets is obtained by multiplying the posterior after T − 1 with the likelihood and normalizing accordingly.
This multi-head strategy has two main drawbacks: ( i ) it introduces an organizational overhead due to the growing number of network heads, and ( ii ) task boundaries must be known at all times, making it unsuitable for more complex continual learning settings.
In the following we argue that [Figure] we can alleviate these problems by combining multiple
To extend VCL to the hierarchical case, we assume that samples are drawn from a set of M independent data generating processes, i.e., the likelihood is given by a mixture model p ( y | x )
= (cid:80)
Mm =1 p ( m | x ) p ( y | m, x ) .
We deﬁne an indicator variable z ∈ Z , where z i,tm is 1 if the output y ti of sample i from task t was generated by expert m and zero otherwise:
[Equation]
where ϑ are the parameters of the selection policy,
ω m the parameters of the m -th expert, and Θ = { ϑ, { ω m } Mm =1 } the combined model parameters.
The posterior after observing T tasks is then given by
[Equation]
The Bayes posterior of an expert p ( ω m |D 1: T ) is recovered by computing the marginal over the selection variables Z .
Again, this forms a recursion, in which the posterior p (Θ |D 1: T ) depends on the posterior after seeing T − 1 tasks and the likelihood p ( D T | Θ) .
Finally, we formulate the HVCL objective for task t as:
[Equation]
where N t is the number of samples in task t , and the likelihood p ( y, x | Θ) is deﬁned as in equation 3.
2. S PARSELY G ATED M IXTURE - OF -V ARIATIONAL L AYERS
As we aim to tackle not only supervised learning problems, but also reinforcement learning problems, we assume in the following a generic scalar utility function U ( x, f θ ( x )) that depends both on the input x and the parameterized agent function f θ ( x ) that generates the agent’s output y .
We assume the agents output function f θ ( x ) is composed of multiple layers.
Our layer design builds on the sparsely gated Mixture-of-Expert (MoE) layers (Shazeer et al., 2017), which in turn draws on the paradigm introduced by Jacobs et al.
MoEs consist of a set of m experts M and a gating network p ( m | x ) whose output is a (sparse) m
-dimensional vector.
All experts have an identical architecture but separate parameters.
Let p ( m | x ) be the gating output and p ( y | m, x ) the response of an expert m given input x .
The layer’s output is then given by a weighted sum of the experts responses.
To save computation time we employ a top- k gating scheme, where only the k experts with
Diverse, but not specialized Specialized, but not diverse Specialized and diverse θθ 1 θ [Figure] θ [Figure]
Posterior p
Priorq {+
 Posterior p+} 1 Priorq [Figure] det ( K ) ( k ( p 0 ,p 0 ) ,k ( p 1 ,p 0 ))
( k ( p 0 ,p 1 ) ,k ( p 1 ,p 1 )) det ( K )=
k ( p 0 ,p 0 ) k
( p 1 ,p 1 )− k
( p 1 ,p 1 ) k
( p 0 ,p 1 )
K =
( k ( p 0 ,p 0 ) k ( p 0 ,p 1 ) k
( p 1 ,p 0 )
k ( p 1 ,p 1 ) ) ( 0,0 ) ( k ( p 0 ,p 0 )
+ k ( p 0 ,p 1 ) ,k ( p 1 ,p 0 )
+ k ( p 1 ,p 1 ))
p 0 opt ,p 1 opt ,p 2 opt ,p 3 opt = argmaxdet ( K ) p 0 ,p 1 ,p 2 ,pk ( p i ,p j )=
exp ( − W 22 ( p i ,p j ) 2 h 2 ) highest gating activation are evaluated and use an additional penalty that encourages gating sparsity (see Section 2.1). In all our experiments we set k = 1 , to drive expert specialization (see Section 2.2.1) and reduce computation time. We implement the learning objective for task t as layer-wise regularization in the following way:
[Equation]
where L is the total number of layers, Θ = { θ, { ϑ m } Mm =1 } the combined parameters,and the temperature parameters β 1 , 2 govern the layer-wise trade-off between utility and information-cost.
Thus, we allow for two major generalizations compared to equation 5: in lieu of the log-likelihood we allow for generic utility functions, and instead of applying the constraint on the gating parameters, we apply it directly on the gating output distribution p ( m | x ) .
This implies, that the weights of the gating policy are not sampled.
Otherwise the gating mechanism would involve two stochastic steps: one in sampling the weights and a second one in sampling the expert index.
This potentially high selection variance hinders expert specialization (see Section 2.2.1).
Encouraging the gating policy to stay close to its prior also ensures that similar inputs are assigned to the same expert.
Next we consider how we could extend objective 6 further by additional terms that encourage diversity between different experts.
2. E NCOURAGING E XPERT D IVERSITY
In the following, we argue that a diverse set of experts may mitigate catastrophic forgetting in continual learning, as experts specialize more easily in different tasks, which improves expert selection.
We present two expert diversity objectives.
The ﬁrst one arises directly from the main learning objective and is designed to act as a regularizer on the gating policy while the second one is a more sophisticated approach that aims for diversity in the expert parameter space.
The latter formulation introduces a new class of diversity measures, as we discuss in more detail in Section 4.
2.2. D IVERSITY THROUGH S PECIALIZATION
The relationship between objectives of the form described by equation 6 with the emergence of expert specialization has been previously investigated for simple learning problems (Genewein et al., 2015) and in the context of meta-learning (Hihn & Braun, 2020b), but not in the context of continual learning.
We assume a two-level hierarchical system of specialized decision-makers where lowlevel decision-makers p ( m | x ) select which high-level decision-maker p ( y | m, x ) serve as experts for a particular input x .
By co-optimizing
[Equation]
7) the combined system ﬁnds an optimal partitioning of the input space X , where I ( ·|· ) denotes the (conditional) mutual information between random variables.
In fact, the hierarchical VCL objective given by equation 5 can be regarded as a special case of the information-theoretic objective given by equation 7, if we interpret the prior as the learning strategy of task t − 1 and the posterior as the strategy of task t , and set β 1 = β 2 = 1 .
All these hierarchical decision systems correspond to a multi prior setting, where different priors associated with different experts can specialize on different sub-regions of the input space .
In contrast, specialization in the context of continual learning can be regarded as the ability of partitioning the task space , where each expert decision-maker m solves a subset of old tasks T m ⊆ T 1: t .
In both cases, expert diversity is a natural consequence of specialization if the gating policy p ( m | x ) partitions between the experts.
In addition to the implicit pressures for specialization already implied by equation 6, here we investigate the effect of an additional entropy cost.
Inspired by recent entropy regularization techniques (Eysenbach et al., 2018; Galashov et al., 2019; Grau-Moya et al., 2019), we aim to improve the gating policy by introducing the entropy cost − β 3 N b (cid:80) x ∈B H ( M ) , where M is the set of experts and X the inputs in a mini-batch B of size N b , and β 3 a weight.
Thus, by minimizing the marginal entropy H ( M ) we prefer solutions that minimize the number of active experts.
We compute these values batch-wise, as the full entropies over p ( x ) are not tractable.
2.2. P ARAMETER D IVERSITY
Our second diversity formulation builds on Determinantal Point Processes (DPPs) (Kulesza et al., 2012), a mechanism that produces diverse subsets by sampling proportionally to the determinant of the kernel matrix of points within the subset (Macchi, 1975).
A point process P on a ground set Y is a probability measure over ﬁnite subsets of Y .
P is a DPP if, when Y is a random subset drawn according to P , we have, for every A ⊂ Y , P ( A ⊂ Y )
=
det ( K A ) for some real, symmetric N × N matrix K indexed by the elements of Y .
Here, K A =
[ K ij ] i,
j ∈
A denotes the restriction of K to the entries indexed by elements of A , and we adopt det ( K ∅ )
= 1 .
If A = { i } is a singleton, then we have P ( i ∈ Y ) =
K i,i .
In this case, the diagonal of K gives the marginal probabilities of inclusion for individual elements of Y .
Since P is a probability measure, all principal minors of K must be nonnegative, and thus K itself must be positive semideﬁnite, which we can achieve by constructing
K by a kernel function k ( x 0 , x 1 ) , such that for any x 0 , x 1 ∈ X :
[Equation]
where X is a vector space and F is a inner-product space such that ∀ x ∈ X : φ ( x ) ∈ F .
Speciﬁcally, we use a exponential kernel based on the Wasserstein-2 distance W ( p, q ) between two probability distributions p and q .
The p th Wasserstein distance between two probability measures p and q in
P p ( M ) is deﬁned as
[Equation]
where Γ( p, q ) denotes the collection of all measures on M × M with marginals p and q on the ﬁrst and second factors.
Let p and q be two isotropic Gaussian distributions and W 22 ( q, p )
the
Wasserstein-2 distance between p and q .
The exponential Wasserstein-2 kernel is then deﬁned by
[Equation]
where h is the kernel width.
We show in Appendix B that equation 10 gives a valid kernel.
This formulation has two properties that make it suitable for our purpose.
Firstly, the Wasserstein distance is symmetric, i.e., W 22 ( p, q )
=
W 22 ( q, p ) , which in turn will lead to a symmetric kernel matrix.
This is not true for other similarity measures on probability distributions, such as D KL (Cover & Thomas, 2012).
Secondly, if p and q are Gaussian and mean-ﬁeld approximations, i.e., covariance matrices are given by diagonal matrices, i.e., Σ p = diag ( d p ) and Σ q = diag ( d q ) , W 22 ( p, q ) can be computed in closed form as
[Equation]
where µ p,q are the means and d p,q the diagonal entries of distributions p and q .
We provide a more detailed derivation of equation 11 in Appendix A. From a geometric perspective, the determinant of the kernel matrix represents the volume of a parallelepiped spanned by feature maps corresponding to the kernel choice.
We seek to maximize this volume, effectively ﬁlling the parameter space – see
Figure 2 for an illustration.
[Table]
E XPERIMENTS
We evaluate our approach in current supervised CL benchmarks in Section 3.1, in a generative learning setting in Section 3.2, and in the CRL setup in Section 3.3.
We give experimental details in
Appendix C and additional ablation studies in Appendix D.
3. C ONTINUAL S UPERVISED L EARNING S CENARIOS
The basic setting of continual learning is deﬁned as an agent which sequentially observes data from a series of tasks while maintaining performance on older tasks.
We evaluate the performance of our method in this setting in split MNIST, permuted MNIST, split CIFAR-10/100 (see Table 1).
We follow the domain incremental setup (van de Ven et al., 2020), where task information is not available, but we also compare against task-incremental methods, where the task information is available, to give a complete overview of current methods.
The ﬁrst benchmark builds on the MNIST dataset.
Five binary classiﬁcation tasks from the MNIST dataset arrive in sequence and at time step t the performance is measured as the average classiﬁcation accuracy on all tasks up to task t .
In permuted MNIST the task received at each time step t consists of labeled MNIST images whose pixels have undergone a ﬁxed random permutation.
The second benchmark is a variation of the CIFAR-10/100 datasets.
In Split CIFAR-10, we divide the ten classes into ﬁve binary classiﬁcation tasks.
CIFAR-100 is like the CIFAR-10, except it has 100 classes and tasks are deﬁned as a 10-way classiﬁcation problem, thus forming ten tasks in total.
We achieve comparable results to current state-of-the-art approaches (see Table 1 on all supervised learning benchmarks.
3. G ENERATIVE C ONTINUAL L EARNING
Generative CL is a simple but powerful paradigm (van de Ven et al., 2020).
The main idea is to learn the data generating distribution and simulate data of previous tasks.
We can extend our approach to the generative setting by modeling a variational autoencoder using the novel layers we propose in this work.
We model the distribution of the latent variable z in the variational autoencoder by using a densely connected MoVE layer with 3 experts.
Using multiple experts enables us to capture a richer class of distributions than a single Gaussian distribution could, as is usually the case in simple VAEs.
We can interpret this as z following a Gaussian Mixture Model, whose components are mutually exclusive and modeled by experts.
We integrate the generated data by optimizing a mixture of the loss on the new task data and the loss of the generated data We were able to improve our results in the supervised settings by incorporating a generative component, as we show in Table 1.
We show additional empirical results in Appendix D.4.
3. C ONTINUAL R EINFORCEMENT L EARNING
In the continual reinforcement learning (CRL) setting, the agent is tasked with ﬁnding an optimal policy in sequentially arriving RL problems.
To benchmark our method, we follow the experimental protocol of Ahn et al.
and use a series of tasks from the PyBullet environments (Ellenberger, 2018–2019).
The environments we selected have different states and action dimensions.
This implies we can’t use a single model to learn policies and value functions.
To remedy this, we pad each state and action with zeros to have equal dimensions.
The Ant environment has the highest dimensionality with a state dimensionality of 28 and an action dimensionality of 8.
All others are zero-padded to have this dimensionality.
Here, we extend SAC (Haarnoja et al., 2018) by implementing all neural networks with MovE layers.
When a new task arrives, the old posterior over the expert parameters and the gating posterior become the new priors.
After each update step in task t , we evaluate the agent in all previous tasks
T 1: t for three episodes each.
We divide the reward achieved during evaluation by the mean reward during training and report the cumulative normalized reward, which gives an upper bound of t in the t -th task.
We compare our approach against a simple continuously trained SAC implementation with dense neural networks, EWC (Kirkpatrick et al., 2017), and the recently published UCL (Ahn et al., 2019) method.
UCL is similar to our approach in that it also employs Bayesian neural networks, but the weight regularization acts on a per-weight basis.
Note that UCL and EWC both require task information to compute task-speciﬁc losses.
Our results (see Figure 3) show that our approach can sequentially learn new policies while maintaining an acceptable performance on previously seen tasks.
Our method outperforms UCL (Ahn et al., 2019) and EWC (Kirkpatrick et al., 2017).
In this setting naively training the agent sequentially (labeled ”Dense”) yields poor performance.
This behavior indicates the complete forgetting of old policies.
D ISCUSSION
The principle we propose in this work falls into a wider class of methods that deal with learning and decision-making problems by integrating information-theoretic cost functions.
Such informationconstrained machine learning methods have enjoyed recent interest in a variety of research ﬁelds, e.g., as reinforcement learning (Eysenbach et al., 2018; Ghosh et al., 2018; Leibfried & Grau-Moya, 2019; Hihn et al., 2019;
Arumugam et al., 2020), MCMC optimization (Hihn et al., 2018; Pang et al.,
, meta-learning (Rothfuss et al., 2018; Hihn & Braun, 2020a), continual learning (Nguyen et al., 2017; Ahn et al., 2019), and self-supervised learning (Thiam et al., 2021;
Tsai et al., 2021).
Currently, there are only few methods that perform well in supervised CL and in CRL (e.g., Ahn et al., 2019; Jung et al., 2020; Cha et al., 2020).
These methods require task information, as they either keep a set of separate task-speciﬁc heads (Ahn et al., 2019;
Jung et al., 2020) or compute task-speciﬁc losses (Cha et al., 2020).
This makes our method one of the ﬁrst task-agnostic CL approaches to drop this requirement while still performing competitively.
The hierarchical structure we employ is a variant of the Mixture-of-Experts (MoE) model (Jacobs et al., 1991), speciﬁcally an extension of the sparsely-gated MoE layers (Shazeer et al., 2017).
Sparsely-gated MoE layers enforce a balanced load between experts.
In our work, we removed the incentive to equally distribute inputs, as we aim to ﬁnd specialized experts, which contradicts a balanced load.
The computational advantage remains, as we still activate only the top- 1 expert.
Our method is similar to the approach described by Hihn & Braun (2020b) but differs in two key aspects.
Firstly, we provide a more stable learning procedure as our layers can readily offer endto-end training.
Secondly, we implement the information-processing constraints on the parameters instead of the output of the experts, thus shifting the information cost from decision-making to learning.
Several methods in the current CL literature rest on modular architectures (e.g., Fernando et al., 2017; Collier et al., 2020; Lin et al., 2019; Lee et al., 2020).
Lin et al.
(2019) propose to condition model parameters on the inputs by learning a (deterministic) grouping function.
Our approach differs in two main ways.
First, our method can capture uncertainty allowing us to learn stochastic tasks.
Second, our design can incorporate up to 2 n paths (or groupings) through a neural net with n layers, making it more ﬂexible than learning a mapping function.
Lee et al.
(2020) propose a MoE model for continual learning, in which the number of experts increases dynamically, utilizing DirichletProcess-Mixtures (Antoniak, 1974) to infer the number of experts.
The authors argue that since the gating mechanism is itself a classiﬁer, training it in an online fashion would result in catastrophic forgetting.
To remedy this, they implement a generative model per expert m to model p ( m | x ) and approximate the output as p ( y | x ) ≈ (cid:80) m p ( y | x ) p ( m | x ) .
In our work, we have demonstrated that it is possible to implement a gating mechanism based only on the input by coupling it with an information-theoretic objective to prevent catastrophic forgetting.
Recently, several diversity measures have been proposed.
Parker-Holder et al.
(2020) introduce a DDP-based method based on the different states a given policy may reach.
Dai et al.
(2021) propose to augment the sampling process in hindsight experience replay (Andrychowicz et al., 2017) with a DPP-diversity bonus.
The method we propose differs from previous methods as we deﬁne diversity in parameter space instead of the policy outcomes or inputs.
Additionally, as we deﬁne it on parameters instead of actions, we can apply it straightforwardly to any problem formulation, as our experiments show.
C ONCLUSION
We introduced a hierarchical approach to task-agnostic continual learning, derived an application, and extensively evaluated this method in supervised CL and CRL.
While we removed the taskinformation limitation, we achieved results competitive to task-aware and to task-agnostic algorithms.
We argued that both VCL and hierarchical VCL have strong connections to an informationtheoretic formulation of bounded rationality.
We designed a diversity objective that stabilizes learning and further reduces the risk of catastrophic forgetting.
Our method builds on generic utility functions, we can apply it independently of the underlying problem, which makes our method one of the ﬁrst to do so.
A CKNOWLEDGMENT
This work was supported by the European Research Council, grant number ERC-StG-2015-ERC,
Project ID: 678082, “BRISC: Bounded Rationality in Sensorimotor Coordination”.
A W ASSERSTEIN -D ISTANCE BETWEEN TWO G AUSSIANS
The W 22 distance between two Gaussians is given by
[Equation]
Proof: Let p = N ( µ p , Σ p ) and q = N ( µ q , σ q ) be two Guassian distributions.
The Wassersteindistance between p and q is then given by
[Equation]
13) where B is the Bures metric between two positive semi-deﬁnite matrices:
[Equation]
[Equation]
where tr ( A ) is the trace of a matrix A and A 1 / 2 is the matrix square root.
Matrix square roots are computationally expensive to compute and there can potentially be and inﬁnite number of solutions.
In the case where p and q are Gaussian mean-ﬁeld approximations, i.e., all dimensions are independent, Σ p and Σ q are given by diagonal matrices, such that Σ p = diag ( d p ) i and Σ q = diag ( d q )
i .
The
Bures metric then reduces to the Hellinger distance between the diagonals d p and d q , and we have:
[Equation]
B W ASSERSTEIN -2 E XPONENTIAL K ERNEL
The exponential Wasserstein-2 kernel between isotropic Gaussian distributions p and q with kernel width h deﬁned by
[Equation]
is a valid kernel function.
Proof: The simplest way to show a kernel function k is valid is by deriving k from other valid kernels.
We can express the Wasserstein distance as the sum of two norms as shown in equation 15.
The euclidean norm and the Hellinger distance both form inner product spaces and are thus valid kernel functions.
Their sum is also a valid kernel function, which makes the Wasserstein distance on isotropic Gaussians a valid kernel.
If k ( p, q ) is a valid kernel, then exp( k ( p, q )) is also a valid kernel.
C E XPERIMENT D ETAILS
To implement variational layers we use Gaussian distributions.
For simplicity we use a
D − dimensional Gaussian mean-ﬁeld approximate posterior q t ( θ ) =
(cid:81)
Dd =1 N ( θ t | p t,d , σ 2 t,d ) .
We use the ﬂip-out estimator (Wen et al., 2018) to approximate the gradients.
In practice, we draw a single sample to approximate the expectation.
C. MNIST E XPERIMENTS
For split MNIST experiments we used dense layers for both the VAE and the classiﬁer.
The VAE encoder contains two layers with 256 units each, followed by 64 units (64 units for the mean and 64 units for log-variance) for the latent variable, and two layers with 256 units for the decoder, followed by an output layer with 28 ∗ 28 = 784 units.
This assumes isotropic Gaussians as priors and posteriors over the latent variable and allows to compute the D KL if closed form.
We used only one expert for the VAE with β 1 = 0 .
002 , β 2 = 0 .
75 , a diversity bonus weight of 0 .
01 and leaky
ReLU activations (Maas et al., 2013) in the hidden layers.
We trained with a batch size 256 forepochs.
The VAE output activation function is a sigmoid and we trained it using a binary crossentropy loss between the normalized pixel values of the original and the reconstructed images.
We used no other regularization methods on the VAE.
We used 10.000 generated samples after each task.
The classiﬁer consists of two dense layers, each with 256 units with leaky ReLU activations (Maas et al., 2013) and dropout (Srivastava et al., 2014) layers, followed by an output layer with two units.
All layers of the classiﬁer have two experts.
We trained with batch size 256 for 150 epochs using
Adam (Kingma & Ba, 2015) with a learning rate of 6 ∗ 10 − 4 .
In the permuted MNIST setting we used the same architecture, but increased the number of units to 512.
C. CIFAR-10 E XPERIMENTS
The VAE encoder consisted of ﬁve convolutional layers with stride 4 with two experts, each with 8, 16, 32, 64, and 128 units, followed by two dense units with two experts, each with 256 units.
The latent variable has 128 dimensions, which we model by two dense layers: one with 128 units for the mean and one with 128 units for the log-variance.
The dense layer modeling the mean has three experts, the layer for the log-variance one expert.
We assume isotropic Gaussian distributions as priors and posteriors over the latent variable, which allows us to compute the D KL if closed form.
The decoder mirrors the encoder and has two dense layers followed by 5 de-convolutional layers with stride 4 (the last layer has stride 3).
All hidden layers use a leaky ReLU activation function (Maas et al., 2013).
The VAE output activation function is a sigmoid and we trained it using a binary cross-entropy loss between the normalized pixel values of the original and the reconstructed images.
We used no other regularization methods on the VAE.
We used 10.000 generated samples after each task.
The classiﬁer architecture is similar to the encoder architecture.
We used ﬁve convolutional layers, followed by two dense layers.
All layers used two experts.
The convolutional layers have 8, 16, 32, 64, and 128 units per experts, while the dense layers both have 256 units per layer.
We used leaky
ReLU as an activation function for the hidden layers and softmax for the output layer.
We trained the classiﬁer using a binary cross-entropy loss between the true and the predicted label.
We trained with batch size 256 for 1000 epochs using the Adam optimizer with a learning rate of 3 ∗ 10 − 4 .
C. R EINFORCEMENT L EARNING E XPERIMENT D ETAILS
Each task was trained for one million time steps.
We use the same network architecture as suggested by the authors UCL: two layer networks (actor and critics) with 16 units each.
Each layer has four experts followed by leaky ReLU (Maas et al., 2013) activation functions.
Each We set each SAC related hyper-parameter as proposed in the original publication (Haarnoja et al., 2018).
For UCL (Ahn et al., 2019), we used the implementation provided by the authors for our experiments and use the hyper-parameters suggested in the publication.
Note that the UCL implementation rests on a
PPO (Schulman et al., 2017) backbone.
Our CRL experiments do not use any form of replay (except for the replay buffer used by SAC).
D A DDITIONAL A BLATION E XPERIMENTS
To further investigate the methods we propose in this work, we designed a set of ablation experiments.
In particular, we aim to demonstrate the importance of each component.
To this effect, we run experiments investigating the generator quality in the generative CL setting, study the diversity bonuses in the supervised CL scenario, and take a closer look at the number of experts and the inﬂuence of the D KL weights in the continual reinforcement learning setup.
D. I NVESTIGATING D IVERSITY B ONUSES
In Section 2.2 we introduced a diversity objective to stabilize learning in a mixture-of-experts system.
Additionally, we argued in favor of an entropy bonus to encourage a selection policy that favors high certainty and sparsity.
To investigate the validity of these additions, we run a set of experiments on the Split CIFAR-10 dataset as described in Section 3, but with different bonuses – see Figure 4.
In the baseline setup, we used no other objectives as those described by Equation equation 5.
Apart from the classiﬁcation accuracy, we are interested in three information-theoretic quantities that allow us to investigate the system closer.
Firstly, the mutual information between the data generating distribution p ( x ) and the expert selection p ( m | x ) as measured by I ( M ; X ) indicates how much uncertainty over m the gating unit can reduce on average after observing an input x .
A higher value means that inputs are differentiated better, which is what we would expect from a more diverse set of experts.
I ( M ; X ) is the highest when we use a DPP-based diversity objective (”DDP”), while the entropy of selection policy H ( M ) is lowest when we use an entropy-based diversity measure (”H-Penalties”), which both show that the objectives we introduced in this study yield the intended results.
Combining both (”DDP+H-Pen.”) enforces a trade-off between both objectives and yields the best empirical results.
We achieve the best results with a DDP diversity bonus combined with an entropy penalty on the expert selection.
We average the results of ten random seeds in each setting.
D. N UMBER OF E XPERTS
Our method builds on a mixture of experts model and it is thus natural to assume that increasing the number of experts improves performance.
Indeed, this is the case as we demonstrate in additional continual reinforcement learning experiments.
As Figure D.2 illustrates, adding experts to layers increases the number of possible information processing paths through the network.
Equipped with a diverse and specialized set of parameters, each path can be regarded as a distinct sub-network that learns to solve tasks.
D. D KL W EIGHTS
As with any hyper-parameter, setting a speciﬁc value for β 1 , 2 has a strong inﬂuence on the outcome of the experiments.
Setting it too small will lead to the regularization term dominating the loss, and the experts can’t learn a new task, as the new parameters remain close to the parameters of the previous task.
A high value will drive the penalty term towards zero, which, in turn, will not preserve parameters from old tasks.
In principle, there are three ways to choose β 1 , 2 .
First, by setting β 1 , 2 such that it satisﬁes an expert information-processing limit.
This technique has the advantage that we can interpret this value, e.g., ”each expert can process 1.57 bits of information on average, i.e., distinguishing between three options”, but shifts the burden from picking β 1 , 2 to setting a target entropy (see, e.g., Haarnoja et al.
and Grau-Moya et al.
for an example of this approach).
Second, employing a schedule for β 1 , 2 , as, e.g., proposed by Fu et al.
Last, another option is to run a grid search over a pre-deﬁned range and choose the one that ﬁts best.
In our supervised learning experiments, we used a cyclic schedule for β 1 and β 2 Fu et al.
(2019) while we kept them ﬁxed in the reinforcement learning experiments.
To systematically investigate the inﬂuence of these parameters, we conducted additional experiments (see Figure D.3).
D. G ENERATIVE CL
We introduce a generative approach to continual learning in Section 3.2 by implementing a Variational Auto-encoder using our proposed layer design.
This addition improved classiﬁcation performance and mitigated catastrophic forgetting, as evidenced by the results shown in Table 1.
We train by integrating artiﬁcially generated data into the process by optimizing a mixture loss:
[-L ( θ ) = 
-][Equation]
where B t is batch of data from the current task, B 1: t a batch of generated data, and (cid:96) ( b ) a loss function on the batch b .
Borrowing methods from generative learning, we investigate the performance of our proposed VAE design further, with the main focus on the quality of the generated images.
We can not measure the accuracy directly, as artiﬁcial images lack labels.
Thus we ﬁrst use the trained classiﬁer to obtain labels and compute metrics based on these self-generated labels.
We opted for the Inception
Score (IS) (Salimans et al., 2016), as it is widely used in the generative learning community.
In this initially proposed formulation, the IS builds on the D KL between the conditional and the marginal class probabilities as returned by a pre-trained Inception model (Szegedy et al., 2015).
To investigate the quality of the generated images concerning the continually trained classiﬁer, we use a different version of the Inception Score, which we deﬁned as
[Equation]
where G 1: T is the data generator trained on tasks up to T , p 1: T ( y | x ) the conditional class distribution returned by the classiﬁer trained up to task T , and p 1: T ( y ) the marginal class distribution up to Task
T .
Note that, IS ( G 1: T ) ≤ log 2 N c , where N c is the number of classes.
We show IS T and the entropy of p ( y ) in the split CIFAR-10 and split CIFAR-100 setting in Figure 7.