C ONTINUAL LEARNING WITH HYPERNETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Artiﬁcial neural networks suffer from catastrophic forgetting when they are sequentially trained on multiple tasks.
To overcome this problem, we present a novel approach based on task-conditioned hypernetworks, i.e., networks that generate the weights of a target model based on task identity.
Continual learning (CL) is less difﬁcult for this class of models thanks to a simple key feature: instead of recalling the input-output relations of all previously seen data, task-conditioned hypernetworks only require rehearsing task-speciﬁc weight realizations, which can be maintained in memory using a simple regularizer.
Besides achieving state-ofthe-art performance on standard CL benchmarks, additional experiments on long task sequences reveal that task-conditioned hypernetworks display a very large capacity to retain previous memories.
Notably, such long memory lifetimes are achieved in a compressive regime, when the number of trainable weights is comparable or smaller than target network size.
We provide insight into the structure of low-dimensional task embedding spaces (the input space of the hypernetwork) and show that task-conditioned hypernetworks demonstrate transfer learning.
Finally, forward information transfer is further supported by empirical results on a challenging CL benchmark based on the CIFAR-10/100 image datasets.
I NTRODUCTION
We assume that a neural network f ( x , Θ) with trainable weights Θ is given data from a set of tasks { ( X (1) , Y (1) ) , . . .
, ( X ( T ) , Y ( T ) ) } , with input samples X ( t ) =
{ x ( t,i ) } n t i =1 and output samples
Y ( t ) =
{ y ( t,i ) } n t i =1 , where n t ≡ | X ( t ) | .
A standard training approach learns the model using data from all tasks at once.
However, this is not always possible in real-world problems, nor desirable in an online learning setting.
Continual learning (CL) refers to an online learning setup in which tasks are presented sequentially (see van de Ven & Tolias, 2019, for a recent review on CL).
In CL, when learning a new task t , starting with weights Θ ( t − 1) and observing only ( X ( t ) , Y ( t ) ) , the goal is to ﬁnd a new set of parameters Θ ( t ) that (1) retains (no catastrophic forgetting) or (2) improves (positive backward transfer) performance on previous tasks compared to Θ ( t − 1) and (3) solves the new task t potentially utilizing previously acquired knowledge (positive forward transfer).
Achieving these goals is non-trivial, and a longstanding issue in neural networks research.
Here, we propose addressing catastrophic forgetting at a meta level: instead of directly attempting to retain f ( x , Θ) for previous tasks, we ﬁx the outputs of a metamodel f h ( e , Θ h ) termed task-conditioned hypernetwork which maps a task embedding e to weights Θ .
Now, a single point has to be memorized per task. To motivate such approach, we perform a thought experiment: we assume that we are allowed to store all inputs { X (1) , . . .
, X ( T ) } seen so far, and to use these data to compute model outputs corresponding to Θ ( T − 1) .
In this idealized setting, we could avoid forgetting by simply mixing data from the current task with data from the past, { ( X (1) , ˆ Y (1) ) , . . .
, ( X ( T − 1) , ˆ Y ( T − 1) ) , ( X ( T ) , Y ( T ) ) } , where ˆ Y ( t ) refers to a set of synthetic targets generated using the model itself f ( · , Θ ( t − 1) ) .
Hence, by training to retain previously acquired input-output mappings, we obtained a sequential algorithm in principle as powerful as multi-task learning.
Multi-task learning, where all tasks are learned simultaneously, can be seen as a CL upper-bound.
The strategy described above has been termed rehearsal (Robins, 1995).
However, storing previous task data violates our CL desiderata.
We introduce a change in perspective and move from the challenge of maintaining individual inputoutput data points to the problem of maintaining sets of parameters { Θ ( t ) } , without explicitly storing them.
To achieve this, we train the metamodel parameters
Θ h analogous to the above outlined learning scheme, where synthetic targets now correspond to weight conﬁgurations that are suitable for previous tasks.
This exchanges the storage of an entire dataset by a single low-dimensional task descriptor, yielding a massive memory saving in all but the simplest of tasks.
Despite relying on regularization, our approach is a conceptual departure from previous algorithms based on regularization in weight (e.g., Kirkpatrick et al., 2017; Zenke et al., 2017) or activation space (e.g., He & Jaeger, 2018).
Our experimental results show that task-conditioned hypernetworks do not suffer from catastrophic forgetting on a set of standard CL benchmarks.
Remarkably, they are capable of retaining memories with practically no decrease in performance, when presented with very long sequences of tasks.
Thanks to the expressive power of neural networks, task-conditioned hypernetworks exploit task-totask similarities and transfer information forward in time to future tasks.
Finally, the task-conditional metamodelling perspective that we put forth is generic, as it does not depend on the speciﬁcs of the target network architecture.
We exploit this key principle and show that the very same metamodelling framework extends to, and can improve, an important class of CL methods known as generative replay methods, which are current state-of-the-art performers in many practical problems (Shin et al., 2017; Wu et al., 2018; van de Ven & Tolias, 2018).
M ODEL 2. T ASK - CONDITIONED HYPERNETWORKS
Hypernetworks parameterize target models.
The centerpiece of our approach to continual learning is the hypernetwork, Fig.
1a.
Instead of learning the parameters
Θ trgt of a particular function f trgt directly (the target model ), we learn the parameters
Θ h of a metamodel.
The output of such metamodel, the hypernetwork, is Θ trgt .
Hypernetworks can therefore be thought of as weight generators, which were originally introduced to dynamically parameterize models in a compressed form (Ha et al., 2017; Schmidhuber, 1992; Bertinetto et al., 2016;
Jia et al., 2016).
[Equation]
x y regularized hypernetwork target network t Θ trgt e (1) y (2) y (3) f
trgt f trgt y (T) f trgt (1) Θ trgt (2) (1) x (2) Θ trgt (3) x (3) Θ trgt ( T ) x ( T ) f
h f h f h (1) e (2) e (3) e
( T ) b chunked hypernetwork ...
y ( t ) x ( t ) Θ trgt ( t )
f trgt c (1) c (2) c
( N c ) c (3) e ( t ) Θ trgt ( t, 1) Θ trgt( t, 2)
Θ trgt ( t, 3) Θ trgt ( t, N c)
f
h f h f h f h
Continual learning with hypernetwork output regularization.
One approach to avoid catastrophic forgetting is to store data from previous tasks and corresponding model outputs, and then ﬁx such outputs.
This can be achieved using an output regularizer of the following form, where past outputs play the role of pseudo-targets (Robins, 1995; Li & Hoiem, 2018; Benjamin et al., 2018):
[Equation]
In the equation above, Θ ∗ is the set of parameters before attempting to learn task T , and f is the learner.
This approach, however, requires storing and iterating over previous data, a process that is known as rehearsing .
This is potentially expensive memory-wise and not strictly online learning.
A possible workaround is to generate the pseudo-targets by evaluating f on random patterns (Robins, 1995) or on the current task dataset (Li & Hoiem, 2018).
However, this does not necessarily ﬁx the behavior of the function in the regions of interest.
Hypernetworks sidestep this problem naturally.
In target network weight space, a single point (i.e., one set of weights) has to be ﬁxed per task.
This can be efﬁciently achieved with task-conditioned hypernetworks, by ﬁxing the hypernetwork output on the appropriate task embedding.
Similar to Benjamin et al.
(2018), we use a two-step optimization procedure to introduce memorypreserving hypernetwork output constraints.
First, we compute a candidate change ∆Θ h which minimizes the current task loss L ( T ) task =
L task (Θ h , e ( T ) , X ( T ) , Y ( T ) ) with respect to Θ .
The candidate ∆Θ h is obtained with an optimizer of choice (we use Adam throughout; Kingma & Ba, 2015).
The actual parameter change is then computed by minimizing the following total loss:
[Equation]
where Θ ∗ h is the set of hypernetwork parameters before attempting to learn task T , ∆Θ h is considered ﬁxed and β output is a hyperparameter that controls the strength of the regularizer.
On Appendix D, we run a sensitivity analysis on β output and experiment with a more efﬁcient stochastic regularizer where the averaging is performed over a random subset of past tasks.
More computationally-intensive algorithms that involve a full inner-loop reﬁnement, or use secondorder gradient information by backpropagating through ∆Θ h could be applied.
However, we found empirically that our one-step correction worked well.
Exploratory hyperparameter scans revealed that the inclusion of the lookahead ∆Θ h in (2) brought a minor increase in performance, even when computed with a cheap one-step procedure.
Note that unlike in Eq.
1, the memory-preserving term L output does not depend on past data.
Memory of previous tasks enters only through the collection of task embeddings { e ( t ) }
T − 1 t =1 .
Learned task embeddings.
Task embeddings are differentiable deterministic parameters that can be learned, just like Θ h .
At every learning step of our algorithm, we also update the current task embedding e ( T ) to minimize the task loss L ( T ) task .
After learning the task, the ﬁnal embedding is saved and added to the collection { e ( t ) } .
2. M ODEL COMPRESSION WITH CHUNKED HYPERNETWORKS
Chunking.
In a straightforward implementation, a hypernetwork produces the entire set of weights of a target neural network.
However, hypernetworks can be invoked iteratively, ﬁlling in only part of the target model at each step, in chunks (Ha et al., 2017; Pawlowski et al., 2017).
This strategy allows applying smaller hypernetworks that are reusable.
Interestingly, with chunked hypernetworks it is possible to solve tasks in a compressive regime, where the number of learned parameters (those of the hypernetwork) is effectively smaller than the number of target network parameters.
Chunk embeddings and network partitioning.
Reapplying the same hypernetwork multiple times introduces weight sharing across partitions of the target network, which is usually not desirable.
To allow for a ﬂexible parameterization of the target network, we introduce a set C
= { c i } N C i =1 of chunk embeddings, which are used as an additional input to the hypernetwork, Fig.
1b.
Thus, the full set of target network parameters Θ trgt =
[ f h ( e , c 1 ) , . . .
, f h ( e , c N C )] is produced by iteration over C , keeping the task embedding e ﬁxed.
This way, the hypernetwork can produce distinct weights for each chunk.
Furthermore, chunk embeddings, just like task embeddings, are ordinary deterministic parameters that we learn via backpropagation.
For simplicity, we use a shared set of chunk embeddings for all tasks and we do not explore special target network partitioning strategies.
How ﬂexible is our approach?
Chunked neural networks can in principle approximate any target weight conﬁguration arbitrarily well.
For completeness, we state this formally in Appendix E.
2. C ONTEXT - FREE INFERENCE : UNKNOWN TASK IDENTITY
Determining which task to solve from input data.
The hypernetwork requires a task embedding input to generate target model weights.
In certain CL applications, an appropriate embedding can be immediately selected as task identity is unambiguous, or can be readily inferred from contextual clues.
In other cases, knowledge of the task at hand is not explicitly available during inference.
In the following, we show that our metamodelling framework generalizes to such situations.
In particular, we consider the problem of inferring which task to solve from a given input pattern, a noted benchmark challenge (Farquhar & Gal, 2018; van de Ven & Tolias, 2019).
Below, we explore two different strategies that leverage task-conditioned hypernetworks in this CL setting.
Task-dependent predictive uncertainty.
Neural network models are increasingly reliable in signalling novelty and appropriately handling out-of-distribution data.
For categorical target distributions, the network ideally produces a ﬂat, high entropy output for unseen data and, conversely, a peaked, low-entropy response for in-distribution data (Hendrycks & Gimpel, 2016; Liang et al., 2017).
This suggests a ﬁrst, simple method for task inference (HNET+ENT).
Given an input pattern for which task identity is unknown, we pick the task embedding which yields lowest predictive uncertainty, as quantiﬁed by output distribution entropy.
While this method relies on accurate novelty detection, which is in itself a far from solved research problem, it is otherwise straightforward to implement and no additional learning or model is required to infer task identity.
Hypernetwork-protected synthetic replay.
When a generative model is available, catastrophic forgetting can be circumvented by mixing current task data with replayed past synthetic data (for recent work see Shin et al., 2017; Wu et al., 2018).
Besides protecting the generative model itself, synthetic data can protect another model of interest, for example, another discriminative model.
This conceptually simple strategy is in practice often the state-of-the-art solution to CL (van de Ven &
Tolias, 2019).
Inspired by these successes, we explore augmenting our system with a replay network, here a standard variational autoencoder (VAE; Kingma & Welling, 2014) (but see Appendix F for experiments with a generative adversarial network, Goodfellow et al., 2014).
Synthetic replay is a strong, but not perfect, CL mechanism as the generative model is subject to drift, and errors tend to accumulate and amplify with time.
Here, we build upon the following key observation: just like the target network, the generator of the replay model can be speciﬁed by a hypernetwork.
This allows protecting it with the output regularizer, Eq. 2, rather than with the model’s own replay data, as done in related work.
Thus, in this combined approach, both synthetic replay and task-conditional metamodelling act in tandem to reduce forgetting.
We explore hypernetwork-protected replay in two distinct setups.
First, we consider a minimalist architecture (HNET+R), where only the replay model , and not the target classiﬁer, is parameterized by a hypernetwork.
Here, forgetting in the target network is obviated by mixing current data with synthetic data.
Synthetic target output values for previous tasks are generated using a soft targets method, i.e., by simply evaluating the target function before learning the new task on synthetic input data.
Second (HNET+TIR), we introduce an auxiliary task inference classiﬁer, protected using synthetic replay data and trained to predict task identity from input patterns.
This architecture requires additional modelling, but it is likely to work well when tasks are strongly dissimilar.
Furthermore, the task inference subsystem can be readily applied to process more general forms of contextual information, beyond the current input pattern.
We provide additional details, including network architectures and the loss functions that are optimized, in Appendices B and C.
R ESULTS
We evaluate our method on a set of standard image classiﬁcation benchmarks on the MNIST, CIFAR10 and CIFAR-100 public datasets 1 .
Our main aim is to (1) study the memory retention capabilities of task-conditioned hypernetworks across three continual learning settings, and (2) investigate information transfer across tasks that are learned sequentially.
Continual learning scenarios.
In our experiments we consider three different CL scenarios (van de
Ven & Tolias, 2019).
In CL1 , the task identity is given to the system.
This is arguably the standard sequential learning scenario, and the one we consider unless noted otherwise.
In CL2 , task identity is unknown to the system, but it does not need to be explicitly determined.
A target network with a ﬁxed head is required to solve multiple tasks.
In CL3 , task identity has to be explicitly inferred.
It has been argued that this scenario is the most natural, and the one that tends to be harder for neural networks (Farquhar & Gal, 2018; van de Ven & Tolias, 2019).
Experimental details.
Aiming at comparability, for the experiments on the MNIST dataset we model the target network as a fully-connected network and set all hyperparameters after van de Ven & Tolias (2019), who recently reviewed and compared a large set of CL algorithms.
For our CIFAR experiments, we opt for a ResNet-32 target neural network (He et al., 2016) to assess the scalability of our method.
A summary description of the architectures and particular hyperparameter choices, as well as additional experimental details, is provided in Appendix C. We emphasize that, on all our experiments, the number of hypernetwork parameters is always smaller or equal than the number of parameters of the models we compare with.
[Figure]
Nonlinear regression toy problem.
To illustrate our approach, we ﬁrst consider a simple nonlinear regression problem, where the function to be approximated is scalar-valued, Fig. 2.
Here, a sequence of polynomial functions of increasing degree has to be inferred from noisy data.
This motivates the continual learning problem: when learning each task in succession by modifying Θ h with the memory-preserving regularizer turned off ( β output = 0 , see Eq. 2) the network learns the last task but forgets previous ones, Fig. 2c.
The regularizer protects old solutions, Fig.
2a, and performance is comparable to an ofﬂine non-continual learner, Fig. 2b.
Permuted MNIST benchmark.
Next, we study the permuted MNIST benchmark.
This problem is set as follows.
First, the learner is presented with the full MNIST dataset.
Subsequently, novel tasks are obtained by applying a random permutation to the input image pixels.
This process can be repeated to yield a long task sequence, with a typical length of T = 10 tasks.
Given the low similarity of the generated tasks, permuted MNIST is well suited to study the memory capacity of a continual learner.
For T = 10 , we ﬁnd that task-conditioned hypernetworks are state-of-the-art on
CL1 , Table 1.
Interestingly, inferring tasks through the predictive distribution entropy (HNET+ENT) works well on the permuted MNIST benchmark.
Despite the simplicity of the method, both synaptic intelligence (SI; Zenke et al., 2017) and online elastic weight consolidation (EWC; Schwarz et al., 2018) are overperformed on CL3 by a large margin.
When complemented with generative replay methods, task-conditioned hypernetworks (HNET+TIR and HNET+R) are the best performers on all three CL scenarios.
Performance differences become larger in the long sequence limit, Fig. 3a.
For longer task sequences ( T = 100 ), SI and DGR+distill (Shin et al., 2017; van de Ven & Tolias, 2018) degrade gracefully, while the regularization strength of online EWC prevents the method from achieving high accuracy (see Fig.
A6 for a hyperparameter search on related work).
Notably, task-conditioned hypernetworks show minimal memory decay and ﬁnd high performance solutions.
Because the hypernetwork operates in a compressive regime (see Fig.
3b and Fig. A7 for an exploration of compression ratios), our results do not naively rely on an increase in the number of parameters.
Rather, they suggest that previous methods are not yet capable of making full use of target model capacity in a CL setting.
We report a set of extended results on this benchmark on Appendix D, including a study of CL2/ ( T = 100 ), where HNET+TIR strongly outperforms the related work.
[Figure]
In the table, EWC refers to online EWC and DGR refers to DGR+distill (results reproduced from van de Ven & Tolias, 2019).
We tested three hypernetwork-based models: for HNET+ENT (HNET alone for CL1 ), we inferred task identity based on the entropy of the predictive distribution; for HNET+TIR, we trained a hypernetwork-protected recognition-replay network (based on a VAE, cf.
Fig.
A1) to infer the task from input patterns; for HNET+R the main classiﬁer was trained by mixing current task data with synthetic data generated from a hypernetwork-protected VAE.
[Table]
[Table]
Split MNIST benchmark.
Split MNIST is another popular CL benchmark, designed to introduce task overlap.
In this problem, the various digits are sequentially paired and used to form ﬁve binary classiﬁcation tasks.
Here, we ﬁnd that task-conditioned hypernetworks are the best overall performers.
In particular, HNET+R improves the previous state-of-the-art method DGR+distill on both CL2 and
CL3 , almost saturating the CL2 upper bound for replay models (Appendix D).
Since HNET+R is essentially hypernetwork-protected DGR, these results demonstrate the generality of task-conditioned hypernetworks as effective memory protectors.
To further support this, in Appendix F we show that our replay models (we experiment with both a VAE and a GAN) can learn in a class-incremental manner the full MNIST dataset.
Finally, HNET+ENT again outperforms both EWC and SI, without any generative modelling.
[Figure]
c y [ [Figure]
[Figure]
benchmark accuracies ± STD, n = 5 ) on the entire CIFAR10 dataset and subsequent CIFAR- splits of ten classes. Our hypernetworkprotected ResNet-32 displays virtually no forgetting; ﬁnal averaged performance (hnet, in red) matches the immediate one (hnet-during, in blue). Furthermore, information is transferred across tasks, as performance is higher than when training each task from scratch (purple).
Disabling our regularizer leads to strong forgetting (in yellow).
On the split MNIST problem, tasks overlap and therefore continual learners can transfer information across tasks.
To analyze such effects, we study task-conditioned hypernetworks with two-dimensional task embedding spaces, which can be easily visualized.
Despite learning happening continually, we ﬁnd that the algorithm converges to a hypernetwork conﬁguration that can produce target model parameters that simultaneously solve old and new tasks, Fig. 4, given the appropriate task embedding.
Split CIFAR-10/100 benchmark.
Finally, we study a more challenging benchmark, where the learner is ﬁrst asked to solve the full CIFAR-10 classiﬁcation task and is then presented with sets of ten classes from the CIFAR-100 dataset.
We perform experiments both with a high-performance
ResNet-32 target network architecture (Fig. 5) and with a shallower model (Fig. A3) that we exactly reproduced from previous work (Zenke et al., 2017).
Remarkably, on the ResNet-32 model, we ﬁnd that task-conditioned hypernetworks essentially eliminate altogether forgetting.
Furthermore, forward information transfer takes place; knowledge from previous tasks allows the network to ﬁnd better solutions than when learning each task individually from initial conditions.
Interestingly, forward transfer is stronger on the shallow model experiments (Fig. A3), where we otherwise ﬁnd that our method performs comparably to SI.
D ISCUSSION
Bayesian accounts of continual learning.
According to the standard Bayesian CL perspective, a posterior parameter distribution is recursively updated using Bayes’ rule as tasks arrive (Kirkpatrick et al., 2017; Huszár, 2018; Nguyen et al., 2018).
While this approach is theoretically sound, in practice, the approximate inference methods that are typically preferred can lead to stiff models, as a compromise solution that suits all tasks has to be found within the mode determined by the ﬁrst task.
Such restriction does not apply to hypernetworks, which can in principle model complex multimodal distributions (Louizos & Welling, 2017; Pawlowski et al., 2017;
Henning et al., 2018).
Thus, rich, hypernetwork-modelled priors are one avenue of improvement for Bayesian CL methods.
Interestingly, task-conditioning offers an alternative possibility: instead of consolidating every task onto a single distribution, a shared task-conditioned hypernetwork could be leveraged to model a set of parameter posterior distributions.
This conditional metamodel naturally extends our framework to the Bayesian learning setting.
Such approach will likely beneﬁt from additional ﬂexibility, compared to conventional recursive Bayesian updating.
Related approaches that rely on task-conditioning.
Our model ﬁts within, and in certain ways generalizes, previous CL methods that condition network computation on task descriptors.
Taskconditioning is commonly implemented using multiplicative masks at the level of modules (Rusu et al., 2016; Fernando et al., 2017), neurons (Serra et al., 2018; Masse et al., 2018) or weights (Mallya & Lazebnik, 2018).
Such methods work best with large networks and come with a signiﬁcant storage overhead, which typically scales with the number of tasks.
Our approach differs by explicitly modelling the full parameter space using a metamodel, the hypernetwork.
Thanks to this metamodel, generalization in parameter and task space is possible, and task-to-task dependencies can be exploited to efﬁciently represent solutions and transfer present knowledge to future problems.
Interestingly, similar arguments have been drawn in work developed concurrently to ours (Lampinen & McClelland, 2019), where task embedding spaces are further explored in the context of few-shot learning.
In the same vein, and like the approach developed here, recent work in CL generates last-layer network parameters as part of a pipeline to avoid catastrophic forgetting (Hu et al., 2019) or distills parameters onto a contractive auto-encoding model store (Camp et al., 2018).
Positive backwards transfer.
In its current form, the hypernetwork output regularizer protects previously learned solutions from changing, such that only weak backwards transfer of information can occur.
Given the role of selective forgetting and reﬁnement of past memories in achieving intelligent behavior (Brea et al., 2014; Richards & Frankland, 2017), investigating and improving backwards transfer stands as an important direction for future research.
Relevance to systems neuroscience.
Uncovering the mechanisms that support continual learning in both brains and artiﬁcial neural networks is a long-standing question (McCloskey & Cohen, 1989;
French, 1999; Parisi et al., 2019).
We close with a speculative systems interpretation (Kumaran et al., 2016; Hassabis et al., 2017) of our work as a model for modulatory top-down signals in cortex.
Task embeddings can be seen as low-dimensional context switches, which determine the behavior of a modulatory system, the hypernetwork in our case.
According to our model, the hypernetwork would in turn regulate the activity of a target cortical network.
As it stands, implementing a hypernetwork would entail dynamically changing the entire connectivity of a target network, or cortical area.
Such a process seems difﬁcult to conceive in the brain.
However, this strict literal interpretation can be relaxed.
For example, a hypernetwork can output lowerdimensional modulatory signals (Marder, 2012), instead of a full set of weights.
This interpretation is consistent with a growing body of work which suggests the involvement of modulatory inputs in implementing context- or task-dependent network mode-switching (Mante et al., 2013; Jaeger, 2014;
Stroud et al., 2018; Masse et al., 2018).
C ONCLUSION
We introduced a novel neural network model, the task-conditioned hypernetwork, that is well-suited for CL problems.
A task-conditioned hypernetwork is a metamodel that learns to parameterize target functions, that are speciﬁed and identiﬁed in a compressed form using a task embedding vector.
Past tasks are kept in memory using a hypernetwork output regularizer, which penalizes changes in previously found target weight conﬁgurations.
This approach is scalable and generic, being applicable as a standalone CL method or in combination with generative replay.
Our results are state-of-the-art on standard benchmarks and suggest that task-conditioned hypernetworks can achieve long memory lifetimes, as well as transfer information to future tasks, two essential properties of a continual learner.
A T ASK - CONDITIONED HYPERNETWORKS : MODEL SUMMARY
In our model, a task-conditioned hypernetwork produces the parameters Θ trgt =
f h ( e , Θ h ) of a target neural network.
Given one such parameterization, the target model then computes predictions ˆ y =
f trgt ( x , Θ trgt )
based on input data.
Learning amounts to adapting the parameters
Θ h of the hypernetwork, including a set of task embeddings { e ( t ) }
Tt =1 , as well as a set of chunk embeddings { c i } N C i =1 in case compression is sought or if the full hypernetwork is too large to be handled directly.
To avoid castastrophic forgetting, we introduce an output regularizer which ﬁxes the behavior of the hypernetwork by penalizing changes in target model parameters that are produced for previously learned tasks.
Variables that need to be stored while learning new tasks.
What are the storage requirements of our model, when learning continually?
Memory retention relies on saving one embedding per task.
This collection { e ( t ) } Tt = therefore grows linearly with T .
Such linear scaling is undesirable asymptotically, but it turns out to be essentially negligible in practice, as each embedding is a single lowdimensional vector (e g., see Fig.
4 for a run with 2D embeddings).
A frozen snapshot of the hypernetwork parameters Θ ∗ h , taken before learning a new task, needs to be kept, to evaluate the output regularizer in Eq. 2.
B A DDITIONAL DETAILS ON HYPERNETWORK - PROTECTED REPLAY MODELS
Variational autoencoders.
For all HNET+TIR and HNET+R experiments reported on the main text we use VAEs as our replay models (Fig.
A1a, Kingma & Welling, 2014).
Brieﬂy, a VAE consists of an encoder-decoder network pair, where the encoder network processes some input pattern x and its outputs f enc ( x ) =
( µ , σ 2 ) comprise the parameters µ and σ 2 (encoded in log domain, to enforce nonnegativity) of a diagonal multivariate
Gaussian p Z ( z ; µ , σ 2 ) , which governs the distribution of latent samples z .
On the other side of the circuit, the decoder network processes a latent sample z and a one-hot-encoded task identity vector and returns an input pattern reconstruction, f dec ( z , 1 t )
=
ˆ x .
VAEs can preserve memories using a technique called generative replay: when training task T , input samples are generated from the current replay network for old tasks t < T , by varying 1 t and drawing latent space samples z .
Generated data can be mixed with the current dataset, yielding an augmented dataset ˜ X used to relearn model parameters.
When protecting a discriminative model, synthetic ‘soft’ targets can be generated by evaluating the network on ˜ X .
We use this strategy to protect an auxiliary task inference classiﬁer in HNET+TIR, and to protect the main target model in HNET+R.
Hypernetwork-protected replay.
In our HNET+TIR and HNET+R experiments, we parameterize the decoder network through a task-conditioned hypernetwork, f h,dec ( e , Θ h,dec ) .
In combination with our output regularizer, this allows us to take advantage of the memory retention capacity of hypernetworks, now on a generative model.
The replay model (encoder, decoder and decoder hypernetwork) is a separate subsystem that is optimized independently from the target network.
Its parameters Θ enc and Θ h,dec are learned by minimizing our regularized loss function, Eq. 2, here with the task-speciﬁc term set to the standard
VAE objective function,
[Equation]
with Θ dec = f h,dec ( e , Θ h,dec ) introducing the dependence on Θ h,dec .
L VAE balances a reconstruction L rec and a prior-matching L prior penalties.
For our MNIST experiments, we choose binary crossentropy (in pixel space) as the reconstruction loss, that we write below for a single example x
[Equation]
where L xent ( t, y ) = − (cid:80) k t k log y k is the cross entropy.
For a diagonal Gaussian p Z , the priormatching term can be evaluated analytically,
[Equation]
Above, z is a sample from p Z ( z ; µ (˜ x ) , σ 2 (˜ x )) obtained via the reparameterization trick (Kingma &
Welling, 2014; Rezende et al., 2014).
This introduces the dependency of L rec on Θ enc .
Task inference network (HNET+TIR).
In the HNET+TIR setup, we extend our system to include a task inference neural network classiﬁer α ( x ) parameterized by Θ TI , where tasks are encoded with a T -dimensional softmax output layer.
In both CL2 and CL3 scenarios we use a growing single-head setup for α , and increase the dimensionality of the softmax layer as tasks arrive.
This network is prone to catastrophic forgetting when tasks are learned continually.
To prevent this from happening we resort to replay data generated from a hypernetwork-protected VAE, described above.
More speciﬁcally, we introduce a task inference loss,
[Equation]
6) where t (˜ x ) denotes the correct task identity for a sample ˜ x from the augmented dataset ˜ X
= { ˜ X (1) , . . .
˜ X ( T − 1) , ˜ X ( T ) } with ˜ X ( t ) being synthetic data f dec ( z , 1 t , Θ dec ) for t = 1 . . .
T − 1 and ˜ X ( T ) =
X ( T ) is the current task data.
Importantly, synthetic data is essential to obtain a well deﬁned objective function for task inference; the cross-entropy loss L TI requires at least two groundtruth classes to be optimized.
Note that replayed data can be generated online by drawing samples z from the prior.
Hypernetwork-protected GANs.
Generative adversarial networks (Goodfellow et al., 2014) have become an established method for generative modelling and tend to produce higher quality images compared to VAEs, even at the scale of datasets as complex as ImageNet (Brock et al., 2019; Lu ˇ ci ´ c et al., 2019; Donahue & Simonyan, 2019).
This makes GANs perfect candidates for powerful replay models.
A suitable GAN instantiation for CL is the conditional GAN (Mirza & Osindero, 2014) as studied by Wu et al.
Recent developments in the GAN literature already allude towards the potential of using hypernetwork-like structures, e.g., when injecting the latent noise (Karras et al., 2019) or when using class-conditional batch-normalization as in (Brock et al., 2019).
We propose to go one step further and use a hypernetwork that maps the condition to the full set of generator a f dec f enc z task id x x ~ ( t ) f h,dec ( t ) θ dec b f gen f disc x ~ ( t ) f h,gen θ gen z task id c α e f TI x , { x } ( T ) T -1 t =1 ( t ) ~ parameters Θ ∗ gen .
Our framework allows training a conditional GAN one condition at the time.
This is potentially of general interest, and goes beyond the scope of replay models, since conditional GANs trained in a mutli-task fashion as in Brock et al.
(2019) require very large computational resources.
For our showcase experiment on class-incremental MNIST learning, Fig.
A8, we did not aim to compare to related work and therefore did not tune to have less weights in the hypernetwork than on the target network (for the VAE experiments, we use the same compressive setup as in the main text, see Appendix C).
The GAN hypernetwork is a fully-connected chunked hypernetwork withhidden layers of size 25 and 25 followed by an output size of 75,000.
We used learning rates for both discriminator and the generator hypernetwork of 0.0001, as well as dropout of 0.4 in the discriminator and the system is trained for 10000 iterations per task.
We use the Pearson Chi 2 Least-Squares GAN loss from Mao et al.
(2017) in our experiments.
C A DDITIONAL EXPERIMENTAL DETAILS
All experiments are conducted using 16 NVIDIA GeForce RTX 2080 TI graphics cards.
For simplicity, we decided to always keep the previous task embeddings e ( t ) , t = 1 , . . .
, T − 1 , ﬁxed and only learn the current task embedding e ( T ) .
In general, performance should be improved if the regularizer in Eq.
2 has a separate copy of the task embeddings e ( t, ∗ ) from before learning the current task, such that e ( t ) can be adapted.
Hence, the targets become f h ( e ( t, ∗ ) , Θ ∗ h ) and remain constant while learning task T .
This would give the hypernetwork the ﬂexibility to adjust the embeddings i.e. the preimage of the targets and therefore represent any function that includes all desired targets in its image.
Nonlinear regression toy problem.
The nonlinear toy regression from Fig.
2 is an illustrative example for a continual learning problem where a set of ground-truth functions { g (1) , . .
, g ( T ) } is given from which we collect 100 noisy training samples per task { ( x , y ) | y
=
g ( t ) ( x ) + (cid:15) with (cid:15) ∼
N (0 , σ 2 I ) , x ∼ U ( X ( t ) ) } , where X ( t ) denotes the input domain of task t .
We set σ = 0 .
05 in this experiment.
We perform 1D regression and choose the following set of tasks:
[Equation]
The target network f trgt consists of two fully-connected hidden layers using 10 neurons each.
For illustrative purposes we use a full hypernetwork f h that generates all 141 weights of f trgt at once, also being a fully-connected network with two hidden-layers of size 10.
Hence, this is the only setup where we did not explore the possibility of a chunked hypernetwork.
We use sigmoid activation functions in both networks.
The task embedding dimension was set to 2.
We train each task for 4000 iterations using the Adam optimizer with a learning rate of 0.01 (and otherwise default PyTorch options) and a batch size of 32.
To test our regularizer in Fig.
2a we set β output to 0.005, while it is set to 0 for the ﬁne-tuning experiment in Fig. 2c.
For the multi-task learner in Fig.
2b we trained only the target network (no hypernetwork) foriterations with a learning rate of 0.05.
Comparable performance could be obtained when training the task-conditioned hypernetwork in this multi-task regime (data not shown).
It is worth noting that the multi-task learner from Fig.
2b that uses no hypernetwork is only able to learn the task since we choose the input domains to be non-overlapping.
Permuted MNIST benchmark.
For our experiments conducted on MNIST we replicated the experimental setup proposed by van de Ven & Tolias (2019) whenever applicable.
We therefore use the same number of training iterations, the same or a lower number of weights in the hypernetwork than in the target network, the same learning rates and the same optimizer.
For the replay model, i.e., the hypernetwork-empowered VAE, as well as for the standard classiﬁer we used 5000 training iterations per task and learning rate is set to 0.0001 for the Adam optimizer (otherwise PyTorch default values).
The batchsize is set to 128 for the VAE whereas the classiﬁer is simultaneously trained on a batch of 128 samples of replayed data (evenly distributed over all past tasks) and a batch of 128 images from the currently available dataset.
MNIST images are padded with zeros, which results in network inputs of size 32 × 32 , again strictly following the implementation of the compared work.
We experienced better performance when we condition our replay model on a
speciﬁc task input.
We therefore construct for every task a speciﬁc input namely a sample from a standard multivariate normal of dimension 100.
In practice we found the dimension to be not important.
This input stays constant throughout the experiment and is not learned.
Note that we use the same hyperparameters for all learning scenarios, which is not true for the reported related work since they have tuned special hyperparameters for all scenarios and all methods.
• Details of hypernetwork for the VAE.
We use one hypernetwork conﬁguration to generate weights for all variational autoencoders used for our PermutedMNIST-10 experiments namely a fully-connected chunked hypernetwork with 2 hidden layers of size 25 andfollowed by an output size of 85,000.
We use ELU nonlinearities in the hidden layers of the hypernetwork.
The size of task embeddings e has been set to 24 and the size of chunk embeddings c to 8.
The parameter β output is 0.05 .
The number of weights in this hypernetwork is 2,211,907 (2,211,691 network weights + 216 task embedding weights).
The corresponding target network (and therefore output of the chunked hypernetwork), as taken from related work, has 2,227,024 weights.
• Details of the VAE for HNET+TIR.
For this variational autoencoder, we use two fullyconnected neural networks with layers of size 1000, 1000 for the encoder and 1000,for the decoder and a latent space of 100.
This setup is again copied from work we compare against.
• Details of the VAE for HNET+R. For this variational autoencoder, we use two fullyconnected neural networks with layers of size 400, 400 for the encoder and 400, 400 for the decoder (both 1000, 1000 in the related work) and a latent space of dimension 100.
Here, we departure from related work by choosing a smaller architecture for the autoencoder.
Note that we still use a hypernetwork with less trainable parameters than the target network (in this case the decoder) in related work.
• Details of the hypernetwork for the target classiﬁer in PermutedMNIST- (HNET+TIR & HNET+ENT).
We use the same setup for the hypernetwork as used for the VAEs above, but since the target network is smaller we reduce the output of the hypernetwork to 78,000.
We also adjust the parameter β output to 0.01, consistent with our PermutedMNIST-100 experiments.
The number of weights in this hypernetwork is therefore 2,029,931 parameters (2,029,691 network weights + 240 task embedding weights). The
[Figure] corresponding target network (from related work) would have 2,126,100 weights for CL and CL3 and 2,036,010 for CL2 (only one output head).
• Details of the hypernetwork for the target classiﬁer for PermutedMNIST-100.
For these experiments we chose an architecture that worked well on the PermutedMNISTbenchmark and did not conduct any more search for new architectures.
For PermutedMNIST100, the reported results were obtained by using a chunked hypernetwork with 3 hidden layers of size 200, 250 and 350 (300 for CL2) and an output size of 7500 (6000 for CL2) (such that we approximately match the corresponding target network size for CL1 / CL2 / CL3 ).
Interestingly, Fig.
A2b shows that even if we don’t adjust the number of hypernetwork weights to the increased number of target network weights, the superiority of our method is evident.
Aside from this, the plots in Fig.
3 have been generated using the PermutedMNIST-10 HNET+TIR setup (note that this includes the conditions set by related work for PermutedMNIST-10, e.g., target network sizes, the number of training iterations, learning rates, etc.).
• Details of the VAE and the hypernetwork for the VAE in PermutedMNIST-100 for CL2 / CL3 .
We use a very similar setup for the VAE and it’s hypernetwork used in HNET+TIR for PermutedMNIST-10 as described above.
Things that we change: Full connected hnet with one hidden layer of size 100, chunk embedding sizes are set to 12, task embedding sizes are set two 128 and the hidden layer sizes of the VAE’s generator are 400,600.
Also we increased the hnet’s regularisation strength β output = 0 .
1 for the VAE’s generator hypernetwork.
• Details of the target classiﬁer for HNET+TIR & HNET+ENT.
For this classiﬁer, we use the same setup as in the study we compare to (van de Ven & Tolias, 2019), i.e., a fully-connected network with layers of size 1000, 1000.
Note that if the classiﬁer is used as a task inference model, it is trained on replay data and the corresponding hard targets, i.e., the argmax of the soft targets.
Below, we report the speciﬁcations for our automatic hyperparameter search (if not noted otherwise, these speciﬁcations apply for the split MNIST and split CIFAR experiments as well):
• Hidden layer sizes of the hypernetwork: (no hidden layer), "5,5" "10,10", "25,25", "50,50", "100,100", "10", "50", "100" • Output size of the hypernetwork: ﬁtted such that we obtain less parameters then the target network which we compare against • Embedding sizes (for e and c ): 8, 12, 24, 36, 62, 96,• β output : 0.0005, 0.001, 0.005, 0.01, 0.005, 0.1, 0.5, 1. • Hypernetwork transfer functions: linear, ReLU, ELU, Leaky-ReLU
Note that only a random subset of all possible combinations of hyperparameters has been explored.
After we found a conﬁguration with promising accuracies and a similar number of weights compared to the original target network, we manually ﬁne-tuned the architecture to increase/decrease the number of hypernetwork weights to approximately match the number of target network weights.
The choice of hypernetwork architecture seems to have a strong inﬂuence on the performance.
It might be worth exploring alternatives, e.g., an architecture inspired by those used in typical generative models.
We note that in addition to the above speciﬁcations we explored manually some hyperparameter conﬁgurations to gain a better understanding of our method
Split MNIST benchmark.
Again, whenever applicable we reproduce the setup from van de Ven &
Tolias (2019).
Differences to the PermutedMNIST-10 experiments are just the learning rate (0.001) and the number of training iterations (set to 2000).
• Details of hypernetwork for the VAE.
We use one hypernetwork conﬁguration to generate weights for all variational autoencoders used for our split MNIST experiments, namely a fully-connected chunked hypernetwork with 2 hidden layers of size 10, 10 followed by an output size of 50,000.
We use ELU nonlinearities in the hidden layers of the hypernetwork.
The size of task embeddings e has been set to 96 and the size of chunk embeddings c to 96.
The parameter β output is 0.01 for HNET+R and 0.05 for HNET+TIR .
The number of weights in this hypernetwork is 553,576 (553,192 network weights + 384 task embedding weights).
The corresponding target network (and therefore output of the chunked hypernetwork), as taken from related work, has 555,184 weights.
For a qualitative analyses of the replay data of this VAE (class incrementally learned), see A8.
• Details of the VAE for HNET+TIR.
For this variational autoencoder, we use two fullyconnected neural networks with layers of size 400, 400 for the encoder and 300, 150 for the decoder (both 400, 400 in the related work) and a latent space of dimension 100.
• Details of the VAE for HNET+R. For this variational autoencoder, we use two fullyconnected neural networks with layers of size 400, 400 for the encoder and 250, 350 for the decoder (both 400, 400 in the related work) and a latent space of dimension 100.
• Details of the hypernetwork for the target classiﬁer in split MNIST (HNET+TIR & HNET+ENT).
We use the same setup for the hypernetwork as used for the VAE above, but since the target network is smaller we reduce the output of the hypernetwork to 42,000.
We also adjust the β output to 0.01 although this parameter seems to not have a strong effect on the performance.
The number of weights in this hypernetwork is therefore 465, parameters (465,192 network weights + 480 task embedding weights).
The corresponding target network (from related work) would have 478,410 weights for CL1 and CL3 and 475,202 for CL2 (only one output head).
• Details of the target classiﬁer for HNET+TIR & HNET+ENT.
For this classiﬁer, we again use the same setup as in the study we compare to (van de Ven & Tolias, 2019), i.e., a fully-connected neural networks with layers of size 400, 400.
Note that if the classiﬁer is used as a task inference model, it is trained on replay data and the corresponding hard targets, i. e ., the argmax the soft targets.
Split CIFAR-10/100 benchmark.
For these experiments, we used as a target network a ResNetnetwork (He et al.
and again produce the weights of this target network by a hypernetwork in a compressive manner.
The hypernetwork in this experiment directly maps from the joint task and chunk embedding space (both dimension 32) to the output space of the hypernetwork, which is of dimension 7,000.
This hypernetwork has 457,336 parameters (457,144 network weights +task embedding weights).
The corresponding target network, the ResNet-32, has 468.540 weights (including batch-norm weights).
We train for 200 epochs per task using the Adam optimizer with an initial learning rate of 0.001 (and otherwise default PyTorch values) and a batch size of 32.
In addition, we apply the two learning rate schedules suggested in the Keras CIFAR-10 example 2 .
Due to the use of batch normalization, we have to ﬁnd an appropriate way to handle the running statistics which are estimated during training.
Note, these are not parameters which are trained through backpropagation.
There are different ways how the running statistics could be treated:
One could ignore the running statistics altogether and simply compute statistics based on the current batch during evaluation.
The statistics could be part of the hypernetwork output.
Therefore, one would have to manipulate the target hypernetwork output of the previous task, such that the estimated running statistics of the previous task will be distilled into the hypernetwork.
The running statistics can simply be checkpointed and stored after every task.
Note, this method would lead to a linear memory growth in the number of tasks that scales with the number of units in the target network.
For simplicity, we chose the last option and simply checkpointed the running statistics after every task.
For the ﬁne-tuning results in Fig.
5 we just continually updated the running statistics (thus, we applied no checkpointing).
D A DDITIONAL EXPERIMENTS AND NOTES
Split CIFAR-10/100 benchmark using the model of Zenke et al.
We re-run the split
CIFAR-10/100 experiment reported on the main text while reproducing the setup from Zenke et al.
Our overall classiﬁcation performance is comparable to synaptic intelligence, which achieves 73.85% task-averaged test set accuracy, while our method reaches 71.29% ± 0.32%, with initial baseline performance being slightly worse in our approach, Fig.
A3.
[Figure]
[Figure]
[Figure] [Figure] Split [Figure]
/ hnet during hnet from scratch ﬁne
[Figure]
tuning
SI
To obtain our results, we use a hypernetwork with 3 hidden-layers of sizes 100, 150, 200 and output size 5500.
size of task embeddings e has been set to 48 and the size of chunk embeddings c to 80.
The parameter β output is 0.01 and the learning rate is set to 0.0001.
The number of weights in this hypernetwork is 1,182,678 (1,182,390 network weights + 288 task embedding weights).
The corresponding target network would have 1,276,508 weights.
In addition to the above speciﬁed hyperparameter search conﬁguration we also included the following learning rates: 0.0001, 0.0005, 0.001 and manually tuned some architectural parameters. [Figure] harder sequentially
Upper bound for replay models.
We obtain an upper bound for the replay-based experiments (Table 2) by sequentially training a classiﬁer, in the same way as for HNET+R and DGR, now using true input data from past tasks and a synthetic, self-generated target.
This corresponds to the rehearsal thought experiment delineated in Sect. [Table]
Robustness of β output -choice.
In Fig.
A2a and Fig.
A2c we provide additional experiments for our method on PermutedMNIST-100.
We show that our method performs comparable for a wide range of β output -values (including the one depicted in Fig. 3a).
Varying the regularization strength for online EWC.
The performance of online EWC in Fig.
3a is closest to our method (labelled hnet, in red) compared to the other methods.
[-cc u r a cy [ % ]     75          78    
-]
Therefore, we take a closer look at this method and show that further adjustments of the regularization strength λ do not lead to better performance.
Results for a wide range of regularization strengths can be seen in Fig.
A5a and Fig.
A5b.
As shown, online EWC cannot attain a performance comparable to our method when tuning the regularization strength only.
The impact of catastrophic forgetting on the hypernetwork and target network.
We have successfully shown that by shifting the continual learning problem from the target network to the hypernetwork we can successfully overcome forgetting due to the introduction of our regularizer in
Eq.
We motivated this success by claiming that it is an inherently simpler task to remember a few input-output mappings in the hypernetwork (namely the weight realizations of each task) rather than the massive number of input-output mappings { ( x ( t,i ) , y ( t,i ) ) } n t i =1 associated with the remembering of each task t by the target network.
Further evidence of this claim is provided by ﬁne-tuning experiments in Fig.
A5c and Fig.
A5d.
Fine-tuning refers to sequentially learning a neural network on a set of tasks without any mechanism in place to prevent forgetting.
It is shown that ﬁne-tuning a target network (no hypernetwork in this setup) has no catastrophic inﬂuence on the performance of previous tasks.
Instead there is a graceful decline in performance.
On the contrary, catastrophic forgetting has an almost immediate affect when training a hypernetwork without protection (i.e., training our method with β output = 0 .
The performance quickly drops to chance level, suggesting that if we weren’t solving a simpler task then preventing forgetting in the hypernetwork rather than in the target network might not be beneﬁcial. Chunking and hypernetwork architecture sensitivity.
In this note we investigate the performance sensitivity for different (fully-connected) hypernetwork architectures on split MNIST and PermutedMNIST-10, Fig.
A7.
We trained thousands of randomly drawn architectures from the following grid (the same training hyperparameters as reported for for CL1 , see Appendix C, were used throughout): possible number of hidden layers 1 , 2 , possible layer size 5 , 10 , 20 , . .
. , 90 , 100 , possible chunk embedding size 8 , 12 , 24 , 56 , 96 and hypernetwork output size in { 10 , 50 , 100 , 200 , 300 , 400 , 500 , 750 , 1 k, 2 k, . .
. , 9 k, 10 k, 20 k, 30 k, 40 k } . Since we realize compression through chunking, we sort our hypernetwork architectures by compression ratio, and consider only architectures with small compression ratios. [Figure] strong regularization prevents the doesn’t prevent
[Figure]
[Figure]
( CL considered
Performance of split MNIST stays in the high 90 percentages even when reaching compression ratios close to 1% whereas for PermutedMNIST-10 accuracies decline in a non-linear fashion.
For both experiments, the choice of the chunked hypernetwork archicture is robust and high performing even in the compressive regime.
Note that the discussed compression ratio compares the amount of trainable parameters in the hypernetwork to its output size, i.e. the parameters of the target network.
Small capacity target networks for the permuted MNIST benchmark.
Swaroop et al.
(2018) argue for using only small capacity target networks for this benchmark.
Speciﬁcally, they propose to use hidden layer sizes [100 , 100] .
Again, we replicated the setup of van de Ven & Tolias (2019) wherever applicable, except for the now smaller hidden layer sizes of [100 , 100] in the target network.
We use a fully-connected chunked hypernetwork with chunk embeddings c having size 12, hidden layers having size 100, 75, 50 and an output size of 2000, resulting in a total number of hypernetwork weights of 122,459 (including 10 × 64 task embedding weights) compared to 122,700 weights that are generated for the target network.
β output is set to 0 .
The experiments performed here correspond to CL1 .
We achieve an average accuracy of 93.91 ± 0.04 for PermutedMNIST-10 after having trained on all tasks.
In general, we saw that the hypernetwork training can beneﬁt from noise injection.
For instance, when training with soft-targets (i.e., we modiﬁed the 1-hot target to be 0.95 for the correct class and 1 − 0 .
95 # classes − 1 for the remaining classes), we could improve the average accuracy to 94.24 ± 0.03.
We also checked the challenging PermutedMNIST-50 benchmark with this small target network as previously investigated by Ritter et al.
Therefore, we slightly adapted the above setup by using a hypernetwork with hidden layer sizes [100 , 100] and a regularization strength of β output = 0 .
This hypernetwork is slightly bigger than the corresponding target network | Θ h ∪{ e ( t ) }| | Θ trgt | = 1 . 37 .
With this conﬁguration, we obtain an average accuracy of 90.91 ± 0.07.
Comparison to HAT. Serra et al. (2018) proposed the hard attention to the task (HAT) algorithm, a strong CL1 method which relies on learning a per-task, per-neuron mask. Since the masks are pushed to become binary, HAT can be viewed as an algorithm for allocating subnetworks (or modules) within the target network, which become specialized to solve a given task.
Thus, the method is similar to ours in the sense that the computation of the target network is task-dependent, but different in spirit, as it relies on network modularity.
In HAT, task identity is assumed to be provided, so that the appropriate mask can be picked during inference (scenario CL1 ).
HAT requires explicitly storing a neural mask for each task, whose size scales with the number of neurons in the target network.
In contrast, our method allows solving tasks in a compressive regime.
Thanks to the hypernetwork, whose input dimension can be freely chosen, only a low-dimensional embedding needs to be stored per task (cf.
Fig. 4), and through chunking it is possible to learn to parameterize large target models with a small number of plastic weights (cf.
Fig.
3b).
Here, we compare our task-conditioned hypernetworks to HAT on the permuted MNIST benchmarks ( T = 10 and T = 100 ), cf.
Table 3.
For large target networks, both methods perform strongly, reaching comparable ﬁnal task-averaged accuracies.
For small target network sizes, task-conditioned hypernetworks perform better, the difference becoming more apparent on PermutedMNIST-100.
We note that the two algorithms use different training setups.
In particular, HAT uses 200 epochs (batch size set to 64) and applies a learning rate scheduler that acts on a held out validation set.
Furthermore, HAT uses differently tuned forgetting hyperparameters when target network sizes change.
This is important to control for the target network capacity used per task and assumes knowledge of the (number of) tasks at hand.
Using the code freely made available by the authors, we were able to rerun HAT for our target network size and longer task sequences.
Here, we used the setup provided by the author’s code for HAT-Large for PermutedMNIST-10 and PermutedMNIST-100.
To draw a fairer comparison, when changing our usual target network size to match the ones reported in Serra et al.
(2018), we trained for 50 epochs per task (no training loss improvements afterwards observed) and also changed the batch size to 64 but did not changed our training scheme otherwise; in particular, we did not use a learning rate scheduler.
[Table]
Efﬁcient PermutedMNIST-250 experiments with a stochastic regularizer on subsets of previous tasks.
An apparent drawback of Eq.
2 is that the runtime complexity of the regularizer grows linearly with the number of tasks.
To overcome this obstacle, we show here that it is sufﬁcient to consider a small random subset of previous tasks.
In particular, we consider the PermutedMNIST-250 benchmark (250 tasks) on CL1 using the hyperparameter setup from our PermutedMNIST-100 experiments except for a hypernetwork output size of 12000 (to adjust to the bigger multi-head target network) and a regularization strength β output = 0 .
Per training iteration, we choose maximally 32 random previous tasks to estimate the regularizer from Eq.
With this setup, we achieve a ﬁnal average accuracy of 94.19 ± 0.16 (compared to an average during accuracy (i.e., the accuracies achieved right after training on the corresponding task) of 95.54 ± 0.05).
All results are across 5 random seeds.
These results indicate that a full evaluation of the regularizer at every training iteration is not necessary such that the linear runtime complexity can be cropped to a constant one.
Combining hypernetwork output regularizers with weight importance.
Our hypernetwork regularizer pulls uniformly in every direction, but it is possible to introduce anisotropy using an EWC-like approach (Kirkpatrick et al., 2017).
Instead of weighting parameters, hypernetwork outputs can be weighted.
This would allow for a more ﬂexible regularizer, at the expense of additional storage.
Task inference through predictive entropy (HNET+ENT).
In this setup, we rely on the capability of neural networks to separate in- from out-of-distribution data.
Although this is a difﬁcult research problem on its own, for continual learning, we face a potentially simpler problem, namely to detect and distinguish between the tasks our network was trained on.
We here take the ﬁrst minimal step exploiting this insight and compare the predictive uncertainty, as quantiﬁed by output distribution entropy, of the different models given an input.
Hence, during test time we iterate over all embeddings and therefore the models our metamodel can generate and compare the predictive entropies which results in making a prediction with the model of lowest entropy.
For future work, we wish to explore the possibility of improving our predictive uncertainty by taking parameter uncertainty into account through the generation of approximate, task-speciﬁc weight posterior distributions.
Learning without task boundaries with hypernetworks.
An interesting problem we did not address in this paper is that of learning without task boundaries.
For most CL methods, it is crucial to know when learning one task ends and training of a new tasks begins.
This is no exception for the methods introduced in this paper.
However, this is not necessarily a realistic or desirable assumption; often, one desires to learn in an online fashion without task boundary supervision, which is particularly relevant for reinforcement learning scenarios where incoming data distributions are frequently subject to change (Rolnick et al., 2018).
At least for discrete changes, with our hypernetwork setup, this boils down to a detection mechanism that activates the saving of the current model, i.e., the embedding e ( T ) , and its storage to the collection of embeddings { e ( t ) } .
We leave the integration of our model with such a hypernetwork-speciﬁc switching detection mechanism for future work.
Interestingly, our task-conditioned hypernetworks would ﬁt very well with methods that rely on fast remembering (a recently proposed approach which appeared in parallel to our paper, He et al., 2019).
E U NIVERSAL FUNCTION APPROXIMATION WITH CHUNKED NEURAL NETWORKS
Proposition 1.
Given a compact subset K ⊂ R m and a continuous function on K i.e. f ∈ C ( K ) , more speciﬁcally, f : K → R n with n = r · N C .
Now ∀ (cid:15) > 0 , there exists a chunked neural network f c h : R m × C
→ R r with parameters Θ h , discrete set C
= { c 1 , . .
. , c N C } and c i ∈
R s such that | ¯ f c h ( x ) − f ( x ) | < (cid:15), ∀ x ∈ K and with ¯ f c h ( x ) =
[ f c h ( x , c 1 ) , . . .
, f c h ( x , c N C )] .
For the following proof, we assume the existence of one form of the universal approximation theorem (UAT) for neural networks (Leshno & Schocken, 1993;
Hanin, 2017).
Note that we will not restrict ourselves to a speciﬁc architecture, nonlinearity, input or output dimension.
Any neural network that is proven to be a universal function approximator is sufﬁcient.
Proof.
Given any (cid:15) > 0 , we assume the existence of a neural network f h : R m → R n that approximates function f on K :
[Equation]
We will in the following show that we can always ﬁnd a chunked neural network f c h : R m × C → R r approximating the neural network f h on K and conclude with the triangle inequality [Equation]
[Equation]
Indeed, given the neural network f h such that (10) holds true, we construct
[Equation]
else
[Equation]
12) by splitting the full neural network f h ( x ) =
[ f c 1 h ( x ) , f c 2 h ( x ) , . . .
, f c N C h ( x )] with ˆ f h : R m × C
→ R r .
Note that ˆ f h is continuous on R m × C with the product topology composed of the topology on R m induced by the metric | · − · | : R m × R m → R and the discrete topology on C .
Now we can make use of the UAT again:
Given the compact K ⊂ R n , the discrete set C = { c 1 , . . .
, c N C } and any (cid:15) 2 N C > 0 , there exists a neural network function f c h : R m × R s
→ R r such that
[Equation]
It follows that
[Equation]
which is equivalent to
[Equation]
We have shown (11) which concludes the proof.
Note that we did not specify the number of chunks N C , r or the dimension s of the embeddings c i .
Despite this theoretical result, we emphasize that we are not aware of a constructive procedure to deﬁne a chunked hypernetwork that comes with a useful bound on the achievable performance and/or compression rate.
We evaluate such aspects empirically in our experimental section.
[Figure] [Figure]