Lifelong Neural Predictive Coding: Learning Cumulatively Online without Forgetting
Author(s)
Abstract
In lifelong learning systems based on artificial neural networks, one of the biggestobstacles is the inability to retain old knowledge as new information is encountered.
This phenomenon is known as catastrophic forgetting.
In this paper, we proposea new kind of connectionist architecture, the Sequential Neural Coding Network,that is robust to forgetting when learning from streams of data points and, unlikenetworks of today, does not learn via the popular back-propagation of errors.
Grounded in the neurocognitive theory of predictive processing, our model adaptssynapses in a biologically-plausible fashion while another neural system learns todirect and control this cortex-like structure, mimicking some of the task-executivecontrol functionality of the basal ganglia.
In our experiments, we demonstrate thatour self-organizing system experiences significantly less forgetting compared tostandard neural models, outperforming a swath of previously proposed methods,including rehearsal/data buffer-based methods, on both standard (SplitMNIST,
Split Fashion MNIST, etc.)
and custom benchmarks even though it is trained in astream-like fashion.
Our work offers evidence that emulating mechanisms in realneuronal systems, e.g., local learning, lateral competition, can yield new directionsand possibilities for tackling the grand challenge of lifelong machine learning.
Introduction
Lifelong learning is a part of machine learning and artificial intelligence research with the goal ofdeveloping a computational agent that can learn over time and continually accumulate new knowledgewhile retaining its previously learned experiences [1, 2].
For example, suppose an agent needs tolearn to classify digits, then types of clothing and then sketches of objects.
As each new task arrivesthe agent is expected to process the accompanying data and learn the new task but still rememberhow to complete the old tasks, at least without significant degradation in performance or loss ofgeneralization.
Modern-day connectionist systems are typically trained on a fixed pool of datasamples, collected in controlled environments, in isolation and random order, and then evaluated on aseparate validation data pool.
This is a far cry from what we really desire from learning machines.
When we look to humans or other animals, we see that they are more than capable of learning in acontinual manner, making decisions based on sensorimotor input throughout their lifespans [3].
Thisability to incrementally acquire and refine knowledge over long periods of time is driven by cognitiveprocesses that come together to create the experience-driven specialization of motor and perceptualskills [4, 3].
Thus, evaluating how neural systems generalize on task sequences, as opposed to single,isolated tasks, proves to be a far greater challenge.
In order to continually adapt, the brain must retainspecific memories of prior tasks.
In working towards the challenge of lifelong machine learning, thispaper makes the following contributions: 1) we propose the sequential neural coding network, aninteractive generative model that jointly reconstructs input and predicts its label, and an algorithm for updating its synapses, 2) we show that memory retention is vastly improved by integrating our model’smulti-step nature with lateral competition that is driven by a task selection function inspired by thebasal ganglia [5], and 3) we compare our model’s performance against state-of-the-art baselines,including both regularization and rehearsal/replay-based methods, on publicly-available benchmarks.
Related Work
It is well-known that when artificial neural networks (ANNs) are trained on more than one tasksequentially, the new information contained in subsequent tasks leads to catastrophic interference(a.k.a.
catastrophic forgetting) with the information acquired in earlier tasks [6, 7, 8].
This happensin connectionist systems when the new data instances to be learned are significantly different frompreviously observed ones.
This causes the new information to overwrite knowledge currently encodedin the system’s synaptic weights, due to the sharing of neural representations over tasks [9, 6] (this isknown as the representational overlap problem).
In isolated, task-specific (offline) learning (though itstill occurs [10]), this type of weight overwriting occurs to a lesser degree because the patterns arepresented to the agent pseudo-randomly and multiple times, i.e., via multiple epochs.
Over the decades, there have been many approaches proposed to mitigate catastrophic forgetting inneural systems.
Some of the earliest attempts proposed memory systems where prior data pointswere stored and regularly replayed to the network in a process called “rehearsal”, which involvedinterleaving these data points with samples drawn from new datasets [11, 12, 13, 14, 15].
Thougheffective, the main drawback of these approaches is that they require explicitly storing old data.
Such amechanism is not known to exist in the brain and, as a practical matter, this leads to exploding working(hardware) memory requirements (inefficiency).
In addition, rehearsal-based approaches do not offerany mechanisms to preserve consolidated knowledge in the face of acquiring new information [4].
Other approaches attempt to allocate additional neural “resources”, i.e., growing the networks whenrequired [16, 17], motivated by earlier findings [18].
However, this leads to dramatically increasingcomputational requirements over time as the networks grow larger.
To compound these issues further,systems with growing capacity cannot know how many resources to allocate at a given time sincethe number of tasks and samples are not known a priori (without imposing strong assumptionson the input distribution).
Other approaches try to block old information from being overwrittenthrough regularization [19].
From this vast collection of research, each approach bearing strengthsand weaknesses, three suggested remedies have emerged: 1) allocate additional neural resources toaccommodate new knowledge, 2) use non-overlapping representations (or semi-distributed ones [20])if resources are fixed, and 3) interleave old patterns with new ones as new information is acquired.
In this work, we consider the setting where the space available to the agent is fixed or grows slowlycompared to the rate of new tasks being presented.
This means that we cannot create a new, separatenetwork for every task observed in the stream.
Furthermore, storing, reshuffling, and re-presenting thedata in the stream is not feasible in this setting.
Concretely, our approach could be classified as taskincremental learning (Domain-IL)
[21] with the exception that no task descriptors are used at bothtraining and test time (an aspect of Class-IL [21]).
In addition to addressing the above, our contributionto lifelong learning is motivated by premise that developing algorithms [22, 23, 24, 25, 26] thatserve as alternatives to back-propagation will lead to the creation of promising architectures withmechanisms better equipped to tackle problems like catastrophic interference.
Cumulative Learning with Neural Coding
Notation:
We start by defining key notation.
⊙ indicates a Hadamard product while · denotesmatrix/vector multiplication.
( v ) T is the transpose of v .
Matrices/vectors are depicted in bold-font,e.g., matrix M or vector v (scalars shown in italic).
Finally, || v || 2 denotes the Euclidean norm of v .
3.1 Sequential Learning and the Data Continuum
This work focuses on learning a neural system in the context of sequential learning.
Starting from anearly definition [27] of this form of learning, we assume there is a sequence of tasks T 1 , T 2 , T 3 , . .
.(with
potentially no end to the sequence) that are presented to a system in order.
When faced with the( N + 1) th task, the system should use the knowledge it has gained from the previous N tasks to aidin learning and performing the current task.
The knowledge of a system is stored in a knowledge base (KB), e.g., the synapses of a neural model.
Each task T i has its own corresponding dataset
D i =
{ ( y 1 , x 1 , t i ) . . .
( y n i , x n i , t i ) } with n i examples.
Here x j ∈ R J x × 1 represents the featurevector of the j th example ( J x is its dimensionality), y j ∈ { 0 , 1 } J y × 1 is the target (label), and t i is thetask descriptor that identifies ( y j , x j ) as being a data point from task i .
The task descriptor is one-hotencoded as t ∈ { 0 , 1 } ( N +1)
× 1 and, when a new task is encountered, the size of the one-hot vector isincreased by one – thus the network adds an extra randomly initialized input node if it accepts t i asan extra input.
Note that while we present the task data continuum with t i explicitly depicted, wewill generalize our models to not depend on t i (making this problem task descriptor-free).
Dynamic Output Units: The output nodes in our setting get re-used for each new task.
For example,output node 1 of the network could represent a prediction for the digit “1” in task T 1 (e.g., digitrecognition), while in task T 2 (e.g., clothing recognition) the same node could represent a predictionfor “pants”.
If a new task has more classes than previous tasks, we add output nodes with randomlyinitialized weights.
For example, if prior tasks were binary and the new task has 4 targets, we add 2more outputs to the network.
This is a difficult form of cumulative learning [28].
Context Units: For the model that we develop in this study, when task t is encountered, for eachlayer ℓ in the network, we make use of a task embedding vector g ℓt (this new memory is much smallerthan creating a new network for task t , which would require new weight matrices per layer rather thanan extra vector).
All g ℓ t ∈ R J ℓ × 1 ( J ℓ is the number of units in layer ℓ ) are stored in a memory matrix M ℓ ∈ R ( N +1) × J ℓ ( M = { M 1 , ... M L } ), where a context can be retrieved using a one-hot encodingof the task descriptor, i.e., M ℓ · t , either explicitly or produced by another system (as in our case).
3.2 The Interactive Generative Model
The sequential neural coding network (S-NCN) is designed to make flexible conditional predictions –e.g., predicting y given x , predicting both y and x , predicting the missing parts of x given y and theobserved parts of x , etc.
In order to do so, it treats inputs and outputs in a non-standard way.
Theinput to the model is a task descriptor t i and the output units represent ( y , x ) .
To predict y i
givenx
i , we clamp the output nodes that are responsible for predicting x and force their output to be x i .
During training, outputs are clamped to both y i and x i , forcing the S-NCN to update latent states andsynapses.
The S-NCN can operate as a probabilistic generative network by feeding in a random noisevector as input, but we leave this extension to future work (our focus here is predicting y given x ).
The full computational process of the S-NCN is defined by three key steps: 1) layer-wise hypothesisgeneration, 2) state error-correction, and 3) synaptic weight evolution.
The first two steps iterativelypredict and correct the representations of the model for the input and target values of the task.
After
K iterations, model weights and the current task descriptor memory are updated.
In this section, weprovide details of the above steps and then describe the objective function that our model optimizes.
3.2.1 Inference: Predicting and Correcting States
Layer-wise State Prediction.
The architecture of the S-NCN can be viewed as a stack of parallel,stateful neural-based prediction layers P 1 , ..., P ℓ , ..., P m , where the goal of each predictor is to guessthe internal state of the predictor in the layer below (i.e., the S-NCN is not a feedforward network).
The state of P ℓ (layer ℓ ) is represented by the vector z ℓ ∈ R J ℓ × 1 ( J ℓ is the number of units in layerℓ ).
P ℓ makes a prediction z ℓ − 1 µ about the current state z ℓ − 1 of P ℓ − 1 (layer ℓ − 1 ).
Furthermore, welet z 0 x and z
y denote clamped outputs.
That is, if we want to predict the label y i given the features x i ,we set z 0 x = x i and if we want to train, we set both z 0 x = x i and z
y = y i .
The values predicted bylayer 1 are denoted by z 0 µ,x and z 0 µ,y .
Note that z 0 =
[ z 0 x , z 0 y ] , where the two inputs are concatenated(if either is missing, the SNCN completes the required values – see Appendix).
With respect to neural structure, the parallel predictors of the S-NCN are locally connected through(forward) generative weights W ℓ ∈ R J ℓ × J ℓ +1 and error (feedback) weights E ℓ ∈ R J ℓ +1 × J ℓ , whichwork to transmit error information to relevant regions of neural processing elements, effectivelycoordinating all of the model’s predictors.
Formally, a predictor, with state z ℓ +1 , that guesses the state of z ℓ , takes on the form (given matrix W ℓ +1 and activation ϕ ℓ +1 ):
[Equation]
where e ℓ is a block of error units.
Error units are paired with each predictor.
Their task is to computethe disagreement or mismatch between the predictor’s output and the target activity pattern z ℓ .
Theerror unit vector e ℓ can also be derived from the total discrepancy reduction presented in Equation5.
Note that for layer 0 , e 0 =
[ e 0 x , e 0 y ]
(there are error neurons for x i as well as others for y i , theconcatenation of which makes up the bottom-most prediction error signal).
Latent State Correction and Context Updating.
Once each layer ℓ has made a prediction aboutthe layer below it and error units have been activated, the state of layer ℓ can be adjusted to take intoaccount the local top-down and bottom-up error information.
Using its current state and the errornodes (along with its task embedding vector g ℓt ), layer ℓ in the S-NCN adjusts its state z ℓ as follows:
[Equation]
where d
[Equation]
= − e ℓ + E
[Equation]
e
[Equation]
−(2) where β is the state correction rate and k marks one step of the S-NCN’s K -step inference process.
Note that d ℓ is the perturbation that adjusts the values of the hidden states – it combines atop-down expectation of layer ℓ w/ a bottom-up pressure from layer ℓ − 1 ).
The error feedbackweights E ℓ are parameters that play a crucial role in this calculation, as they are responsible fortransmitting the error for ℓ back up to layer ℓ + 1 .
Notably, part of the state-correction requirescompetition among the individual units in a given layer through the function f ℓ ( z ℓ , g ℓt ) .
There arevarious ways in which this function can be implemented and, in this study, we implemented it as:f ℓ ( z ℓ , g ℓt ) =
(cid:0)
I ⊙ V ℓ (cid:1) · z ℓ , where, V ℓ =
BK WTA ( g ℓt , K ) · BK WTA (( g ℓt ) T , K ) where BK WTA ( v , K ) is the binarized K winners-take-all function, yielding a binary vector with 1at the index of each of the K winning units, or formally:
BK WTA ( v , K ) = (cid:8) 1 if v j ∈ { K largest elements of v } and ϕ ( v ) = 0 otherwise (cid:9) .
In the Appendix, we study other forms of the competition function (and find that the above performedbest, so we report this in the main paper).
Note that this lateral inhibition is a function of evolvingcontext g ℓ , triggered by a task pointer t i (produced by a task selector model, which we define later).
In real neural systems, competition between units within a layer is thought to facilitate contextualprocessing [29], where only some neuronal signals are strengthened while the activity of others is suppressed.
Moreover, lateral competition, often modeled in classical models with anti-Hebbian learning[30], encourages the formation of sparse codes [31, 32].
Since the S-NCN is an interactive/iterativemodel, incorporating lateral competition (in the form of a task-driven memory matrix) is natural andcomputed online, highlighting the model’s flexibility compared to an ANN – to incorporate lateralsynaptic activity, recurrent weights could be introduced to an ANN but training it would requiringunrolling via backprop through time.
More importantly, a lateral mechanism in the S-NCN works asan inductive bias that leads it to acquire task-dependent representations, which can aid in memoryretention since the system has to store information on multiple, disjoint tasks.
In a sense, this taskspecialization that we build into the neural dynamics is similar in spirit to activation sharpening [33].
3.2.2 Updating Synaptic Parameters
Given that the S-NCN is an interactive network [34], inferring its states requires running Equations 1and 2 K times, where the model alternates between making predictions and then correcting statesonce error units have been computed.
Once latent states have been inferred, the S-NCN is then ableto adjust its synaptic values.
The synaptic updates take the form of Hebbian-like rules:
[Equation]
where α is a scaling factor, usually set to < 1 . 0 , that makes the error feedback weights changeat a slower rate than the forward weights (this improves convergence [35]).
S ℓW ∈ R J ℓ × J ℓ +1 and S ℓ E ∈ R J ℓ +1 × J ℓ are modulation factors that provide stability to the weight updates (see Appendix).
An important property of the above weight update rules is that they are local – to compute changesin the synapses, all we require is the information immediately available to the neuron(s) of interest(making these rules function similarly to classical Hebbian updates [36], although there are importantdifferences to them, as discussed in [35]).
Since a neuron is able to immediately generate a hypothesis
1: Input: sample ( y , x , t ) , β , M , & Θ 2: function I NFER S TATES ( ( y , x , t ) , Θ ) 3: ( g 1 ,..., g ℓ ,..., g L ) ← getContexts ( t , M ) 4:
Set z 1 ,..., z ℓ ,..., z L to 0 5:
e L = 0 , z 0 x = x , z
y =
y 6: for k = 1 to K do 7: for ℓ =
L to 1 do ▷ Run layerwise predictors 8: if ℓ > 1 then ▷ Latent prediction layer 9: z ℓ − 1 µ = W ℓ · ϕ ℓ ( z ℓ ) 10: e ℓ − 1 = ( z ℓ − 1 − z ℓ − 1 µ ) 11: else ▷ Sensory prediction layer 12: z ℓ − 1 µ,x =
W ℓx · ϕ ℓ ( z ℓ ) 13: e ℓ − 1 x = ( z ℓ − 1 x − z ℓ − 1 µ,x ) 14: z ℓ − 1 µ,y =
W ℓy · ϕ ℓ ( z ℓ ) 15: e ℓ − 1 y =
( z ℓ − 1 y − z ℓ − 1 µ,y ) 16: for ℓ =
L to 1 do ▷ Correct internal states 17: if ℓ == 1 then 18: d ℓ = − e + E ℓx · e ℓ − 1 x + E ℓy · e ℓ − 1 y 19: else 20: d ℓ = − e + E ℓ · e ℓ −21: z ℓ = f ℓ (cid:16) ϕ ℓ ( z ℓ ) + β d ℓ , g ℓ (cid:17) 22: Λ = ( z 1 ,..., z ℓ ,..., z L , d 1 ,..., d ℓ ,..., d L , 23: e 0 ,..., e ℓ ,..., e L − 1 ) 24:
Return Λ
1: Input: Λ , λ , γ , M , & Θ 2: function U PDATE W EIGHTS ( Λ , Θ ) 3: //
Calculate weight displacements 4: for ℓ =
L to 1 do 5: ∆ W ℓ = (cid:0) e ℓ − 1 · ( ϕ ℓ ( z ℓ )) T (cid:1) ⊙ S ℓW 6: ∆ E ℓ = γ ( d ℓ · ( e ℓ − 1 ) T ) ⊙ S ℓE 7: ∆ W 1 x = (cid:0) e 0 x · ( ϕ 1 ( z 1 )) T (cid:1) ⊙ S 1 W,x 8:
∆ E 1 x = γ ( d 1 · ( e 0 x ) T ) ⊙ S 1 E,x 9: ∆ W 1 y = (cid:0) e 0 y · ( ϕ 1 ( z 1 )) T (cid:1) ⊙ S 1 W,y 10:
∆ E 1 y = γ ( d 1 · ( e 0 y ) T ) ⊙ S 1 E,y 11:12: // Update current weights 13: for ℓ =
L to 1 do 14:
W ℓ = W ℓ + λ ∆ W ℓ 15:
E ℓ = E ℓ + λ ∆ E ℓ 16:
W 1 x = W 1 x + λ ∆ W 1 x 17:
E 1 x = E 1 x + λ ∆ E 1 x 18:
W 1 y =
W 1 y + λ ∆ W 1 y 19:
E 1 y
=
E 1 y + λ ∆ E 1 y 20:
Update ( g 1 , ··· , g L , M ) via Eqn. 4 21: //
Return new weights 22: Θ = { W 1 x , W 1 y ,..., W ℓ ... W L , 23:
E 1 x , E 1 y ,..., E ℓ ,..., E L } 24:
Return Θ given its own internal state, without requiring the active generation of other predictors, and its errorcan be readily computed after prediction by comparing to the current state of the target neuronstate, the weight updates of any predictor layer may be computed in parallel to others.
This wouldallow us to allocate dedicated computing cores to particular predictors, or “pieces”, of the S-NCN.
Observe that the S-NCN does not require activation derivatives in any of its computations (thisneurobiologically more realistic and favorable for specialized hardware implementations).
Furthermore, during the learning process, each context vector g ℓt and the corresponding memorymatrix is adjusted according to the following simple contrastive rule:
[Equation]
where η e modulates a long-term memory update using the current perturbation to be applied to layerℓ .
η g controls the repulsion term, which “pushes” context codes away from each other (for diversity).
These adapted codes, which influence inter-neuronal competition in a task-sensitive manner, could beviewed as a simplification of distributed temporal context [37], where contiguity, i.e., recall/generationof one item is influenced by the presence of another, is introduced into S-NCN distributed processing.
The full pseudocode illustrating how all the elements described so far are combined to run and updatethe S-NCN network is presented n Algorithms 1 and 2.
The transmission of bottom-up and top-downerrors in the S-NCN is motivated by the theory of predictive processing [38, 39, 40, 41, 42, 43, 44]and classical work related to interactive networks [34, 45, 46], where neural models undergo a settlingprocess to process input stimuli more than once.
Though this requires extra computation, this processendows the network with desirable properties, such as the ability to conduct constraint satisfaction[47, 48].
By using a multi-step processing, laterally-competitive processing scheme, the S-NCNis able to “select” subnetworks (portions of neurons) for specific tasks, reducing representationaloverlap and, ultimately, forgetting.
This selection is driven by the task pointer, produced by the taskselection model, T ( x ) , the final piece of the S-NCN neural system, which we describe next.
3.2.3 Objective Function
During training, when presented with stimulus ( y i , x i , t i ) , the S-NCN adjusts its internal statesand synapses so that the output z 0 µ of layer 1 ( P 1 ) is as close as possible to ( y i , x i ) .
It does thisby minimizing total discrepancy [49] – a measure of its total internal disorder (and one that can
C o rr ec t
i on
C o rr ec t
i on zzz
𝝁 z
𝝁
[x y] z= =
Prediction
Prediction W
W
V
Vzzt = T(x)
Modulation from task selection system, T ( x )
K steps of iterative processing to find correct subnetwork
E
E
Context ( x 1 , y 1 )
Generative Model ( x 2 , y 2 )
[Figure] Continuum
Context
Generative Model (x) (x) be shown to approximate free energy [50]), which is the sum of all mismatches between layerwiseguesses and actual states.
In its general form, total discrepancy for an S-NCN is formally:
[Equation]
[Equation]
[Equation]
Θ contains all of the synaptic parameters, i.e., Θ = { W 1 , E 1 , · · · , W L , E L } and Θ ℓ = { W ℓ , E ℓ } .
The above loss decomposes the problem of credit assignment in the S-NCN into several sub-problemsthat each focuses on the comparison between the prediction z
ℓµ made by layer ℓ + 1 and the actualstate value z ℓ of layer ℓ .
The resulting updates to each state z ℓ , as well as the relevant parameters, willthen depend on a bottom-up transmitted error signal and the top-down influence of the mismatch withthe expectation of the predictor immediately above [49, 35].
Note that, while we motivate aspects ofour model from a neuro-cognitive perspective, the error units and weight updates can be derived fromthe total discrepancy function above [35] (and cast as approximately minimizing free energy).
3.3 The Neural Task Selection Model
In the last section, we described the S-NCN as taking in a task signal that drives the choice of the rightinternal task context memory g ℓ for layer ℓ .
To create this task pointer, we design a second neuralmodel that we call the task selector T ( x ) – this will allow the S-NCN to automatically decide when anew task has been encountered and to determine, at test time, what task data points belong to.
Themotivation behind T ( x ) comes from the neuroscientific idea that the basal ganglia plays the role ofinformation routing (among other roles), which serves as a form of task selection [51, 52]).
In otherwords, it selects or enables various cognitive programs stored in other cortical regions [5].
As a result,we develop a sort of complementary learning system (CLS) (different than that of [53], which sets upan interaction between a hippocampus and a neocortex model), which pairs our cortex-like modelwith a basal ganglia-like model.
We refer to our task selector T ( x ) as the “functional neural basalganglia” (FNBG) in order to emphasize the fact that the actual basal ganglia in the human brain is farmore complex and does far more than what our simple computational model does.
Our FNBG model,built with competitive learning, has two roles: 1) task shift detection - deciding whether data froman input stream is indicating the presence of a new task, and 2) task recognition - deciding whetherincoming data requires switching to an existent task context or creating a new context knowledge.
Task Shift Detection:
In order to detect the occurrence of a new task while processing data fromthe pattern stream, T ( x ) utilizes the output error neurons of the generative model described earlierto detect spikes in their values that might indicate distributional/context drift.
Specifically, T ( x )maintains an exponential running mean µ L and variance σ 2 L of the norm of the cortex model’s labelerror neurons e 0 y .
The necessary statistics are calculated as follows:
[Equation]
with η = 0 .
1 (determined by preliminary experiments).
Using the above dynamic statistics, a shiftis detected by determining if the following inequality evaluates to true (repeatedly for a series of5 consecutive batches): µ L ( t ) > µ L ( t − 1) + 2 (cid:112) σ 2 L ( t − 1) .
Upon detection of a boundary, wesuppress the check until 1000 samples have been seen after the last detected shift (creating a refractoryperiod to allow the competitive learning model, described next, to acquire enough data to learn).
Task Recognition through Competitive Learning:
To conduct task recognition, T ( x ) firstrandomly projects the input x to a low-dimensional space (“key”)
k
= R · x ( R is initialized from a
Gaussian distribution).
We then update a rolling average estimate of the streaming input using thisgenerated key as follows: k µ = (1 − τ ) k µ + τ k (with τ = 0 . 65 ).
Finally, with the matrix Q , FNBGmaps this rolling representation k µ to a decision as to which task context the S-NCN generativecortex is to utilize, i.e., ˆt = BK WTA ( Q · ( k µ / || k µ ) || 2 , K = 1)
.The
task pointer produced for the
S-NCN model is then t i = arg max( ˆt ) ( arg max( ˆt ) returns the index of the maximum value in ˆt ).
To update the FNBG weights, while also avoiding catastrophic interference in T ( x ) itself, we proposea biologically-inspired learning rule based on competitive learning.
Specifically, we develop what wecall “guided competitive learning”, since during the act of processing a stream of certain samplesfrom the task we know we are operating on, we know which neuron out of a set of T − 1 task outputneurons should be selected.
This leads to the following update rule formally defined as:
[Equation]
where ρ is the competitive weight adjustment factor (a value we found works well in the range of[0 .
Note that t =
ˆt recovers an unsupervised classical competitive Hebbian learning.
However,we force the model to a specific task pointer value by using t , the one-hot encoding of the dynamicinteger variable t maintained by the FNBG itself, initialized as t = 0 .
Every time a task shift isdetected according to Equations 6-8, this dynamic variable is incremented by one, i.e., t ← t + 1 .
For both task recognition and the FNBG’s synaptic update, note that the rolling representation k µ isnormalized by its Euclidean norm so that we may utilize a vectorizable form of competitive learningbased on dot products (taking advantage of GPU-based matrix multiplication).
In short, we takethe (normalized) rolling average representation of the input stream for a given task, compute its dotproduct simultaneously with all currently-available task weight vectors, and choose the dot productwith maximal value as the winner.
Finally, we re-normalize the matrix Q by its column Euclideannorms after each update, i.e., Q [: , i ] =
( Q [: , i ]+∆ Q [: , i ]) /
|| Q
[: , i ] || 2 where Q [: , i ] indicates extractingall values in column i from Q (this normalization is similar to that of adaptive resonance theory [54]).
3.4 Putting It All Together: A Complementary System
At a high level, given the above, the full S-NCN complementary system, depicted in Figure 1 (Right),consists of:1) a task selection model (inspired by the information routing/executive control behaviorof the basal ganglia [5]) which creates the task contexts that laterally inhibit/gate the activities ofthe generative S-NCN, and 2) a generative model that learns to predict inputs/labels given a taskcontext.
In essence, the FNBG T ( x ) takes in x j to produce ˆ t (a one-hot encoding of the task pointert i ) which is then fed into the generative S-NCN (along with x and y ) to compute predictions.
Experiments
Simulation Setup:
In our experiments, we train models with three hidden layers, whether they bemultilayer perceptrons (MLPs) or sequential neural coding networks (S-NCNs) and compare againstbaselines from the literature.
All models were restricted to contain (a maximum of) 500 units perlayer.
For the S-NCN, weights were initialized from a Gaussian distribution scaled by each layer’sfan-in and were optimized using stochastic gradient descent with a learning rate of λ = 0 . 01 .
Baselinemodels were trained on each task dataset for 40 epochs while the S-NCN only made a single pass(mini-batches were of size 10 ).
The output layer for each MLP was a maximum entropy classifier andthe objective was to minimize Categorical cross entropy – in the S-NCN, this was encoded in its labelerror neurons e [-
-]0 y .
(See Appendix for experimental details, computing infrastructure, hyper-parameterdetails, and code details.)
[Table] [Table] Evaluation Metrics: To measure model generalization over the sequence of tasks, we make useof the resulting task matrix R (as in [55]), an N × N matrix of task accuracy scores (normalized to[0 , 1] ), where in this study N = T .
We measure average accuracy (ACC) (mean performance acrosstasks) and backward transfer (BWT).
BWT measures the influence that learning a task T t has on theperformance of task
T k < T t .
A positive BWT indicates that a learning task T t increases performanceon a preceding task T k .
As such, a higher BWT is better and a strongly negative BWT means there isstronger (more catastrophic) forgetting.
Mean and standard deviation ( 10 trials) are reported for ACCand mean ( 10 trials) for BWT (see Appendix for standard deviation).
The formulas for ACC, BWT,and for a new set of metrics created to analyze memory retention, are provided in the Appendix.
4.1 The Multi-Dataset Task Stream
To start, we tested the S-NCN model on a complex task sequence composed of several learningbenchmarks.
We create task sequences by breaking apart MNIST ( M ), Fashion MNIST ( FM ), and
Google Draw ( GD ) each into two “sub-tasks” (e.g., for MNIST, M1 and M2 ), or portions of data witha particular subset of the original dataset’s classes.
See the Appendix for details on sub-tasks/tasksequence creation.
In Table 1, we present two task orderings (Ordering #1 is “High Color Sim.” and
Ordering #2 is “Low Color Sim.”)
each under two conditions: sub-tasks that have 1) an equal number of classes ( 5 each), and 2) an unequal number of classes (number of classes was chosen randomly,omitting 5 as an option).
The number of classes was sampled once and held constant for all trials.
We evaluate our proposed S-NCN system (as well as four variations of it in the Appendix) – hyperparameters were β = 0 .
05 , K = 10 , η g = 0 .
9 , η e = 0 .
01 , α = 0 . 98 ).
The baselines include an MLPtrained only with backprop (Backprop), Elastic Weight Consolidation (EWC)
[19], the Mode-IMMmethod [56], the Md-IMM method combined with either DropTransfer (DT+Md-IMM) or both
L2-Transfer and DropTransfer (L2-DT-Mode-IMM)
[56], and the competitive model, hard attentionto task (HAT) [57].
For each baseline, we tuned hyper-parameters based on their accuracy on eachtask’s development set.
See Appendix for extra baseline results.
Discussion:
Results are reported in Table 1 (see Appendix for more results).
Each simulationwas run 10 times (each trial used a unique seed) – we report both mean and standard deviation.
Asobserved in our results, we see that our S-NCN model outperforms all baselines consistently,in terms of ACC and BWT, exhibiting improved memory retention over the baselines, suchas backprop, and more notably, EWC (and, in the Appendix, our expanded results show thatthe FNBG-driven lateral inhibition is key to improving memory retention the most).
This result isrobust across both task sequences and equal/unequal class settings.
Meta-parameters for the S-NCNswere only tweaked minorly, with the same values used in all scenarios.
The observation that lateralinhibition improves the neural computation (in our model) also corroborates the result of [48].
4.2 Continual Learning Benchmarks
To connect our model to current lifelong learning results, we experimented with a wide swatchof approaches on three benchmarks – Split MNIST, Split NotMNIST, and Split Fashion MNIST(FMNIST).
Furthermore, we compare to multi-head (below dashed line in Table 2) and single-headmodels (above dashed line).
We compare the S-NCN to replay/rehearsal and non-replay methods:na¨ıve rehearsal with memory (NR+M), EWC, synaptic intelligence (SI), MAS [58], Lwf [59], GEM[60], ICarl [61], Lucir [62], and Mnemonics [63] (additional baselines can be found in the Appendix).
Discussion
In Table 2, we report ACC and BWT, averaged over 10 trials, offering an extensive, comprehensive comparison of methods and demonstrating that, for all three benchmarks, theproposed S-NCN outperforms all of them in terms of ACC , and nearly all in terms of BWT(and on par with GEM, the difference in BWT being negligible) demonstrating the power affordedby designing models with stronger grounding in neurobiology (see Appendix for a discussion oflimitations).
Furthermore, it is promising to see that the S-NCN outperforms/matches performancewith not only the single-head models but also with the multi-head models (except GEM), which enjoyan easier version of the problem since they can utilize a different classifier per task.
Finally, notethat the S-NCN, due to the FNBG-driven lateral competition, learns to compose task contexts ina data-dependent manner.
Desirably, the S-NCN is a single-head model, meaning that it does notgrow out a separate softmax classifier for each task (as in multi-headed approaches, e.g., HAT, IMM,
GEM), which means that the S-NCN is tackling the harder form of the forgetting problem, learningrepresentations that preserve knowledge across disjoint tasks.
Furthermore, note that our model isonline, whereas models such as IMM or SI require multiple passes per task dataset, and does notrequire validation data in order to run an expensive neural architecture search (NAS) as in [64].
Conclusion
In this paper, we proposed the sequential neural coding network (S-NCN), an interactive generativemodel, and its local learning procedure for lifelong machine learning.
As demonstrated on severalbenchmarks/setups, this model retains the knowledge acquired from prior tasks when learning newones in task data streams, primarily when lateral inhibition, driven by a self-organizing task selectionmodel, sharpens neural activities within its layers.
In terms of negative social impact, S-NCN-basedlifelong learning models could lead to the development of better-performing systems, i.e., roboticsystems (drones), that might result in the loss of life (see Appendix for an expanded discussion).
Tosafeguard against this, it will be important to develop an ethical framework to guide the design andtraining of general intelligent systems to ensure safe integration into human society.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] Limitations are discussed in the
Appendix/Supplementary Material.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Potentialnegative societal impacts of the work are discussed in the conclusion and in the Appendix/Supplementary Material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] The code and
the data are proprietary.
We will provide complete codebase to support this paper uponacceptance.
We do provide detailed pesudocode of our model, describing overall flow
which can be used to re-create our model.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report one standard deviation of uncertainty ( 10trials).(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Yes in appendix we provide details
about the compute used to run all these experiments.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Yes, we do cite all
the existing assests in our paper as well as in our codebase.
(b) Did you mention the license of the assets?
[Yes] We do mention licensing in our
codebase.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]In the appendix, we provide details for creating the special challenge benchmark of
this paper.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]