Lifelong Neural Predictive Coding: Learning Cumulatively Online without Forgetting
Abstract
In lifelong learning systems based on artificial neural networks, one of the biggest obstacles is the inability to retain old knowledge as new information is encountered.
This phenomenon is known as catastrophic forgetting.
In this paper, we propose a new kind of connectionist architecture, the Sequential Neural Coding Network, that is robust to forgetting when learning from streams of data points and, unlike networks of today, does not learn via the popular back-propagation of errors.
Grounded in the neurocognitive theory of predictive coding, our model adapts its synapses in a biologically-plausible fashion while another neural system learns to direct and control this cortex-like structure, mimicking some of the task-executive control functionality of the basal ganglia.
In our experiments, we demonstrate that our self-organizing system experiences significantly less forgetting compared to standard neural models, outperforming a swath of previously proposed methods, including rehearsal/data buffer-based methods, on both standard (SplitMNIST, Split Fashion MNIST, etc.) and custom benchmarks even though it is trained in a stream-like fashion.
Our work offers evidence that emulating mechanisms in real neuronal systems, e.g., local learning, lateral competition, can yield new directions and possibilities for tackling the grand challenge of lifelong machine learning.
Introduction
Lifelong learning is a part of machine learning and artificial intelligence research with the goal of developing a computational agent that can learn over time and continually accumulate new knowledge while retaining its previously learned experiences [1, 2].
For example, suppose an agent needs to learn to classify digits, then types of clothing, and then sketches of objects.
As each new task arrives the agent is expected to process the accompanying data and learn the new task but still remember how to complete the old tasks, at least without significant degradation in performance or loss of generalization.
Modern-day connectionist systems are typically trained on a fixed pool of data samples, collected in controlled environments, in isolation and random order, and then evaluated on a separate validation data pool.
This is a far cry from what we really desire from learning machines.
When we look to humans or other animals, we see that they are more than capable of learning in a continual manner, making decisions based on sensorimotor input throughout their lifespans [3].
This ability to incrementally acquire and refine knowledge over long periods of time is driven by cognitive processes that come together to create the experience-driven specialization of motor and perceptual skills [4, 3].
Thus, evaluating how neural systems generalize on task sequences, as opposed to single, isolated tasks, proves to be a far greater challenge.
In order to continually adapt, the brain must retain specific memories of prior tasks.
In working towards the challenge of lifelong machine learning, this paper makes the following contributions: 1) we propose the sequential neural coding network, an interactive generative model that jointly reconstructs input and predicts its label, and an algorithm for updating its synapses, 2) we show that memory retention is vastly improved by integrating our model’s multi-step nature with lateral competition that is driven by a task selection function inspired by the basal ganglia [5], and 3) we compare our model’s performance against state-of-the-art baselines, including both regularization and rehearsal/replay-based methods, on publicly-available benchmarks.
Related Work
It is well-known that when artificial neural networks (ANNs) are trained on more than one task sequentially, the new information contained in subsequent tasks leads to catastrophic interference (a.k.a.
catastrophic forgetting) with the information acquired in earlier tasks [6, 7, 8].
This happens in connectionist systems when the new data instances to be learned are significantly different from previously observed ones.
This causes the new information to overwrite knowledge currently encoded in the system’s synaptic weights, due to the sharing of neural representations over tasks [9, 6] (this is known as the representational overlap problem).
In isolated, task-specific (offline) learning (though it still occurs [10]), this type of weight overwriting occurs to a lesser degree because the patterns are presented to the agent pseudo-randomly and multiple times, i.e., via multiple epochs.
Over the decades, there have been many approaches proposed to mitigate catastrophic forgetting in neural systems.
Some of the earliest attempts proposed memory systems where prior data points were stored and regularly replayed to the network in a process called “rehearsal”, which involved interleaving these data points with samples drawn from new datasets [11, 12, 13, 14, 15].
Though effective, the main drawback of these approaches is that they require explicitly storing old data.
Such a mechanism is not known to exist in the brain and, as a practical matter, this leads to exploding working (hardware) memory requirements (inefficiency).
In addition, rehearsal-based approaches do not offer any mechanisms to preserve consolidated knowledge in the face of acquiring new information [4].
Other approaches attempt to allocate additional neural “resources”, i.e., growing the networks when required [16, 17], motivated by earlier findings [18].
However, this leads to dramatically increasing computational requirements over time as the networks grow larger.
To compound these issues further, systems with growing capacity cannot know how many resources to allocate at a given time since the number of tasks and samples are not known a priori (without imposing strong assumptions on the input distribution).
Other approaches try to block old information from being overwritten through regularization [19].
From this vast collection of research, each approach bearing strengths and weaknesses, three suggested remedies have emerged: 1) allocate additional neural resources to accommodate new knowledge, 2) use non-overlapping representations (or semi-distributed ones [20]) if resources are fixed, and 3) interleave old patterns with new ones as new information is acquired.
In this work, we consider the setting where the space available to the agent grows slowly compared to the rate of new tasks being presented.
This means that we cannot just create a new, separate network for every task observed in the stream.
Furthermore, storing, reshuffling, and re-presenting data in the stream is not feasible in this setting.
Concretely, our approach could be classified as class incremental learning (Class-IL)
[21] given that we do not use task identifiers at both training and test time.
In addition to addressing the above, our contribution to lifelong learning is motivated by the premise that developing algorithms [22, 23, 24, 25, 26] that serve as alternatives to back-propagation will lead to the creation of promising architectures with mechanisms equipped to tackle catastrophic interference.
Cumulative Learning with Neural Coding
Notation:
We start by defining key notation.
⊙ indicates a Hadamard product while · denotes matrix/vector multiplication.
( v ) T is the transpose of v .
Matrices/vectors are depicted in bold-font, e.g., matrix M or vector v (scalars shown in italic).
Finally, || v |
3.1 Sequential Learning and the Data Continuum
This work focuses on adapting a neural system in the context of sequential learning.
Starting from an early definition [27] of this form of learning, we assume that there is a sequence of tasks T 1 , T 2 , T 3 , . . .
(with potentially no end) with each task presented to a system in order.
When faced with the ( N +1) th task, the system should use the knowledge that it has gained from the previous N tasks to aid in learning and performing the current task.
The knowledge of a system is stored in a knowledge base (KB), e.g., the synapses of a neural model.
Each task T i has its own corresponding dataset D i =
{ ( y 1 , x 1 , t i ) . . .
( y n i , x n i , t i ) } with n i examples.
Here x j ∈ R J x × 1 represents the feature vector of the j th example ( J x is its dimensionality), y j ∈ { 0 , 1 } J y × 1 is the target (label), and t i is the task descriptor that identifies ( y j , x j ) as being a data point from task i .
The task descriptor is one-hot encoded as t ∈ { 0 , 1 } ( N +1)
× 1 and, when a new task is encountered, the size of the one-hot vector is increased by one – thus the network adds an extra randomly initialized input node if it accepts t i as an extra input.
Note that while we present the task data continuum with t i explicitly depicted, we will generalize our models to not depend on t i (making this problem task descriptor-free).
Dynamic Output Units: The output nodes in our setting get re-used for each new task.
For example, output node 1 of the network could represent a prediction for the digit “1” in task T 1 (e.g., digit recognition), while in task T 2 (e.g., clothing recognition) the same node could represent a prediction for “pants”.
If a new task has more classes than previous tasks, we add output nodes with randomly initialized weights.
For example, if prior tasks were binary and the new task has 4 targets, we addmore outputs to the network.
Note this is a difficult form of cumulative learning [28].
Context Units: For the model that we develop in this study, when task t is encountered, for each layer ℓ in the network, we make use of a task embedding vector g ℓt (this new memory is much smaller than creating a new network for task t , which would require new weight matrices per layer rather than an extra vector).
All g ℓt ∈ R J ℓ × 1 ( J ℓ is the number of units in layer ℓ ) are stored in memory matrices M ℓ ∈ R ( N +1)
× J ℓ ( M = { M 1 , ...
M L } ), where a context can be retrieved using a one-hot encoding of the task descriptor, i.e., M ℓ · t ( t will be produced by another system – see Section 3.3).
3.2 The Interactive Generative Model
The sequential neural coding network (S-NCN) is designed to make flexible conditional predictions – e.g., predicting y given x , predicting both y and x , predicting missing parts of x given y and the observed parts of x , etc.
In order to do so, it treats inputs/outputs in a non-standard way.
The input to the model is the set of task contexts { g 1 t , ..., g Lt } and the output units represent ( y , x ) .
To predict y i given x i , we clamp the output nodes responsible for predicting x , forcing their output to be x i .
During training, outputs are clamped to both y i and x i , forcing the S-NCN to update latent states and synapses.
The S-NCN can also operate as a probabilistic generative network by feeding in a random noise vector as input, but we leave this extension to future work (and focus on predicting y given x ).
The full computational process of the S-NCN is defined by three key steps: 1) layer-wise hypothesis generation, 2) state error-correction, and 3) synaptic weight evolution.
The first two steps iteratively predict and correct the representations of the model for the input and target values of the task.
After K iterations, model weights and the current task context memory are updated.
In this section, we provide details of the above steps and then describe the objective function that our model optimizes.
3.2.1 Inference: Predicting and Correcting States
Layer-wise State Prediction.
The architecture of the S-NCN can be viewed as a stack of parallel, stateful neural-based prediction layers P 1 , ..., P ℓ , ..., P m , where the goal of each predictor is to guess the internal state of the predictor in the layer below (i.e., the S-NCN is not a feedforward network).
The state of P ℓ (layer ℓ ) is represented by the (zero-initialized) vector z ℓ ∈ R J ℓ × 1 ( J ℓ is the number of units in layer ℓ ).
P ℓ makes a prediction z ℓ − 1 µ about the current state z ℓ − 1 of P ℓ − 1 (layer ℓ − 1 ).
Furthermore, we let z 0 x and z
y denote clamped outputs.
That is, if we want to predict the label y i given the features x i , we set z 0 x = x i and if we want to train, we set both z 0 x = x i and z
y = y i .
The values predicted by layer 1 are denoted by z 0 µ,x and z 0 µ,y .
Note that z 0 =
[ z 0 x , z 0 y ] , where the two are concatenated (if either is missing, the SNCN completes the required values – see Appendix).
With respect to neural structure, the parallel predictors of the S-NCN are locally connected through (forward) generative weights W ℓ ∈ R J ℓ × J ℓ +1 and error (feedback) weights E ℓ ∈ R J ℓ +1 × J ℓ , which work to transmit error information to relevant regions of neural processing elements, effectively coordinating all of the S-NCN’s predictors.
Formally, a predictor (which embodies the computation of deep excitatory pyramidal neurons), with state z ℓ +1 , that guesses the state of z ℓ , takes on the following form (given matrix W ℓ +1 and activation ϕ ℓ +1 ):
[Equation]
where e ℓ is a block of error units.
Error units are paired with each predictor.
Their task is to compute the disagreement or mismatch between the predictor’s output and the target activity pattern z ℓ (embodying the mismatch computation carried out by superficial pyramidal neurons).
The error unit vector e ℓ can also be derived from the total discrepancy reduction presented in Equation 5 (see Appendix).
Note that for layer 0 , e 0 =
[ e 0 x , e 0 y ]
(there are error neurons for x i as well as others for y i , the concatenation of which makes up the bottom-most prediction error signal).
Latent State Correction and Context Updating.
Once each layer ℓ has made a prediction about the layer below it and error units have been activated, the state of layer ℓ can be adjusted to take into account the local top-down and bottom-up error information.
Using its current state and the error nodes (along with its task context embedding g ℓt ), layer ℓ in the S-NCN adjusts its state z ℓ as follows:
[Equation]
where β is the state correction rate and k marks one step of the S-NCN’s K -step inference process.
Note that d ℓ is the perturbation that adjusts the values of the hidden states – it combines a top-down expectation of layer ℓ with a bottom-up pressure from layer ℓ − 1 . The error feedback weights E ℓ are parameters that play a crucial role in this calculation, as they are responsible for transmitting the error at ℓ back up to layer ℓ + 1 . Notably, part of the state-correction requires competition among the individual units in a given layer through the function f ℓ ( z ℓ , g ℓt ) . There are various ways in which this function can be implemented and, in this study, we implemented it as follows:
[Equation]
where BK WTA ( v , K ) is the binarized K winners-take-all function, yielding a binary vector withat the index of each of the K winning units, or formally:
[Equation]
In the Appendix, we study other forms of the competition function (we found that the above performed best, so we report this in the main paper).
Note that this lateral inhibition is a function of evolving context g ℓ , triggered by a task pointer t i (produced by a task selector model, which we define later).
In real neural systems, intra-layer competition between units is thought to facilitate contextual processing [29], where only some neuronal signals are strengthened while the activity of others is suppressed.
Moreover, lateral competition, often classically modeled with anti-Hebbian learning [30], encourages the formation of sparse codes [31, 32].
Since the S-NCN is an interactive model, incorporating lateral competition is natural and computed online, highlighting its flexibility compared to ANNs.
2 Crucially, this lateral activity is an inductive bias that leads the S-NCN to acquire taskdependent representations that encode information for multiple, disjoint tasks.
In a sense, this task specialization that we build into the neural dynamics is similar in spirit to activation sharpening [33].
3.2.2 Updating Synaptic Parameters
Given that the S-NCN is an interactive network [34], inferring its states requires running Equationsand 2 K times, where the model alternates between making predictions and then correcting states once error units have been computed.
Once latent states have been inferred, the S-NCN then adjusts its synaptic values.
The synaptic updates take the form of Hebbian-like rules:
[Equation]
1: Input: sample ( y , x , t ) , β , M , & Θ 2: function I NFER S TATES ( ( y , x , t ) , Θ ) 3: ( g 1 ,..., g ℓ ,..., g L ) ← getContexts ( t , M ) 4:
Set z 1 ,..., z ℓ ,..., z L to 0 5:
e L = 0 , z 0 x = x , z
y =
y 6: for k = 1 to K do 7: for ℓ =
L to 1 do ▷ Run layerwise predictors 8: if ℓ > 1 then ▷ Latent prediction layer 9: z ℓ − 1 µ = W ℓ · ϕ ℓ ( z ℓ ) 10: e ℓ − 1 = ( z ℓ − 1 − z ℓ − 1 µ ) 11: else ▷ Sensory prediction layer 12: z ℓ − 1 µ,x =
W ℓx · ϕ ℓ ( z ℓ ) 13: e ℓ − 1 x = ( z ℓ − 1 x − z ℓ − 1 µ,x ) 14: z ℓ − 1 µ,y =
W ℓy · ϕ ℓ ( z ℓ ) 15: e ℓ − 1 y =
( z ℓ − 1 y − z ℓ − 1 µ,y ) 16: for ℓ =
L to 1 do ▷ Correct internal states 17: if ℓ == 1 then 18: d ℓ = − e ℓ + E ℓx · e ℓ − 1 x + E ℓy · e ℓ − 1 y 19: else 20: d ℓ = − e ℓ + E ℓ · e ℓ −21: z ℓ = f ℓ (cid:16) ϕ ℓ ( z ℓ ) + β d ℓ , g ℓ (cid:17) 22: Λ = ( z 1 ,..., z ℓ ,..., z L , d 1 ,..., d ℓ ,..., d L , 23: e 0 ,..., e ℓ ,..., e L − 1 ) 24:
Return Λ
: Input: Λ λ M & Θ 2: function U PDATE W EIGHTS ( Λ , Θ ) 3: //
Calculate weight displacements 4: for ℓ =
L to 2 do 5: ∆ W ℓ = (cid:0) e ℓ − 1 · ( ϕ ℓ ( z ℓ )) T (cid:1) ⊙ S ℓW 6: ∆ E ℓ = γ ( d ℓ · ( e ℓ − 1 ) T ) ⊙ S ℓE 7: ∆ W 1 x = (cid:0) e 0 x · ( ϕ 1 ( z 1 )) T (cid:1) ⊙ S 1 W,x 8:
∆ E 1 x = γ ( d 1 · ( e 0 x ) T ) ⊙ S 1 E,x 9: ∆ W 1 y = (cid:0) e 0 y · ( ϕ 1 ( z 1 )) T (cid:1) ⊙ S 1 W,y 10:
∆ E 1 y = γ ( d 1 · ( e 0 y ) T ) ⊙ S 1 E,y 11:12: // Update current weights 13: for ℓ = L to 2 do 14:
W ℓ = W ℓ + λ ∆ W ℓ 15:
E ℓ = E ℓ + λ ∆ E ℓ 16:
W 1 x = W 1 x + λ ∆ W 1 x 17:
E 1 x = E 1 x + λ ∆ E 1 x 18:
W 1 y =
W 1 y + λ ∆ W 1 y 19:
E 1 y
=
E 1 y + λ ∆ E 1 y 20:
Update ( g 1 , ··· , g L , M ) via Eqn. 4 21: //
Return new weights 22: Θ = { W 1 x , W 1 y ,..., W ℓ ... W L , 23:
E 1 x , E 1 y ,..., E ℓ ,..., E L } 24:
Return Θ where α is a scaling factor, usually set to < 1 . 0 , that makes the error feedback weights change at a slower rate than the forward weights (this improves convergence [35]).
S ℓW ∈ R J ℓ × J ℓ +1 and
S ℓE ∈ R J ℓ +1 × J ℓ are modulation factors that provide stability to the weight updates (see Appendix).
An important property of the above weight update rules is that they are local – to compute changes in the synapses, all we require is the information immediately available to the neuron(s) of interest (making these rules function similarly to classical Hebbian updates [36], although there are important differences to them, as discussed in [35]).
Since a neuron is able to immediately generate a hypothesis given its own internal state, without requiring the active generation of other predictors, and its error can be readily computed after prediction by comparing to the current state of the target neuron state, the weight updates of any predictor layer may be computed in parallel to others.
This would allow us to allocate dedicated computing cores to particular predictors, or “pieces”, of the S-NCN.
Observe that the S-NCN does not require activation derivatives in any of its computations (this is neurobiologically more realistic and favorable for specialized hardware implementations).
Furthermore, during the learning process, each context vector g ℓt and its corresponding memory matrix is adjusted according to the following simple contrastive rule:
[Equation]
where η e modulates a long-term memory update using the current perturbation to be applied to layer ℓ .
η g controls the repulsion term, which “pushes” context codes away from each other (for diversity).
These adapted codes, which influence inter-neuronal competition in a task-sensitive manner, could be viewed as a simplification of distributed temporal context [37], where contiguity, i.e., recall/generation of one item is influenced by the presence of another, is introduced into S-NCN distributed processing.
The pseudocode illustrating how the elements described so far are combined in an S-NCN system is presented in Algorithms 1 and 2.
The transmission of bottom-up and top-down errors in the S-NCN is motivated by the theory of predictive processing [38, 39, 40, 41, 42, 43, 44] and classical work on interactive networks [34, 45, 46], where models undergo a settling process to process input stimuli more than once (see Appendix for intuition).
Though this requires extra computation, the process endows the network with desirable properties, e.g., the ability to conduct constraint satisfaction [47, 48].
By using a multi-step, laterally-competitive processing scheme, the S-NCN is able to “select” subnetworks (portions of neurons) for specific tasks, reducing representational overlap and,
C o rr ec t
i on
C o rr ec t
i on zzz
𝝁 z
𝝁
[x y] z= =
Prediction
Prediction W
W
V
Vzzt = T(x)
Modulation from task selection system, T ( x )
K steps of iterative processing to find correct subnetwork
E
E
Context ( x 1 , y 1 )
Generative Model ( x 2 , y 2 )
Task Stream / Data Continuum
Context
Generative Model (x) (x) ultimately, forgetting.
This selection is driven by the task pointer t , produced by the task selector,
T ( x ) , the final piece of the S-NCN system, which we describe in Section 3.3.
3.2.3 Objective Function
During training, when presented with stimulus ( y i , x i , t i ) , the S-NCN adjusts its internal states and synapses so that the output z 0 µ of layer 1 ( P 1 ) is as close as possible to ( y i , x i ) .
It does this by minimizing total discrepancy (ToD) [49] – a measure of its total internal disorder,which is the sum of all mismatches between layerwise guesses and actual states.
Formally, the ToD for an S-NCN is: [Equation] (5) Θ contains all of the synaptic parameters, i.e., Θ = { W 1 , E 1 , · · · , W L , E L } and Θ ℓ = { W ℓ , E ℓ } . The above loss decomposes the problem of credit assignment in the S-NCN into sub-problems with each sub-problem focusing on the comparison between the prediction z
ℓµ made by layer ℓ + 1 and the actual state value z ℓ of layer ℓ .
The resulting updates to each state z ℓ , as well as relevant synapses, will then depend on a bottom-up transmitted error signal and the top-down influence of the mismatch with the expectation of the predictor immediately above [49, 35].
While we have motivated aspects of our model from a neuro-cognitive perspective, the error units and weight updates can be derived from the total discrepancy function above [35] (and cast as approximately minimizing free energy [50]).
3.3 The Neural Task Selection Model
Earlier, we described the S-NCN as taking in a task signal t that drives the system’s task context memory g ℓ for layer ℓ (i.e., the routine getContexts ( t , M ) in Algorithm 1).
To create this task pointer, we design a second neural circuit that we call the task selector T ( x ) – this will allow the S-NCN to automatically decide when a new task has been encountered and to determine, at test time, what task observed data points belong to.
The motivation behind T ( x ) comes from the neuroscientific idea that the basal ganglia plays the role of information routing (among other roles), which serves as a form of task selection [51, 52]).
In other words, it selects/enables various cognitive programs stored in other cortical regions [5].
As a result, we develop a type of complementary learning system (CLS) (different than that of [53], which models an interaction between the hippocampus and neocortex), which pairs our cortex-like model with a basal ganglia-like model.
We refer to our task selector T ( x ) as the “functional neural basal ganglia” (FNBG) in order to emphasize that the actual basal ganglia in the human brain is more complex and does more than what our computational model does.
Our FNBG model, built with competitive learning, has two roles: 1) task shift detection - deciding whether data from an input stream indicates the presence of a new task, and 2) task recognitiondeciding whether incoming data requires switching to an existent task context or creating a new one.
Task Shift Detection:
In order to detect the occurrence of a new task while processing data from the pattern stream, T ( x ) utilizes the output error neurons of the generative model described earlier to detect spikes in their values which might indicate distributional drift.
Specifically, T ( x ) maintains an exponentially weighted running mean µ L and variance σ 2 L of the norm of the cortex model’s label error neurons e 0 y .
The necessary statistics are calculated as follows:
[Equation]
with η = 0 .
1 (determined by preliminary experiments).
Using the above dynamic statistics, a shift is detected by determining if the following inequality evaluates to true (repeatedly for a series of 5 consecutive batches): µ L ( t ) > µ L ( t − 1) + 2 (cid:112) σ 2 L ( t − 1) .
Upon detection of a boundary, we suppress the check until 1000 samples have been seen after the last detected shift (creating a refractory period to allow the competitive learning model, described next, to acquire enough data to learn).
Task Recognition through Competitive Learning:
To conduct task recognition, T ( x ) first randomly projects the input x to a low-dimensional space (“key”)
k
= R · x ( R is initialized from a Gaussian distribution).
We then update a rolling average estimate of the streaming input using this generated key as follows: k µ = (1 − τ ) k µ + τ k (with τ = 0 . 65 ).
Finally, with the matrix Q , the FNBG maps this rolling representation k µ to a decision as to which task context the S-NCN generative cortex is to utilize, i.e., ˆ t = BK WTA ( Q · ( k µ / || k µ ) || 2 , K = 1) .The
task pointer is then used to retrieve contexts { g 1 t , ..., g Lt } where g ℓ = M · ˆt (implementing getContexts ( t = ˆt , M ) ).
To update the FNBG weights, while also avoiding catastrophic interference in T ( x ) itself, we propose a biologically-inspired learning rule based on competitive learning.
Specifically, we develop what we call “guided competitive learning”, since during the act of processing a stream of certain samples from the task that we know that we are operating on, we also know which neuron out of a set of T −task output neurons should be selected.
This leads to the following update rule formally defined as:
[Equation]
where ρ is the competitive weight adjustment factor (a value we found works well in the range of [0 . 5 , 1] ).
Note that t =
ˆt recovers an unsupervised classical competitive Hebbian learning.
However, we force the model to a specific task pointer value by using t , the one-hot encoding of the dynamic integer variable t maintained by the FNBG itself, initialized as t = 0 .
Every time a task shift is detected according to Equations 6-8, this dynamic variable is incremented by one, i.e., t ← t + 1 .
For both task recognition and the FNBG’s synaptic update, note that the rolling representation k µ is normalized by its Euclidean norm so that we may utilize a vectorizable form of competitive learning based on dot products (taking advantage of GPU-based matrix multiplication).
In essence, we take the (normalized) rolling average representation of the input stream for a given task, compute its dot-product with all currently-available task weight vectors, and choose the dot product with maximal value as the winner.
Finally, we re-normalize the matrix Q by its column Euclidean norms after each of its updates, i.e., Q [: , i ] =
( Q [: , i ] + ∆ Q [: , i ]) /
|| Q
[: , i ] || 2 where Q [: , i ] indicates the extraction of all values in column i from Q (this normalization is similar to that of adaptive resonance theory [54]).
3.4 Putting It All Together: A Complementary System
At a high level, given the above, the full S-NCN complementary system, depicted in Figure 1 (Right), consists of: 1) a task selection model (inspired by the information routing/executive control behavior of the basal ganglia [5]) which creates the task contexts that laterally inhibit/gate the activities of the generative S-NCN, and 2) a generative model that learns to predict inputs/labels given a task context.
In essence, the FNBG T ( x ) takes in x j to produce ˆt (a one-hot representation of a task pointer t i ) which is then fed into the generative S-NCN (along with x and y ) to compute predictions.
Experiments
Simulation Setup:
In our experiments, we train models with three hidden layers, whether they be multilayer perceptrons (MLPs) or S-NCNs and compare against baselines from the literature.
All models were restricted to contain (a maximum of) 500 units per layer.
For the S-NCN, weights were initialized from a Gaussian distribution scaled by each layer’s fan-in and were optimized using stochastic gradient descent with learning rate of λ = 0 . 01 .
Baseline models were trained on each task for 40 epochs while the S-NCN only made a single pass.
The output layer for each MLP was a maximum entropy classifier and the objective was to minimize Categorical cross entropy – in the S-NCN, this was encoded in its label error neurons e 0 y .
(See Appendix for experimental details, computing infrastructure, hyper-parameter details, and code details.)
Evaluation Metrics: To measure model generalization over the sequence of tasks, we make use of the resulting task matrix R (as in [55]), an N × N matrix of task accuracy scores (normalized to [0 , 1] ), where in this study N = T .
We measure average accuracy (ACC) (mean performance across tasks) and backward transfer (BWT).
BWT measures the influence that learning a task T t has on the performance of task
T k < T t .
A positive BWT indicates that a learning task T t increases performance on preceding task T k .
As such, higher BWT is better and a strongly negative BWT means there is stronger (more catastrophic) forgetting.
Mean and standard deviation ( 10 trials) are reported for ACC and mean ( 10 trials) for BWT (see Appendix for its standard deviation).
The formulas for ACC, BWT, and a new set of metrics we created to analyze memory retention, are provided in the Appendix.
4.1 The Multi-Dataset Task Stream
To start, we tested the S-NCN model on a complex task sequence composed of several learning benchmarks 4 .We create task sequences by breaking apart MNIST ( M ), Fashion MNIST ( FM ), and
Google Draw ( GD ) each into two “sub-tasks” (e.g., for MNIST, M1 and M2 ), or portions of data with a particular subset of the original dataset’s classes.
See the Appendix for details on sub-tasks/task sequence creation.
In Table 1, we present two task orderings (Ordering #1 is “High Color Sim.” and Ordering #2 is “Low Color Sim.”)
each under two conditions: sub-tasks that have 1) an equal number of classes ( 5 each), and 2) an unequal number of classes (number of classes was chosen randomly, omitting 5 as an option).
The number of classes was sampled once and held constant for all trials.
We evaluate our proposed S-NCN system (as well as four variations of it in the Appendix) – hyperparameters were β = 0 .
05 , K = 10 , η g = 0 .
9 , η e = 0 .
01 , α = 0 . 98 ).
The baselines include an MLP trained only with backprop (Backprop), Elastic Weight Consolidation (EWC)
[19], the Mode-IMM method [56], the Md-IMM method combined with either DropTransfer (DT+Md-IMM) or both L2-Transfer and DropTransfer (L2-DT-Mode-IMM) [56], and the competitive model, hard attention to task (HAT) [57].
For each baseline, we tuned hyper-parameters based on their accuracy on each task’s development set.
See Appendix for additional baseline results.
Discussion: Results are reported in Table 1 (see Appendix for more results).
Each simulation was run 10 times (each trial used a unique seed).As observed in our results, we see that the S-NCN outperforms all baselines consistently, in terms of ACC and BWT, exhibiting improved memory retention over modern, backprop-centric baselines (and, in the Appendix, our expanded results show that the FNBG-driven lateral inhibition is key to improving memory retention the most).
This result is robust across both task sequences and equal/unequal class settings.
Meta-parameters for the S-NCNs were only tweaked minorly, with the same values used in all scenarios.
The observation that lateral inhibition improves the neural computation (in our model) also corroborates the result of [48].
4.2 Continual Learning Benchmarks
To connect our model to current learning results, we experimented with a wide swath of approaches on three benchmarks – Split MNIST, Split NotMNIST, and Split Fashion MNIST (FMNIST).
Furthermore, we compare to multi-head (below dashed line in Table 2) and single-head models (above dashed line).
We compare the S-NCN to both replay/rehearsal and non-replay methods: na¨ıve rehearsal with memory (NR+M), EWC, synaptic intelligence (SI) [58], MAS [59], Lwf [60], GEM [61], ICarl [62], Lucir [63], and Mnemonics [64] (additional baselines can be found in the Appendix).
Discussion
In Table 2, we report ACC and BWT, averaged over 10 trials, offering a comprehensive comparison of methods and demonstrating that, for all three benchmarks, the proposed S-NCN outperforms all of them in terms of ACC , and nearly all in terms of BWT (and on par with GDumb and GEM, the difference in BWT being negligible) demonstrating the power afforded by designing [Table]
[Table]
models with stronger grounding in neurobiology. Furthermore, it is promising to see that the S-NCN outperforms/matches performance with not only the single-head models but also with multi-head models (except GEM), which enjoy an easier version of the problem, i.e., they can utilize a different classifier per task.
Finally, note that the S-NCN, due to the FNBG-driven competition, learns to compose task contexts in a data-dependent manner.
Desirably, the S-NCN is a single-head model, meaning that it does not grow out a separate softmax classifier per task (as in multi-headed models, e.g., HAT, IMM, GEM), which means that it tackles the harder form of the forgetting problem, learning representations that preserve knowledge across disjoint tasks.
Furthermore, our model is online, whereas models such as IMM or SI require multiple passes per dataset, and does not require validation data in order to run an expensive neural architecture search (NAS) as in [65]. 4.3 Discussion: On Model Limitations
Although our results with respect to catastrophic forgetting are promising, the S-NCN does have several limitations that are important to discuss.
First and foremost, the S-NCN attempts to learn a generative model of the joint distribution over inputs and labels, i.e., p ( y , x ) , which has (classically) been shown to be a much more difficult problem compared to directly learning a conditional mapping between inputs and labels [66].
Second, to learn p ( y , x ) , the S-NCN adapts to data through an expectation-maximization process where, given an input, it must iteratively infer a maximum a posteriori (MAP) estimate of its latent neural activities over a K -step stimulus window.
Despite the fortunate fact that, for the benchmarks studied in this work, values for K were fairly low (only 10 to 20 steps were needed per sample/mini-batch), for more complex data types, such as natural images with multiple objects and background scenery, the value of K will quite likely need to be much higher, increasing the S-NCN’s computation time when conducting its online inference. We remark that this drawback could be mitigated by integrating mechanisms to support amortized inference, e.g., as in predictive sparse decomposition [67], and by designing custom software/hardware implementations that exploit the S-NCN’s layer-wise parallelism (which would also work in asynchronous settings) in both its inference and weight updating cycles. Future work should explore adapting the S-NCN to only learning a conditional mapping as opposed to a full joint distribution as well as develop mechanisms for pre-training the generative side of the system (which would allow freezing of the generative synapses and only require updating discriminative ones – this could reduce the value of K for more complex sensory inputs). While the S-NCN’s objective of minimizing ToD is important for breaking free of backprop and its limitations, offering benefits such as resolving weight transport, eschewing the need for derivatives, facilitating local learning without the need for a global feedback pathway, inferring richer stateful representations, it also creates a more challenging optimization problem, i.e., the model must match the values imposed by input data as well as ensure that its internal activities and local predictions are aligned.
While the full complementary system largely mitigates catastrophic interference, this primarily benefits measurements of BWT and could potentially damage the model’s per-task performance on more complex datasets, i.e., the main diagonal of task matrix
R .
Since we do not impose any distributional assumptions over the latent activities (such as a clean
Gaussian prior as in variational autoencoders), if the S-NCN’s estimated value of the latent activities is far from their true posterior distribution, then the S-NCN might experience sub-optimal performance in complex setups.
Even though all continual learning systems suffer from this issue (especially most modern-day ANN-based ones), our model’s dual optimization nature could experience this problem more frequently.
We believe that integrating memory-aware retrieval driven by (a brain-like form of) replay, which would exploit the S-NCN’s generative nature, could be a plausible solution moving forward (helping the system come closer to its true posterior and avoiding bad local optima).
4.4 On Negative Societal Impacts
The potential negative societal impact of the proposed S-NCN complementary system is indirect – while the model and algorithmic framework we develop is foundational in nature, it could potentially affect the myriad of applications/systems currently in development today or those to be developed in the future.
As a result, at best, the same negative consequences that result from using backprop-based
ANNs are still present when using the S-NCN training process instead.
At worst, given that we have presented promising results across several lifelong learning benchmarks, the S-NCN could facilitate the development of better-performing robotic agents that might be used in military applications that might result in the loss of life, such as drones.
Despite the benefits that S-NCN offers to the statistical learning and cognitive neuroscience communities, one should consider the drawbacks of building powerful neural systems to drive applications.
To safeguard against its potential negative impact, we emphasize that developing an ethical framework to guide the design and training of general intelligent systems will be important for ensuring safe integration into human society.
Conclusion
In this paper, we proposed the sequential neural coding network (S-NCN), an interactive generative model, and its local learning procedure for lifelong machine learning.
As demonstrated on several benchmarks and setups, this model retains the knowledge it acquires from prior tasks when learning new ones in the face of task data streams, primarily when lateral inhibition, driven by a self-organizing task selection model, sharpens neural activities within its layers.
As a result, this work marks an important step towards developing brain-inspired agents that are capable of robustly combating catastrophic forgetting, especially ones that do not require task descriptors to guide the learning process, offering a promising pathway towards achieving the ultimate goal of designing agents that act and adapt in ways more similar to animals and humans.
Acknowledgements
We would like to thank Alexander Ororbia (Sr.) for useful discussions related to cognitive types, a concept that served as key motivation for the task contexts that drove lateral inhibition in this work. Checklist For all authors.. (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Limitations are discussed in the Appendix/Supplementary Material.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Potential negative societal impacts of the work are discussed in the conclusion and in the Appendix/Supplementary Material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] The code and the data are proprietary. We will provide complete codebase to support this paper upon acceptance. We do provide detailed pesudocode of our model, describing overall flow which can be used to re-create our model. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report one standard deviation of uncertainty (trials).
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Yes in appendix we provide details about the compute used to run all these experiments.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Yes, we do cite all the existing assests in our paper as well as in our codebase.
(b) Did you mention the license of the assets?
[Yes] We do mention licensing in our codebase.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] In the appendix, we provide details for creating the special challenge benchmark of this paper.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]