Lifelong Neural Predictive Coding: Learning Cumulatively Online without Forgetting (Supplementary Material)
Computational Resource Setup:
All experiments were performed with 128GB RAM on an intel Xeon server with 3.5GHZ processors, consisting of 4 1080Ti GPUs.
Any of our models can easily fit into single 1080Ti GPUs – a multi
GPU setup was only used to speed up the computation.
All models are coded in Tensorflow 2.2 and we only use basic parallelism provided by the Tensorflow library to speed up computation.
Experimental Code: Code to support this work can be found at the following URL/repository: https://github.com/ago109/lifelong pc.git .
Theoretical Foundations and Technical Details of Sequential Neural Coding
In this appendix section, we present definitions and mathematical derivations, answers to key questions, and further detail core intuitions related to sequential neural coding (i.e., the S-NCN system presented in the main paper).
Definition Table:
In Table 1, we explain what each mathematical symbol/operation/abbreviation in the main paper represents.
Derivation of State & Weight Updates: As mentioned in the main paper, the S-NCN’s generative circuit minimizes an objective function known as total discrepancy (ToD) when it is presented with input stimuli ( x i , y i )) .
The ToD is formally:
L (Θ) =
[Equation]
(cid:88) ℓ
[Equation]
[-Furthermore, we let z 0 x and z 
 y denote clamped outputs. That is, if we want to predict the label y i given the features x i , we set z 0 x = x i and if we want to train, we set both z 0 x = x i and z 
 y = y i .-][Equation]
( |
[Equation]
z
[Equation]
||
[Equation]
[Equation]
[-e L = 0 , z 0 x = x , z 
-]
=
L
[Equation]
[-i on C o rr ec t i on zzz 
 𝝁 z 
-](cid:88) ℓ
[Equation]
(cid:88) j (cid:0)
z
[Equation]
[ j ]
[Equation]
z ℓµ [ j
[Equation]
(cid:1)
[Equation]
where z ℓ [ j ] means that we extract the j -th element of vector z ℓ (and we have simplified the expression by squaring the square root operator of the L2 norm, giving us a sum of squared dimensions).
Since all of the latent states of the generative circuit are continuous, the updates will follow the form of the exact gradient, i.e., differentiation (which would permit the use of gradient descent), to optimize the latent variables and the synaptic weight parameters.
Given this, the partial derivative of Equation
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
with respect to any layer of neural activities (or latent state)
z ℓ would be:
[Equation]
where we notice that the error neurons are derived directly from the ToD objective as well,
i
[Table]
e., e ℓ = ∂∂ L (Θ) ∂
z ℓµ =
z ℓ − z ℓµ (allowing us to write Equation 3 in terms of error neurons as in Equation 4).
Alternatively, by replacing the term ∂ z ℓ − 1 ∂ z ℓ with a learnable error matrix E ℓ instead, Equation 3 can be simplified to the following:
[Equation]
[Equation]
which is a stable derivative-free perturbation d ℓ (so long as the activation function ϕ ℓ () is monotonically increasing) to the neural activities (and as noted in [1], the dampening effect that the activation derivative ∂ϕ ℓ ( z ℓ ) ∂ z ℓ would have can be approximated with biologically-plausible dampening functions if needed).
The final update to the latent neural activities is then performed using a gradient-ascent like operation, i.e., z ℓ ( k ) =
f ℓ ( z ℓ ( k − 1) + β d ℓ ) (this was presented in the main paper).
Deriving the updates to the synaptic generative parameters is also done in a similar fashion as above, i.e., by taking the gradient of ToD with respect to W ℓ .
[Equation]
If we are using E ℓ feedback/error matrices (as we do in this paper), we can leverage a simple Hebbian update ∆ E ℓ = α (cid:16) ϕ ℓ +1 ( z ℓ +1 ) · ( e ℓ ) T (cid:17) [1] (which, if applied to E ℓ every time that Equation 7 is applied to W ℓ , allows E ℓ to converge to the approximate transpose of W ℓ ).
Much as was done for the states, synaptic weight matrices are updated via gradient ascent: W ℓ = W ℓ + λ ∆ W ℓ and
E ℓ = E ℓ + λ ∆ E ℓ ( λ is the learning rate/step size).
How would a model with symmetric connections behave?
A model without separate feedback connections (in contrast to the S-NCN we experiment that uses asymmetric forward/feedback weights) would behave quite similarly yet favorably offer a reduction in memory cost (one does not need to store separate feedback/error matrices in memory).
In other words, one could certainly swap out E ℓ with ( W ℓ ) T if this memory cost reduction was desired/necessary.
However, by utilizing separate learnable feedback synapses, the S-NCN in the form presented in this study resolves the weight transport problem, a well-known biological criticism of backprop where error/teaching information is carried backwards along the same synapses that were used to forward propagate information.
Interestingly enough, in preliminary experimentation, we found that using separate learnable feedback synapses improved the generative modeling/reconstruction ability of the generative cortex (particularly in the online case).
Although we will investigate this effect in future work, we note this change in generative performance did not really seem to impact the classification accuracy (arguably because discrimination is easier than generation).
Initializing Latent States:
In the S-NCN’s generative circuit, there are several layers of neural activities that are not clamped to data, e.g., z 1 , z 2 , ..., z L .
These activity vectors, as mentioned in the main paper, are initialized to zero vectors (i.e., z ℓ = 0 ) before they are updated/modified by the message passing that occurs over K steps of processing input stimuli ( x , y ).
While these initially zero vectors will eventually become non-zero vectors, particularly after a minimum of K =
L steps (for example: after k = 1 , z 1 will be non-zero given that the error neurons in layer 0 will be non-zero and thus the layer 1 state perturbation vector
d 1 will contain non-zero entries; after k = 2 , z 2 will contain non-zero values, and so on and so forth), it is entirely possible to randomly initialize these states with non-zero vectors (though it is recommended to keep the magnitude of the randomly chosen initial numbers relatively small).
We leave investigation of alternative initialization schemes for future work.
Relationship to Surrogate Gradients:
A particular line work that shares interesting relationships with the generative circuit of the S-NCN is that of surrogate gradients, such as decoupled neural interfaces (DNIs) [2, 3].
In essence, this class of methods aims to resolve one of backprop’s central issues – the update-locking problem (where updates to one layer’s synaptic parameters must for other layer’s updates to be computed as error/teaching information is backwards propagated down a serial feedback pathway).
The key module driving this class of methods is the introduction of a gradient predictor, which can be adapted/taught to approximate actual gradients as produced by backprop, ultimately, after training the predictor’s well enough removing the need for backprop later in training and permitting parallel, asynchronous updates to be made to deep, even recurrent, network architectures.
In contrast to surrogate gradient-based approaches, the S-NCN works to compute synaptic updates without resorting to predicting gradients given that its generative circuit is naturally layer-wise parallelizable.
In effect, each layer-wise prediction is made independently of the others (unlike the typical forward passes in modern-day deep networks) and the synaptic updates for each layer (both forward and error/feedback synapses) can be made without others having been computed/completed.
This opens the door to potential parallel asynchronous implementations of the
S-NCN that could drastically speed up its computation further.
In contrast to DNIs, the S-NCN’s generative cortex does not require training gradient predictors (DNIs typically require access to true gradients provided by backprop in order to train them properly) and, without incurring synthetic approximation errors (as in a fully-unlocked network using DNI, where now even the layer activities require additional modules to be trained to predict actual layer-wise activities) furthermore, resolves both update and forward locking problems.
Crucially, the S-NCN’s updates are biologically plausible – it only requires simple (multi-term)
Hebbian updates for the generative cortex and competitive
Hebbian updates for the basal ganglia.
General S-NCN System Process Intuition: From a high-level intuitive point-of-view, the S-NCN system described in the main paper is composed of two complementary neural circuits: 1) an interactive generative (cortical) circuit that learns to predict its input stimuli (pixel images and their respective labels), and 2) the functional neural basal ganglia (FNBG) which is a specialized circuit that learns to group pixel images into unique “task” contexts.
When presented with a sample or mini-batch at any point within the task stream being sampled, the S-NCN does the following:
The FNBG task selection model determines if the currently sampled data is coming from the same task that the S-NCN has currently been processing or if it comes from either new/different task.
(Note, not mentioned in the paper, the FNBG can also determine that the current data belongs to a task that it has previously seen by letting its set of neurons compete and determining if the winner has a very high dot product score – usually checked against a threshold).
Before letting the S-NCN generative circuit process/adapt to current data, if the FNBG determines that the data is coming from a novel task, it will create a new task context memory g ℓt for each M ℓ to subsequently drive the S-NCN as it processes current data.
Given the task context provided by the FNBG, the S-NCN generative circuit will then process the current data over a fixed stimulus window (or for K discrete time steps), adjusting its synapses so that it may better generate the input sensory sample better in the future as well as predict its label.
Note that the update to the S-NCN’s synapses will also trigger an update to the task context memory vectors as well as an update to the FNBG’s synapses (which are themselves adjusted through competitive Hebbian learning).
Task Shift Detection Intuition: To provide further intuition as to how the FNBG task selector operates when detecting the occurrence of a novel task, i e., “task shift detection”, we explain what the key equations presented in the main paper are doing:
The FNBG lets the S-NCN generative circuit continue as it normally would and have it make a prediction of both x j and y j .
Then, it extracts the label error neuron vector e
y from the generative model and computes its squared Euclidean norm ||
e
y ||
The FNBG maintains two particular scalar parameters µ L and σ 2 L , which are the mean and variance of the squared Euclidean norm of the S-NCN’s label error neuron activities.
The key Equations 7-9 in the main paper depict how the FNBG updates these mean µ L ( t ) and variance σ 2 L ( t ) parameters (based on an exponentially weighted moving mean and variance).
Once the FNBG updates these two statistical parameters, it then performs a check of its current new mean µ L ( t ) against the previous value ( t − 1 ) of its mean and variance parameters, i.e., µ L ( t − 1) and σ 2 L ( t − 1) .
This check is specifically (as in the main paper) the following: µ L ( t ) > µ L ( t − 1) + 2 (cid:112) σ 2 L ( t − 1) which says that, if the current mean of the label error neuron activity (at time t ) is greater than the sum of the previous mean (at time t − 1 ) plus two times the previous standard deviation, the S-NCN system has encountered a shift in task , which means that, if this inequality evaluates to true, the currently encountered data sample comes from either a novel task or a previously encountered task (but not from the current task).
If this task shift inequality evaluates to true, the FNBG will not allow the generative circuit to update its synapses and instead force it to recompute its prediction of the current data using a newly generated task context.
Otherwise, if the inequality evaluates to false, then the FNBG will let the generative circuit continue and update its synapses.
General Design Motivation/Benefits: The key motivations behind the S-NCN’s design are: 1) to develop a neurobiologically-inspired online approach to learn a generative model of p ( x , y )
(one motivated by perceptual cortical circuits), 2) to develop an online information routing model (one motivated/inspired by the basal ganglia) that suppresses/drives the neural activities in the generative model of item (1) depending on the task that it decides that the system is operating on – this crucially removes the need for user-provided task descriptors.
In addition, by design, the S-NCN: 1) exhibits no need for activation function derivatives, 2) exhibits no update-locking (i.e., it is layerwise parallelizable), 3) does not require/need a global feedback pathway to drive/facilitate credit assignment (i.e., side-stepping exploding/vanishing gradient problems), and 4) it resolves the weight transport problem by using asymmetric generative and error-correction synapses (those these could be tied/shared to reduce the S-NCN’s memory footprint if need be).
In contrast to many current lifelong learning approaches, the S-NCN differs in that it is a complementary system that focuses on the relationship between the basal ganglia (as an information router) and (generative/predictive) cortical regions.
This is what allows the S-NCN system to offer the advantage of internal, automatic task selection and task boundary detection, which few modern methods provide.
Although the S-NCN shares predictive processing’s (PP’s) iterative inference/learning (note that [4] focused on PP’s auto-association abilities and further note that [1] focused on PP’s generative/sampling abilities); its ability to combat forgetting comes from the interaction between the task selector and the generative circuit (the former drives lateral suppression/excitation in the latter).
Note that the S-NCN’s generative model could be useful for inducing a form of (internal) layer-wise generative replay (each layer could refresh itself in a sleep phase).
This we observe is additional, untapped potential for using the S-NCN’s learned directed generative model in a spirit similar to the backprop-based model of [5].
We argue that this self-induced form of replay could prove useful for longer task sequences with far more tasks.
Empirically, we note that asymmetric forward and backwards (error) weights were found to work best for NCN systems without activation derivatives in their state updates, yielding fewer inference steps.
However, since backwards weights converge to the approximate transpose of the forward ones, using symmetric weights would reduce memory usage (see earlier discussion/section “How would a model with symmetric connections behave?”).
We further remark that, although we do not explore them in this work, fixed, random backward synapses (as in feedback alignment) would work as well, which we found, in preliminary experimentation, reduced/harmed the S-NCN’s ACC by only a few points.
Relationship to Temporal Neural Coding: Prior related on temporal neural coding [6, 7] has investigated the design and development of predictive coding frameworks for handling the modeling of time-varying data points (such as frames in a video).
However, in contrast to the generative circuit in this work, these previous models were either not layer-wise parallelizable [6] or were created under a specialized recurrent formulation of PC for processing data over multiple epochs [7] (such as bouncing ball or digit videos).
These models would not be naturally resistant to forgetting like the S-NCN is, specifically given that the S-NCN is a complementary system where one circuit aids the other in learning task-sensitive/dependent sub-networks.
Furthermore, this earlier related work [6, 7] focused on unsupervised sequence modeling and did not investigate discriminative forms of learning (e.g., classification) where forgetting occurs, according to our experience, more quickly and its effects are stronger/more easily observed.
Note that in [7], although some improved memory retention across the three sequence modeling tasks was observed, forgetting over tasks was still apparent, and any improvement in memory retention could be be viewed more as a pleasant side-effect rather than the result of a mechanism specialized for safe-guarding against forgetting.
On Weight Update Modulation
As presented in the main paper, the synaptic weight updates for the generative circuit of the S-NCN applies modulation matrices to improve learning stability over time (specifically invoking a form of synaptic scaling – note that these modulation matrices are not meant to mitigate the occurrence of forgetting).
The modulation factors are (locally) computed as a function of the synapses as follows:
[Equation]
where W [: , j ] denotes the extraction of the j th column of W , max( (cid:98)
m ℓW ) returns the maximum scalar value of (cid:98) m ℓW , and γ s = 2 .
We note that the first two formulae collapse the forward matrix to a column vector of normalized multiplicative weighting factors and the third formula converts the column vector to a tiled matrix of the same shape as W ℓ .
The error weight modulation factor is computed in fashion similar to that of the forward weights:
[Equation]
where we observe that modulation factors are computed across the pre-synaptic dimension/side of either matrix W ℓ or E ℓ .
The multiplicative modulation terms come from the insight in neuroscience that synaptic scaling, driven by competition across synapses, serves as a global (negative) feedback mechanism for regulating the magnitude of synaptic adjustments [8, 9, 10].
From a practical standpoint, we found that using the above modulation/scaling factors meant we did not have to craft a synaptic normalization scheme (such as in sparse coding schemes, where the columns/rows of a synaptic matrix must be normalized such that they of unit length each time the matrix is updated).
We remark that the modulation factors we introduce could instead be adapted such that they are useful for mitigating forgetting instead, as has been done in other continual learning approaches [11, 12, 13, 14].
This could help to reduce the cost for growing out new task contexts each time a new task is encountered.
For example, one could adapt the modulation factor matrices to instead be conditioned on the output of the S-NCN’s functional neural basal ganglia (or another type of task selector circuit, such as one that mimics the cognitive control capabilities of the prefrontal cortex).
On Partial Pattern Completion
In the event that incomplete input x i is provided to the S-NCN, i.e., portions of x i are masked out by the variable m ∈ { 0 , 1 } J 0 × 1 , as mentioned in the main paper, we may infer the remaining portions of x i by using the relevant output error neurons of the S-NCN and treating the bottom sensory/input layer z 0 x as a partial latent state.
Specifically, we update the missing portions, i.e., 1 − m , of z 0 as:
[Equation]
(12) where e 0 x
= z 0 x − z 0 µ,x (error neuron signals related to x i ).
Parameter Optimization Setup and Baseline Details
S-NCN Optimization: For the S-NCN, we used SGD with a learning rate of λ = 0 .
01 (this rate was only minorly tuned on the validation set of the first task in preliminary investigation) and mini-batches each containing 10 samples.
Based on preliminary experiments, the S-NCN, in general, was found to be robust w.r.t.
such hyper-parameters.
However, the final meta-parameter values used, i.e., β = 0 .
05 , K = 10 , η g = 0 .
9 , η e = 0 .
01 , α = 0 .
98 , were obtained by conducting a grid search (using the validation sets to find best generalization).
This meant that we searched over the ranges: β = [0 . 01 , 0 . 1] , K = [5 , 30] , η g = [0 . 5 , 1 . 0] , η e = [0 . 005 , 0 . 25] , α = [0 . 5 , 1 . 0] .
Baseline Descriptions: The baselines include an MLP trained exclusively with backprop (Backprop), an MLP trained by backprop but regularized by drop-out (Backprop+DO), Elastic Weight
Consolidation (EWC) [15], EWC further regularized by drop-out (EWC-DO), the mean incremental moment matching method (IMM) or Mean-IMM [16], the Mode-IMM method [16], the Mean-IMM method combined with either DropTransfer (DT+Mean-IMM) or L2-transfer (L2-Mean-IMM) or both (L2+DT+Mean-IMM) [16], the Md-IMM method combined with either DropTransfer (DT+MdIMM) or L2-Transfer (L2+Md-IMM) or both (L2-DT-Mode-IMM) [16], and the state-of-the-art competitive model, hard attention to task (HAT) [17].
Furthermore, as mentioned in the main paper, we examine other methods including those based on replay/rehearsal: na ¨ ıve rehearsal with memory (NR+M), EWC, synaptic intelligence (SI), MAS [18], Lwf [19], GEM [20], ICarl [21], Lucir [22], and Mnemonics [23]. With respect to very modern baselines, we also include, in the main paper, results for the greedy sampler and dumb learner (GDumb) [24, 25], experience replay (ER) [26, 25], and average gradient episodic memory (A-GEM) [27, 25]. Baseline Meta-parameter Tuning:
For all baselines we take/use the hyper-parameters from each model/algorithm’s source work as a starting point and tuned each, using grid search, the batch size, learning rate, number of hidden units in each layer of the target MLP classifier, and optimizer choice. We tuned across the following ranges: learning rate range was [3 e − 5 , 0 . 1] , the optimizer choice was tuned across the discrete set [“SGD”, “momentum”, “Adam”, “AdamW”], the hidden layer size range was [128 , 512] , and number of layers range was [2 − 5] . For each baseline, we tuned hyper-parameters based on their accuracy on each task’s development set (as mentioned in the next sub-section). For
IMM, we used the same settings proposed in the original paper as a starting point [16].
However, we found that HAT [17] was quite sensitive to the choice of its two key hyper-parameters: 1) the stability parameter s max , and 2) the “compressibility” parameter c .
After extensive tuning, we used s max = 450 and c = 0 . For other baseline-specific hyperparameters, e.g., A-GEM has a gamma and soft-constraint meta-parameter, we used the best-practice values reported in the literature (as we found that these values worked well in general, even after some preliminary experimentation). Limitations (Expanded Discussion): Our model jointly predicts the target label and learns to generate the sensory input, further driven/modulated by a simple complementary neural system that mitigates neural cross-talk. The dual nature of our model/system helps to uncover distributed representations that facilitate robust learning and adaptation over sequences of tasks/datasets.
Even though this design scheme provides flexibility and seems to offer many advantages compared to other backprop-based models, it does come with several limitations. Mainly, finding the true posterior distribution over latent neural activities is harder than just learning a forward mapping between inputs and output targets and, notably, it can be expensive to find a good set of neural activity values as the problem complexity increases (notably the K hyper-parameter, which controls the amount of steps taken per data point/mini-batch by the S-NCN to iteratively infer a potential maximum a posterior estimate of its state variables). Currently, the S-NCN conducts inference and learning through a sort of expectation-maximization process and, fortunately, in the problems we studied, the value of K was fairly low (only 10 to 20 steps at most were needed to find useful state values per sample/minibatch). However, for more complex data types, such as natural images with multiple objects and even background scenery, the value of K will quite likely need to be much higher, increasing
[Table] MNIST FMNIST NotMNIST
Model
Configuration Configuration
Configuration
EWC lr =
1e-
, SGD
NH = 1024, NL =lr = 1e-4, SGD+M
NH = 512, NL =lr
=
1e-3, SGD
NH = 1024, NL =
VCL lr = 2e-4, Adam
NH = 512, NL =lr = 2e-4, SGD+M
NH = 512, NL =lr = 2e-4, Adam
NH = 512, NL =
IMM lr
= 2e-3, SGD+M
NH = 512, NL =lr
= 1e-5, SGD
NH = 512, NL =lr = 1e-4, SGD+M
NH = 512, NL =
HAT lr = 1e-4, SGD
NH = 512, NL =lr = 2e-4, SGD NH = 512, NL =lr = 2e-4, SGD+M
NH = 512, NL =
A-GEM lr = 1e-4, SGD+M
NH = 512, NL =lr
= 2e-5, SGD
NH = 512, NL =lr = 2e-4, SGD+M
NH = 512, NL =
ER lr = 2e-4, Adam
NH = 512, NL =lr = 1e-4, AdamW NH = 512, NL =lr = 1e-4, AdamW
NH = 512, NL =
EWC lr = 1e-3, SGD+M
NH = 1024, NL =lr
= 1e-3, SGD+M
NH = 1024, NL =lr = 1e-3, SGD
NH = 512, NL =
NR+M lr = 1e-4, SGD+M
NH = 512, NL =lr
=
1e-3, SGD+M
NH = 512, NL =lr = 1e-4, SGD
NH = 512, NL =
SI lr = 1e-3, SGD
NH = 512, NL =lr
=
1e-3, SGD+M
NH = 512, NL =lr
=
1e-3, SGD
NH = 512, NL =
MAS lr = 1e-4, Adam
NH = 512, NL =lr = 1e-4, Adam NH = 512, NL =lr = 2e-4, Adam
NH = 512, NL =
Lwf lr = 1e-3, SGD
NH = 1024, NL =lr = 1e-3, SGD NH = 1024, NL =lr = 1e-3, SGD
NH = 1024, NL =
ICarl lr = 1e-3, SGD+M
NH = 1024, NL =lr
= 1e-3, SGD+M
NH = 1000, NL =lr = 1e-4, SGD+M
NH = 1000, NL =
Lucir lr = 1e-4, Adam
NH = 512, NL =lr = 2e-4, AdamW
NH = 512, NL =lr
= 2e-5, AdamW
NH = 512, NL =
GDumb lr = 2e-4, AdamW
NH = 512, NL =lr = 2e-5, Adam
NH = 512, NL =lr = 2e-4, AdamW
NH =512 , NL =
Mnem lr = 1e-4, Adam
NH = 512, NL =lr = 2e-5, Adam
NH = 500, NL =lr = 2e-4, Adam
NH = 512, NL =
S-NCN lr = 0.0105, SGD
NH = 500 , NL =lr = 0.01, SGD
NH = 500 , NL =lr = 0.011, SGD
NH = 500 , NL =
the computation time needed to conduct online inference.
This drawback could be mitigated by integrating amortized inference, e.g., predictive sparse decomposition [28], and by designing custom software/hardware implementations to exploit the S-NCN’s layer-wise parallelism.
Furthermore, the fact that the S-NCN (in its current form/implementation in this study) must solve a dual optimization problem that entails jointly learning to predict the target label and generate the sensory input (image) might compromise the model’s overall accuracy when tested on large-scale images.
It is often an easier problem to directly learn a conditional mapping between the input and label as opposed to learning a full generative model as the S-NCN does [29].
Future work should explore adapting the S-NCN to only learn a conditional mapping as opposed to a full joint distribution over inputs and labels as well as potential mechanisms for pre-training the generative side of the system (which would allow freezing of the synaptic weights for generation and only require updating discriminative parameters – this could potentially reduce the value of K even for more complex sensory inputs).
Another drawback, yet also simultaneously a strength, is the fact that the S-NCN is attempting to optimize (online) total discrepancy as opposed to a single, global surface loss.
While total discrepancy is one important key to breaking free of backprop and its limitations it also creates a more challenging optimization problem in general, i.e., the neural system must now not only match the values created by data but also ensure its internal activities and its local predictions of each of them are aligned.
While the overall complementary system mitigates catastrophic interference (or the neural cross-talk that would trigger the loss/deletion of previously acquired knowledge), this primarily affects measurements of backward transfer (BWT) but could potentially damage the model’s per-task performance, i.e., the main diagonal of its task matrix.
Since we do not impose any strong distributional assumptions over the latent activities (such as a clean Gaussian prior as is often done in variational autoencoders), if the S-NCN’s estimated value of the latent activities are far from the true posterior, then the S-NCN might produce sub-optimal performance, especially for complex problems.
Even though all continual learning systems suffer from this issue (especially most modern-day continual learning ANNs), our model’s dual optimization nature could experience this problem more frequently.
We believe that integrating memory-aware retrieval from synapses, a mechanism guided by (a brain-like form of) replay, could help direct the system to be closer to the true posterior by avoiding bad local minima when learning continuously.
Additionally, with respect to our proposed task selection mechanism (the functional neural basal ganglia circuit), one notable drawback is that a small refractory period is imposed in order to ensure that enough data is accumulated from the stream to update the competitive task selector’s weights.
This would be an issue for task streams that constantly introduce tasks with fewer samples than the pre-set refractory period.
A subject of our future work is to improve the power /adaptability of the task selection model in the face of more volatile task sequence streams.
Another limitation is that the S-NCN is, in effect, a dynamically-expanding architecture: there is an overhead for the task-context memory – one new task context vector would need to be generated/grown for each new (disjoint) task is encountered.
While this required parameter growth/generation is not as high as other dynamically-expanding architecture approaches (such as progressive networks [30] or dynamicallyexpanding networks [31], where many new parameters must be created per task).
While the inclusion of relatively few, additional context vector parameters is more desirable, requiring the growing out of parameters at a rate far less than methods such as [30, 31].
To mitigate the cost that even the
S-NCN imposes
, we remark that the S-NCN’s generative cortex could be adapted to induce its own form of layer-wise replay as one alternative, similar to [5], or that another circuit could be designed to potentially learn how to compress these task contexts by reducing redundancy exploiting overlap/redundancy between contexts (serving as a form of efficient long-term memory).
Finally, a more obvious drawback is that the S-NCN’s error synapses also increase the memory footprint of the overall model.
It would be advisable, when using/applying a model like the S-NCN on other continual learning problems, to select the number of hidden layers and number of neurons in each layer based on model capacity, i.e., compute the total number of (generative and error) synapses that would result from making the neural structure more complex or deeper.
A more long-term, promising means of mitigating the increased memory footprint would be to design error units further inspired by actual neurons – instead of assuming a one-to-one mapping (one error neuron per state neuron), design small pools of neurons that are responsible for computing the mismatch activities for large groups of state neurons.
This is a key solution to investigate in future work.
Creating Task Orderings #1 and #
To create our sequential learning benchmarks, we utilize the MNIST, Fashion MNIST, and Google
Draw datasets to create various sets of “subtasks”, or rather, classification problems that involve different classes of the original set of each full dataset.
In this paper, we create a 6 task sequence, { T 1 , T 2 , T 3 , T 4 , T 5 , T 6 } , from these datasets, where two tasks are generated from each specific dataset.
To create the task splits, we create data subsets based on minimizing the amount of knowledge transfer across data splits, specifically by examining the amount of stroke overlap in the images across classes, yielding a challenging problem.
For equal number of classes, the splits we created were: MNIST set #1, M1 = { 0 , 8 , 3 , 5 , 2 } , MNIST set #2, M2 = { 1 , 4 , 6 , 7 , 9 } , Fashion MNIST set #1, FM1 = { top, trouser, pullover, dress, coat } , Fashion MNIST set #2, FM2 = { sandal, shirt, sneaker, bag, ankle boot } , Google Draw set #1, GD1 = { objects that were car or bike variants } , and Google Draw set #2, GD2 = { objects belong to variants of airplanes or submarines } .
For a task sequence, we create two scenarios: 1) where number of classes are equal for all tasks (i.e., 5 classes in our setup), and 2) where number of classes are unequal (number of classes per task was chosen randomly, omitting the number 5 as an option).
In our experiments, we investigate two task orderings (Ordering #1 and
Ordering #2).
We compute the color index similarity [32] between every pair of tasks (as a proxy for task similarity) and randomly chose Orderings # 1 and # 2 so that the color similarity between adjacent tasks was higher for Ordering #1 (“High Color Sim.”
for high color similarity) than for
Ordering #2 (“Low Color Sim.”
for low color similarity), hence task transfer should be easier for
Ordering #1 than #2.
These particular orderings could be considered to be “harder“ and “easier” task orderings, respectively, since it is possible that a the difference in color-index would make it easier to differentiate the tasks (the diversity of inputs from the first few tasks might even improve the performance, as it would be in the case of Ordering # 2).
We can see this reflected by the fact that all models (the S-NCN and the baselines) perform a bit better in general on Ordering # 2 (low color similarity or “easier” ordering) than on Ordering # 1 (high color similarity or “harder” ordering).
Expanded Results for Task Orderings # 1 and #
Metrics for Quantifying Memory Retention: The formulas for ACC and BWT are:
[Equation]
In addition, we propose two additional, complementary metrics , with the motivation that these metrics examine aspects of forgetting and capacity not clearly captured by ACC or BWT.
Our two measures, True BWT (TBWT) and Cumulative BWT for task T t (CBWT(t)), are defined as follows: [Equation]
[Equation]
(15) where G i,i is the performance of an independent classifier trained on task
i (in our experiments, this was a full capacity MLP trained via backprop).
TBWT relates the degradation in prior task performance by replacing the diagonal of task matrix R with a “gold standard”, which is the performance of a model that, in isolation, is able to allocate its full capacity to a particular task.
CBWT(t) is a task-specific metric, where we instead examine a particular column of R , and measure the total amount of forgetting throughout the sequential learning process, instead of simply examining the final performance at the end (bottom row of R ) as BWT can only do.
CBWT(t) would punish models that suffer large dips in performance in the middle of learning (but not necessarily at the end), and would be better suited for characterizing forgetting in stream settings than BWT.
Discussion:
Results are reported in Tables 3 and 4 (an expanded version of the original one in the main paper).
Each simulation was run 10 times, each trial using a unique seed for pseudorandom number generation, we report both mean and standard deviation.
As we observe in our experimental results, we see that all of our S-NCN models exhibit improved memory retention over simple baselines, such as backprop, and more notably, EWC.
However, we see that incorporating task-driven lateral inhibition in facilitating gradual forgetting as opposed to catastrophic forgetting, as evidenced by the very competitive performance of both Lat1-S-NCN and Lat2-S-NCNs, with
Lat1-S-NCN outperforming all baselines consistently
, in terms of both ACC and BWT.
This result is robust across both task sequences and equal/ unequal class settings.
It is further important to note that the meta-parameter settings used for the various S-NCNs were only tweaked minorly with the same values across all of the settings/scenarios.
The observation that lateral inhibition improves the neural computation of our interactive network further corroborates the result of [33], though it focused on models trained via contrastive Hebbian learning.
Upon examination of Table 3, in terms of TBWT and CBWT(1) 1 , the proposed lateral S-NCNs still outperform the baselines.
The key difference is that we see that the lateral S-NCNs actually do retain prior information throughout learning and do not simply just recover it at the end.
T a b l e 3 :
A lt e r n a ti v e m e t r i c
s r e po r t e d f o r t a s k
s e qu e n ce o r d e
r
i ng s # a nd #
( h i gh e r v a l u e s a r e b e tt e r) .
O r d er i n g #1 (
H i g h
C
o l o r S
i m . )
O r d er i n g # 2 (
L o w
C
o l o r S
i m . )
E qu a l
U n e qu a l
E qu a l
U n e qu a l
TB
W T
C B
W T
TB
W T
C B
W T
TB
W T
C B
W T
TB
W T
C B
W T
B ac kp r op - 0 .
E W
C - 0 .
M d -I
MM - 0 .
D T +
M d -I
MM - 0 .
L 2 +
D T +
M d -I
MM - 0 .
HA T - 0 .
S - N
C N − 0 .
− 0 . − 0 .
− 0 . − 0 . − 0 .
− 0 . − 0 .
L a t-
S - N
C N − 0 .
− 0 . − 0 .
− 0 . − 0 . − 0 .
− 0 . − 0 .
L a t-
S - N
C N − 0 .
− 0 . − 0 .
− 0 . − 0 . − 0 .
− 0 . − 0 .
[Table]
To start
[Table] we describe the three forms of the lateral competition we designed for the S-NCN. They were specifically as follows: f ℓ ( z ℓ , g ℓt ) =
I · z ℓ , which means that the lateral inhibitory matrix is fixed to a diagonal matrix I and forces the model to ignore the task embedding (in the Appendix, we denote this as “NoLat-S-NCN”), 2. f ℓ ( z ℓ , g ℓt ) =
(cid:0) I ⊙ V ℓ (cid:1) · z ℓ , where the matrix V ℓ = BK WTA ( g ℓt , K ) · BK WTA (( g ℓt ) T , K ) and BK WTA ( v , K ) is the binarized K winners-take-all function, yielding a binary vector with 1 at the index of each of the K winning units (in the Appendix, we denote this as “Lat1-S-NCN”), 3.
f ℓ ( z ℓ , g ℓt ) = max (cid:0) 0 , z ℓ − (cid:0) ( V ℓ ⊙ ( g ℓt · ( g ℓt ) T )) · z ℓ (cid:1)(cid:1) , where: V ℓi,j = { α , if i ̸ = j , else, 0 } (in the Appendix, we denote this as “Lat2-S-NCN” In the last two forms of the competition function, we see that lateral inhibition is a function of an evolving context vector g ℓ , triggered by the presence of the task signal t i . The above competition functions correspond to different designs of lateral suppression patterns: (1) corresponds to no lateral suppression, (2) corresponds to shutting off neurons that are not task-relevant driven by a task selector, (3) corresponds to a task-driven, real-valued lateral matrix that scales neural activities.
T a b l e 5 :
G e n e r a li za ti on m e t r i c s ( 10 t r i a l s )
f o r S p lit
M N I
S T ,
S p lit
F a s h i on
M N I
S T ( F M
N I S
T ) a nd
N o tM N I
S T b e n c h m a r k s .
N o t e f o r
I
MM , w e e m p l oy t h e b e s t p e rf o r m i ng v a r i a n t , L 2 + D
T +
M d -I
MM .
A bov e d a s h e d li n e a r e
s i ng l e - h ea d m od e l s & b e l o w a r e m u lti - h
ea d m o d e l s ( e x ce p t t h e
S - N
C N ) .
M N I
S T
F M
N I S
T
N o t
M N I
S T
ACC
B W
T
ACC
B W
T
ACC
B W
T
E W
C 0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
V C L 0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
I
MM 0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
HA T 0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
G E
M 0 .
± 0 .
+ 0 .
± 0 . ± 0 .
+ 0 .
± 0 . ± 0 .
− 0 .
± 0 .
DG
R 0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
R t f 0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
E W
C 0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
N R +
M0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
N R +
M0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
S I 0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
M A
S 0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
L w f 0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
I C a r l 0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
L u c i r 0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
B i c 0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
M n e m 0 .
± 0 .
− 0 .
± 0 . ± 0 .
+ 0 .
± 0 . ± 0 .
− 0 .
± 0 .
S - N
C N 0 .
± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 . ± 0 .
− 0 .
± 0 .
On the custom benchmarks (including both orderings #1 and #2), we evaluate four variations of the S-NCN: 1) an S-NCN, with hyperbolic tangent activations and no context-dependent lateral inhibition (S-NCN), 2) an S-NCN with sparsity created by the use of a linear rectifier activation function and no lateral inhibition (S-NCN-relu), 3) an S-NCN with the second variant of our proposed lateral inhibition (Lat1-S-NCN), and 4) an S-NCN with the third variant of our proposed lateral inhibition (Lat2-S-NCN) (All variants used: β = 0 . 05 , K = 10 , η g = 0 .
9 , η e = 0 . 01 , α = 0 . 98 ). The last two S-NCN models (“Lat1-S-NCN” and “Lat2-S-NCN”) were driven by the FNBG model that we described in Section 3.3 (“The Neural Task Selection Model”). We compare our S-NCN model (specifically, our best-performing one from the experiment in the last section – the Lat1-S-NCN) to the following approaches that have been proposed over the years to combat catastrophic interference: Naive rehearsal with memory (NR+M1 & NR+M2), EWC, synaptic intelligence (SI), MAS [18], Lwf [19], GEM [20], DGR [34], Rtf [35], ICarl [21], Lucir [22], Bic [36], and Mnemonics [23].
In Table 5, we report model ACC and BWT, averaged over 10 trials, offering not only an extensive and comprehensive comparison of competitive methods, but also demonstrating that, for all three benchmarks, our proposed S-NCN outperforms all of them , demonstrating the power afforded by challenging the very assumptions underlying modern-day artificial neural systems and designing models with stronger grounding in neuro-biology. Furthermore, it is astounding to see that the S-NCN outperforms/matches performance with not only the single-head models but also with the multi-head models (except GEM), which enjoy an easier version of the problem since they are permitted to utilize a different classifier per task. Finally, it is critical to note that our proposed SNCN is a complementary neural system that learns without explicitly-provided task descriptors , i.e., in other words, the model learns to compose its own task contexts in a data-dependent manner. Note that, in the online setting, split FashionMNIST appears to be simpler, given that the S-NCN readily learns to generate rough forms of bigger objects (shirt/shoes/pants) and associate these with labels early whereas digits/characters are a bit more intricate and take longer to learn.
See Figure 1 (below) for a visual depiction of the S-NCN’s task accuracy over time across tasks for all three continual learning benchmarks investigated in this paper.
T ask 1 : F r ac .
C
o rr ec t
T ask 2 : F r ac .
C
o rr ec t
T ask 3 : F r ac .
C
o rr ec t
T ask 4 : F r ac .
C
o rr ec t
T ask 5 : F r ac .
C
o rr ec t
Training Time
Training Time
Training
[Figure]
Training Time
Training Time
Training
[Figure]
Training Time
Training Time
Training
[Figure]
Training Time
Training Time
Training
[Figure]
Training Time
Training Time
Training
[Figure]