Biologically-plausible backpropagation through arbitrary timespans via local neuromodulators
Abstract
The spectacular successes of recurrent neural network models where key parameters are adjusted via backpropagation-based gradient descent have inspired much thought as to how biological neuronal networks might solve the corresponding synaptic credit assignment problem [1, 2, 3].
There is so far little agreement, however, as to how biological networks could implement the necessary backpropagation through time, given widely recognized constraints of biological synaptic network signaling architectures.
Here, we propose that extra-synaptic diffusion of local neuromodulators such as neuropeptides may afford an effective mode of backpropagation lying within the bounds of biological plausibility.
Going beyond existing temporal truncation-based gradient approximations [4, 5, 6], our approximate gradient-based update rule, ModProp, propagates credit information through arbitrary time steps.
ModProp suggests that modulatory signals can act on receiving cells by convolving their eligibility traces via causal, time-invariant and synapse-type-speciﬁc ﬁlter taps.
Our mathematical analysis of ModProp learning, together with simulation results on benchmark temporal tasks, demonstrate the advantage of ModProp over existing biologically-plausible temporal credit assignment rules.
These results suggest a potential neuronal mechanism for signaling credit information related to recurrent interactions over a longer time horizon.
Finally, we derive an in-silico implementation of ModProp that could serve as a low-complexity and causal alternative to backpropagation through time.
Introduction
Recurrent connectivity is a hallmark of neuronal circuits.
While this feature enables rich and ﬂexible computation, mechanisms enabling efﬁcient task learning in large circuits remain a central problem in neuroscience and artiﬁcial intelligence research.
Fundamentally, the problem stems from the fact that potentially all history of all neurons, including synaptically far away ones, can affect neuronal activity and contribute to task output.
Motivated by the success of gradient descent learning, several biological learning models approximate the exact gradient in recurrent neural networks using known biological processes and gain insights into computational principles of how the brain might learn [4, 5, 6].
By ignoring dependencies beyond a few recurrent steps — thus severely truncating the gradient computational graph — these existing models succeed in representing their approximate gradientbased update rule as a combination of terms that resemble known synaptic physiological processes: "eligibility trace", which maintains a fading memory of coincidental activation of presynaptic and Systems ( postsynaptic neurons [7, 8, 9, 10, 11], combined with a third “modulatory” factor — top-down learning or “reward” signals [12, 13, 14, 15], for which dopamine is a prominent candidate [16].
Despite the impressive performance of such approximations on a variety of tasks, truncating relevant credit information results in a signiﬁcant performance gap compared to algorithms using exact gradient information; backpropagation-through-time (BPTT) and real time recurrent learning (RTRL)
How might neural circuits account for long-term recurrent interactions in order to assign credit (or blame) to neuronal ﬁring that happened arbitrary steps before the presentation of reward?
Given the rich repertoire of dynamical and signaling elements in the brain, one avenue could be to examine biological processes that have been underexplored in existing models.
Dopamine — whose cellular actions are exerted by activation of G protein-coupled receptors (GPCRs), which can greatly impact
STDP — is not the only neuromodulator involved in learning [18, 19, 20, 21].
More recently, transcriptomic studies have uncovered strong evidence for many other neuromodulatory pathways throughout the brain that also act via the activation of GPCRs, leading to similar downstream actions as dopamine [22, 23].
This suggests that, similar to dopamine, they could also play a role in shaping synaptic credit assignment.
A conspicuous family within these pathways is neuropeptide signaling because peptidergic genes are densely and abundantly expressed in the forebrains of divergent species, including the human, in a cell-type-speciﬁc fashion [24], suggesting widespread interaction between synaptic and peptidergic modulatory networks for synaptic credit assignment [25, 6].
Moreover, intra-cortical expression allows neuropeptides to potentially carry information local to the cortical network, cell type speciﬁcity enables sculpted signals for different recipient cells, and their diffusive nature could enable communication between neurons that are not synaptically connected.
Finally, peptidergic signals have time-scales much longer than the time scales for axonal propagation of action potentials or synaptic delays [26].
Taken together, these properties make cell-type-speciﬁc local neuromodulation seem promising for propagating credit signal over multiple recurrent steps.
Developing explicit computational principles of how these local modulatory elements could propagate credit signal over arbitrary recurrent steps could advance our understanding of biological learning and may inspire more efﬁcient low complexity bio-inspired learning rules.
Motivated by the question above as well as shortcomings of gradient approximations based on severe temporal truncations, we investigate how biological credit signals could be propagated through arbitrary recurrent steps via widespread cell-type-speciﬁc neuropeptidergic signaling [23, 25].
While
Ref.
[6] recently introduced a framework exploiting properties of neuropeptidergic signaling for temporal credit assignment, similar to [4] and [5], their approach performs severe temporal truncation of the error gradient and does not consider credit propagation beyond disynaptic connections.
Our main contributions are summarized as follows:
• We derive a theory that provides mechanism and intuition for the effectiveness of synapsetype-speciﬁc modulatory backpropagation (through time) weights (Theorem 1).
• We develop a model predicting how modulatory signaling could be the basis for biological backpropagation through arbitrary time spans (Figures 1 and 2).
Unlike temporal truncationbased approximations [4, 5, 6], our model enables each neuron to receive ﬁltered (rather than precise) credit signals regarding its contribution to the outcome via neuromodulation.
• We demonstrate the effectiveness of modulatory signaling via synapse-type-speciﬁc, rather than synapse-speciﬁc, modulatory weights on learning tasks that involve temporal credit assignment (Figure 4).
In particular, we demonstrate an online learning setting where weights are updated causally and in real-time (Figure 5).
• We also derive a low-complexity in silico implementation of our algorithm suitable for online learning (Proposition 1).
Related works
Neuromodulatory factors in synaptic plasticity: One of the most fundamental learning rules,
Hebbian plasticity, attributes lasting change in synaptic strength and memory formation to correlations of spike timing between particular presynaptic and postsynaptic neurons [27, 28].
However, multiple experimental and theoretical investigations now indicate that the Hebbian rule alone is insufﬁcient.
First, there have been numerous suggestions that some persistent “eligibility trace” must exist to bridge the temporal gap between correlated ﬁrings at millisecond timescales and behavioral timescales lasting seconds [7, 8, 10, 9, 29, 30].
Moreover, impacts of correlated spike timing must be augmented by one or more additional modulatory factors to steer weight updates toward desired outcome [31, 16, 32, 15, 33, 7, 1, 34, 35, 36, 8, 37, 38].
This is commonly known as learning with three factors.
A prominent candidate for such a modulatory factor is dopamine [13].
Dopamine inﬂuences receiving neurons via activation of G protein-coupled receptors (GPCRs), which can regulate membrane excitability and key parameters of synaptic plasticity rules.
Besides dopamine, recent transcriptomic evidence has uncovered genes encoding a plethora of other neuromodulatory pathways throughout the brain, including neuropeptide signaling genes that are abundantly expressed in the forebrains of tetrapods, including the human [23, 24].
Like dopamine, their cellular actions are exerted by the activation of G protein-coupled receptors (GPCRs), which can persistently modulate Hebbian synaptic plasticity [15, 39, 40].
This suggests that they too, could play a role in shaping cortical learning and synaptic credit assignment.
Moreover, nearly all neurons express one or more neuropeptide signaling gene [23], which suggests a dense interplay between synaptic and peptidergic modulatory networks to shape synaptic credit assignment [25].
Approximate gradient descent learning: Standard algorithms for gradient descent learning in recurrent neural networks (RNNs), real time recurrent learning (RTRL) and backpropagation through time (BPTT), are not biologically-plausible [41] and have vast computational and storage demands [42].
However, multiple studies have shown that learning algorithms that approximate the gradient, while mitigating some of the problems of exact gradient computation, can lead to satisfactory learning outcomes [41, 43, 44].
In feedforward networks, plenty of biologically-plausible learning rules have been proposed and demonstrated impressive performance that rival backpropagation on many different tasks [33, 41, 14, 44, 34, 45, 46, 47, 48, 49].
For efﬁcient online learning in RNNs, approximations to RTRL have been proposed [17, 50, 51, 52, 4, 53, 54].
For instance, a recent inﬂuential study [5] conceived how the three-factor learning framework could approximate the gradient.
Among the biologically-plausible proposals [4, 5, 6], approximations have mainly been based on temporal truncations of the gradient computation graph and because of that, their ability to learn dependencies across arbitrary recurrent steps has been limited.
Lastly, non-truncation based approximations have been proposed outside of bio-plausible research [50, 51].
For further discussions on related algorithms, please refer to Appendix C.
Results 3.1 Learning rule overview
Biological plausibility is a guiding principle in developing our model.
Hence, the model choices are based either on established constraints of neurobiology such as the locality of synaptic transmission, causality, and Dale’s Law, or on emerging evidence from large-scale datasets, such as the cell-typespeciﬁcity of certain neuromodulators [23] or the hierarchical organization of cell types [22].
We consider a discrete-time rate-based RNN similar to the form in [56] with observable states, i.e. ﬁring rates, as z t at time t , and the corresponding internal states as s t .
W denotes the recurrent weight matrix with ( pq ) th entry W pq representing the synaptic connection strength between presynaptic neuron q and postsynaptic neuron p .
See Supp.
Section A for the neuron model.
Gradient descent learning in RNNs : RNNs are typically trained by gradient descent learning on the task error (or negative reward) E .
However, the two equivalent factorizations of error gradient in
RNNs, BPTT and RTRL, both involve nonlocal information that is inaccessible to neural circuits.
This is due to recurrent connectivity: a synaptic weight W pq can affect loss E through many other neurons in the other network in addition to its pre- and postsynaptic neurons.
To see this more concretely, the following is the exact gradient at time t using RTRL factorization, on which we base our approximation for online learning: d E d W pq
[Equation]
depends on all weights W jl
[Equation]
[Figure] where ∂ denotes direct dependency and d accounts for all (direct and indirect) dependencies, following the notation in [5].
(See Appendix A.2 for further details on the notation.)
While ∂E∂s j,t , which considers only the direct contribution of the internal state of neuron j at time t to the loss, is easy to compute, the factor d s j,t d W pq is a memory trace of all inter-cellular dependencies and requires O ( N 3 ) memory and O ( N 4 ) computations.
This makes RTRL expensive to implement for large networks.
Moreover, the last factor d s j,t d W pq poses a serious problem for biological plausibility: the nonlocal terms in Eq.
2 requires knowledge of all other weights in the network to update the weight W pq .
Existing biologically-plausible solutions to this problem apply severe truncations: references [4] and [5] completely ignore the third nonlocal term (Figure 2), whereas reference [6] restores terms within one recurrent step through local modulatory signaling but truncates further terms.
ModProp approximation overview: Derivation of gradient descent-based weight updates involving neuromodulatory signaling from Eq.
1 suggests two approximations to Eq.
S4 and 2 to move beyond severe truncations of the exact gradient while retaining biological plausibility.
Approximationreplaces the activation derivative with a constant:
[Equation]
where µ represents average activity of neurons in a ReLU network.
This approximation assumes stationarity in neuron activity and uncorrelatedness of such activity with the weight, as explained in derivations leading up to Appendix Eq.
S14 and Eq.
S21 (Methods in Appendix A).
While neuronal activity and synaptic weights are not necessarily uncorrelated, considering that a single neuron may have thousands of synaptic partners, the activity of the neuron or its time derivative is typically weakly correlated to any one synaptic weight.
This also indicates that such approximation might work better for large networks with many neurons and synaptic partners for each neuron, as the case for biological neural networks.
We take advantage of this phenomenon in our model and ignore these weak correlations.
This approximation enables the ﬁlter taps (Eq. 6) to be time-invariant , a p q w pq p q w pq p q w pq j
A ii A i Δw pq depends on activity of cells q, p, j, i, k, j, ...
k l ... i j k l i j k l i
A iii
A iii Backpropagation through time Three-factor rule
ModProp p q w pq ...
j k l i
Hebbian
No feedback
Feedback on direct influence
Feedback on direct and indirect influence
B ii B
i
[-t+
 z-]B iii
Spatial view
[Figure]
[Figure]
E z t z t+ z t+2 z t- ... ..
[Figure]
E z t [Figure]
+
[Figure]
E z t z t+ z t+2 z t-
[Figure]
ingredients:
[Figure]
[Figure]
of indirectly view propagates the precise intercellular dependencies in an acausal manner. Bii) Three-factor learning rule neglects all the intercellular dependencies in the temporal propagation of the credit signal.
Biii) ModProp approximates such spatiotemporal dependencies through local neuromodulatory signals (Eq. 6).
ModProp approximates the exact gradient by assuming similar connectivity among cells of the same type, and ﬁltering the indirect effects on loss from neurons that are potentially many synapses away.
Figure S1 provides a summary of approximations made by ModProp. property likely required for biological plausibility. We also deﬁne S , the arbitrary number of credit propagation steps, and it will become clear later that this corresponds to the number of modulatory ﬁlter taps.
With this, the estimated gradient becomes:
[Equation]
where e pq,t ≈ dz p,t dW pq ( e pq,t is deﬁned precisely in Appendix Eq.
S24) can be interpreted as a persistent
Hebbian “eligibility trace” [10, 9, 8] that keeps a fading memory of past coincidental pre- and postsynaptic activity [5].
Here, ( W s ) jp represents the ( jp ) th entry of W s , W raised to the power of s , and ( W 1 ) jp
= W jp .
Details of the derivation can be found in Appendix A.
It is important to note that Approximation 1 is applied only to the nonlocal gradient terms, i.e. exact activation derivative is used during the computation of the eligibility trace that is local to the preand postsynaptic neurons.
Also, we treat µ as a hyperparameter in our simulations and explain how this is tuned in Appendix F. One should note that tuning the hyperparameter µ properly is important, because the estimated gradient can explode when µ is too large and ModProp approaches the three-factor rule when µ is too small.
Future work involves improving ModProp with an adaptive µ for better numerical stability and accuracy.
Approximation 2 replaces synapse-speciﬁc feedback weights with type-speciﬁc weights:
[Equation]
Here, cell j belongs to type α , cell p is of type β and C denotes the set of cell types.
(e.g.,
W αβ =
E j ∈ α,p ∈ β
[ W jp ] , α, β ∈ C .)
This approximation is due to the type-speciﬁc nature of modulatory channels [23].
We call these modulatory weights synapse-type-speciﬁc (as opposed to cell-type-speciﬁc) to emphasize connectivity-based grouping.
Details of how these modulatory weights were obtained can be found in Appendix A.
ModProp: ﬁltering credit signals via local neuromodulation: Substituting Approximation 1 and
Approximation 2 into Eq.
S4 and 2 leads to the ModProp update:
[Equation]
where L and e denote top-down learning signal and eligibility trace, respectively.
We postulate that F αβ represents type-speciﬁc ﬁlter taps of GPCRs expressed by cells of type β to precursors secreted by cells of type α ; ∗ is the convolution operation with S as the number of ﬁlter taps.
Note that the matrix powers ( W s ) αβ appearing in the values of different ﬁlter taps F αβ,s could be genetically pre-determined as part of the cell identity and optimized over evolutionary time scales.
(Appendix Figure S2 shows successful learning using ﬁxed modulatory weights.)
Details of a biological interpretation of Eq.
6 can be found in Appendix D.1.
Brieﬂy, secretion of top-down (TD) learning signals can selectively activate a biochemical process at the post-synaptic neuron, which can then act as a temporal ﬁlter on the eligibility trace.
Observing that neurons of the same type demonstrate consistent properties across a range of features (e.g. connectivity, physiology, gene expression) [23, 57], we use two cell types with consistent wiring, neuromodulation, and type of synaptic action (excitatory/inhibitory) in our relatively simple models.
In summary, we propose a synaptic weight update rule, where the eligibility trace is compounded not only with top-down learning signals – as in modern biologically-plausible learning rules [7, 8] – but also with local modulatory pathways through convolution (Figure 1).
This modulatory mechanism allows the propagation of credit signal through arbitrary number of recurrent steps.
3.2 Properties of ModProp
Remark.
The biologically plausible implementation (Eq. 6) of ModProp complexity scales as
O ( SN 2 ) per time step t .
Here, N and S denote the number of recurrent units and number of ﬁlter taps.
As seen in Eq.
4, the number of ﬁlter taps corresponds to the number of recurrent steps for which the credit information is propagated.
We also present an alternative implementation with potentially lower cost later in
Proposition 1.
Details for cost analysis and biological interpretation can be found in Appendix D.1.
We next show through Theorem 1 that learning with synapse-type-speciﬁc weights leads to reduction in loss at every step on average.
The mechanistic intuition behind Theorem 1 is that, in the presence of a statistical connection between synaptic weights (forward path) and modulatory weights (feedback path), the modulatory feedback signal each neuron receives can be a good estimate of how its activity contributes to the overall task error, which can be used for effective tuning of incoming synaptic weights to reduce the error.
Figure S6 compares the angle with the exact gradient across different learning algorithms to demonstrate that the direction of the approximate gradient computed by
ModProp is similar to (aligned with) the direction of exact gradient, thereby reducing the loss.
In the following, we deﬁne the “residual weights” away from the cell-type averages as (cid:15) ij :=
[Equation] and ( (cid:15) s ) ij :
= ( W s ) ij − ( W s ) αβ , where i ∈ α, j ∈ β , α, β ∈ C .
We consider these terms as stochastic so that the circuit output and the eligibility traces are also stochastic, as functions of (cid:15) s .
We consider the cell type averages and the output connection strengths as deterministic. (Merely modifying the uncorrelatedness assumption below would extend our results to stochastic output connection strengths.)
Below, E is short for E (cid:15) , ∆ E | t denotes the change in the task error at time t and ∆ E | pq,t denotes the contribution of the synapse ( pq ) to it.
Theorem 1.
Consider linear RNNs N , (cid:98) N with weight matrices W , (cid:99) W , respectively and identical architectures.
Let (cid:99) W ij =
W αβ , ∀ i ∈ α, j ∈ β .
Assume a small enough learning rate such that the remainder term of the ﬁrst order Taylor expansion of loss is negligible.
For network N , if E [ (cid:15) ij ] = 0 , ( (cid:15) s )
ik and (cid:15) kj are uncorrelated for i (cid:54) =
j , and (cid:15) s and ( y t − y ∗ t ) 2 e pq,t (cid:48)
e pq,t − s are uncorrelated for any s, t, t (cid:48) , then E [∆ E | pq,t ] ≤ 0 and E [∆ E | t ] ≤ 0 .
Moreover, E [∆ E | t ] <
if gradient descent is possible for network (cid:98) N .
(Proof is in Appendix E.)
Note that the uncorrelatedness assumption above is relatively mild because any single connection strength is typically a very poor predictor of network activity, especially as the network size grows.
Note also that gradient update can introduce a drift in residual weights, requiring a similar drift in cell-type averages for the strict inequality to be applicable over multiple update steps.
3.3 Simulation results
[Figure]
c) Pixel-by-pixel MNIST task [58].
Note that this task is unlikely to be solved effectively by humans.
(See text.)
Consistent with the original MDGL paper [6], we also that MDGL confers little advantage over the three-factor rule (e-prop) under dense connectivity
To test the ModProp formulation, we study its performance in well-known tasks involving temporal processing: pattern generation, delayed XOR, and sequential MNIST.
We compare the learning performance of ModProp with the state-of-the-art biologically-plausible learning rules (MDGL [6] and e-prop [5] (labelled as “three-factor”), as well as BPTT to provide a lower bound on task error.
Note, BPTT and RTRL both compute the exact gradient, so they should be identical in terms of performance.
Here, we chose BPTT due to computational efﬁciency.
Consistent with [6], we also found MDGL to confer little advantage over e-prop when all neurons are connected to the readout (Figure 3), so we focus our analysis on that case.
We ﬁrst study pattern generation with RNNs, where the aim is to produce a one-dimensional target output, generated from the sum of ﬁve sinusoids, given a ﬁxed Gaussian input realization.
We change the target output and the random input along with initial weights for different training runs, and show the learning curve in Figure 3A across seven such runs.
By using a densely connected network, we observe that MDGL confers little advantage over the three-factor rule (e-prop), as reported in the original paper [6].
Moreover, communicating longer indirect effects, despite being ﬁltered, leads to improved learning outcomes, as demonstrated by the superior performance of ModProp over MDGL.
[Figure] Next, to study how RNNs learn to process discrete cues that impact delayed outcomes, we consider a delayed XOR task: two cue alternatives, 1 or 0 , are encoded by the presence/absence of input.
The network is trained to remember the ﬁrst cue and learn to compare it with the second cue delivered at a later time to determine if the two cues match.
Figure 3B illustrates the learning curves for this task.
We observe the same general conclusion as the previous task.
Some learning curves have a standard deviation, which indicates that the network struggled to learn the task with these rules for some seeds; based on the standard deviation of the curves, it seems possible for ModProp to perform similarly as three-factor for some seeds, but the focus here is to examine performance across many runs.
In Appendix Figure S4, we also examined the learning performance of the same set of learning rules for a longer delay period.
Interestingly, the performance of ModProp approaches that of BPTT for this task and the previous one, further closing the gap between artiﬁcial network training and biological learning mechanisms in performing credit assignment over a long period.
Finally, we study the pixel-by-pixel MNIST [58] task, which is a popular machine learning benchmark.
Although it is not a task that the brain would solve well (i.e., humans would struggle to predict the digit with only one pixel presented at a time), we investigate it to test the limits of spatiotemporally ﬁltered credit signals for tasks that demand temporally precise input integration.
Figure 3C illustrates the learning curves for this task.
While the performance ordering of learning rules is still the same as previous tasks, we observe a wider gap between ModProp and BPTT.
Since the time-invariant ﬁlter approximation (Eq 3) restricts the spatiotemporal resolution of the credit signal, this is in line with our expectation that ModProp will struggle with tasks that demand highly precise spatiotemporal integration, such as the pixel-by-pixel MNIST task that even the brain would struggle to solve.
As a proof-of-concept study, we initially focused our analysis on the approximation performance by imposing time invariant ﬁlter taps (Eq. 3).
Next, in Figure 4, we investigate the learning performance using type-speciﬁc rather than synapse-speciﬁc feedback modulatory weights (Eq. 5).
These typespeciﬁc weights were calculated using weight averages as in [6].
Later, we repeat these simulations using ﬁxed random modulatory weights in Appendix Figure S2; we also demonstrate the superior performance of ModProp with ﬁxed random modulatory weights for the “copy task” [59] in Appendix
Figure S5.
Little performance degradation is observed for only two modulatory types mapped onto the two main cell classes, indicating the effectiveness of cell-type discretization.
Other than the sequential MNIST task, cells are divided into two main cell classes — with 80% of the cells being excitatory (E) and 20% being inhibitory (I) — and obeyed connection sign constraints.
Lastly, we demonstrate the advantage of our weight update rule in an online learning setting of the pattern generation task, where weights are updated in real time (Figure 5).
For that, we also derive a cost and storage efﬁcient in silico implementation of ModProp in Proposition 1.
[Figure]
Proposition 1.
ModProp has an online in-silico (not necessarily biologically-plausible) implementation with O ( CN 2 ) storage and O ( C 2 N 2 ) computational complexity, where C is the number of cell types.
(Proof is in Appendix D.2.)
Discussion
A central question in the study of biological and artiﬁcial intelligence is how the temporal credit assignment problem is solved in neuronal circuits [3].
Motivated by recent genomic evidence on the widespread presence of local modulatory networks [23, 24], we demonstrated how such signaling could complement Hebbian learning and global neuromodulation (e.g., via dopamine) to achieve biologically plausible propagation of the credit signal through arbitrary time steps.
Going beyond the scalar feedback provided by global top-down modulation, our study proposes how detailed vector feedback information [1] can be delivered in neuronal circuits.
Instead of the severe temporal truncations of the gradient information proposed by the state-of-the-art [4, 5, 6], ModProp offers a framework where the full temporal feedback signal can be received, albeit via low-pass ﬁltering at the post-synaptic neuron due to speciﬁcity at the level of neuronal types, and not individual neurons.
Moreover, predictions generated by ModProp on the role of a family of signaling molecules (e.g. neuropeptides) could potentially be tested experimentally: physiology of multiple individual cells can be monitored in modern neurobiology experiments [25, 60].
Blocking certain peptidergic receptors of the neurons that are involved with learning a task and comparing the performance to that without blocking can provide a test for the role of peptidergic communication.
While feedback alignment [44] addresses the weight transport problem in feedforward networks, it is not clear in RNNs which biological pathways would implement temporal feedback.
Our model suggests that such pathways could come from synapse-type-speciﬁc local neuromodulation.
In addition to improved performance compared to the state-of-the-art across all of our simulations, our theoretical and experimental results show that ModProp can be implemented efﬁciently for online learning.
Figure S1 brieﬂy mentions some of the connections of ModProp to feedback alignment.
Together, these ﬁndings suggest that synapse-type-speciﬁc local modulatory signaling could be a neural mechanism for propagating credit information over more than just a few recurrent steps.
Among the many future directions, a natural extension could be to investigate the performance of
ModProp across a broader range of tasks.
This could include situations where the assumptions in deriving the rule (e.g., stationarity of activity) are severely violated.
This might improve the approximations introduced here.
Similarly, we observed a signiﬁcant gap between ModProp and BPTT for the pixel-by-pixel MNIST task that demands precise temporal integration of input (Figure 3C), but we discussed earlier that this is a challenging task for the brain.
Additionally, this study focuses on dense networks with ReLU activation; future directions include investigating two biologically relevant paradigms: sparse connectivity and diverse set of activation functions (spike-based in particular).
Although ModProp can be applied in theory to temporal credit assignment over an arbitrary duration, the presented learning rule accounts for dependencies at every single step (as for BPTT).
This means that, similar to BPTT, it is ill-suited for very long-term credit assignment [61].
An interesting line of research attends to the issue of extracting relevant, rather than full, memories [61, 62].
Investigating how our approximations could potentially be combined with memory sparsiﬁcation techniques to perform very long-term biologically-plausible credit assignment can be fruitful [61].
While our paper advances the basic science of learning and we do not foresee immediate societal impacts of our work, its beneﬁts to both neuroscience and deep learning research could have long term (positive or negative) societal impacts.
It is hard to overstate the philosophical implications of understanding how the brain works (and, in particular, learns).
Moreover, such understanding could guide us towards curing certain diseases of the brain.
Nevertheless, in the same vein, such future tools could also enable abuse if left unregulated.
On the machine learning side, our method can be considered as a low-cost alternative to the ubiquitous BPTT algorithm.
In this sense, our algorithm or a future method that builds on it can make tasks that are currently approachable by just a few wealthy entities available to many more practitioners.
On the ﬂip side, as with many other capable machine learning tools, such entities are also capable of utilizing low-cost learning to conquer even harder tasks, which is potentially a reason for concern.
Finally, data-driven tools will ultimately reﬂect the various biases in their training data and our method is no exception. Acknowledgements
We thank the Allen Institute founder, Paul G Allen, for his vision, encouragement and support.
YHL is supported by the NSERC PGS-D program.
This work was facilitated through the use of the UW
Hyak supercomputer system.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] We have supported our key contribution points (see Introduction) with simulations and mathematical proofs.
(b) Did you describe the limitations of your work?
[Yes] Please see discussion on future direction in the Discussion section.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Please see the Discussion section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We have carefully read the ethics review guidelines and conﬁrm that this paper conforms to them.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] We have made sure that all assumptions are precisely stated in the Theorem statement.
Details for where these assumptions come into the proof can be found in Appendix E. (b) Did you include complete proofs of all theoretical results?
[Yes] We have provided complete and detailed proofs in Appendix E. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Anonymized code link is provided in Appendix F.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Please see Appendix F. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We ran experiments across random seeds and reported uncertainty for all applicable ﬁgures.
We explained the plotting convention in the caption.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] Please see Appendix F. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Please see Appendix F. (b) Did you mention the license of the assets?
[Yes] Please see Appendix F. (c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Methods A.1 Network Model
We consider a discrete-time implementation of a rate-based recurrent neural network (RNN) similar to the form in [56].
We denote the observable states, i.e. ﬁring rates, as z t at time t , and the corresponding internal states as s t .
The dynamics of those states are governed by
[Equation]
where η = e − dt/τ m denotes the leak factor for simulation time step dt and membrane time constant τ m , W lj denotes the weight of the synaptic connection from neuron j to l , W IN jm denotes the strength of the connection between the m th external input and neuron j and x t denotes the external input at time t .
Threshold adaptation is not used here in order to focus on capacity of the temporal credit propagation mechanism.
We focused on ReLU activation due to its wide adoption in both deep learning and computational neuroscience communities; as discussed, we leave extension to other activation functions (spike-based in particular) for future work.
The readout y is a linear transformation of the hidden state
[Equation]
where W OUT kj denotes the strength of the connection from neuron j to output neuron k , b OUT k denotes the bias of the k -th output neuron.
We quantify how well the network output matches the desired target using loss function E :
[Equation]
where y ∗ k,t is the time-dependent target, π ∗ k,t is the one-hot encoded target and π k,t =
softmax k
( y 1 ,t , . . .
, y N OUT ,t ) =
exp( y k,t ) / (cid:80) k (cid:48) exp( y k (cid:48) ,t ) is the predicted category probability.
A.2 Gradient descent learning in RNNs
Notation for Derivatives : There are two types of computational dependencies in RNNs; direct and indirect dependencies.
We distinguish direct dependencies versus all dependencies (including indirect ones) using partial derivatives ( ∂ ) versus total derivatives ( d ), respectively.
Without loss of generality, consider a function f
( x, y ) , where y itself may depend on x .
The partial derivative ∂ of f at x 0 considers y as a constant, and evaluates as ∂f ( x,y ) ∂x | x 0 ,y ( x 0 ) ; i.e. the derivative calculation only considers how x directly affects f .
The total derivative d , on the other hand, may not treat y as a constant and evaluates as df
[Equation]
x
[Equation]
i [Equation] e. the derivative calculation also takes into account how x can indirectly affect f through y .
As an example in our network, variable W pq can impact state s p,t directly through Eq.
S1, i.e. ∂s p,t W pq =
(1 − η ) z q,t − 1 .
On the other hand, W pq can also impact s p,t indirectly through other cells in the network: i.e. the dependence of s p,t on W pq and all s j,t (cid:48)
( t (cid:48) <
t , j ∈ { 1 , ..., N } ) affected by W pq are taken into account for the derivative calculation, which leads to the recursive equation in Eq. S8.
Exact gradient computation and locality issue : In gradient descent learning, all weight parameters (input weights W IN , recurrent weights W and output weights W OUT ) are adjusted iteratively according to the error gradient.
This error gradient can be calculated with classical machine learning algorithms, backpropagation through time (BPTT) and real time recurrent learning (RTRL)
[42], which uses different factorization but yield equivalent results.
However, the BPTT factorization depend on future activity, which poses an obstacle for online learning and biological plausibility.
Our learning rule derivation follows the RTRL factorization because it is causal.
RTRL factors the error gradient across time and space as
[Equation]
depends on all weights W jl .
[Equation]
S8) following the derivative notation explained above.
The factor ∂E∂z j,t in Eq.
S4 can be interpreted as the top-down learning signal, which is deﬁned as L j,t := (cid:80) k W OUTkj ( y k,t − y ∗ k,t ) for regression tasks [5].
It is straightforward to compute.
However, the triple tensor d s j,t d W pq requires O ( N 3 ) memory and O ( N 4 ) computation costs.
It keeps track of all the paths that z j,t can affect W pq (for every j, p, q ).
Moreover, it poses a signiﬁcant challenge to biological plausibility: updating each weight
W pq requires knowing all other weights W jl (for every j and l ) in the network, and that information should be inaccessible to neural circuits.
To address this, references [4] and [5] dropped the problematic terms so that the updates to weight
W pq would only depend on pre- and post-synaptic activity, and applied this truncation to train rateand spike-based networks, respectively.
However, such truncation results in limited performance.
A.3 Derivation of ModProp
We ask how intercellular neuromodulation might communicate the expensive spatiotemporal dependency in the factor
d s j,t d W pq .
Along with the interpretation of L j,t := ∂E∂z
j,t as top-down learning signal, let e pq,t denote the eligibility trace of coincidental activation between presynaptic cell q and postsynaptic cell p [5].
The following derivation leads to our learning rule.
The leak factor is omitted in the derivation below ( η = 0 ) for readability, and we substitute Eq.
S8 into Eq.
S4 and repeatedly expand the expensive d s d W factor using Eq. S8: [Equation] = .
[Equation]
d
[Equation]
S14) where H ( t, 1) = 1 , H ( t, s ) =
1 N s − 1 (cid:80)
i 1 ,...,i s − 1 h
i 1 ,t − 1 h i 2 ,t − 2 . . .
h
i s − 1 ,t − s +1 for s = 1 , . . .
, S and
S , as explained later, is the number of ﬁlter taps.
Again, we neglected the leak factor in the derivation for readability but included in the actual simulations.
The only approximation step above, ( a ) , is made by using a point estimate assuming that the W and h chains are uncorrelated and the central limit theorem applies.
We note that in a linear network, all the activation derivatives h would be 1 , making the approximation exact.
We expand on our explanation for step ( a ) approximation.
We deﬁne a W -chain (of length l ) as
[Equation]
for any indices i 1 , . .
, i l +1 ∈ { 1 , ..., N } .
Similarly, we deﬁne an h -chain (of length l (cid:48) ) as
[Equation]
for any indices j 1 , . . .
, j l (cid:48) ∈ { 1 , ..., N } .
With these deﬁnitions, we call the W -chain W i 1 ,...,i s − 1 =
W ji 1 W i 1 i 2 . . .
W i s − 1 p and the h
-chain h
i 1 ,...,i s −
= h
i 1 ,t − 1 . . .
h
i s − 1 ,t − s +1 uncorrelated if
[Equation]
Considering W i 1 ,...,i s − 1 and h i 1 ,...,i s − 1 as random i.i.d.
samples indexed by i 1 , .
, i s − 1 , the central limit theorem states that (cid:88) i 1 ,...,i s −
[Equation]
as the sum tends to inﬁnity.
Here, we simply use the i.i.d. assumption even though stronger version of the Central Limit Theorem need weaker assumptions than i.i.d.
When the W - and h -chains are uncorrelated, we take the mean of this distribution as a point estimate (note, however, the growing variance) to arrive at the following approximation:
[Equation]
Since E W i 1 ,...,i s − 1 =
1 N s − [Equation] W s )
jp (by the same application of central limit theorem as above) when i 1 , . .
, i s − 1 are distributed uniformly over valid index ranges, we conclude that
[Equation]
where h -chain with their empirical estimates.
To put this derivation in terms of biological components, we make the following further approximations.
First, we link modulatory weights to type-speciﬁc GPCR efﬁcacies, which means they are type-speciﬁc, i.e. ( W s ) jp
≈ ( W s ) αβ , for type-indicies α and β in a set of possible classes C .
Second, H ( t, s ) should be time-invariant, i.e. H ( t, s )
=
H ( s ) , since biological ﬁlter properties should not vary rapidly across time.
Interestingly, we observe that H ( s ) is an average of activation history across time and cells (Eq. S14).
In particular, when the activation function is ReLU, one can think of H ( s ) as approximating the number of activation chains with length s (divided by the total number of possible chains).
Thus, a crude starting point would be to assume ﬁrst-order stationarity, i.e., assume the average activity level remains invariant ( 1 N (cid:80)
i h i,t :
= µ t ≈ µ, ∀ t ).
Then
[Equation]
where µ is a scalar constant and represents global average neuron activation.
( a ) is because in the case of ReLU activation (where activation derivative h is binary), total number of different activation chain combinations, (cid:80)
i 1 ,...,i s − 1 h i 1 ,t − 1 . . . h
i s − 1 ,t − s +1 , would equal to the product of number of activation at each time step, (cid:80)
i 1 h
i 1 ,t − 1 (cid:80) i 2 h
i 2 ,t − 2 . . .
(cid:80) i s − 1 h
i s − 1 ,t − s +1 .
This is because to choose a chain of activated neurons from all possible combinations, the number of possible indices to choose from for each step is equal to the number of activated neuron at that step.
Indeed, intercellular signaling level is activity-dependent [26].
For implementation, µ can be treated as a hyperparameter or adapted on a separate timescale.
It is also important to note that this activation derivative approximation is only applied to the term (in the equation below) additional to the e-prop term, ∂E∂z p,t e pq,t , for which the exact activation derivative is still used.
By substituting these further approximations into Eq.
S14, the approximated gradient becomes:
[Equation]
and this leads to the ModProp update:
[Equation]
where cell j is of type α , cell p is of type β and C denotes the set of cell types.
L and e denote topdown learning signal and eligibility trace, respectively.
Again, activation derivative h j is closely linked to activity level of neuron j . F represents the modulatory ﬁlter, F αβ ∗ e pq =
(cid:80) Ss =1 F αβ,s e pq,t − s is the convolution operation with S as the number of ﬁlter taps, and scaling factor µ is a hyperparameter.
For calculating the modulatory weights, the weights were calculated using matrix powers for s > 1 . (See beginning of Theorem 1 proof.)
For s = 1 , we ﬁrst examined W 1 αβ = <
W 1 jp > j ∈ α,p ∈ β in the main text.
This assumes modulatory weights and synaptic weights co-adapt throughout training and to what extent they co-adapt in neural circuits is unclear.
Thus, we also set modulatory weights to ﬁxed random type-speciﬁc values and demonstrate the resulting learning performance in Appendix
Figure S2.
These ﬁxed random type-speciﬁc modulatory weights were generated randomly from the distribution of averages of random initial weights.
Thus, these ﬁxed random type-speciﬁc modulatory weights would be close to the initial synaptic weight averages and could stay close depending on how much these synaptic weight averages change throughout training.
Eligibility trace implementation : We here explain the implementation of eligibility trace e pq,t :
[Equation]
S [Equation] which tracks the coincidence of postsynaptic activity h p,t = ∂z p,t ∂s p,t and a low pass ﬁltering of presynaptic activity stored in (cid:15) pq,t ( ∂s p,t ∂w pq =
z q,t − 1 and ∂s p,t +1 ∂s p,t =
η following Eq. S1).
Reference [5] provides a comprehensive discussion on how eligibility traces can be interpreted as derivatives. p [Equation] j k l i w
[Equation]
Exact gradient p q w pq j k l i
B jp µ
B ij µ
B ki µ
B kl µ
B ij
[Equation]
<W> i ε α, j ε β
ModProp (
[Equation]
[Equation]
[Equation]
[Figure]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
modulatory weights), rather than cell-speciﬁc feedback weights (Eq. 3).
Second, ModProp approximates the activation derivative with a constant (Eq. 5).
Similar to feedback alignment for feedforward networks [44], different weights are used for the feedback path than the feedforward computation due to the well-known weight transport problem.
On top of that, time-invariant activation derivative approximation was applied for time-invariant ﬁltering.
Unspeciﬁed symbols were deﬁned in Figure 2.
[Figure] [Figure] We discussed how the basic form of ModProp completely neglects any spatiotemporal speciﬁcity in the activation derivative.
We ask how much performance gain could we get if we lose only temporal speciﬁcity, i e. only average activation derivative across time points.
This would see different neurons as having different average activity.
To put this more concretely we approximate the corresponding factor in Eq.
S12 as (cid:88)
[Equation]
where W := W (cid:12) h with h — a 1 − by − N vector with each entry corresponding to a neuronspeciﬁc mean activation — broadcasted for the element-wise multiplication with W .
In other words, this restores spatial speciﬁcity and the only approximation being made here is to remove temporal speciﬁcity of activation derivative.
As a practical note, by the famous AM-GM inequality, the estimation ( Π s h s ≈ h ) would yield an upper bound of the actual.
Thus, we multiply a dampening
We
.5 times the delay period used in Figure 3B. Although ModProp (with cell-type approximation) still outperforms other bio-plausible learning rules, the performance degrades (compared to ModProp_Wab in Figure 4B).
Moreover, we found all rules (including BPTT) struggle to learn if we increased the delay period to twice of that in Figure 3 without changing other task or network parameters (e.g.
This connects nicely to our discussion point on the limitation of ModProp in addressing very long temporal assignment problems the a
[Figure]
C
[Figure]
Figure S
Sequences of binary cues are presented to an RNN.
For each sequence, once the full sequence has been presented, the network should output the original sequence (with the same value and duration) without any further information [59].
A) Examples of input/output pairs at different sequence lengths.
Instead of having each cue lasting just 1 step, we have each cue lasting 100 steps (100ms) to mimic the duration of a quick cue ﬂash in biological settings.
Superior performance of ModProp even with ﬁxed and random modulatory weights (compared to other biologically plausible rules) is demonstrated for the copy task with a sequence length of B) ﬁve cues ( nc = 5 ) and C) seven cues ( nc = 7 ).
Average loss denotes the binary cross entropy loss computed on target and actual output averaged across time steps.
Solid lines/shaded regions mean/standard deviation of loss curves across ﬁve runs.
factor µ to every W for stability, and treat µ as a hyperparameter.
We name this variant of ModProp as "ModProp_nSpeciﬁc", and the most basic form we investigated in the
Figure S3 shows that this restoration of neuron-speciﬁcity in activation derivative did not lead to signiﬁcant performance improvement.
C Further discussion on related algorithms
For efﬁcient online learning in RNNs, approximations to RTRL have been proposed [17, 50, 51, 52, 4, 53, 54].
For biological realism (use only local information for local updates) and reduced computational cost, e-prop [5] and RFLO [4] proposes severe truncation such that the weight update would only depend on pre- and postsynptic neuron activity as well as a top-down error signal that only tells how a neuron directly contributes to the overall network outcome.
MDGL [6, 63] proposes a less severe truncation than e-prop and RFLO, but it only addresses the contributions to the task error of neurons that are at most 2 synapses away (note, Ref [6] can be considered as a special case of our work, where the ﬁlter length is constrained to a single tap).
ModProp shows a way of removing this signiﬁcant limitation and enables the communication and calculation of credit from neurons that can be arbitrarily many synapses away.
It also experimentally demonstrates the beneﬁt of this key contribution.
Along the approach of truncations [64], reference [53] proposed the SnAp-n algorithm that allows the user to customize the amount of truncation by deciding on n .
SnAP-n stores d s j,t d w pq (seen in Eq. S8) only for j such that parameter w pq inﬂuences the activity of unit j within n time steps.
SnApis closely related to e-prop/RFLO (assuming no autapses).
However, starting at n = 2 (SnAp-2), d s j,t d w pq will be stored for every { j, p, q } such that w pq inﬂuences j in two steps.
This would require the storage of kN 3 traces, where k is a constant that equals the connection density squared.
To our knowledge, there is no evidence on how neural circuits can accommodate such O ( kN 3 ) storage.
Therefore, SnAp-n ( n ≥ 2 ) still poses a signiﬁcant biological plausibility issue while SnAp-1 reduces to e-prop/RFLO in certain circumstances.
Moving from temporal truncation, KeRNL [52] approximates long term dependencies by assuming the dependency to be ﬁrst order low-pass and learn the parameters using node perturbation.
However, the algorithm poses signiﬁcant implementation and biologically plausibility issues: (1) it uses node perturbation to ﬁnd the meta-parameters (e.g. ﬁrst order decay constant), which is not scalable; (2) meta-parameters are updated per step on the same timescale as synaptic weight update.
Their idea of approximating long term dependencies by assuming it follows a certain structure rather than truncating it is what partially inspired our rule.
Unlike KeRNL, our approximation can lead to successful learning with ﬁxed meta parameters that are likely updated on the evolutionary timescale in biology (Figure S2).
D Cost analysis and biological implementation D.1 Cost and interpretation for biologically-plausible implementation of ModProp in Eq.
Recall for ModProp, the eligibility trace is combined with total modulatory signals detected:
[Equation]
We see that the eligibility trace (ET) is brought outside of the double summation of local modulatory (LM) signals.
A biological interpretation is that secretion of top-down (TD) learning signals can selectively activate a biochemical process at the post-synaptic neuron, which can then act as a temporal ﬁlter on the eligibility trace.
The number of ﬁlter taps for the underlying biochemical process, S , determines the number of steps for credit information propagation.
Here is the computational cost breakdown for Remark 3.2:
• a j,t = TD j × ( activity j ) for all j = 1 , .., N has O ( N ) operations.
• (cid:80) j ∈
α a j,t for j = 1 , ..., N α has O ( N α ) operations (assuming a j,t already available, from the last step), where N α is the number of cells in type α .
• LM β,s := (cid:80) α ∈ C W sαβ ( (cid:80) j ∈
α a j,t ) for all α, β = 1 , ...
C and s = 1 , ..., S has O ( SC 2 ) operations.
Note, this step is the modulatory communication step, where type-speciﬁcapproximation of weights can reduce the cost.
• (cid:80) Ss =1 ET pq,t − s × LM β,s for all p, q = 1 , ..., N has N 2 element-wise multiplications per s = 1 , ..., S , leading to a total of O ( SN 2 ) .
Since β can be determined from p , there is no need to loop over β in this step.
Since the cost of the last item dominates, the computational cost scales as O ( SN 2 ) .
For storage cost, ModProp stores e pq,t − S , . . .
e pq,t for every ( pq ) , leading to a storage cost of O ( SN 2 ) .
D.2 Cost for biologically-implausible implementation of ModProp
We prove Proposition 1 next, where we discussed a potentially biologically-implausible in silico implementation with lower computational and storage costs than the biologically-plausible version above.
Proof.
Let us ﬁrst introduce the following notations:
• N α denotes the number of cells in type α • [( W s ) αβ ] ∈ R N α × N β is a matrix repeating the value of scalar ( W s ) αβ •
Thus, [( W 1 ) αγ ][( W s ) γβ ] =
[ N γ ( W 1 ) αγ ( W s ) γβ ] • G tα,pq := (cid:80) ts =1 ( W s ) αβ µ s − 1 e pq,t − s
By properties of block matrix product:
[Equation]
Now
[Equation]
let’s ﬁnd a recursive expression to calculate Z t +1 α,pq online:
[Equation]
And the overall update is:
[Equation]
The second term dominates the cost, for which we need to store G tα,pq for every α, p, q .
This amounts to O ( CN 2 ) storage cost.
To update and attain G t +1 α,pq , we need O ( C ) summations and multplications per G t +1 α,pq , which amounts to O
( C 2 N 2 ) computational cost.
The ﬁnal step of combining G and (cid:80) j ∈ α ∂E∂z j,t h j,t , requires O ( CN 2 ) computational cost, which does not dominate the cost.
We note that the speciﬁc implementation outlined in the proof of Proposition 1 can signiﬁcantly reduce the implementation cost, but is likely biologically-implausible, because each synaptic weight update requires the knowledge of all modulatory weights in the network (Appendix D).
Moreover, it reduces the cost compared to RTRL ( O ( N 3 ) storage and O ( N 4 ) computational complexity) as well as SnAP-2 ( O ( d 2 N 3 ) storage and O ( d 2 N 4 ) computational complexity for connection density d ) [53] signiﬁcantly if only a few cell types are used.
In this work, we used only two cell types ( C = 2 ) that map onto the two main cell classes: excitatory and inhibitory.
However, ModProp is more expensive (by a constant factor) than e-prop, RFLO and MDGL, which all have O ( N 2 ) storage and O ( N 2 ) computational complexity.
However, as mentioned, the performance of these rules are limited due to their severe temporal truncation.
E Unreasonable effectiveness of synapse-type-speciﬁc modulatory backpropagation (through time) weights
We provide the proof for Theorem 1 below:
Proof.
We ﬁrst show that E [( (cid:15) s ) ij ] = 0 for all s ≥ 1 .
Note ( W s ) αβ
= (cid:80) γ N γ ( W s − 1 ) αγ W γβ and ( W s ) ij = (cid:80)
k
( W s − 1 )
ik W kj can be calculated recursively.
The base case s = 1 is already given in the condition.
Suppose E [( (cid:15) s ) ij ] = 0 , for s + 1 :
E [( (cid:15) s +1 ) ij ] =
E [( W s +1 ) ij − ( W s +1 ) αβ ] =
[Equation]
] +
[Equation]
= (cid:88)
[Equation]
(cid:88) k ∈ γ
E [( (cid:15) s )
ik ]
[Equation]
We now prove the Theorem statement for the scalar output case.
The extension to multiple output signals follows identically.
Consider the loss decrement after one update, under the (locally) ﬁrst order loss assumption:
[Equation]
(S32) where ( a ) follows from the uncorrelatedness condition and ( b ) follows from the result of (S31).
Here, we deﬁned Γ pq :=
η ( y t − y ∗ t ) (cid:
) W OUT p e pq [Equation]
t + (cid:80) s (cid:80) α (cid:80) j ∈
[Equation]
W OUT j (
[Equation]
s
[Equation]
Then,
E
[Equation]
S33)
Moreover, if gradient descent is possible for a network ˆ N with weight W ij =
W αβ , ∀ i ∈ α, j ∈ β , then E [ (cid:80) p,q Γ pq ] < 0 by deﬁnition and E [∆ E | t ] < 0 .
We note that with the linear RNN assumption in Theorem 1, Approximation 1 becomes exact when µ = 1 because the activation derivative is a constant 1 for linear networks.
Thus, the proof only only examines the effect of Approximation 2 (type-speciﬁc feedback weight approximation).
Also, the
Theorem assumes uncorrelatedness for residual weights (cid:15) , which may not be the case for networks that are not Erd˝os–Rényi [65].
Despite that, ModProp still lead to performance improvement over existing rules for tasks examined in this study.
Nevertheless, it is important to investigate ModProp across a broad range of tasks in the future. [Figure] Alignment angle comparison shows that gradients approximated by ModProp (with or without type-speciﬁc modulatory weights) are more similar (than MDGL) to the exact gradients . We quantify the similarity between approximated and exact gradients via the alignment angle, which describes the similarity in the direction of the two update vectors (Appendix F) for various tasks.
In all top-panels, the alignment angles between approximate rules and BPTT are less than 90 ◦ , which indicate that the approximated gradients are aligned with the exact gradient, despite the high-dimensionality of the update vectors.
All bottom panel plots show that ModProp variants achieve smaller alignment angles (hence better alignment) with BPTT than MDGL does.
To ensure a fair comparison, we examine the statistics of pairwise differences, so that the point on the loss landscape — where the comparison is done — is matched.
This is achieved by training the network using BPTT across seven different runs and sampling the approximated gradient once every 50 training iterations.
Alignment analysis illustrated here is for recurrent weight gradients, and similar trends are observed for the input weights as well.
F Simulation details
All weight updates were implemented using Adam with default parameters [66].
All Adam learning rates are optimized by picking the best one within { 5 e − 5 , 1 e − 4 , 2 e − 4 , 5 e − 4 , 1 e − 3 , 2 e − 3 , 5 e − 3 } for every learning rule and task.
For ModProp, the best value of hyperparameter µ (Eq. 3) was picked within { 0 . 2 , 0 . 25 , 0 . 3 , 0 . 35 , 0 . 4 , 0 . 45 , 0 . 5 } .
For every learning rule and task, we removed the worst performing run quantiﬁed by area under the learning curve.
We note that while input, recurrent and output weights are all being trained, the nonlocality issue (Eq. S8) only applies to training input and recurrent weights.
Thus, all approaches update output weights using backpropagation, and approximations apply to training input and recurrent weights.
As stated, we repeated runs with different random initialization to quantify uncertainty and weights were initialized similarly as in [67].
We used alignment angle to quantify the similarity between two vectors.
The alignment angle θ between two vectors, a and b, was computed by θ = acos ( (cid:107) a T b (cid:107) / (cid:107) a (cid:107)(cid:107) b (cid:107) ) .
The alignment between two 2D matrices was computed by ﬂattening the matrices into vectors.
For the pattern generation task, our network consisted of 400 neurons described in Eq.
S1.
All neurons had a membrane time constant of τ m = 30 ms. Input to this network was provided by 50 units each producing a different random Gaussian input.
The ﬁxed target signal had a duration of 2000 ms and given by the sum of ﬁve sinusoids, with ﬁxed frequencies of 0 .
5 Hz, 1 Hz, 2 Hz, 3 Hz and 4 Hz.
For learning, we used mean squared loss function and for visualization, we used normalized mean squared error NMSE = (cid:80) k,
t ( y ∗ k,t − y k,t ) {+
+} (cid:80) k,t ( y ∗ k,t )for zero-mean target output y ∗ k,t .
For the delayed XOR task, our implementation of the task involved the presentation of two sequential cues, each lasting 100 ms and separated by a 700 ms delay.
There was only one input unit involved and two cue alternatives were presented by setting the input unit to 1 or 0 , and the unit was set to to 0 during the delay period.
In addition, a Gaussian noise with σ = 0 .
01 was added to the input.
The network was trained to output 1 (resp. 0) at the last time step when the two cues have matching (resp.
non-matching) values.
Our network consisted of 120 neurons.
All neurons had a membrane time constant of τ m = 100 ms.
For learning, we used cross-entropy loss function and the target corresponding to the correct output was given at the end of the trial.
A batch size of 32 was used and the gradients were accumulated during those trials additively.
For the copy task, we presented a input sequence of seven binary cues (taking on the value of 0 or 1 ) on one set of runs and ﬁve cues on another.
Each cue lasts 100 ms to mimic duration of a quick cue ﬂash in biological setting.
After the full sequence presentation, the network is tasked to output the same sequence (same value and duration) without further instruction.
Our network consisted of 120 neurons.
All neurons had a membrane time constant of τ m = 100 ms.
For learning, we used cross-entropy loss function and the target corresponding to the correct output was given at the end of the trial.
We used full batch training: a batch size of 8 (resp. 128) was used for the three (resp.
seven) cue sequence runs due to 8 (resp.
128) possible permutations.
For the pixel-by-pixel MNIST task [58], our network consisted of 200 neurons.
All neurons had a membrane time constant of τ m = 20 ms. Input to this network was provided by a single unit that represented the grey-scaled value of a single pixel, with a total of 784 steps and the network prediction was made at the last step.
For learning, we used the cross-entropy loss function and the target corresponding to the correct output was given at the end of the trial.
A batch size of 256 was used and the gradients were accumulated during those trials additively.
We used TensorFlow [68] version 1.14 and based it on top of [67].
1 We performed simulations on a computer server with x2 6-core Intel Xeon E5-2640, 2.5GHz, 32 GB RAM.
Regardless of the learning rule, our implementation takes approximately one hour to complete one run of pattern generation or delayed XOR task training (for Figures 3 and 4) on the server.