Biologically-plausible backpropagation through arbitrary timespans via local neuromodulators
Abstract
The spectacular successes of recurrent neural network models where key parameters are adjusted via backpropagation-based gradient descent have inspired muchthought as to how biological neuronal networks might solve the correspondingsynaptic credit assignment problem [1, 2, 3].
There is so far little agreement,however, as to how biological networks could implement the necessary backpropagation through time, given widely recognized constraints of biological synapticnetwork signaling architectures.
Here, we propose that extra-synaptic diffusionof local neuromodulators such as neuropeptides may afford an effective modeof backpropagation lying within the bounds of biological plausibility.
Going beyond existing temporal truncation-based gradient approximations [4, 5, 6], ourapproximate gradient-based update rule, ModProp, propagates credit informationthrough arbitrary time steps.
ModProp suggests that modulatory signals can act onreceiving cells by convolving their eligibility traces via causal, time-invariant andsynapse-type-speciﬁc ﬁlter taps.
Our mathematical analysis of ModProp learning,together with simulation results on benchmark temporal tasks, demonstrate theadvantage of ModProp over existing biologically-plausible temporal credit assignment rules.
These results suggest a potential neuronal mechanism for signalingcredit information related to recurrent interactions over a longer time horizon.
Finally, we derive an in-silico implementation of ModProp that could serve as alow-complexity and causal alternative to backpropagation through time.
Introduction
Recurrent connectivity is a hallmark of neuronal circuits.
While this feature enables rich and ﬂexiblecomputation, mechanisms enabling efﬁcient task learning in large circuits remain a central problem inneuroscience and artiﬁcial intelligence research.
Fundamentally, the problem stems from the fact thatpotentially all history of all neurons, including synaptically far away ones, can affect neuronal activityand contribute to task output.
Motivated by the success of gradient descent learning, several biologicallearning models approximate the exact gradient in recurrent neural networks using known biologicalprocesses and gain insights into computational principles of how the brain might learn [4, 5, 6].
By ignoring dependencies beyond a few recurrent steps — thus severely truncating the gradientcomputational graph — these existing models succeed in representing their approximate gradientbased update rule as a combination of terms that resemble known synaptic physiological processes:"eligibility trace", which maintains a fading memory of coincidental activation of presynaptic andpostsynaptic neurons [7, 8, 9, 10, 11], combined with a third “modulatory” factor — top-downlearning or “reward” signals [12, 13, 14, 15], for which dopamine is a prominent candidate [16].
Despite the impressive performance of such approximations on a variety of tasks, truncating relevantcredit information results in a signiﬁcant performance gap compared to algorithms using exact gradient information; backpropagation-through-time (BPTT) and real time recurrent learning (RTRL) [4, 5, 6,17].
How might neural circuits account for long-term recurrent interactions in order to assign credit (orblame) to neuronal ﬁring that happened arbitrary steps before the presentation of reward?
Given therich repertoire of dynamical and signaling elements in the brain, one avenue could be to examinebiological processes that have been underexplored in existing models.
Dopamine — whose cellularactions are exerted by activation of G protein-coupled receptors (GPCRs), which can greatly impact
STDP — is not the only neuromodulator involved in learning [18, 19, 20, 21].
More recently,transcriptomic studies have uncovered strong evidence for many other neuromodulatory pathwaysthroughout the brain that also act via the activation of GPCRs, leading to similar downstream actionsas dopamine [22, 23].
This suggests that, similar to dopamine, they could also play a role in shapingsynaptic credit assignment.
A conspicuous family within these pathways is neuropeptide signalingbecause peptidergic genes are densely and abundantly expressed in the forebrains of divergent species,including the human, in a cell-type-speciﬁc fashion [24], suggesting widespread interaction betweensynaptic and peptidergic modulatory networks for synaptic credit assignment [25, 6].
Moreover,intra-cortical expression allows neuropeptides to potentially carry information local to the corticalnetwork, cell type speciﬁcity enables sculpted signals for different recipient cells, and their diffusivenature could enable communication between neurons that are not synaptically connected.
Finally,peptidergic signals have time-scales much longer than the time scales for axonal propagation ofaction potentials or synaptic delays [26].
Taken together, these properties make cell-type-speciﬁclocal neuromodulation seem promising for propagating credit signal over multiple recurrent steps.
Developing explicit computational principles of how these local modulatory elements could propagatecredit signal over arbitrary recurrent steps could advance our understanding of biological learningand may inspire more efﬁcient low complexity bio-inspired learning rules.
Motivated by the question above as well as shortcomings of gradient approximations based on severetemporal truncations, we investigate how biological credit signals could be propagated througharbitrary recurrent steps via widespread cell-type-speciﬁc neuropeptidergic signaling [23, 25].
While
Ref.
[6] recently introduced a framework exploiting properties of neuropeptidergic signaling fortemporal credit assignment, similar to [4] and [5], their approach performs severe temporal truncationof the error gradient and does not consider credit propagation beyond disynaptic connections.
Ourmain contributions are summarized as follows:
• We derive a theory that provides mechanism and intuition for the effectiveness of synapsetype-speciﬁc modulatory backpropagation (through time) weights (Theorem 1).
• We develop a model predicting how modulatory signaling could be the basis for biologicalbackpropagation through arbitrary time spans (Figures 1 and 2).
Unlike temporal truncationbased approximations [4, 5, 6], our model enables each neuron to receive ﬁltered (ratherthan precise) credit signals regarding its contribution to the outcome via neuromodulation.•
We demonstrate the effectiveness of modulatory signaling via synapse-type-speciﬁc, ratherthan synapse-speciﬁc, modulatory weights on learning tasks that involve temporal creditassignment (Figure 4).
In particular, we demonstrate an online learning setting whereweights are updated causally and in real-time (Figure 5).
• We also derive a low-complexity in silico implementation of our algorithm suitable for
online learning (Proposition 1).
Related works
Neuromodulatory factors in synaptic plasticity: One of the most fundamental learning rules,
Hebbian plasticity, attributes lasting change in synaptic strength and memory formation to correlationsof spike timing between particular presynaptic and postsynaptic neurons [27, 28].
However, multipleexperimental and theoretical investigations now indicate that the Hebbian rule alone is insufﬁcient.
First, there have been numerous suggestions that some persistent “eligibility trace” must existto bridge the temporal gap between correlated ﬁrings at millisecond timescales and behavioraltimescales lasting seconds [7, 8, 10, 9, 29, 30].
Moreover, impacts of correlated spike timing mustbe augmented by one or more additional modulatory factors to steer weight updates toward desiredoutcome [31, 16, 32, 15, 33, 7, 1, 34, 35, 36, 8, 37, 38].
This is commonly known as learning withthree factors.
A prominent candidate for such a modulatory factor is dopamine [13].
Dopamine inﬂuences receiving neurons via activation of G protein-coupled receptors (GPCRs), which canregulate membrane excitability and key parameters of synaptic plasticity rules.
Besides dopamine, recent transcriptomic evidence has uncovered genes encoding a plethora of otherneuromodulatory pathways throughout the brain, including neuropeptide signaling genes that areabundantly expressed in the forebrains of tetrapods, including the human [23, 24].
Like dopamine,their cellular actions are exerted by the activation of G protein-coupled receptors (GPCRs), whichcan persistently modulate Hebbian synaptic plasticity [15, 39, 40].
This suggests that they too, couldplay a role in shaping cortical learning and synaptic credit assignment.
Moreover, nearly all neuronsexpress one or more neuropeptide signaling gene [23], which suggests a dense interplay betweensynaptic and peptidergic modulatory networks to shape synaptic credit assignment [25].
Approximate gradient descent learning: Standard algorithms for gradient descent learning in recurrent neural networks (RNNs), real time recurrent learning (RTRL) and backpropagation through time(BPTT), are not biologically-plausible [41] and have vast computational and storage demands [42].
However, multiple studies have shown that learning algorithms that approximate the gradient, whilemitigating some of the problems of exact gradient computation, can lead to satisfactory learningoutcomes [41, 43, 44].
In feedforward networks, plenty of biologically-plausible learning ruleshave been proposed and demonstrated impressive performance that rival backpropagation on manydifferent tasks [33, 41, 14, 44, 34, 45, 46, 47, 48, 49].
For efﬁcient online learning in RNNs, approximations to RTRL have been proposed [17, 50, 51,52, 4, 53, 54].
For instance, a recent inﬂuential study [5] conceived how the three-factor learningframework could approximate the gradient.
Among the biologically-plausible proposals [4, 5, 6],approximations have mainly been based on temporal truncations of the gradient computation graphand because of that, their ability to learn dependencies across arbitrary recurrent steps has beenlimited.
Lastly, non-truncation based approximations have been proposed outside of bio-plausibleresearch [50, 51].
For further discussions on related algorithms, please refer to Appendix B.
Results
3.1 Learning rule overview
Biological plausibility is a guiding principle in developing our model.
Hence, the model choices arebased either on established constraints of neurobiology such as the locality of synaptic transmission,causality, and Dale’s Law, or on emerging evidence from large-scale datasets, such as the cell-typespeciﬁcity of certain neuromodulators [23] or the hierarchical organization of cell types [22].
Weconsider a discrete-time rate-based RNN similar to the form in [56] with observable states, i.e. ﬁringrates, as z t at time t , and the corresponding internal states as s t .
W denotes the recurrent weightmatrix with ( pq ) th entry W pq representing the synaptic connection strength between presynapticneuron q and postsynaptic neuron p .
See Supp.
Section A for the neuron model.
Gradient descent learning in RNNs: RNNs are typically trained by gradient descent learning onthe task error (or negative reward) E .
However, the two equivalent factorizations of error gradient in
RNNs, BPTT and RTRL, both involve nonlocal information that is inaccessible to neural circuits.
This is due to recurrent connectivity: a synaptic weight W pq can affect loss E through many otherneurons in the other network in addition to its pre- and postsynaptic neurons.
To see this moreconcretely, the following is the exact gradient at time t using RTRL factorization, on which we baseour approximation for online learning:
[Equation]
depends on all weights W
[Equation]
2) where ∂ denotes direct dependency and d accounts for all (direct and indirect) dependencies, followingthe notation in [5].
(See Appendix A.2 for further details on the notation.)
While ∂E∂s j,t , whichconsiders only the direct contribution of the internal state of neuron j at time t to the loss, is easy to
[Figure]
k
[Figure]
compute, the factor d s j,t d W pq is a memory trace of all inter-cellular dependencies and requires O ( N 3 )memory and O ( N 4 ) computations.
This makes RTRL expensive to implement for large networks.
Moreover, the last factor d s j,t d W pq poses a serious problem for biological plausibility: the nonlocalterms in Eq.
2 requires knowledge of all other weights in the network to update the weight W pq .
Existing biologically-plausible solutions to this problem apply severe truncations: references [4]and [5] completely ignore the third nonlocal term (Figure 2), whereas reference [6] restores termswithin one recurrent step through local modulatory signaling but truncates further terms.
ModProp approximation overview: Derivation of gradient descent-based weight updates involvingneuromodulatory signaling from Eq.
1 suggests two approximations to Eq.
S4 and 2 to move beyondsevere truncations of the exact gradient while retaining biological plausibility.
Approximation 1replaces the activation derivative with a constant:
[Equation]
where µ represents average activity of neurons in a ReLU network.
This approximation assumesstationarity in neuron activity and uncorrelatedness of such activity with the weight, as explained inderivations leading up to Appendix Eq.
S14 and Eq.
S21 (Methods in Appendix A).
While neuronalactivity and synaptic weights are not necessarily uncorrelated, considering that a single neuron mayhave thousands of synaptic partners, the activity of the neuron or its time derivative is typically weaklycorrelated to any one synaptic weight.
We take advantage of this phenomenon in our model andignore these weak correlations.
In our simulations, we treat µ as a hyperparameter and explain howthis is tuned in
Appendix F.
This approximation enables the ﬁlter taps (Eq. 6) to be time-invariant , aproperty likely required for biological plausibility.
We also deﬁne S , the arbitrary number of creditpropagation steps, and it will become clear later that this corresponds to the number of modulatory p q w pq p q w pq p q w pq dEd z t Flow of (exact) computation for z t
[Figure]
propagation (
[Figure]
j
A ii A i Δw pq depends on activity of cells q, p, j, i, k, j, ...
k l ... i j k l i j k l i
A iii
A iii Backpropagation through time Three-factor rule
ModProp p q w pq ...
j k l i
Hebbian
No feedback
Feedback on direct influence
Feedback on direct and indirect influence
[Figure]
[Figure]
z t z t+ z t+
[Figure]
E Backpropagation through time z t z t+ z t+2 z
t- ... ...
E ModProp z t z t+ z
t+
z
t-
[Figure]
ii
[Figure]
i
[Figure]
iii
[Figure]
Temporal view
weight
Hebbian learning, where weight update depends only on pre-/postsynaptic activities
(ii
Three-factor learning [55, 4, 5], which updates weights using additional top-down learning signals severely truncates the exact gradient
(iii
ModProp also accounts for (ﬁltered) distant feedback information delivered through synapse-type-speciﬁc neuromodulation; (iv) BPTT computes the exact gradient: weight update involves nonlocal information, i.e. activities of indirectly connected units.
B)
A temporal view.
Bi)
Three factor learning rule neglects all the intercellular dependencies in the temporal propagation of the credit signal.
Bii) BPTT propagates the precise intercellular dependencies in an acausal manner.
Biii) ModProp approximates such spatiotemporal dependencies through local neuromodulatory signals (Eq. 6).
ModProp approximates the exact gradient by assuming similar connectivity among cells of the same type, and ﬁltering the indirect effects on loss from neurons that are potentially many synapses away.
ﬁlter taps.
With this, the estimated gradient becomes:
[Equation]
where e pq,t ≈ dz p,t dW pq ( e pq,t is deﬁned precisely in Appendix Eq.
S24) can be interpreted as a persistent
Hebbian “eligibility trace” [10, 9, 8] that keeps a fading memory of past coincidental pre- andpostsynaptic activity [5].
Here, ( W s ) jp represents the ( jp ) th entry of W s , W raised to the power ofs , and ( W 1 )
jp
=
W jp Details of the derivation can be found in Appendix A.
Approximation 2 replaces synapse-speciﬁc feedback weights with type-speciﬁc weights:(
W s )
jp
≈ ( W s ) αβ (approximation 2) .
Here, cell j belongs to type α , cell p is of type β and C denotes the set of cell types.
(e.g.,
W αβ =
E j ∈ α,p ∈ β
[ W jp ] , α, β ∈ C .)
This approximation is due to the type-speciﬁc nature ofmodulatory channels [23].
We call these modulatory weights synapse-type-speciﬁc (as opposedto cell-type-speciﬁc) to emphasize connectivity-based grouping.
Details of how these modulatoryweights were obtained can be found in Appendix A.
ModProp: ﬁltering credit signals via local neuromodulation: Substituting Approximation 1 and
Approximation 2 into Eq.
S4 and 2 leads to the ModProp update:
[Equation]
where L and e denote top-down learning signal and eligibility trace, respectively.
We postulatethat F αβ represents type-speciﬁc ﬁlter taps of GPCRs expressed by cells of type β to precursorssecreted by cells of type α ; ∗ is the convolution operation with S as the number of ﬁlter taps.
Note that the matrix powers ( W s ) αβ appearing in the values of different ﬁlter taps F αβ,s couldbe genetically pre-determined as part of the cell identity and optimized over evolutionary timescales.
(Appendix Figure S1 shows successful learning using ﬁxed modulatory weights.)
Details of abiological interpretation of Eq.
6 can be found in Appendix D.1.
Brieﬂy, secretion of top-down (TD)learning signals can selectively activate a biochemical process at the post-synaptic neuron, which canthen act as a temporal ﬁlter on the eligibility trace.
Observing that neurons of the same type demonstrate consistent properties across a range of features(e.g.
connectivity, physiology, gene expression) [23, 57], we use two cell types with consistent wiring,neuromodulation, and type of synaptic action (excitatory/inhibitory) in our relatively simple models.
In summary, we propose a synaptic weight update rule, where the eligibility trace is compounded notonly with top-down learning signals – as in modern biologically-plausible learning rules [7, 8] – butalso with local modulatory pathways through convolution (Figure 1).
This modulatory mechanismallows the propagation of credit signal through arbitrary number of recurrent steps.
3.2 Properties of ModProp
Remark.
The biologically plausible implementation (Eq. 6) of ModProp complexity scales as
O ( SN 2 ) per time step t .
Here, N and S denote the number of recurrent units and number of ﬁlter taps.
As seen in Eq.
4, thenumber of ﬁlter taps corresponds to the number of recurrent steps for which the credit informationis propagated.
We also present an alternative implementation with potentially lower cost later in
Proposition 1.
Details for cost analysis and biological interpretation can be found in Appendix D.1.
We next show through Theorem 1 that learning with synapse-type-speciﬁc weights leads to reductionin loss at every step on average.
The mechanistic intuition behind Theorem 1 is that, in the presenceof a statistical connection between synaptic weights (forward path) and modulatory weights (feedbackpath), the modulatory feedback signal each neuron receives can be a good estimate of how its activitycontributes to the overall task error, which can be used for effective tuning of incoming synapticweights to reduce the error.
Figure S5 compares the angle with the exact gradient across differentlearning algorithms to demonstrate that the direction of the approximate gradient computed by
ModProp is similar to (aligned with) the direction of exact gradient, thereby reducing the loss.
In the following, we deﬁne the “residual weights” away from the cell-type averages as (cid:15) ij :=
W ij − W αβ and ( (cid:15) s ) ij :=
( W s ) ij − ( W s ) αβ , where i ∈ α, j ∈ β , α, β ∈ C .
We consider theseterms as stochastic so that the circuit output and the eligibility traces are also stochastic, as functionsof (cid:15) s .
We consider the cell type averages and the output connection strengths as deterministic.(Merely modifying the uncorrelatedness assumption below would extend our results to stochasticoutput connection strengths.)
Below, E is short for E (cid:15) , ∆ E | t denotes the change in the task error attime t and ∆ E | pq,t denotes the contribution of the synapse ( pq ) to it.
Theorem 1.
Consider linear RNNs N , (cid:98) N with weight matrices W , (cid:99) W , respectively and identicalarchitectures.
Let (cid:99) W ij =
W αβ , ∀ i ∈ α, j ∈ β .
Assume a small enough learning rate such that theremainder term of the ﬁrst order Taylor expansion of loss is negligible.
For network N , if E [ (cid:15) ij ] = 0 ,( (cid:15) s )
ik and (cid:15) kj are uncorrelated for i (cid:54) =
j , and (cid:15) s and ( y t − y ∗ t ) 2 e pq,t (cid:48)
e pq,t − s are uncorrelated forany s, t, t (cid:48) , then E [∆ E | pq,t ] ≤ 0 and E [∆ E | t ] ≤ 0 .
Moreover, E [∆ E | t ] <
if gradient descent ispossible for network (cid:98) N .
(Proof is in Appendix E.)
Note that the uncorrelatedness assumption above is relatively mild because any single connectionstrength is typically a very poor predictor of network activity, especially as the network size grows.
Note also that gradient update can introduce a drift in residual weights, requiring a similar drift incell-type averages for the strict inequality to be applicable over multiple update steps.
3.3 Simulation results
B
C A y*y
Cue 1 0:
Cue 2 Delay 1:
Pattern Generation
Delayed XOR
Sequential MNIST
Classify z tz T z t ... ...
[Figure]
In addition to Approximation 1 studied in Figure 3, this ﬁgure investigates the effect of Approximation 2 (labeled as ModProp_Wab), which uses type-speciﬁc, rather than synapse-speciﬁc feedback weights for signaling credit information (Eq. 5).
Here, ModProp_Wab uses only two modulatory types mapped to the two main cell classes.
The cell type approximation does not result in any signiﬁcant performance degradation in A) the pattern generation task and B) the delayed XOR task.
This analysis is not done for the sequential MNIST task, where neurons were not divided into E and I types.
Solid lines/shaded regions: mean/standard deviation of loss curves ﬁve runs.
To test the ModProp formulation, we study its performance in well-known tasks involving temporalprocessing: pattern generation, delayed XOR, and sequential MNIST.
We compare the learningperformance of ModProp with the state-of-the-art biologically-plausible learning rules (MDGL [6]and e-prop [5] (labelled as “three-factor”), as well as BPTT to provide a lower bound on task error.
Note, BPTT and RTRL both compute the exact gradient, so they should be identical in terms ofperformance.
Here, we chose BPTT due to computational efﬁciency.
Consistent with [6], we also found MDGL to confer little advantage over e-prop when all neurons are connected to the readout(Figure 3), so we focus our analysis on that case.
We ﬁrst study pattern generation with RNNs, where the aim is to produce a one-dimensional targetoutput, generated from the sum of ﬁve sinusoids, given a ﬁxed Gaussian input realization.
We changethe target output and the random input along with initial weights for different training runs, and showthe learning curve in Figure 3A across seven such runs.
By using a densely connected network, weobserve that MDGL confers little advantage over the three-factor rule (e-prop), as reported in theoriginal paper [6].
Moreover, communicating longer indirect effects, despite being ﬁltered, leads toimproved learning outcomes, as demonstrated by the superior performance of ModProp over MDGL.
Next, to study how RNNs learn to process discrete cues that impact delayed outcomes, we consider adelayed XOR task: two cue alternatives, 1 or 0 , are encoded by the presence/absence of input.
Thenetwork is trained to remember the ﬁrst cue and learn to compare it with the second cue delivered ata later time to determine if the two cues match.
Figure 3B illustrates the learning curves for this task.
We observe the same general conclusion as the previous task.
Some learning curves have a wideregion of standard deviation, which indicates that the network struggled to learn the task with theserules for some seeds.
In Appendix Figure S3, we also examined the learning performance of the sameset of learning rules for a longer delay period.
Interestingly, the performance of ModProp approachesthat of BPTT for this task and the previous one, further closing the gap between artiﬁcial networktraining and biological learning mechanisms in performing credit assignment over a long period.
Finally, we study the pixel-by-pixel MNIST [58] task, which is a popular machine learning benchmark.
Although it is not a task that the brain would solve well (i.e., humans would struggle to predict thedigit with only one pixel presented at a time), we investigate it to test the limits of spatiotemporallyﬁltered credit signals for tasks that demand temporally precise input integration.
Figure 3C illustratesthe learning curves for this task.
While the performance ordering of learning rules is still the same asprevious tasks, we observe a wider gap between ModProp and BPTT.
Since the time-invariant ﬁlterapproximation (Eq 3) restricts the spatiotemporal resolution of the credit signal, this is in line withour expectation that ModProp will struggle with tasks that demand highly precise spatiotemporalintegration, such as the pixel-by-pixel MNIST task that even the brain would struggle to solve.
As a proof-of-concept study, we initially focused our analysis on the approximation performance byimposing time invariant ﬁlter taps (Eq. 3).
Next, in Figure 4, we investigate the learning performanceusing type-speciﬁc rather than synapse-speciﬁc feedback modulatory weights (Eq. 5).
These typespeciﬁc weights were calculated using weight averages as in [6].
Later, we repeat these simulationsusing ﬁxed random modulatory weights in Appendix Figure S1; we also demonstrate the superiorperformance of ModProp with ﬁxed random modulatory weights for the “copy task” [59] in Appendix
Figure S4.
Little performance degradation is observed for only two modulatory types mapped ontothe two main cell classes, indicating the effectiveness of cell-type discretization.
Other than thesequential MNIST task, cells are divided into two main cell classes — with 80% of the cells beingexcitatory (E) and 20% being inhibitory (I) — and obeyed connection sign constraints.
[Figure]
Lastly, we demonstrate the advantage of our weight update rule in an online learning setting of thepattern generation task, where weights are updated in real time (Figure 5).
For that, we also derive acost and storage efﬁcient in silico implementation of ModProp in Proposition 1.
Proposition 1.
ModProp has an online in-silico (not necessarily biologically-plausible) implementation with O ( CN 2 ) storage and O ( C 2 N 2 ) computational complexity, where C is the number of celltypes.
(Proof is in Appendix D.2.)
Discussion
A central question in the study of biological and artiﬁcial intelligence is how the temporal creditassignment problem is solved in neuronal circuits [3].
Motivated by recent genomic evidence on thewidespread presence of local modulatory networks [23, 24], we demonstrated how such signalingcould complement Hebbian learning and global neuromodulation (e.g., via dopamine) to achievebiologically plausible propagation of the credit signal through arbitrary time steps.
Going beyondthe scalar feedback provided by global top-down modulation, our study proposes how detailedvector feedback information [1] can be delivered in neuronal circuits.
Instead of the severe temporaltruncations of the gradient information proposed by the state-of-the-art [4, 5, 6], ModProp offers aframework where the full temporal feedback signal can be received, albeit via low-pass ﬁltering atthe post-synaptic neuron due to speciﬁcity at the level of neuronal types, and not individual neurons.
Moreover, predictions generated by ModProp on the role of a family of signaling molecules (e.g.neuropeptides) could potentially be tested experimentally: physiology of multiple individual cells canbe monitored in modern neurobiology experiments [25, 60].
Blocking certain peptidergic receptorsof the neurons that are involved with learning a task and comparing the performance to that withoutblocking can provide a test for the role of peptidergic communication.
While feedback alignment [44] addresses the weight transport problem in feedforward networks, itis not clear in RNNs which biological pathways would implement temporal feedback.
Our modelsuggests that such pathways could come from synapse-type-speciﬁc local neuromodulation.
Inaddition to improved performance compared to the state-of-the-art across all of our simulations, ourtheoretical and experimental results show that ModProp can be implemented efﬁciently for onlinelearning.
Together, these ﬁndings suggest that synapse-type-speciﬁc local modulatory signaling couldbe a neural mechanism for propagating credit information over more than just a few recurrent steps.
Among the many future directions, a natural extension could be to investigate the performance of
ModProp across a broader range of tasks.
This could include situations where the assumptions inderiving the rule (e.g., stationarity of activity) are severely violated.
This might improve the approximations introduced here.
Similarly, we observed a signiﬁcant gap between ModProp and BPTT forthe pixel-by-pixel MNIST task that demands precise temporal integration of input (Figure 3C), but wediscussed earlier that this is a challenging task for the brain.
Additionally, this study focuses on densenetworks with ReLU activation; future directions include investigating two biologically relevantparadigms: sparse connectivity and diverse set of activation functions (spike-based in particular).
Although ModProp can be applied in theory to temporal credit assignment over an arbitrary duration,the presented learning rule accounts for dependencies at every single step (as for BPTT).
This meansthat, similar to BPTT, it is ill-suited for very long-term credit assignment [61].
An interesting line ofresearch attends to the issue of extracting relevant, rather than full, memories [61, 62].
Investigatinghow our approximations could potentially be combined with memory sparsiﬁcation techniques toperform very long-term biologically-plausible credit assignment can be fruitful [61].
While our paper advances the basic science of learning and we do not foresee immediate societalimpacts of our work, its beneﬁts to both neuroscience and deep learning research could have longterm (positive or negative) societal impacts.
It is hard to overstate the philosophical implications ofunderstanding how the brain works (and, in particular, learns).
Moreover, such understanding couldguide us towards curing certain diseases of the brain.
Nevertheless, in the same vein, such futuretools could also enable abuse if left unregulated.
On the machine learning side, our method can beconsidered as a low-cost alternative to the ubiquitous BPTT algorithm.
In this sense, our algorithm ora future method that builds on it can make tasks that are currently approachable by just a few wealthyentities available to many more practitioners.
On the ﬂip side, as with many other capable machinelearning tools, such entities are also capable of utilizing low-cost learning to conquer even hardertasks, which is potentially a reason for concern.
Finally, data-driven tools will ultimately reﬂect thevarious biases in their training data and our method is no exception.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes]
[No] or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .•
Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes] We have supported our key contribution points (seeIntroduction) with simulations and mathematical proofs.
(b) Did you describe the limitations of your work?
[Yes] Please see discussion on futuredirection in the Discussion section.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Please see
the Discussion section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes] We have carefully read the ethics review guidelines and conﬁrm that this
paper conforms to them.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We have made
sure that all assumptions are precisely stated in the Theorem statement.
Details for
where these assumptions come into the proof can be found in Appendix E.
(b) Did you include complete proofs of all theoretical results?
[Yes] We have provided
complete and detailed proofs in Appendix E.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Anonymizedcode link is provided in Appendix F.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Please see Appendix F.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We ran experiments across random seeds and reported
uncertainty for all applicable ﬁgures.
We explained the plotting convention in the
caption.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] Please see Appendix F.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] Please see Appendix F.
(b) Did you mention the license of the assets?
[Yes] Please see Appendix F.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A MethodsA.1 Network Model
We consider a discrete-time implementation of a rate-based recurrent neural network (RNN) similarto the form in [56].
We denote the observable states, i.e. ﬁring rates, as z t at time t , and thecorresponding internal states as s t .
The dynamics of those states are governed by
[Equation]
where η = e − dt/τ m denotes the leak factor for simulation time step dt and membrane time constantτ m , W lj denotes the weight of the synaptic connection from neuron j to l , W IN jm denotes the strengthof the connection between the m th external input and neuron j and x t denotes the external input attime t .
Threshold adaptation is not used here in order to focus on capacity of the temporal creditpropagation mechanism.
We focused on ReLU activation due to its wide adoption in both deeplearning and computational neuroscience communities; as discussed, we leave extension to otheractivation functions (spike-based in particular) for future work.
The readout y is a linear transformation of the hidden state
[Equation]
where W OUT kj denotes the strength of the connection from neuron j to output neuron k , b
OUT k denotesthe bias of the k -th output neuron.
We quantify how well the network output matches the desired target using loss function E :
[Equation]
tasks
[Equation]
tasks
[Equation]
where y ∗ k,t is the time-dependent target, π ∗ k,t is the one-hot encoded target and π k,t =
softmax k
( y 1 ,t , . . .
, y N OUT ,t ) =
exp( y k,t ) / (cid:80) k (cid:48) exp( y k (cid:48) ,t ) is the predicted category probability.
A.2 Gradient descent learning in RNNs
Notation for Derivatives : There are two types of computational dependencies in RNNs; direct andindirect dependencies.
We distinguish direct dependencies versus all dependencies (including indirectones) using partial derivatives ( ∂ ) versus total derivatives ( d ), respectively.
Without loss of generality, consider a function f
( x, y ) , where y itself may depend on x .
The partialderivative ∂ of f at x 0 considers y as a constant, and evaluates as ∂f ( x,y ) ∂x | x 0 ,y ( x 0 ) ; i.e. the derivativecalculation only considers how x directly affects f .
The total derivative d , on the other hand, maynot treat y as a constant and evaluates as df ( x,y ) dx
[Equation]
x
[Equation]
∂y∂x | x 0 ; i.e. thederivative calculation also takes into account how x can indirectly affect f through y .
As an example in our network, variable W pq can impact state s p,t directly through Eq.
S1, i.e.∂s p,t W pq =
(1 − η ) z q,t − 1 .
On the other hand, W pq can also impact s p,t indirectly through other cells inthe network: i.e. the dependence of s p,t on W pq and all s j,t (cid:48)
( t (cid:48) <
t , j ∈ { 1 , ..., N } ) affected by W pqare taken into account for the derivative calculation, which leads to the recursive equation in Eq. S8.
Exact gradient computation and locality issue : In gradient descent learning, all weight parameters(input weights W IN , recurrent weights W and output weights W OUT ) are adjusted iterativelyaccording to the error gradient.
This error gradient can be calculated with classical machine learningalgorithms, backpropagation through time (BPTT) and real time recurrent learning (RTRL)
[42],which uses different factorization but yield equivalent results.
However, the BPTT factorizationdepend on future activity, which poses an obstacle for online learning and biological plausibility.
Ourlearning rule derivation follows the RTRL factorization because it is causal.
RTRL factors the error gradient across time and space as
[Equation]
depends on all weights W jl .
[Equation]
S8) following the derivative notation explained above.
The factor ∂E∂z j,t in Eq.
S4 can be interpreted asthe top-down learning signal, which is deﬁned as L j,t := (cid:80) k W OUTkj ( y k,t − y ∗ k,t ) for regressiontasks [5].
It is straightforward to compute.
However, the triple tensor d s j,t d W pq requires O ( N 3 ) memoryand O
( N 4 ) computation costs.
It keeps track of all the paths that z j,t can affect W pq (for everyj, p, q ).
Moreover, it poses a signiﬁcant challenge to biological plausibility: updating each weight
W pq requires knowing all other weights W jl (for every j and l ) in the network, and that informationshould be inaccessible to neural circuits.
To address this, references [4] and [5] dropped the problematic terms so that the updates to weight
W pq would only depend on pre- and post-synaptic activity, and applied this truncation to train rateand spike-based networks, respectively.
However, such truncation results in limited performance.
A.3 Derivation of ModProp
We ask how intercellular neuromodulation might communicate the expensive spatiotemporal dependency in the factor
d s j,t d W pq .
Along with the interpretation of L j,t := ∂E∂z
j,t as top-down learningsignal, let e pq,t denote the eligibility trace of coincidental activation between presynaptic cell q andpostsynaptic cell p [5].
The following derivation leads to our learning rule.
The leak factor is omittedin the derivation below ( η = 0 ) for readability, and we substitute Eq.
S8 into Eq.
S4 and repeatedlyexpand the expensive d s d W factor using Eq. S8: [Equation] d [Equation]

[Equation]
j ∂E ∂z
j
[Equation]
t h j
[Equation]
t
[Equation]
jp 
 e pq,t − 1 + (cid:88)
j
[Equation]
j
[Equation]
t h j
[Equation]
t
[Equation]
s
[Equation]
(cid:88) i 1 ,...,i s −
W ji 1 W i 1 i 2 . . .
W i s − 1 p h
i 1 ,t − 1 . . .
h
i s − 1 ,t − s +1 e pq,t − s (
[Equation]
[Equation]
a
[Equation]
t
[Equation]
 (cid:88)
j
[Equation]
j,t h j,t W
[Equation]

 e pq,t −
[Equation]
e pq
[Equation]
t − s
N s −(cid:88)
i 1 ,...,i s
[Equation]
h i 1 ,t − 1 h i 2 ,t − 2 . . . h i s
[Equation]
1 ,t
[Equation]
s
[Equation]
[Equation]
[Equation]
(S14) where H ( t, 1) = 1 , H ( t, s )
=
1 N s [Equation] [Equation]
i 1 ,...,i s − 1 h i 1 ,t − 1 h i 2 ,t − 2 . . .
h
i s − 1 ,t − s +1 for s = 1 , . . .
, S and
S , as explained later, is the number of ﬁlter taps.
Again, we neglected the leak factor in the derivationfor readability but included in the actual simulations.
The only approximation step above, ( a ) , ismade by using a point estimate assuming that the W and h chains are uncorrelated and the centrallimit theorem applies.
We note that in a linear network, all the activation derivatives h would be 1 ,making the approximation exact.
We expand on our explanation for step ( a ) approximation.
We deﬁne a W -chain (of length l ) as
[Equation]
(S15) for any indices i 1 , . . .
, i l +1 ∈ { 1 , ..., N } .
Similarly, we deﬁne an h -chain (of length l (cid:48) ) as
[Equation]
for any indices j 1 , . . .
, j l (cid:48) ∈ { 1 , ..., N } .
With these deﬁnitions, we call the W -chain W i 1 ,...,i s − 1 =
W ji 1 W i 1 i 2 . . .
W i s − 1 p and the h
-chain h
i 1 ,...,i s −
= h
i 1 ,t − 1 . . .
h
i s − 1 ,t − s +1 uncorrelated if
[Equation]
Considering W i 1 ,...,i s − 1 and h i 1 ,...,i s − 1 as random i.i.d.
samples indexed by i 1 , .
, i s − 1 , the central 618 limit theorem states that (cid:88)
[Equation]
as the sum tends to inﬁnity.
Here, we simply use the i.i.d. assumption even though stronger versionof the Central Limit Theorem need weaker assumptions than i
[Equation]
i
[Equation]
d. When the W - and h -chains areuncorrelated
[Equation]
we take the mean of this distribution as a point estimate (note, however, the growingvariance) to arrive at the following approximation:
[Equation]
Since E W i 1 ,...,i s − 1 = 1 N s − 1 ( W s )
jp (by the same application of central limit theorem as above) wheni 1 , . .
, i s − 1 are distributed uniformly over valid index ranges, we conclude that [Equation] S
[Equation]
where h -chain with their empirical estimates.
To put this derivation in terms of biological components, we make the following further approximations.
First, we link modulatory weights to type-speciﬁc GPCR efﬁcacies, which means theyare type-speciﬁc, i.e. ( W s ) jp
≈ ( W s ) αβ , for type-indicies α and β in a set of possible classes C .
Second, H ( t, s ) should be time-invariant, i.e. H ( t, s )
=
H ( s ) , since biological ﬁlter propertiesshould not vary rapidly across time.
Interestingly, we observe that H ( s ) is an average of activation history across time and cells (Eq. S14).
In particular, when the activation function is ReLU, one can think of H ( s ) as approximating thenumber of activation chains with length s (divided by the total number of possible chains).
Thus, acrude starting point would be to assume ﬁrst-order stationarity, i.e., assume the average activity levelremains invariant ( 1 N (cid:80)
i h i,t :
= µ t ≈ µ, ∀ t ).
Then
[Equation]
where µ is a scalar constant and represents global average neuron activation.
( a ) is because in the caseof ReLU activation (where activation derivative h is binary), total number of different activation chaincombinations, (cid:80) i 1 ,...,i s − 1 h
i 1 ,t − 1 . . .
h
i s − 1 ,t − s +1 , would equal to the product of number of activationat each time step, (cid:80)
i 1 h
i 1 ,t − 1 (cid:80) i 2 h
i 2 ,t − 2 . . .
(cid:80) i s − 1 h
i s − 1 ,t − s +1 .
This is because to choose a chainof activated neurons from all possible combinations, the number of possible indices to choose fromfor each step is equal to the number of activated neuron at that step.
Indeed, intercellular signalinglevel is activity-dependent [26].
For implementation, µ can be treated as a hyperparameter or adaptedon a separate timescale.
It is also important to note that this activation derivative approximation isonly applied to the term (in the equation below) additional to the e-prop term, ∂E∂z p,t e pq,t , for whichthe exact activation derivative is still used.
By substituting these further approximations into Eq.
S14, the approximated gradient becomes:
[Equation]
and this leads to the ModProp update:
[Equation]
where cell j is of type α , cell p is of type β and C denotes the set of cell types.
L and e denote topdown learning signal and eligibility trace, respectively.
Again, activation derivative h
j is closely linkedto activity level of neuron j .
F represents the modulatory ﬁlter, F αβ ∗ e pq =
(cid:80) Ss =1 F αβ,s e pq,t − s isthe convolution operation with S as the number of ﬁlter taps, and scaling factor µ is a hyperparameter.
For calculating the modulatory weights, the weights were calculated using matrix powers for s > 1 .(See beginning of Theorem 1 proof.)
For s = 1 , we ﬁrst examined W 1 αβ = <
W 1 jp > j ∈ α,p ∈ β in themain text.
This assumes modulatory weights and synaptic weights co-adapt throughout training andto what extent they co-adapt in neural circuits is unclear.
Thus, we also set modulatory weights toﬁxed random type-speciﬁc values and demonstrate the resulting learning performance in Appendix
Figure S1.
Eligibility trace implementation : We here explain the implementation of eligibility trace e pq,t :
[Equation]
which tracks the coincidence of postsynaptic activity h p,t = ∂z p,t ∂s p,t and a low pass ﬁltering ofpresynaptic activity stored in (cid:15) pq,t ( ∂s p,t ∂w pq =
z q,t − 1 and ∂s p,t +1 ∂s p,t =
η following Eq. S1).
Reference [5]provides a comprehensive discussion on how eligibility traces can be interpreted as derivatives.
[Figure] [Figure]
We discussed how the basic form of ModProp completely neglects any spatiotemporal speciﬁcity inthe activation derivative.
We ask how much performance gain could we get if we lose only temporalspeciﬁcity, i.e. only average activation derivative across time points.
This would see different neuronsas having different average activity.
To put this more concretely, we approximate the correspondingfactor in Eq.
S12 as(cid:88)
[Equation]
where W := W (cid:12) h with h — a 1 − by − N vector with each entry corresponding to a neuronspeciﬁc mean activation — broadcasted for the element-wise multiplication with W .
In other words,this restores spatial speciﬁcity and the only approximation being made here is to remove temporalspeciﬁcity of activation derivative.
As a practical note, by the famous AM-GM inequality, theestimation ( Π s h s ≈ h ) would yield an upper bound of the actual.
Thus, we multiply a dampening
We simulate
1.5 times the delay period used in Figure 3B. Although ModProp (with cell-type approximation) still outperforms other bio-plausible learning rules, the performance degrades (compared to ModProp_Wab in Figure 4B).
Moreover, we found all rules (including BPTT) struggle to learn if we increased the delay period to twice of that in Figure 3 without changing other task or network parameters (e.g. width and
This connects nicely to our discussion point on the limitation of ModProp in addressing very long temporal credit assignment problems in the absence of a
[Figure]
C
[Figure]
Figure S
Sequences of binary cues are presented to an RNN.
For each sequence, once the full sequence has been presented, the network should output the original sequence (with the same value and duration) without any further information [59].
A) Examples of input/output pairs at different sequence lengths.
Instead of having each cue lasting just 1 step, we have each cue lasting 100 steps (100ms) to mimic the duration of a quick cue ﬂash in biological settings.
Superior performance of ModProp even with ﬁxed and random modulatory weights (compared to other biologically plausible rules) is demonstrated for the copy task with a sequence length of B) three cues ( nc = 3 ) and C) seven cues ( nc = 7 ).
Average loss denotes the binary cross entropy loss computed on target and actual output averaged across time steps.
Solid lines/shaded regions: mean/standard deviation of loss curves across ﬁve runs.
factor µ to every W for stability, and treat µ as a hyperparameter.
We name this variant of ModProp as"ModProp_nSpeciﬁc", and the most basic form we investigated in
Figure S2 shows that this restoration of neuron-speciﬁcity in activation derivative did not lead tosigniﬁcant performance improvement.
C Further discussion on related algorithms
For efﬁcient online learning in RNNs, approximations to RTRL have been proposed [17, 50, 51,52, 4, 53, 54].
For biological realism (use only local information for local updates) and reducedcomputational cost, e-prop [5] and RFLO [4] proposes severe truncation such that the weight updatewould only depend on pre- and postsynptic neuron activity as well as a top-down error signal thatonly tells how a neuron directly contributes to the overall network outcome.
MDGL [6] proposes aless severe truncation than e-prop and RFLO, but it only addresses the contributions to the task errorof neurons that are at most 2 synapses away (note, Ref [6] can be considered as a special case ofour work, where the ﬁlter length is constrained to a single tap).
ModProp shows a way of removingthis signiﬁcant limitation and enables the communication and calculation of credit from neurons thatcan be arbitrarily many synapses away.
It also experimentally demonstrates the beneﬁt of this keycontribution.
Along the approach of truncations, reference [53] proposed the SnAp-n algorithm that allows theuser to customize the amount of truncation by deciding on n .
SnAP-n stores d s j,t d w pq (seen in Eq.
S8)only for j such that parameter w pq inﬂuences the activity of unit j within n time steps.
SnAp-1is closely related to e-prop/RFLO (assuming no autapses).
However, starting at n = 2 (SnAp-2),d s j,t d w pq will be stored for every { j, p, q } such that w pq inﬂuences j in two steps.
This would requirethe storage of kN 3 traces, where k is a constant that equals the connection density squared.
To ourknowledge, there is no evidence on how neural circuits can accommodate such O ( kN 3 ) storage.
Therefore, SnAp-n ( n ≥ 2 ) still poses a signiﬁcant biological plausibility issue while SnAp-1 reducesto e-prop/RFLO in certain circumstances.
Moving from temporal truncation, KeRNL [52] approximates long term dependencies by assumingthe dependency to be ﬁrst order low-pass and learn the parameters using node perturbation.
However,the algorithm poses signiﬁcant implementation and biologically plausibility issues: (1) it uses nodeperturbation to ﬁnd the meta-parameters (e.g. ﬁrst order decay constant), which is not scalable;(2) meta-parameters are updated per step on the same timescale as synaptic weight update.
Theiridea of approximating long term dependencies by assuming it follows a certain structure ratherthan truncating it is what partially inspired our rule.
Unlike KeRNL, our approximation can lead tosuccessful learning with ﬁxed meta parameters that are likely updated on the evolutionary timescalein biology (Figure S1).
D Cost analysis and biological implementation
D.1 Cost and interpretation for biologically-plausible implementation of ModProp in Eq. 6
Recall for ModProp, the eligibility trace is combined with total modulatory signals detected:
[Equation]
modulatory signal j
[Equation]
We see that the eligibility trace (ET) is brought outside of the double summation of local modulatory(LM) signals.
A biological interpretation is that secretion of top-down (TD) learning signals canselectively activate a biochemical process at the post-synaptic neuron, which can then act as atemporal ﬁlter on the eligibility trace.
The number of ﬁlter taps for the underlying biochemicalprocess, S , determines the number of steps for credit information propagation.
Here is the computational cost breakdown for Remark 3.2:
• a j,t = TD j × ( activity j ) for all j = 1 , .., N has O ( N ) operations.
• (cid:80) j ∈
α a j,t for j = 1 , ..., N α has O ( N α ) operations (assuming a j,t already available, fromthe last step), where N α is the number of cells in type α .
• LM β,s := (cid:80) α ∈ C W sαβ ( (cid:80) j ∈
α a j,t ) for all α, β = 1 , ...
C and s = 1 , ..., S has O ( SC 2 )
operations.
Note, this step is the modulatory communication step, where type-speciﬁc-
approximation of weights can reduce the cost.
• (cid:80) Ss =1 ET pq,t − s × LM β,s for all p, q = 1 , ..., N has N 2 element-wise multiplications pers = 1 , ..., S , leading to a total of O ( SN 2 ) .
Since β can be determined from p , there is noneed to loop over β in this step.
Since the cost of the last item dominates, the computational cost scales as O ( SN 2 ) .
For storagecost, ModProp stores e pq,t − S , . . .
e pq,t for every ( pq ) , leading to a storage cost of O ( SN 2 ) .
D.2 Cost for biologically-implausible implementation of ModProp
We prove Proposition 1 next, where we discussed a potentially biologically-implausible in silicoimplementation with lower computational and storage costs than the biologically-plausible versionabove.
Proof.
Let us ﬁrst introduce the following notations:
• N α denotes the number of cells in type α•
[( W s ) αβ ] ∈ R N α × N β is a matrix repeating the value of scalar
( W s ) αβ• Thus, [( W 1 ) αγ ][( W s ) γβ ] =
[ N γ ( W 1 ) αγ ( W s ) γβ ] • G tα,pq := (cid:80) ts =1 ( W s ) αβ µ s − 1 e pq,t − s
By properties of block matrix product:
[Equation]
Now, let’s ﬁnd a recursive expression to calculate Z t +1 α,pq online:
[Equation]
And the overall update is:
[Equation]
The second term dominates the cost, for which we need to store G tα,pq for every α, p, q .
This amountsto O ( CN 2 ) storage cost.
To update and attain G t +1 α,pq , we need O ( C ) summations and multplicationsper G t +1 α,pq , which amounts to O
( C 2 N 2 ) computational cost.
The ﬁnal step of combining G and(cid:80)
j ∈
α ∂E∂z j,t h j,t , requires O ( CN 2 ) computational cost, which does not dominate the cost.
We note that the speciﬁc implementation outlined in the proof of Proposition 1 can signiﬁcantlyreduce the implementation cost, but is likely biologically-implausible, because each synaptic weightupdate requires the knowledge of all modulatory weights in the network (Appendix D).
Moreover,it reduces the cost compared to RTRL ( O ( N 3 ) storage and O ( N 4 ) computational complexity) aswell as SnAP-2 ( O ( d 2 N 3 ) storage and O ( d 2 N 4 ) computational complexity for connection densityd ) [53] signiﬁcantly if only a few cell types are used.
In this work, we used only two cell types( C = 2 ) that map onto the two main cell classes: excitatory and inhibitory.
However, ModProp ismore expensive (by a constant factor) than e-prop, RFLO and MDGL, which all have O ( N 2 ) storageand O ( N 2 ) computational complexity.
However, as mentioned, the performance of these rules arelimited due to their severe temporal truncation.
E Unreasonable effectiveness of synapse-type-speciﬁc modulatory
backpropagation (through time) weights
We provide the proof for Theorem 1 below:
Proof.
We ﬁrst show that E [( (cid:15) s ) ij ] = 0 for all s ≥ 1 .
Note ( W s ) αβ
= (cid:80) γ N γ ( W s − 1 ) αγ W γβ and( W s ) ij = (cid:80)
k
( W s − 1 )
ik W kj can be calculated recursively.
The base case s = 1 is already given in the condition.
Suppose E [( (cid:15) s ) ij ] = 0 , for s + 1 :
E [( (cid:15) s +1 ) ij ] =
E [( W s +1 ) ij − ( W s +1 ) αβ ]
=
E (cid:34)(cid:88) k
( W s )
ik W kj (cid:35) − (cid:88) γ
N γ ( W s )
αγ W γβ =
E  (cid:88) γ (cid:88)
k ∈ γ
(( W s ) αγ + ( (cid:15) s ) ik )( W γβ + (cid:15) kj ) 
 − (cid:88) γ
N γ ( W s )
αγ W γβ
=
(cid:88)
γ
N γ ( W s ) αγ W γβ + (cid:88) γ (cid:88)
k ∈ γ
E [( (cid:15) s )
ik ] W γβ + (cid:88) γ (cid:88)
k ∈ γ
( W s ) αγ E [ (cid:15) kj ] + (cid:88) γ (cid:88)
k ∈ γ
E [( (cid:15) s ) ik (cid:15) kj ] − (cid:88) γ
N γ ( W s )
αγ W γβ
=
(cid:88) γ (cid:88)
k ∈ γ
E
[Equation]
We now prove the Theorem statement for the scalar output case.
The extension to multiple outputsignals follows identically.
Consider the loss decrement after one update, under the (locally) ﬁrstorder loss assumption:
[Equation]
where ( a ) follows from the uncorrelatedness condition and ( b ) follows from the result of (S31).
Here,we deﬁned Γ pq :=
η ( y t − y ∗ t ) (cid:
) W OUT p e pq,t + (cid:80)
s (cid:80)
α (cid:80) j ∈
α W OUT j ( W s ) αβ e pq,t − s (cid:105) .
Then,
[Figure]
E [∆ E | t ] = −
[Equation]
E (cid:104)(cid:100) ∇ E
[Equation]
E (cid:105) = − η (cid:88) p,q
E (cid:34) (cid:92) dE dW
[Equation]
dE dW pq (cid:35) ≤ 0 .
(S33)
Moreover, if gradient descent is possible for a network ˆ N with weight W ij =
W αβ , ∀ i ∈ α, j ∈ β ,then E [ (cid:80) p,q Γ pq ] < 0 by deﬁnition and E [∆ E | t ] < 0 .
[Figure]
ensure a fair comparison, statistics of pairwise differences, that the is the network using BPTT across seven different runs and sampling the approximated gradient once every 50 training iterations
Alignment analysis illustrated here is for recurrent weight gradients, and similar trends are observed for the input weights as well.
F Simulation details
All weight updates were implemented using Adam with default parameters [63].
All Adam learningrates are optimized by picking the best one within { 5 e − 5 , 1 e − 4 , 2 e − 4 , 5 e − 4 , 1 e − 3 , 2 e − 3 , 5 e − 3 }for every learning rule and task.
For ModProp, the best value of hyperparameter µ (Eq. 3) was pickedwithin { 0 . 2 , 0 .
For every learning rule and task, we removed the worstperforming run quantiﬁed by area under the learning curve.
We note that while input, recurrent andoutput weights are all being trained, the nonlocality issue (Eq. S8) only applies to training inputand recurrent weights.
Thus, all approaches update output weights using backpropagation, andapproximations apply to training input and recurrent weights.
We used alignment angle to quantify the similarity between two vectors.
The alignment angle θbetween two vectors, a and b, was computed by θ = acos ( (cid:107) a T b (cid:107) / (cid:107) a (cid:107)(cid:107) b (cid:107) ) .
The alignment betweentwo 2D matrices was computed by ﬂattening the matrices into vectors.
For the pattern generation task, our network consisted of 400 neurons described in Eq.
S1.
Allneurons had a membrane time constant of τ m =
30 ms. Input to this network was provided by 50units each producing a different random Gaussian input.
The ﬁxed target signal had a duration of2000 ms and given by the sum of ﬁve sinusoids, with ﬁxed frequencies of 0 .
5 Hz, 1 Hz, 2 Hz, 3 Hz and4 Hz.
For learning, we used mean squared loss function and for visualization, we used normalizedmean squared error NMSE = (cid:80) k,
t
( y ∗ k,t − y k,t ) 2 (cid:80) k,t ( y ∗ k,t )for zero-mean target output y ∗ k,t .
For the delayed XOR task, our implementation of the task involved the presentation of two sequentialcues, each lasting 100 ms and separated by a 700 ms delay.
There was only one input unit involvedand two cue alternatives were presented by setting the input unit to 1 or 0 , and the unit was set toto 0 during the delay period.
In addition, a Gaussian noise with σ = 0 .
01 was added to the input.
The network was trained to output 1 (resp. 0) at the last time step when the two cues have matching(resp.
non-matching) values.
Our network consisted of 120 neurons.
All neurons had a membranetime constant of τ m = 100 ms.
For learning, we used cross-entropy loss function and the targetcorresponding to the correct output was given at the end of the trial.
A batch size of 32 was used andthe gradients were accumulated during those trials additively.
For the copy task, we presented a input sequence of seven binary cues (taking on the value of 0 or1 ) on one set of runs and three cues on another.
Each cue lasts 100 ms to mimic duration of a quickcue ﬂash in biological setting.
After the full sequence presentation, the network is tasked to outputthe same sequence (same value and duration) without further instruction.
Our network consisted of120 neurons.
All neurons had a membrane time constant of τ m = 100 ms.
For learning, we usedcross-entropy loss function and the target corresponding to the correct output was given at the end ofthe trial.
We used full batch training: a batch size of 8 (resp. 128) was used for the three (resp.
seven)cue sequence runs due to 8 (resp.
128) possible permutations.
For the pixel-by-pixel MNIST task [58], our network consisted of 200 neurons.
All neurons hada membrane time constant of τ m =
20 ms. Input to this network was provided by a single unitthat represented the grey-scaled value of a single pixel, with a total of 784 steps and the networkprediction was made at the last step.
For learning, we used the cross-entropy loss function and thetarget corresponding to the correct output was given at the end of the trial.
A batch size of 256 wasused and the gradients were accumulated during those trials additively.
We used TensorFlow [64] version 1.14 and based it on top of [65].
We performed simulations on acomputer server with x2 6-core Intel Xeon E5-2640, 2.5GHz, 32 GB RAM.
Regardless of the learningrule, our implementation takes approximately one hour to complete one run of pattern generation ordelayed XOR task training (for Figures 3 and 4) on the server.