Distinguishing Learning Rules with Brain Machine Interfaces
Anonymous Author(s)
AffiliationAddress
Abstract
Despite extensive theoretical work on biologically plausible learning rules, it hasbeen difficult to obtain clear evidence about whether and how such rules areimplemented in the brain.
We consider biologically plausible supervised- andreinforcement-learning rules and ask whether changes in network activity duringlearning can be used to determine which learning rule is being used.
Supervisedlearning requires a credit-assignment model estimating the mapping from neuralactivity to behavior, and, in a biological organism, this model will inevitably be animperfect approximation of the ideal mapping, leading to a bias in the directionof the weight updates relative to the true gradient.
Reinforcement learning, on theother hand, requires no credit-assignment model and tends to make weight updatesfollowing the true gradient direction.
We derive a metric to distinguish betweenlearning rules by observing changes in the network activity during learning, giventhat the mapping from brain to behavior is known by the experimenter.
Becausebrain-machine interface (BMI) experiments allow for perfect knowledge of thismapping
, we focus on modeling a cursor-control BMI task using recurrent neuralnetworks, showing that learning rules can be distinguished in simulated experimentsusing only observations that a neuroscience experimenter would plausibly haveaccess to.
Introduction
In order to update synaptic weights effectively during learning, a biological or artificial neural networkmust solve the problem of credit assignment.
That is, a neuron must infer whether it should increaseor decrease its activity in order to decrease the error in the behavior that it is driving.
Supervisedlearning (SL) provides one approach for solving the credit assignment problem by endowing thesynaptic learning rule with a credit assignment mapping , i e. an estimate of how each neuron’sactivity affects the behavioral readout.
Indeed, gradient-based methods such as backpropagationfor feedforward networks and backpropagation through time for recurrent networks do exactly thisby multiplying a (possibly multi-dimensional) readout error by the downstream readout weights inorder to update hidden-layer synaptic weights.
However, because neurons in a biological neuralnetwork likely lack complete information about their downstream projections, ideal credit assignmentmappings such as those used in gradient-based methods are biologically implausible–an issue knownas the weight transport problem [1, 2, 3].
Recent work has shown that effective supervised learningin both feedforward networks and RNNs can still take place as long as there is a positive overlapbetween the true readout weights and the credit assignment matrix used for learning, providing onepossible solution to this problem [4, 3, 5, 6, 7].
Reinforcement learning (RL) is another approach for solving the credit assignment problem.
Nodeperturbation algorithms correlate noise injections with scalar reward signals in
order
to update networkweights [8, 9, 10].
Such algorithms avoid the weight transport problem, as the update rule does notexplicitly require a credit assignment matrix mapping the vector error back to the recurrent weights.
While the weight updates for algorithms in this family tend to be noisy, the policy-gradient theoremguarantees that they tend to follow the true gradient of the objective function (Appendix B.2, [8, 11]).
Brain-machine interfaces (BMI) provide an ideal paradigm for investigating questions about learningand credit assignment in real brains, as they enable the experimenter to define the mapping (the“decoder") from neural activity onto behavior.
For example, monkeys can be trained to control acursor using a BMI in the motor cortex, and different manipulations of the BMI decoder can belearned in the course of a single day [12, 13] or over the course of multiple days [14, 15, 16].
In thecase of BMIs, the weight transport problem is particularly acute, since the readout weights mappingthe neural activity onto the cursor position may be changed abruptly by the experimenter, and thereis no plausible way for the agent to generate an instantaneous estimate of the new readout weightsin order to then apply a learning rule.
If an agent is to use a credit-assignment model at all in sucha task, it must consist of a weight matrix that imperfectly approximates the readout mapping, i.e. abiased internal model of credit assignment.
We refer to this as the decoder alignment problem.
The key insight in our work is that, under an SL rule, the agent will invariably perform creditassignment using an imperfect internal model of how its neural activity is mapped onto the kinematicsof the BMI readout, whereas no such internal model is used under a policy gradient-like RL rule.
Weshow that, for SL but not for RL, this imperfect internal model introduces a systematic bias in theweight updates, which in turn leads to a bias in the direction in which neural activity evolves duringlearning, thereby providing a means by which to distinguish these different learning scenarios.
In this work, we develop a framework for modeling BMI experiments with recurrent neural networks (RNNs) using biologically plausible versions of SL and RL.
We then show how, under certain experimental designs, different learning rules can lead to distinct changes in neural activity during the course of learning.
We derive a statistical metric for comparing observed changes in neural activity with thechanges predicted assuming knowledge of the BMI decoder, and we show how this metric can beused to detect learning bias in our simulated experiments.
Related work.
While there has recently been a renaissance of research on biologically plausiblelearning rules [4, 10, 6, 17, 5, 3], there has been relatively little work on how such proposals might beexperimentally verified in neural experiments.
Lim et al.
[18] inferred hyperparameters of a Hebbianlearning rule based on neural activity from a visual cortical area in monkeys.
Their work, however,assumed a single class of learning rules rather than trying to distinguish between multiple classes.
Nayebi et al.
[19] showed that convolutional neural networks trained with different optimizers andhyperparameters can in principle be distinguished by a classifier based on both weight changes andneural activations.
However, the classifier was trained using ground-truth data about the identity of thelearning rule, which is the very thing that we aim to infer in our model.
Kepple et al.
[20] proposed acurriculum-based method for discerning the objective being optimized with SL and showed via RNNmodeling that different objectives lead to distinct task-acquisition timescales.
While their approachanalyzed evidence accumulation and decision tasks and focused on distinguishing different objectives,we focus primarily on continuous control tasks and focus on distinguishing learning rules.
Feulnerand Clopath [21] simulated BMI learning with RNNs to compare learning that required changeswithin vs. outside of the intrinsic manifold of neural activity, while Feulner et al.
[22] used a similarapproach to analyze how learning at upstream or recurrent synapses affected the covariance of neuralactivity, but neither of these studies attempted to distinguish learning rules.
To our knowledge, ourstudy is the first to propose a framework for distinguishing between learning rules using BMIs.
Theory
The overall approach for our work is shown schematically in Fig.
We train RNNs to perform a
BMI-inspired cursor-control task using different learning rules, with the goal of using the observed
[Figure]
predict predict ? ?
learningruleB vectorerror scalarreward ...
activity to infer the learning rule that was used to train the network.
Inspired by numerous BMIexperiments [12, 23, 15], we pre-train the RNN using a particular choice of decoder weights, thenrequire the RNN to relearn the task using a new set of decoder weights.
Our main result is to predictthe different ways in which the RNN activity is expected to evolve through learning, then to comparethese predictions with the observed changes in the activity in order to infer the learning rule.
In thesections that follow, we first introduce the SL and RL update rules that are used for training ournetworks.
We then show that they lead to distinct predictions about how the RNN activity is expectedto evolve with training.
The remainder of the paper will present empirical results showing that thelearning rules can be inferred by observing the network activity, first for a relatively simple case andthen under more generalized assumptions.
While our main results can be obtained using either a feedforward or recurrent circuit model, wemostly focus on the latter case because BMI experiments typically involve control tasks driven bytime-varying activity in motor cortex, for which an RNN model is more appropriate.
The feedforwardversion of our results is provided in Appendix A.
2.1 Modeling BMI learning with RNNs
We model motor cortex as a vanilla RNN characterized by input weights W in , recurrent weights
W rec , feedback weights W fb , and BMI decoder weights W bmi .
Since the experimenter has completecontrol over the decoder in a BMI experiment, we treat W bmi as fixed and assume that learningoccurs in recurrent weights W rec alone.
The activity of the recurrent units h t is given by
[Equation]
where τ is the RNN time constant, ϕ ( · ) is a nonlinear activation function tanh( · ) , x t is an inputsignal, and ξ ti ∼ N (0 , σ 2 ) is isotropic noise added to the recurrent units.
This activity is read outby W bmi to obtain the cursor position y tk
= (cid:80)
i W bmi ki h ti .
The error at each time step is definedas ϵ tk = y ∗ tk − y tk , where y ∗ t is the target output for each time step.
The loss to be minimized isthen L = 12 T (cid:80)
Tt =1 (cid:80) N y k
=
( ϵ tk ) 2 where T is the trial duration and N y
= 2 is the number of readoutdimensions.
Supervised learning.
Standard supervised algorithms for RNNs such as Backpropagation Through
Time [24] and Real-Time Recurrent Learning [25] are widely acknowledged to be biologicallyimplausible.
The reasons for this are two-fold.
First, both algorithms assume knowledge of thereadout weights for purposes of credit assignment, which we referred to above as the weight transport problem.
Second, updating a particular recurrent weight with one of these algorithms requiresknowledge of all of the other weights in the network, which is not information that a biologicalneuron would likely have access to.
A recently proposed alternative, Random Feedback Local Online (RFLO) learning [5], is an approximate gradient-based algorithm with ∆ W rec ≈ − ∂L/∂ W rec that addresses these problems by (i)assuming that error information is projected back into the network with weights that imperfectlyapproximate the ideal credit assignment mapping, and (ii) dropping nonlocal terms from the updaterule.
The recurrent weight update with this learning rule is
[Equation]
with u ti = (cid:80) k W rec ik h t − 1 k + (cid:80) k W in ik x tk
+ (cid:80)
k
W
fb
ik y tk and p tij
≈ ∂h
ti /∂W rec ij .
Further details areprovided in Appendix B.1.
The learning rule that we implement differs slightly from RFLO in thatthe readout weights, as in a BMI experiment, are fixed during training rather than learned.
In RFLO,learning of the readout weights was required in order to align the readout weights with the creditassignment matrix.
Here we will instead assume that M is fixed and partially aligned with ( W bmi ) ⊤ ,so that the cosine similarity of the flattened matrices satisfies sim( M , ( W bmi ) ⊤ ) > 0 .
In (2), the credit assignment mapping M ik assigns responsibility for the readout error ϵ tk to neuron i .
While the ideal mapping would be M =
( W bmi ) ⊤ , learning can still be effective as long as thereis a positive correlation between M and ( W bmi ) ⊤ [5] (as shown earlier in the case of feedforwardnetworks [4]).
In the case where they are not equal, as we presume they cannot be in a BMI experimentunless the monkey is very highly overtrained, the matrix M constitutes a biased internal model ofcredit assignment.
Below, we show that this leads to a predictable bias in the change of the RNNactivity as the task is learned.
Example output trajectories from an RNN trained on the cursor-control task with the SL rule describedabove are shown in Fig.
2A.
The results in Fig.
2B show that learning the task is possible wheneverthere is partial alignment between M and ( W bmi ) ⊤ , with faster learning as this alignment increases.
Reinforcement learning.
To train RNNs with RL, we use a node perturbation learning rule closelyrelated to REINFORCE [8].
We first define a scalar reward R t =
−| ϵ t | 2 .
Node perturbation seeks toincrease this reward by trial and error using the noise appearing in the population activity in (1).
Ifthe noise to a given neuron causes the reward to increase, then the weights will be updated so thatthat neuron’s input will be greater in subsequent trials.
Including an eligibility trace in the learningrule further makes it possible to assign credit for changes in reward to noise events at previous timesteps.
Finally, we can subtract off a baseline from the reward so that learning is guided by a rewardprediction error, which somewhat decreases the variance of the weight updates without introducingbias [11].
The resulting learning rule, which we derive using the policy gradient theorem [26, 11] in
Appendix B.2, is
[Equation]
Very similar RL rules for RNNs have been previously proposed and studied in Refs.
Importantly for our purposes, the fact that this learning rule is derived using a policy gradientapproach guarantees that, upon averaging over noise, the weight updates follow the true gradient ofthe loss function (in our case given by L = − 12 (cid:80) t R t ), making this an unbiased learning rule.
Like(2), (3) does not have nonlocal terms.
Example output trajectories and the learning curve from an
RNN trained on the cursor-control task using (3) are shown in Fig.
2D,E.
2.2 Characterizing changes in neural activity with vector flow fields
Because observing changes in neural activity is much more straightforward than observing changesin synaptic weights in neuroscience experiments, we aim to identify experimentally observable signatures in the neural activity that might distinguish biased SL from unbiased RL.
Because the
RNN activity consists of time-dependent trajectories, and changes in the weights will generally affectentire trajectories cumulatively from one time step to the next, calculating the change ∆ h t due to anupdate ∆ W rec is not entirely straightforward (though cf.
the feedforward case described in Appendix
A, where this can be done).
A more useful approach is to quantify the learning-induced change in thevector flow field that shapes the RNN activity, where the flow field can be estimated empirically byobserving neural activity trajectories.
To do this, we linearize (1), assuming ϕ ( u ) =
u , to obtain anexpression for a vector flow field function F ( h ) that maps activity h t −→ h t +1 and is defined for allpoints h in neural activity space:
[Equation]
where, in our simulations, ∆ t = 1 .
Thus, from (1), we see that F ( h t )
≈ h t
+∆ t − h t .
If the recurrentweights change according to W rec → W rec + ∆ W rec , then the flow field exhibits a correspondingchange F → F + ∆ F , where ∆ F ( h ) = 1 τ ∆ W rec h .
This equation enables us to relate changes in
W rec to changes in neural activity (note that this assumes that plasticity is occurring in the recurrentweights W rec alone, and that W in and W fb do not change).
As illustrated in Fig.
1D, our approachwill be to (i) predict the expected change in flow field ∆ F pred ( h ) under either SL or RL, (ii) estimate∆ F obs ( h ) empirically using activity observed early and late in learning, and (iii) determine whichlearning rule was used to train the network by comparing the empirically observed vs. predicted flowfield changes.
In order to predict the flow field change in the SL case for a point h in neural activity space, wepredict the direction of weight change using an approximation of (2):
[Equation]
where n refers to trials, and n ∈ mid denotes that we sample from trials during the middle part oftraining, when most of the learning is presumed to take place.
Importantly for our purposes, thepredicted change in flow field depends entirely on quantities that are known or can be estimatedfrom experimental data, including W bmi , as well as samples of h n,t and ϵ n,t .
As shown in Section3.4 below, M can also be estimated from data.
Because we are asking only about changes in theflow field rather than the flow field itself, these quantities do not depend explicitly on the recurrentconnectivity W rec itself, which is presumably unobtainable experimentally.
In the RL case, we similarly predict the direction of weight change using an approximation of (3):
[Equation]
In this equation, Σ is the noise covariance, which, under conditions of isotropic noise, reduces to aprefactor σ 2 .
The key distinction between (5) and (6) is that the change in flow field in the SL casedepends on M , while the change in flow field in the RL case depends on ( W bmi ) ⊤ .
We motivate(5) and (6) by computing ⟨ ∆ W rec ⟩ for each learning rule (averaging over noise) in Appendix B.1and Appendix B.2, respectively.
We provide empirical evidence that updates approximately follow aconsistent direction for each rule in Appendix C.
Next, we wish to estimate the empirical change in the flow field given neural activity observed atdifferent points in learning.
The approach is somewhat simplified if we assume that the trials duringwhich learning takes place are preceded and followed by “early” and “late” blocks of trials, in whichthe performance is observed but the weights are not updated.
The assumption that the amount oflearning during these early and late blocks is negligible is justified in the limit where the learningrate is small and the number of training trials is large.
(Evidence that our main results do not dependstrongly on this assumption is given in Appendix A, where learning was allowed to take place duringthe early and late blocks.)
Then, from (4), we define the observed flow field from time step t in trial nas either F early ( h ) =
( A early − I ) h or F late ( h )
=
( A late − I ) h , where A early is the least-squaressolution to the autoregression equation h n,t +1 =
A early h n,t for all time steps t in all early trials n ,
[Figure]
and A late is the corresponding solution for late trials.
The empirical change in the flow field due tolearning for any point h is then given by ∆ F obs ( h ) =
F late ( h ) − F early ( h ) .
Having defined the observed and predicted changes in the flow fields for arbitrary h , we can definethe correlation between these quantities as
[Equation]
We refer to this as the flow field change correlation (FFCC).
One important point to note is that,because this quantity is computed only at observed values of h n,t , it is not the case that the flowfield needs to be estimated everywhere in the high-dimensional space of neural activity.
If theactivity tends to occupy a low-dimensional manifold within this space, then the flow field needs tobe determined only within this manifold.
Another point to note is that this metric is independent ofthe magnitudes of ∆ F obs and ∆ F pred , so that it does not depend on overall prefactors such as thelearning rates and network time constant.
In the simulations that follow, the correlation metric (7)will be computed separately under our two hypothesized learning rules, i.e. with ∆ F pred = ∆ F predSLand with ∆ F pred = ∆ F predRL .
While the absolute magnitudes of this quantity are not informative ontheir own, we show below that the relative values can be compared to infer which of the two learningrules is more likely to have generated the data.
Simulation results
In this section, we use the theory outlined above to show that the SL and RL rules we consider can bedistinguished in simulations, first in a basic case with simplifying assumptions, then in more realisticcases where these assumptions are relaxed.
3.1 Distinct neural signatures of biased SL and unbiased RL
In order to emulate monkey BMI experiments (e.g., [12, 15, 14, 13]), we first pretrain the recurrentweights W rec of an RNN with fixed, random readout weights W bmi0 to perform a center-out cursorcontrol task, in which a cursor must be moved to one of four target locations specified by the input tothe RNN.
This results in low dimensional recurrent dynamics, analogous to the “intrinsic manifold”of activity observed in neural recordings [12, 21].
We then change the BMI decoder to W bmi1 , makeidentical copies of the network, and train one with SL and the other with RL.
For these simulations,shown in Fig.
2, we make the simplifying assumptions (to be generalized in later sections) that (i) thecredit assignment mapping M is fixed and known; (ii) the feedback weights W fb are zero; and (iii)the noise ξ t ∼ N (0 , σ 2 I ) is isotropic.
We show in Appendix C that the results shown in Fig.
2 arerobust to changes in hyperparameters.
For the networks trained with biased SL, we find that a comparison of the observed and predictedflow field changes correctly detects the bias (Fig.
2C).
After pretraining, we change the readoutweights to W bmi1 such that the alignment between the decoders (i.e. the cosine similarity of theflattened matrices) is sim( W bmi0 , W bmi1 ) = 0 .
We next train networks with SL using a creditassignment mapping M , then calculate the correlation between the change in flow field from thesimulated data and the predicted change in flow field based on either the RL prediction (whichimplicitly depends on the true decoder W bmi1 ) or the SL prediction (which depends on M ) using (7).
As we vary the alignment between M and ( W bmi1 )
⊤ , we find that the correlation metric correctlydetects the bias present during learning whenever these two matrices are sufficiently distinct (e.g.sim( M , ( W bmi1 ) ⊤ ) < 0 . 9 ).
For the networks trained with RL using the same procedure describedabove, we find that the change in activity is aligned more closely to the prediction using true decoder
W bmi1 than when using a random matrix
ˆM that is partially aligned to W bmi1 .
As we increase thealignment sim( ˆM , ( W bmi1 ) ⊤ ) , this distinction is less discernible (Fig. 2F).
In the empirical results that follow, we will generalize the assumptions made thus far and show thatthere is a significant range of parameters where the results still hold.
In particular, we will considerthe roles played by (i) estimation and/or learning of the credit-assignment model M , (ii) drivingfeedback via the weights W fb , and (iii) non-isotropic noise.
3.2 Learning the credit assignment mapping
In order to optimize learning in a supervised setting, a learner’s credit assignment mapping couldimprove over time to align more closely with the decoder (Fig. 3A).
To investigate the effect ofsuch learning in our model, we applied our analysis to an RNN in which recurrent weights W recwere trained with SL, with a biased credit assignment mapping M that slowly improved via weightmirroring , a perturbation-based algorithm for learning a “backward-pass” credit assignment matrix[6, 17].
In our simulations (Fig.
3B), the alignment between M and ( W bmi1 )
⊤ is initially set at0 .
5 , then slowly increases as M is learned simultaneously along with the recurrent weights W rec .
Instead of predicting the change in flow field before and after learning, we now apply our analysisover discrete windows during learning.
As expected, activity changes are biased—and hence the SLand RL learning rules are distinguishable—early in learning.
However, these differences disappear as
M approaches ( W bmi1 ) ⊤ later in learning.
3.3 Incorporating driving feedback
So far, we have not included driving feedback, which can strongly influence the dynamics ofpopulation activity.
We next included fixed driving feedback weights W fb = γ M and varied thestrength of these weights by a scalar factor γ (Fig.
3C).
M was randomly generated such thatsim( M , ( W bmi1 ) ⊤ ) = 0 .
Note that the networks trained with SL therefore used the same mapping
M for both the learning rule and for the driving feedback.
We found that bias in an RNN trained with
SL is correctly identified for a wide range of γ (Fig.
3D).
For RNNs trained with RL, we found thatthe correct learning rule is identified for weak feedback, but that the FFCC metric fails for strong
[Figure]
driving feedback (Fig.
3E).
This is likely because neural activity in this limit is dominated by thecontribution from W fb , which is fully aligned with M and less aligned to ( W bmi1 )
⊤ .3.
Estimating the credit assignment mapping from observed activity
Our approach thus far estimates the extent to which the activity updates follow either the directionof the gradient defined by the decoder weights or the (possibly biased) gradient defined by thecredit assignment mapping.
This estimation assumes knowledge of the credit assignment mapping
M .
In a neuroscience experiment, however, this quantity would not be known a priori and wouldneed to be estimated.
To obtain an estimate ˆM of M , we follow an approach previously used inexperiments, in which dimensionality reduction is first applied to the neural data, followed by linearregression to relate the network activity to the cursor movements [12, 15, 21].
More specifically,we (i) pretrain the RNN with decoder W bmi0 and biased credit assignment mapping M (wheresim( M , ( W bmi0 ) ⊤ ) = 0 . 6 ), (ii) observe the activity and cursor trajectories produced by the trainednetwork, (iii) perform principal component analysis on the network activity and relate the principalcomponents to the cursor positions with linear regression, and (iv) use the regression coefficientsto define the estimated credit assignment mapping
ˆM
= DC , where C is the matrix of principalcomponent eigenvectors and D is the matrix of regression coefficients.
Fig.
4A shows that the ˆMobtained in this way is similar to the true M used to train the RNN.
It is important to note for our aimof distinguishing learning rules that ˆM is more similar to M than to W bmi0 and correctly identifiesthe alignment between M and ( W bmi0 ) ⊤ .
We also see from Fig.
4A that this result depends onlyweakly on the number of principal components used to describe the network activity.
After estimating ˆM , we changed the decoder to W bmi1 and trained the network to proficiency witheither
SL (using M in the learning update) or RL (which doesn’t use M ).
For an RNN trained with
SL (Fig. 4B), the correct learning rule is identified using either the true M (red curve) or the estimatedˆM (brown curve) for low and intermediate degrees of similarity between W bmi0 and W bmi1 .
For an
RNN trained with RL (Fig. 4C), correct identification of the learning rule does not depend on the
[Figure]
B
C
SL( M ) SL( M )
RL 0.
sim ( W bmi0 , W bmi1 ) 0.
Train with SL
C o rr(
F ob s ,
F p r ed ) 0.
sim ( W bmi0 , W bmi1 ) 0.
C o rr(
F ob s ,
F p r ed ) 0.
Train with RL
SL( M )
RL
[Figure]
similarity between W bmi0 and W bmi1 .
Together, the above show that, for a relatively widerange of parameters, the credit assignment mapping can be estimated from observed data, and thatthis estimate is sufficient to distinguish between learning rules used to train different networks.
Discussion .
In this work, we have proposed a method for distinguishing biased SL from unbiased RLunder the assumption that the mapping from brain to behavior is known, as in a BMI experiment.
There is ample experimental evidence for plasticity in motor cortex during motor learning [28].
Both
SL and RL are plausible candidates for how motor learning might be implemented in the brain.
Cerebellum in particular has been hypothesized as a classic system for SL, both in classification andcontinuous motor-control settings [29, 30, 31], while basal ganglia are classically viewed as the seatof RL in the brain [32, 33].
These areas are both part of the brain’s motor-control circuitry and areconnected by various direct and indirect pathways to the motor cortex, where BMI experiments aremost often performed.
One possible limitation of our approach is that the number of learning rules that might conceivablybe implemented in recurrent circuits in the brain is much greater than the two that we have focusedon here.
Nevertheless, we view the SL and RL rules considered above as representative examplesof two fundamental classes of learning rules: one in which learning is based on a model for creditassignment, where this model is necessarily imperfect, and another which uses no such model.
Whilewe cannot test every learning algorithm from these classes, we conjecture that the basic fact thatallows us to distinguish our SL and RL rules—namely, that the expected flow field change is in theimage of the credit-assignment matrix for model-based rules but in the image of the true readout fornon-model-based rules—is a fairly generic feature of these two classes.
To support this, we show in
Appendix C that our main results hold for different versions of the biased and unbiased learning rulesthat we have studied.
While we have shown that SL and RL are in principle distinguishable in simulation, there will bechallenges in extending this analysis to real data.
Reliable estimation of our flow field correlationmetric might require large populations of neurons recorded for long periods of time.
Other potentialchallenges that will need to be addressed include accounting for the possibility that multiple types oflearning might be occurring simultaneously at different locations, as well as considering cognitivemechanisms such as attention and contextual inference, which we have not included in our model.
Finally, there are possible ambiguities to consider in mapping our model onto the brain’s motorcircuitry.
While the most straightforward interpretation is that the RNN in our model corresponds tomotor cortex, a more elaborate interpretation could be made by noting that both the cerebellum andthe basal ganglia have well-established roles in motor learning [34] and are likely to play roles in
BMI learning as well [33, 35].
A Distinguishing Supervised Learning from Reinforcement Learning in aFeedforward Model
In order to illustrate the main idea from our paper in a simplified context, we show in this sectionhow observed hidden-layer activity in a linear feedforward network can be used to infer the learningrule that is used to train the network.
Consider the simple feedforward network shown in Fig.
S1.
In this network, random binary input patterns x t , where x ti ∈ {− 1 , 1 } and t = 1 , . . .
, T , areprojected onto a hidden layer h t =
Wx t + ξ t , where ξ t ∼ N (0 , Σ ) is noise injected into thenetwork.
The readout is then given by y t
= W bmi h , and the goal of the network is to minimize
L = (cid:80) t L t = 1 T (cid:80) t | y ∗ t − y t | 2 , where y ∗ are randomly chosen target patterns with y ∗ i ∼ N (0 , 1) .
With W bmi fixed, we can train the network using either supervised learning (SL) or reinforcementlearning (RL).
In the case of SL, the learning rule performs credit assignment using a model M thatapproximates the ideal credit assignment model ( W bmi ) ⊤ to project the error ϵ t =
y ∗ t − y t back tothe hidden layer (Fig.
S1A)
, giving the following weight update:
[Equation]
In the case where M = ( W bmi ) ⊤ , this rule would be implementing gradient descent.
In the casewhere M ̸ = ( W bmi ) ⊤ but the two matrices have positive alignment, it instead implements a biasedversion of gradient descent.
This is similar to learning with Feedback Alignment [4], except that herewe do not assume that the readout weights are being learned.
An alternative learning algorithm is policy gradient learning [8], which gives the following updateequation:
[Equation]
where we define the reward as R t = − L t , and we subtract off the baseline ¯ R t = ⟨ L t ⟩ ξ .
Our goal will be to train the network to minimize the loss using either the SL or RL learning rule,then, assuming W bmi and M are known, to use the observed output and hidden-layer activity duringtraining to infer which of the two algorithms was used to train the network.
In the case of SL,averaging (8) over noise ξ gives
[Equation]
The expected change in the hidden-layer activity due to learning is then
[Equation]
In the case of RL, the average of the weight update from (9) is
[Equation] [Equation]
The expected change in the hidden-layer activity in this case is then
[Equation]
Equations (11) and (13) provide predictions for how the hidden-layer activity is expected to evolveunder either SL or RL.
Since we do not presume to have knowledge of either the learning rate or theupstream activity x t , we can concern ourselves only with the direction, but not the magnitude, of⟨ ∆ h t ⟩ .
In addition, we must make an assumption about correlations between the input patterns.
In
W
W bmi
[Equation]
A
B
C
D
E
F sim( M ,( W bmi ) ⊤ ) c
o rr( ⟨ Δ h ⟩ ob s , ⟨ Δ h ⟩ p r e d ) sim
( M ,( W bmi ) ⊤ )
c o rr( ⟨ Δ h ⟩ ob s , ⟨ Δ h ⟩ p r e d ) general, if the input patterns are correlated with one another, we can let x t · x t ′ =
C ( | t − t ′ | ) , where
C is a function with a peak near zero.
For the simulations shown in Fig.
S1, we have used uncorrelated inputs x t , so we can assume thatx t · x t ′ ∼ δ tt ′ .
With this assumption, Equations (11) and (13) give
[Equation]
SL
[Equation]
Thus, if we assume that the quantities M , Σ , and ( W bmi ) ⊤ are known or can be estimated, (14)provides a prediction for how the hidden-layer activity is expected to change due to learning.
If thisquantity is estimated from data, then the average over noise can be replaced with an empirical averageover observed trials: ⟨ . . .
⟩ = 1 N trials (cid:80) N trials n =1 ( . . . ) .
If we next suppose that we observe the hidden-layer activity empirically without necessarily knowingthe learning rule being used to train the network, then we can define the observed change in hiddenlayer activity as the difference between activity observed in early and late trials
[Equation]
where h n,t is the activity observed at time t in trial n .
We can then compare the similarity of ⟨ ∆ h t ⟩ obswith ⟨ ∆ h t ⟩ pred by computing their correlations:
[Equation]
where ( . . . ) denotes an average over neurons.
In Fig.
S1, we show results from a linear feedforwardnetwork trained with either SL (Fig.
S1A-B) or RL (Fig.
S1D-E)
to map random input patterns ontorandom output patterns.
Then, using (16), we ask whether the change in the hidden-layer activityduring learning in each of these cases is more similar to ⟨ ∆ h ⟩ pred predicted assuming SL or RL.
Theresults in Fig.
S1C,F show that, whenever M and ( W bmi ) ⊤ are sufficiently different, this metric isable to correctly identify whether the network was trained with SL (Fig.
S1C) or RL (Fig.
S1F).
B Biologically Plausible Learning Rules for Recurrent Neural Networks
In this section we provide derivations of the two learning rules studied in our paper.
The RNN updateequation is
[Equation]
(17) where u t =
W rec h t − 1 + W fb y t − 1 + W in x t , and ξ ti ∼ N (0 , σ 2 )
is i.i.d.
noise injected into thenetwork.
The readout is given by y = W bmi h t .
The goal of both learning rules is to iterativelyupdate the recurrent weights W rec such that the readout matches a target function y ∗ t , minimizingthe magnitude of the error ϵ t = y ∗ t − y t .
The learning rules that we consider below depend on amultiplicative combination of pre- and postsynaptic activity, as well as a third factor related to error orreward.
Evidence for such “three-factor" learning rules has been found in a number of neuroscienceexperiments [36].
B.1 Random Feedback Local Online (RFLO) Learning
In this section, we briefly recapitulate the derivation of RFLO [5], a supervised learning algorithmfor RNNs that uses local weight updates to approximate gradient descent.
We then compute theexpectation of the weight update by averaging over noise and show that this expected weight updateis determined by the credit-assignment matrix M .
The loss function to be minimized is
[Equation]
Taking the derivative with respect to the recurrent weights gives
[Equation]
Using (17), we obtain the following recursion relation:
[Equation]
To obtain a learning rule for W rec ab that is local, i.e. that depends only on pre- and postsynapticactivity from units b and a , respectively, we can discard the second line of this equation and write∂h ti /∂W rec ab ≈ δ ia p tab , where the eligibility trace p tab follows the recursion relation
[Equation]
and p 0 ab = 0 .
With this approximation, we arrive at the RFLO learning rule:
[Equation]
In order to compute the expected change in the RNN flow fields, we wish to compute the expectationof the weight update (22) by averaging over noise.
In order to obtain a result for ∆ F that does notdepend explicitly on W rec , we assume a linear network with ϕ ( u ) = u .
For convenience, we alsoswitch to the continuous-time limit.
In this case, the above eligibility trace in (21) is given by
[Equation]
Thus, the expected weight update is given by
[Equation]
where we have defined the covariance matrix Q t,t ′ = ⟨ h t ( h t ′ )
⊤ ⟩ .
In the case where τ is unknown(as it may be in experimental data) or sufficiently small relative to the timescale of RNN dynamics,this expression can be simplified by taking the limit τ → 0 , leading to
[Equation]
In the limit where the noise is small, the first term in parentheses will be much larger than the second,which can be dropped to obtain a simplified expression.
B.2 Reinforcement Learning in Recurrent Neural Networks
In this section, we derive a local RNN update rule using policy gradient learning.
The resulting“node perturbation” learning algorithm is essentially equivalent to previously proposed RL rules forrecurrent circuits from Refs.
We then compute the expectation of the weight update (34) byaveraging over noise and show that this expected weight update is determined by the decoder W bmi ,following the unbiased gradient direction.
In policy gradient learning, a policy π (action | state) is optimized with respect to its parameters inorder to maximize a scalar performance measure R t .
In our case, we interpret h t as the action, h t − 1as the state, and W rec as the parameters to be optimized.
We take the policy to be
[Equation]
26) where
[Equation] [Equation]
is the deterministic part of the update equation (17).
The policy gradient theorem allows us to updatethe policy parameters in a way that ensures improvement of the objective R t .
The REINFORCEalgorithm [8] is based on the policy gradient theorem and updates the parameters W rec according to
[Equation]
[Equation]
where the gradient is with respect to W rec .
The reward baseline ¯ R t , to be defined below, is notrequired for policy gradient learning but can decrease the variance of the updates [11].
The gradient can be computed as follows:
[Equation]
[Equation]
In order to address temporally delayed credit assignment, we can additionally incorporate an eligibilitytrace in the gradient appearing in (28), replacing ∇ ln π −→ ∇ ln π , where the bar denotes low-pass filtering [11].
This allows credit for rewards at time t to be assigned to the RNN activity at earliertime steps.
This leads to the following update rule:
[Equation]
where the eligibility trace is given by
[Equation]
with q 0 ab = 0 .
In our simulations, we set the timescale for the eligibility trace to be equal to thenetwork time constant, i.e. τ e = τ .
In order to compute the expected change ∆ F pred in the RNN flow fields, we next wish to compute theexpectation of the weight update (30) by averaging over noise.
As in the SL case from the previoussection, we assume that the RNN is linear and switch to the continuous-time limit.
In this case, theeligibility trace is given by
[Equation]
The reward R t itself is given by
[Equation]
[Equation]
We assume that the expected reward ¯ R t is independent of the noise ξ , so it doesn’t contribute to⟨ ∆ W rec ⟩ .
Then we can use Wick’s theorem to calculate the expected weight update:
[Equation]
For the linear network we are considering, we have ⟨ ξ si h tl ⟩ = (cid:80) m Σ
im (cid:0) e W ( t − s ) (cid:1) lm , where ξ s ∼
N (0 , Σ ) and W = W rec − I + W fb W bmi .
Then
[Equation]
In the case where τ e is unknown (as it may be in experimental data) or sufficiently small, thisexpression can be simplified by taking the limit τ e → 0 , leading to
[Equation]
As in the SL case, in the limit where the noise is small, the first term in parentheses will be muchlarger than the second, which can be dropped to obtain a simplified expression.
C SimulationsC.1 Experimental Details
The code used to run these simulations will be made available.
As stated in the main text, we first pretrain the recurrent weights W rec of an RNN with fixed, randomreadout weights W bmi0 to perform a center-out cursor-control task, in which a cursor must be moved to one of four target locations specified by the input to the RNN.
We then change the BMI decoder to
W bmi1 , make identical copies of the network, and train one with SL and the other with RL.
Multipleseeds ( n = 4 ) were selected for W bmi1 and M and applied to network copies in the training phase,and correlation metric results Corr(∆ F obs , ∆ F pred ) were averaged over these seeds.
Error bars are
S.E.M. Unless stated otherwise, all simulations involved pretraining.
Simulations for Figures 2 used 4 input dimensions, N = 50 recurrent units, and 2 output dimensions,with a trial duration of 20 timesteps.
The scale for the variance of the noise injected at eachlayer was σ in = 0 , σ rec = 0 .
25 , σ bmi = 0 .
01 for input, recurrent, and output units respectively.
Constant learning rate for recurrent weights was η rec = 0 . 1 , and the RNN time constant was setto τ = 10 .
Recurrent weights W rec were initialized with W rec ∼ N (0 , g 2 / √ N )
where g
= 1 . 5 .
Input weights W in and decoder weights W bmi were initialized randomly and uniformly over [ − 2 , 2]and [ − 2 / √ N, 2 / √ N ] respectively.
For all networks, the activation function was ϕ ( · ) = tanh( · ) .
Pretraining was run for 2,500 trials using SL, and training was run for 1,500 trials for SL and 15,000trials for RL.
Alignment between M and W bmi0 was fixed at 0.5.
Input signals for each targetconsisted of a step function that was 1 for 20% of the trial duration (i.e. 4 timesteps) and 0 for theremainder.
Input to the network at each timestep was therefore a 4 dimensional vector with one entryequal to 1 and other entries equal to zero.
For both SL and RL algorithms, weights were updated atthe end of each trial (i.e. “offline”).
For RL simulations using (3), a separate reward baseline ¯ R t waskept for each target.
In order to control alignment α between decoder weights ( W bmi ) ⊤ and matrices M , we generated a matrix M by randomly changing a subset of matrix entries from ( W bmi )
⊤ such thatsim
( M , ( W bmi ) ⊤ )
=
α .
Networks trained with SL were able to consistently learn the task withfour targets for α > 0 .
Throughout this study, analyses were only performed on networks thatsuccessfully learned the center-out reach task.
When calculating the flow field metric in (7), “early” (before learning) and “late” (after learning)blocks consisted of 500 trials.
Activity during learning was split into training trials and test trials;predictions of the (direction of) weight change ∆ W pred | SL and ∆ W pred | RL were constructed usingactivity h n,t and error ϵ n,t from the training trails, and the full metric Corr(∆ F obs ( h ) , ∆ F pred ( h ))was evaluated on activity h n,t from the test trials.
Fig.
S2 shows that the flow field correlation metric successfully distinguishes the learning rulesacross hyperparameters, including recurrent noise σ rec and number of recurrent units, for both SLand RL.
For networks with a large number of recurrent units, we found that the RL node perturbationalgorithm was more effective when the noise was low-dimensional.
In Fig.
S2D, therefore, therecurrent noise is 50-dimensional, and isotropic within those dimensions.
For Fig.
3A, weight mirroring [6] was used as a convenient way to update M while also learningrecurrent weights
W rec with SL.
Network parameters were the same as Fig. 2, except that η rec waslowered to 0.05 and the weight mirroring learning rate was η WM = 0 .
The weight mirroringalgorithm applied to our context simply correlates presynaptic recurrent noise ξ j ∼ N (0 , σ 2 ) withpostsynaptic activity y i
= W bmi ij ξ j and then updates M with a update rule ∆ M ji = η WM ξ j y i .
Onaverage, this pushes M in the direction of E [ ξ j y i ]
= σ 2 W bmi ji , i.e. the transpose of the decoderweights W bmi .
For Fig.
3C-E, we included driving feedback weights W fb = γ M and varied the strength of thedriving feedback weights by a scalar factor γ between 0.5 and 5.
For these simulations, M was keptfixed at sim( M , ( W bmi ) ⊤ ) = 0 .
Training ran for 5,000 trials for networks using SL, and 10,000trials for networks using RL.
Network parameters were the same as Fig.
2, except with η rec = 0 .
1and σ rec = 0 .
For Fig.
4, the stimulus signal was equal to 1 for the full trial, and the feedback weights were set to
W fb = γ M with γ = 5 .
Pretraining lasted for 2,500 trials, while training was set to 1,000 trialsand the overlap between M and W bmi0 was set to 0.5.
Network parameters were the same as Fig.
2,except with η rec = 1 , and σ rec = 0 . 2 .
[Figure]
For Fig.
S
[Figure]
we trained networks to learn a cursor-control task with non-isotropic noise via either
SL or RL.
In the SL case, we chose a biased credit mapping M that has partial overlap with the newdecoder, with sim( M , ( W bmi1 ) ⊤ ) = 0 .
Network parameters were the same as Fig.
2, except withη rec = 0 .
Training was run for 1,000 trials.
The noise covariance was chosen to be d -dimensional,with rank( Σ ) =
d , and isotropic within those dimensions.
Simulations were run for d = 5 , 10 , 25 , 50with three seeds for each dimension.
The first two of these dimensions were selected to lie inthe subspace spanned by M for the SL-trained RNNs or by W bmi1 for the RL-trained RNNs via
QR decomposition.
Other components of Σ were added in random dimensions orthogonal to thissubspace and to one another.
In Fig.
S1, the network size was 20-20-2, and the task was to map T = 5 random binary input patternsonto random output targets.
For SL, the parameters were η SL = 0 .
001 , σ = 0 .
1 , N trials = 500 , and
N early =
N late = 10 .
For RL, the parameters were η SL = 0 .
003 , σ = 0 .
1 , N trials = 5000 , and
N early =
N late = 100 .
In both cases, the results shown in Fig.
S1C,F were computed by averagingover 100 different networks for each condition.
C.2 Alternative Supervised Learning Rules: “Biased” Backpropagation Through Time(BPTT)
As described in the main text, RFLO [5] is an approximate gradient-based algorithm with ∆ W rec ≈− ∂L/∂ W rec .
The main results of this study depend on the key idea that the matrix M used in the(supervised) learning rule is not identical to the transpose of the decoder weights ( W bmi ) ⊤ .
Thisidea can be applied to BPTT, leading to a learning rule we call “biased” BPTT.
The standard BPTT
[Figure]
RL
SL update is
[Equation]
with u ti = (cid:80) k W rec ik h t − 1 k + (cid:80) k W in ik x tk
+ (cid:80)
k
W
fb
ik y tk , as in the main text.
For a careful comparisonof BPTT, RFLO, and other related gradient-based algorithms for training RNNs, see Refs.
A biased BPTT learning rule according to our framework would simply replace the ( W bmi )
⊤ ϵ t termwith M ϵ t :
[Equation] [Equation]
We show in Fig.
S3 that our results hold when using a biased BPTT learning rule instead of thesupervised RFLO learning rule.
Network parameters and simulation details were the same as Fig.
2,except that a biased BPTT learning rule was used.
For lower similarity between M and ( W bmi ) ⊤ ,the correlation metric is able to correctly identify bias in the change in flow field.
Compared with theresults shown in Fig. 2, the correlation metric values are somewhat lower for biased BPTT.
This islikely because additional nonlocal recurrent terms are contained in the true weight update that are notaccounted for in the expression for ∆ F predSL that uses ∆ W pred | SL from (5).
[Figure]
RL
SL 0.4 0.5 0.6 0.7 0.8 0.9sim( M , ( W bmi1 ) ) 0.
C o rr ( F ob s ,
F p r ed )
Train with SL A
B
C
SL( M )
RL 0.
sim( M , ( W bmi1 ) ) 0.
C o rr ( F ob s ,
F p r ed )
Train with RL
C.3 Linear RNNs
We show that the RNN nonlinearity ϕ ( · ) = tanh( · ) is not a crucial architectural choice for our mainconclusions.
In Fig.
S4 we apply the same approach (and hyperparameters) as shown in Fig.
2C tolinear RNNs and find similar results.
C.4 Velocity-based cursor control
In the main text, we modeled a task in which a BMI readout maps neural activity directly onto cursorposition.
While this is a conceptually simple way to illustrate our main ideas, actual BMI experimentsmore commonly use the readout of neural activity to control cursor velocity rather than position(e.g.
Refs.
In this section we show that our main results can also be obtained forthis case.
Let the RNN readout y t
= W bmi h t correspond to cursor velocity rather than to cursor position, andlet the cursor position be given by r t =
(1 − 1 /τ r ) r t − 1 + y t /τ r .
For simplicity, we take τ r = τ inour simulations.
Let the target velocity at time t be given by y ∗ t = r ∗ − r t , where r ∗ is the targetposition, and the error be given by ϵ t
= y ∗ t − y t .
Using velocity-based cursor control leads to smoother cursor trajectories, as shown in Fig.
S5A.
Figures S5B-C show that the learning rules can be correctly identified in RNNs that are trained underthese assumptions using either SL or RL.
[Figure]
C.5 Learning with a BMI that samples from a subset of the neural population
The simulations in this study have assumed that the decoder W bmi reads out the neural activityfrom all the neurons in the RNN.
While this assumption is not realistic with respect to neuroscienceexperiments, we show here that it does not affect our conclusions.
We ran simulations for RNNs with 200 recurrent units using SL and varied the number of units read outby the decoder between 25 and 200, setting the decoder weights of all non-readout units to zero.
Afterpretraining the RNN, the decoder W bmi1 was randomly selected such that sim( W bmi0 , W bmi1 )
=0 .
5 , while reading out from the same units that were read out from during pretraining.
The creditassignment mapping M was randomly chosen such that sim( M , ( W bmi1 ) ⊤ )
= 0 .
5 , and was notnecessarily restricted to the same subset of readout units.
This was repeated with different numbersof readout units, each across 5 random seeds.
Network and training hyperparameters were otherwisethe same as in Figures 2 and S2.
Fig.
S6 shows that, in this more realistic scenario where a BMIonly decodes a subset of the neurons in the neural population, our correlation metric continues todistinguish between the SL and RL training algorithms.
C.6 Weight updates are distinct for biased SL and RL
In order to build an intuition for how the SL and RL rules affect weight updates, we analyzed theseupdates directly (rather than the changes in flow fields, which was the focus in the main text).
In the SL case we use (5):
[Equation]
In the RL case we use (6):
[Equation]
with isotropic noise Σ = σ 2 I .
Fig.
S7B shows that, for an RNN trained with SL, the observed weight updates ∆ W obs for each trialare highly correlated with the predicted weight updates ∆ W pred | SL .
This is particularly pronouncedearly in learning, as the loss is still decreasing.
This analysis is applied to a network trained with SLwith sim( M , ( W bmi ) ⊤ ) = 0 .
6 , and is one way of building intuition for the results in Fig.
2C and the
FFCC metric Corr(∆ F obs , ∆ F pred ) .
The RL updates look quite different.
Principal component analysis on the weight changes for anetwork trained with RL shows that weight updates are much more spread out across PCs (Fig.
S7A),which also indicates that individual weight updates don’t follow one direction in the loss landscape.
In
Fig.
S7C, however, the average of the observed weight updates over trials during learning ⟨ ∆ W obs ⟩is correlated with ∆ W pred | RL predicted after each trial throughout learning.
C.7 Statistical significance of simulations
To confirm the statistical significance of our results and their dependence on the level of noise in the
RNN, we repeat the simulations from Figure 2 with different levels of noise in Figure S8 and showthat predictions of the two learning rules are statistically distinct when the alignment between the
BMI decoder and the credit assignment weights is sufficiently small.
C.8 RL with non-isotropic noise
In the theoretical and empirical results above, we have assumed that the noise covariance is isotropic.
If we instead allow for non-isotropic noise covariance Σ , we find that this covariance appears in theexpected RL weight update (Appendix B.2).
As in feedforward networks [38], this introduces a biasin the weight updates, so that the noise-averaged weight updates no longer follow the true gradient.
[Figure]
l a i ned v a r i
an c e
[Figure] [Figure]
C o rr ( F ob s ,
F p r ed )
Train with SL, σ rec =0.
RL
SL *** *** *** ** **
[Figure]
Given that RL is no longer unbiased in this case, we thus asked under what conditions we might stillbe able to distinguish biased SL from RL.
We first established that the RNNs can learn a cursor-control task with non-isotropic noise via either
SL (trivially) or RL (as long as some of the noise is in the subspace of the decoder).
To do this, wepretrained networks with W bmi0 , changed the decoder to W bmi1 , and then trained using either SLor RL.
In the SL case, we chose a biased credit mapping M that has partial overlap with the newdecoder, with sim( M , ( W bmi1 ) ⊤ ) = 0 .
The noise covariance was chosen to be d -dimensional,with rank( Σ ) =
d , and isotropic within those dimensions.
The first two of these dimensions wereselected to lie in the subspace spanned by M for the SL-trained RNNs or by W bmi1 for the RL-
[Figure]
trained RNNs.
Other components of Σ were added in random dimensions orthogonal to this subspaceand to one another.
The results in Fig.
S9 show that, when training with SL, we cannot reliably distinguish the learningrule used to train the network when the noise is very low-dimensional (Fig.
S9B).
When training with
RL, however, the learning rule can be distinguished even with low-dimensional noise (Fig. S9C).
Asthe noise dimensionality increases, the learning rules become more easily distinguishable in bothcases.
This remains true in the RL case even if we do not presume to know the true noise covariancematrix, but instead use a naive assumption of isotropic noise in order to generate the predictions (darkblue lines in Fig.
S9B,C).