Stability and Scalability of Node Perturbation Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
To survive, animals must adapt synaptic weights based on external stimuli andrewards.
And they must do so using local, biologically plausible, learning rules – ahighly nontrivial constraint.
One possible approach is to perturb neural activity (oruse intrinsic, ongoing noise to perturb it), determine whether performance increasesor decreases, and use that information to adjust the weights.
This algorithm – knownas node perturbation – has been shown to work on simple problems, but little isknown about either its stability or its scalability with respect to network size.
Weinvestigate these issues both analytically, in deep linear networks, and numerically,in deep nonlinear ones.
We show analytically that in deep linear networks withone hidden layer, both learning time and performance depend very weakly onhidden layer size.
However, unlike stochastic gradient descent, when there is modelmismatch between the student and teacher networks, node perturbation is alwaysunstable.
The instability is triggered by weight diffusion, which eventually leads tovery large weights.
This instability can be suppressed by weight normalization, atthe cost of bias in the learning rule.
We conﬁrm numerically that a similar instability,and to a lesser extent scalability, exist in deep nonlinear networks trained on botha motor control task and image classiﬁcation tasks.
Our study highlights thelimitations and potential of node perturbation as a biologically plausible learningrule in the brain.
Introduction
The immense success of deep learning in recent years has revived interest in the backpropagationalgorithm (known simply as “backprop”) as a learning mechanism in the brain [1, 2].
Althoughin its pure form this algorithm is incompatible with biological constraints (most notably, the factthat synapses do not know the weights of other synapses in the brain), many biologically plausibleapproximations have been proposed [3, 4, 5, 6, 7, 8, 9].
However, it remains unclear how robust theseapproximate learning rules are or how they perform on large neural networks [10], partially becausewe do not have a clear analytical understanding of them.
In the node perturbation (NP) algorithm, synaptic weights are updated according to how the errorchanges when a small perturbation is added to each node of a neural network [11, 4, 12].
If aperturbation improves performance, the weights are modiﬁed so that the perturbation is consolidated,and vice versa (see §3).
This algorithm has two advantages over more standard biologically plausible learning rules.
First, synaptic updates rely only on a global error signal and pre- and postsynapticactivity.
Such a global error signal, mediated by neuromodulators, is commonly observed across thebrain [13], and is known to modulate synaptic plasticity [14].
This is in contrast to many biologically plausible credit assignment algorithms, which require vector-valued supervising signals (what the true label was), not just scalar error signals (how accurate the network output was).
Second, neural activity in the brain is inherently stochastic [15, 16], which NP can make use of – making noise afeature rather than a bug.
Because of its biological plausibility and algorithmic simplicity, NP has been investigated as apotential learning mechanism of several brain regions, including the vocal production circuits ofsongbirds [17, 18], motor cortex [19], and the cerebellum [20].
In particular, experimental workon birdsong learning suggest that perturbation of neural activity in the song production circuit viaa projection from a region called LMAN is crucial for song acquisition [21, 22], supporting thepresence of perturbation-based learning.
Here we analyze the learning dynamics of node perturbation.
We develop a mean-ﬁeld theory of
NP learning in one hidden layer linear networks, and we show analytically, and conﬁrm empirically,that the minimum training time of NP depends linearly on the number of output nodes, as suggestedpreviously [12].
Perhaps surprisingly, we show that the training time is independent of the hiddenlayer width in the over-parameterized regime.
Finally, we show that NP is always unstable when thesupervised signal contains noise.
This instability can be mitigated by regularizing the weight norms,but that induces bias in the update.
To verify that these results are not speciﬁc to deep linear networks, we investigate the scalability andstability of NP in nonlinear deep neural networks applied to the SARCOS [23] and MNIST tasks [24].
In both tasks, the minimum learning time required to reach high performance scales sub-linearlywith the hidden layer size.
These observations support the relative robustness of NP against hiddenlayer width expansion, a prediction from linear networks.
However, NP dynamics is unstable athigh learning rate, with the instability preceded by an expansion of the weight norm, as predicted.
Weight normalization prevents this instability, but impairs performance, especially for SARCOS.
For MNIST, stabilization by weight normalization improves the convergence time of NP against thedepth expansion.
We also conﬁrmed the stabilization of learning dynamics by weight regularizationin convolutional neural networks solving the CIFAR-10 task [25].
The minimum learning time of NP is about one hundred times longer than that of stochastic gradientdescent (SGD) in both tasks.
However, a portion of this discrepancy is explained by the fact that NP isa reinforcement learning algorithm while SGD uses supervising signals.
Our study thus suggests that,depending on the task and the architecture, NP may play a role as a credit assignment mechanism inthe brain.
However, our results also indicate that NP alone is not enough to account for the learningability of the brain. In particular, NP is too slow to be practical in supervised image recognition tasks.
Related work
NP has been previously proposed as a learning mechanism of birdsong [17, 18], motor cortex [19],and cerebellum [20].
Nonetheless, the practicality of NP as a credit assignment mechanism in thebrain remains unclear, as all of these studies considered simpliﬁed tasks, and scarcely providedanalytical insight.
Werfel and colleagues did provide analytical insight into the scalability of NP
ina linear regression setting: they showed that the convergence of NP becomes progressively slowerthan SGD as the output layer size increases [12].
However, they could not tell whether the slowerconvergence was due to an increase in the number of perturbed neurons or the number of outputneurons, as they were the same.
In this work, we distinguish these two possibilities by introducinga hidden layer (see §4 for the details).
Fiete and colleagues numerically demonstrated that whenthe loss function is deﬁned on a low dimensional projection of output neurons, the training timedoesn’t depend on the output layer size [17].
However, it remains unclear how general this result is,as their analysis is limited to a linear regression setting.
Lansdell and colleagues used NP to trainfeedback weights of a deep neural network that back-propagates the error [26].
The performanceof their algorithm was comparable to SGD on CIFAR-10.
However, it required supervising signals,unlike the vanilla NP, which only needs reinforcement signals.
In addition, it remains unclear howmuch performance gain in their algorithm was brought by the weight alignment rather than from NP.
If perturbations are added to synaptic weights instead of neural activity, the algorithm is called weightperturbation [3].
This algorithm is noiser than node perturbation in the vanilla setting [12], but a recentresult suggests that the weight perturbation algorithm is competitive with, or better than, NP whenthe feedback signal is sparse [27].
It should also be noted that weight perturbation is a variation ofzeroth order optimization, which is known to scale badly to high-dimensional optimization problems
However, recent works found that when there is a low-dimensional latent structure in theparameter space, this algorithm works efﬁciently [30, 31].
Scalability of noisy weight update rules has also been investigated in the bandit literature [32].
Inbandit problems, the gradient needs to be estimated from insufﬁcient feedback, thus its estimationinevitably becomes either biased or noisy.
This limitation raises the question of whether one shouldchoose a biased update or a noisy one, for which the following property is known [33].
Consider anoptimization of a strongly convex loss function L ( w ) with a biased noisy gradient.
Suppose thereis a parameter  such that, the update rule is noisy but unbiased in the limit  ! 0 , and biased butnoise free in the limit  !
Then, under some mild conditions, the value of  that achieves theminimax regret under a ﬁxed number of iterations n , denoted  ⇤ , is given by  ⇤ /
q cdn where dis the number of parameters.
Consequently, when the number of iterations n is small while thenumber of parameters d is large, as in the brain, the system should choose a biased update, not a noisyone.
However, the results from convex optimization are often not directly applicable to deep neuralnetworks, because the effective number of degrees of freedom is typically much smaller than thenumber of parameters in these networks [34], and the loss function is non-convex.
Thus, it remainsunclear if the brain should always choose a biased learning rule over a noisy one, such as NP.
Node perturbation is unbiased, but noisy
We formulate node perturbation in the context of deep learning.
Let us consider a vanilla deepfeedforward network
[Equation]
where x 0 and x K are the input and the output respectively, and f k ( · ) is an element-wise activationfunction.
Throughout the paper, we use bold italic lower case letters to denote column vectors andbold italic upper case letters to denote matrices.
Adding a small Gaussian perturbation  ⇠ k to eachlayer except the input layer gives us a perturbed network,
[Equation]
where h ⇠ k ⇠ Tl i =
kl
I k , with I k the identity matrix in the appropriate dimension, and  ⌧ 1 .
Undera loss function ` ( x K , x 0 ) , the node perturbation update is
[Equation]
In words, if the perturbation decreases the error (i.e., ˜ `  ` < 0 ), the weights are shifted towards thedirection of the perturbation ( ⇠ k ), and vice versa.
Importantly, in this update rule, the network onlyneeds to know how much the loss changes when a perturbation is added to the network.
This is incontrast to SGD and most of its biologically plausible variants, which require a supervised signaltelling what the correct answer was.
It is straightforward to show that, at the small perturbation limit(ie.
! 0 ; see Appendix A.1),
[Equation]
where g k ⌘ @` @ h k , and h k ⌘ W k x k  1 . [Equation] Taking the expectation over ⇠ , we recover the SGD update rule,
[Equation]
Consequently, NP is unbiased relative to SGD. However, because the weight update is driven byrandom perturbation to the nodes, NP is much more noisy than SGD.
For instance, the cosinesimilarity between SGD and NP update at k -th layer is given by
[Equation]
where L k is the number of neurons in the k -th layer (see Appendix A.2).
Thus, for wide networks,the NP and SGD updates becomes nearly perpendicular (see Figs.
4A and S4A).
Another way to characterize how noisy NP is to look at the noise covariance.
Because both SGD and
NP are unbiased against the true gradient, both update rules can be written as
[Equation]
where Q = np or sgd , and Z Qk is a (not necessarily Gaussian) random matrix.
We show in Appendix
A.3 that the covariance of Z Qk under NP, denoted C npkl , is written as,
[Equation]
where, I k is a size- L k identity matrix, C sgdkl is the noise-covariance under SGD, and “ ⌦ ” denotesa tensor product between two matrices (i.e., [ A ⌦ B ] ijkl = A ij B kl ).
Because the second term iszero unless two weights share the same postsynaptic neuron, the noise correlation of NP updatesacross neurons in different layers, or across different neurons in the same layer, is only a factor oftwo larger than that of SGD.
However, the NP update at each synapse contains additional, mostlyindependent, noise from the second term that scales with the total gradient norm
[Equation]
This 136 term is typically much larger than the SGD variance, which only depends on the gradient amplitudeat each weight (see Eq. 30).
NP in one hidden layer linear networks: scalable but unstable
We have so far observed that NP is unbiased but noisy, meaning that a one-step update by NP is justa noisy version of SGD.
However, it is not clear what that implies for the global stability and thescalability of the algorithm.
Because it is very difﬁcult to address these issues in general, here weﬁrst focus on deep linear networks in a student-teacher setting [35, 36].
In the absence of hidden layers (i.e., in a linear regression setting), it is known that the minimumnumber of training steps required to reach a target error level ✏ tg from an initial error ✏ o is
[Equation]
where L x and L y are input and output layer sizes, respectively (see [12] and Appendix B.6).
Thus, NPis linearly slower than SGD as the number of output units increases.
Note that at a small learning rate⌘ , the learning trajectories of NP and SGD behave similarly (left edge of Fig.
S1, in the Appendix).
However, the optimal learning rate – the learning rate that minimizes the training time – is smallerunder NP than SGD, so the minimum training time becomes longer (Fig. S1; here, the optimallearning rates are the points at which the curves touch the horizontal dotted lines).
Nevertheless,in the linear regression setting, NP learning dynamics is simple, and monotonic: the error eitherdecreases or increases monotonically throughout learning.
As we will see, however, this is not thecase in deep neural network; they display non-trivial, and non-monotonic, dynamics.
Equation (9) tells us that NP is slower than SGD by a factor of L y , the number of output units.
However, it doesn’t tell us in general whether training time scales with the number of output units orthe number of perturbed units, because the two are the same in linear regression.
To disambiguatebetween these two possibilities, we analyze a linear network with one hidden layer,
[Equation]
[Equation]
where the matrices W 2 and W 1 are L y ⇥
L h and L h ⇥ L x , respectively.
We consider a student-teachersetting, in which the target output y ⇤ is generated from a teacher network,
[Equation]
(11) where  t ⇣ is zero mean Gaussian noise that creates a mismatch between the student and the teachernetworks.
We implement NP in this network by applying small perturbations to both the hiddenand output layers simultaneously (see Eq.
3 and Appendix B.1).
Although the exact learningdynamics is not tractable for this network, in the limit of large hidden layer size, and under theassumption that weight updates are dominated by noise, we can use a mean-ﬁeld approximation togain considerable insight into the learning dynamics as described below. Our derivation proceeds intwo steps.
First, we introduce four order parameters capturing the weight norms k W 1 k 2 F and k W 2 k 2 F ,error k W 2 W 1  A k 2 F , and their inner product tr [( W 2 W 1  A ) T W 2 W 1 ] , and approximately writedown the learning dynamics in terms of these order parameters (B.2).
Secondly, we reparameterize thelearning dynamics, which leads us to two-variable description of the dynamics (B.3).
The derivationof the mean-ﬁeld approximation is detailed in Appendix B.
Scalability of NP at the zero teacher noise limit.
We ﬁrst consider zero teacher noise (  t = 0 in Eq. 11).
In a network satisfying L h  L x  1 and L h  L y  1 , and using the Xavier-Glorotinitialization (see [37] and Eq. 69), for the ﬁrst O ( L 1 / 3 h ) iterations, the learning dynamics of NP isdescribed by the following coupled dynamics (see Appendix B.3),
[Equation]
(12) where ✏ ⌘ 1 L x L y k W 2 W 1  A k 2 F is the loss, a ⌘ 1 L x L 1 / 3 h k W 1 k 2 F is the normalized input weightnorm, and c o is a positive coefﬁcient proportional to the learning rate (see Eq. 76).
This simpledynamical system captures the learning dynamics of NP well (Fig.
1A; solid lines are simulations,and dotted lines are Eq. 12).
Because ✏  0 by deﬁnition, the norm a increases monotonically withtime.
By contrast, the loss ✏ either decreases or increases depending on whether a is smaller than2 /c
o or not.
When the learning rate ⌘ is ﬁxed to a small value, c o is also small.
Therefore, the weightnorm a increases slowly (purple and blue lines in Fig.
1B), and the weight norm threshold 2 /c
o isrelatively large (purple and blue vertical dotted lines).
Thus, the error converges to zero before theweight norm crosses the threshold.
In contrast, when the learning rate is larger than the critical value⌘ ⇤ (deﬁned by Eq. 83), the weight norm surpasses the threshold 2 /c
o before the error convergesto zero; at that point the error starts to go up again (green lines in Fig.
1B).
Similarly, if the initialweight norm is set to a large value, the error rises before converging to zero (green vs. blue linesin Fig.
1C).
This non-monotonic learning dynamics is in contrast to the linear regression setting inwhich the error either decreases or increases monotonically throughout learning.
When we optimize the learning rate, we ﬁnd that the minimum number of training steps required toreach a target error level, ✏ tg , from an initial error, ✏ o , under SGD and NP are given by
[Equation]
where  o , c o , ✏ tot are O (1) coefﬁcients, and a o and a tg are the weight norms under the initial error✏ o and the target error ✏ tg , respectively (see Appendix B.3).
Notably, the training time T np doesn’tdepend on the hidden layer size L h except for small-size effects on a o and  o (see Eq. 71), indicatingthat NP doesn’t slow down linearly with the number of perturbed nodes.
Indeed, simulations showthat the minimum training time is almost ﬂat when plotted against the hidden layer size (Figs.
2Aand S2A).
Moreover, the optimal learning rate in the simulations scales as ⌘ / L  1 / 3 h , as predicted(Fig. S2B).
In contrast, the training time increases linearly with the output layer size L y (Fig.
2B).
It may appear puzzling that training time saturates in the large hidden layer size limit (Fig.
2A and
S2A), considering that weight updates under NP are almost random in that limit (Eqs. 6 and 8).
However, as illustrated in Fig.
2C, while the noise in the weight updates (gray circles) increases the loss for some directions (red arrows), it decreases it for others (green arrows).
When the loss function is strongly convex, there are always more bad directions (red arrows) than good directions (green arrows), as illustrated in the left panel of Fig.
2C.
Therefore, and also because the loss associated with red arrows is larger than the gain with green arrows in a convex loss, noise is detrimental to learningon average.
Nevertheless, in overparameterized networks, there are a large number of directionsthat don’t affect the loss (because the Hessian of an over-parameterized neural network is typicallylow rank [34]); such directions are shown in the right panel of Fig.
2C. Since NP noise is mostlyhomogeneous, the more overparameterized the network is, the lower the volume of weights space inwhich the loss increases.
Thus, in an over-parameterized network, NP noise is not as detrimental as anticipated from convex optimization theory.
Instability of NP in the presence of teacher noise.
So far we have focused on the zero-noise limit(  t = 0 ).
We now investigate the effect of non-zero noise (  t 6 = 0 ) in the teacher network, whichintroduces a mismatch between the student and the teacher networks.
Our goal is to understand howthis mismatch affects the stability of the learning dynamics.
In this setting, the error ✏ decreases onlywhen (see Eq. 84a)
[Equation] [Equation]
Thus, even at a very small learning rate (proportional to c 0 ), eventually Eq.
14 will not be satisﬁed, andthe error will begin to increase.
This process is illustrated in Fig.
3A: the loss decreases until it crossesthe dotted lines, which are the nullclines, given by equality in Eq. 14: ✏ =
c o a  2 t / ( L x (2  c o a )) .
Because weight norm a is a monotonically increasing function of time, the right-hand side of Eq.14 is also monotonically increasing.
Thus, this instability occurs faster under a larger learning rate(green vs blue lines in Fig.
3A).
The black line in Fig.
3B shows the number of iterations it takes until the error starts to increase.
At alearning rate ⌘ > ⌘ ⇤ , the number of iterations scales roughly as ⌘  3 (right side of vertical line), butthe scaling becomes more moderate ( / ⌘  1 . 5 ) at a lower learning rate (left side).
Moreover, the erroreventually surpasses the initial error level, though that takes a long time when the learning rate isbelow the critical value (gray line).
Under SGD, the teacher noise affects the learning dynamics in aqualitatively different manner.
Up to a certain learning rate, SGD dynamics is always unstable fromthe initial update (ﬁlled circles in Fig.
3C), whereas below the critical learning rate, the error keepsdecreasing almost monotonically at least for 10 6 iterations (open circles).
Our analysis so far indicates that the instability of NP is triggered by the expansion of the weightnorm.
Thus, we expect the system to regain stability if we regularize the weight.
Indeed, if wekeep the norm of the incoming weights of each neuron constant, the error converges – unlike for theun-regularized network (Fig.
3D; solid vs. dotted lines).
However, because the weight normalizationinduces bias in the weight update, the error no longer goes to zero; instead, it saturates at a ﬁnitevalue.
The effect of this bias is particularly detrimental when the hidden layer size is large, becausestrong weight normalization is necessary for a larger network (yellow vs purple lines in Fig.
3D).
NP in nonlinear networks solving motor and visual tasks
Our analysis in deep linear networks revealed that NP scales well against over-parameterization,but the learning dynamics is susceptible to an instability because noise expands the weight normmonotonically.
Weight normalization suppresses this instability, but the normalization also introduces a bias that impairs scalability.
To see if these results generalize to more complex tasks that bettermimic typical credit assignment problems the brain faces, we analyze nonlinear networks solving the
SARCOS [23], MNIST [24] and CIFAR-10 tasks [25].
We ﬁrst applied NP to the SARCOS dataset, a kinematics dataset collected from a seven degree-offreedom SARCOS robotic arm [23, 38, 39].
The task is to predict the torques at the joints (sevendimensional output) given their positions, velocities and accelerations as inputs (21-dimensionalinput).
We chose this task because NP is previously proposed as a learning algorithm for motorsystems [17, 19, 20].
As expected, NP is indeed noisy, even with mini-batch implementation, resultinga low cosine similarity with SGD (Fig.
4A; here mini-batch size is 100).
In particular, in one-hiddenlayer networks the cosine similarity between the NP and SGD weight updates in the hidden layerfall off as 1 / p L h , while the similarity in the output layer stays roughly constant, as predicted by thetheory (Fig.
4A and Eq. 6).
Moreover, learning under NP is indeed susceptible to an instability, as predicted.
Under a smalllearning rate, the learning dynamics of NP is stable for more than 10 4 epochs (purple and blue linesin Fig.
4B), but as the learning rate increases, the error starts to diverge (green lines).
Because theweight norm increases monotonically, we expect NP dynamics to eventually become unstable evenunder a small learning rate, though it is difﬁcult to conﬁrm numerically due to the high computationalcost.
The learning-rate dependence of this instability is also similar to what we observed in deeplinear networks (compare Fig.
4C with Fig.
3B).
At a relatively high learning rate, the error, measuredby the test error, surpasses the initial error level immediately after the error starts to go up (right halfof Fig.
4C; black and gray lines are close to each other).
On the other hand, at a lower learning rate,the error increases slowly after it reaches the minimum (left half of Fig. 4C).
This instability is absentfrom SGD learning; in that case, learning is either stable or unstable from the beginning (Fig.
S3A).
Plotting the test error against the squared weight norm of the input layer k W 1 k 2 F , we see U-shapedcurves as in deep linear networks (Fig.
4D vs. Fig.
1B and 3A).
This result suggests that expansionof the input weight norm underlies the observed NP instability, even in nonlinear networks.
We next estimated the scalability of NP against the hidden layer width.
Under NP, the minimumtraining time required for reaching the target (here set to the test error, ✏ = 5 . 0 , which corresponds to R 2 = 0 . 986 , roughly one hundredth of the initial error) depends only weakly on the hidden layerwidth (Fig.
4E vs. Fig.
2A; see Fig.
S3B for the choice of the optimal learning rates).
However,we also observed that its learning efﬁciency is about one hundred times worse than SGD (orange vs. blue lines in Fig. 4E).
One key difference between NP and SGD, which may partially explainthis discrepancy, is that NP is a reinforcement learning algorithm that doesn’t use a vector-valuedsupervised signal, unlike SGD.
To examine its effect, we consider the following reinforcementlearning algorithm [40].
By adding perturbations only to the output layer, the gradient with respect tothe last layer is estimated as
[Equation]
By back-propagating the gradient, e g K , the network can approximately perform SGD based on thereinforcement signals ` ( e x K ) and ` ( x K ) .
The minimum training time under this reinforcementlearning algorithm is about three times longer than SGD (solid green versus blue lines in Fig.
4E),though much better than NP. Moreover, assuming that the perturbation originated from intrinsic noisein the system, we added perturbations to the hidden layer as well.
This additional perturbation impairsthe performance of the reinforcement learning algorithm (dashed vs solid green lines).
However, NPis still ﬁve to ten times slower than this noisy reinforcement learning (orange vs. dashed green lines).
Our analysis of deep linear networks indicates that weight normalization prevents the instability in NP learning dynamics.
Indeed, in the presence of neuron-wise weight normalization (see Appendix C.1), learning dynamics becomes stable even when the hidden layer size is large (dotted vs solid linesin Fig.
4F), but this prevents the error from converging to the target level ( ✏ = 5 . 0 ).
At a low learning rate, NP and SGD show similar error curves (Fig. S3C), but it doesn’t mean that thetwo algorithms learn the same representation.
We found that the linear dimensionality of the hiddenlayer representation, measured by the participation ratio of PCA eigenvalues (see Appendix C.1),was lower under NP than SGD, even at a low learning rate (Fig. S3D).
This result implies that it maybe possible to distinguish the two learning rules, just by observing activity in the hidden layer.
NP exhibits similar learning dynamics when applied to MNIST (Figs. 5 and S4).
Under a largelearning rate, learning becomes unstable before reaching high performance (green lines in Fig.
5A).
The minimum training time depends only sub-linearly on the hidden layer width (orange line in
Fig.
5B).
Compared to SGD, NP learning is one hundred to one thousand times slower, but theperformance deﬁcit against reinforcement learning is smaller (Fig. 5B).
Although the analysis ofthe depth dependence is complicated even in the linear setting, we empirically observed that theminimum learning time of NP scales supra-linearly with respect to depth (orange line in Fig.
5C).
However, this effect can be ameliorated by introducing weight normalization (red line).
Lastly, weapplied NP to a convolutional network learning CIFAR-10, and found that weight regularizationstabilizes NP learning dynamics (Fig. S5), supporting generalizability of our results.
Discussion
We analyzed the learning dynamics of NP in deep neural networks and revealed its limitationand potential.
This study provides several insights into biologically plausible credit assignmentmechanisms.
Firstly, considering biological implementation, it is often useful to measure the performance by theminimum number of iterations required to reach a target – which could, for instance, be human-levelperformance.
This is because an algorithm is biologically implausible if it takes an inordinately longtime to reach good performance.
For instance, under a sufﬁciently small learning rate, NP eventuallyachieves good performance, since it is unbiased (§3).
However, convergence is signiﬁcantly slowerthan SGD (§4 and §5), which limits its utility, especially in the presence of supervising signals.
Though we found the minimum training time of NP is relatively robust against over-parameterization,its scalability against complex supervised learning tasks is clearly limited due to this slowness.
In the brain, learning often needs to rely on scalar-valued reward signals rather than supervisedsignals.
Although NP is much slower than SGD, its performance deﬁcit is smaller, when compared toa reinforcement learning rule using the error backpropagation, though NP is still more than one orderof magnitude slower (Figs.
4E and 5B).
Thus, NP might be utilized in a noisy reinforcement learningproblem, such as motor learning, as a building block of learning mechanism.
Our work highlights the importance of theoretical investigations in an idealized scenario, such as adeep linear network, not only for understanding how the proposed learning algorithm works, but alsofor obtaining insight into what kind of regularization is needed for making a learning algorithm worksefﬁciently.
We revealed that neuron-wise weight normalization stabilizes the learning dynamics.
Bycontrast, a constant-rate weight decay doesn’t work robustly, because the change in the weight normis not constant (black vs colored lines in Fig.
S4F).
Finally, it is important to analyze the representation in the hidden layers in the learned circuit forobtaining an experimentally testable prediction [41, 42].
Although the representation in the outputlayer is almost invariant among any successful learning rules by deﬁnition, hidden layer representationmay vary depending on the learning rule, particularly in over-parameterized neural networks.
Here,we showed that the linear dimensionality of the hidden layer representation is different between NPand SGD, even when they achieve similar performance (Figs.
S3CD and S4E).
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
This is aneuroscience-oriented theoretical work with limited direct societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See AppendixB.
(b) Did you include complete proofs of all theoretical results?
[Yes] We included thedetailed derivation of the theoretical results in Appendices A and B.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Ap-
pendix C and anonymized repository: https://anonymous.4open.science/r/
node_perturbation_E2F9 .
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they
were chosen)?
[Yes] See Appendix C.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] We depicted variability over random seeds in Figs.
1BC, 4BF, and 5A. For the rest of ﬁgures, we used the average trajectory over ten (linear networks) or ﬁve (nonlinear networks) random seeds, to capture the average behavior.
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C.1.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]