Original: A Theoretical Framework for Inference Learning
Revised: A Theoretical Framework for Inference Learning

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Backpropagation (BP) is the most successful and widely used algorithm in deeplearning.
Revised: Backpropagation (BP) is the most successful and widely used algorithm in deeplearning.

Original: However, the computations required by BP are challenging to reconcilewith known neurobiology.
Revised: However, the computations required by BP are challenging to reconcilewith known neurobiology.

Original: This difficulty has stimulated interest in more biologically plausible alternatives to BP.
Revised: This difficulty has stimulated interest in more biologically plausible alternatives to BP.

Original: One such algorithm is the inference learningalgorithm (IL).
Revised: One such algorithm is the inference learningalgorithm (IL).

Original: IL has close connections toneurobiological models of corticalfunction and has achieved equal performance to BP on supervised learning andauto-associative tasks.
Revised: IL has close connections to models of cortical circuits and hasachieved equal performance to BP on supervised and auto-associative tasks.

Original: In contrast to BP, however, the mathematical foundationsof IL are not well-understood.
Revised: In contrast to BP, however, the mathematical foundations of IL are not well-understood.

Original: Here, we develop a novel theoretical frameworkfor IL.
Revised: Here, we develop a novel theoretical framework for IL.

Original: Our main result is that IL closely approximates an optimization methodknown as implicit stochastic gradient descent (implicit SGD), which is distinct fromthe explicit SGD implemented by BP.
Revised: Our main result is that ILclosely approximates an optimization method known as implicit stochastic gradientdescent (implicit SGD), which is distinct from the explicit SGD implemented by BP.

Original: Our results further show how the standardimplementation of IL can be altered to better approximate implicit SGD.
Revised: Our results further show how the standard implementation of IL can be altered tobetter approximate implicit SGD.

Original: Our novelimplementation considerably improves the stability of IL across learning rates,which is consistent with our theory, as a key property of implicit SGD is its stability.
Revised: Our novel implementation considerably improvesthe stability of IL across learning rates, which is consistent with our theory, as a keyproperty of implicit SGD is its stability.

Original: We provide extensive simulation results that further support our theoretical interpretations and also demonstrate IL achieves quicker convergence when trained withsmall mini-batches while matching the performance of BP for large mini-batches.
Revised: We provide extensive simulation resultsthat further support our theoretical interpretations and find IL achieves quickerconvergence when trained with mini-batch size one while performing competitivelywith BP for larger mini-batches when combined with Adam.

Original:  Introduction
Revised:  Introduction

Original: Backpropagation (BP) [37], the most successful and ubiquitously used learning algorithm in deeplearning has long been criticized as being incompatible with known neurobiology [10, 23].
Revised: Backpropagation (BP) [34], the most successful and ubiquitously used learning algorithm in deeplearning, has long been criticized as being incompatible with known neurobiology [11, 22].

Revised: Forexample, unlike the highly interconnected, recurrent circuits in the brain, BP performs credit assignment through a separate feedback stream that does not alter feedforward (FF) signals [11, 22].

Revised: Relatedly, standard implementations of BP are largely incompatible with energy efficient, neuromorphic hardware, due, in part, to BP’s non-local gradient computations.

Original: Thiscriticism has inspired researchers to search for alternatives to BP that may better fit biologicalconstraints.
Revised: These concerns have motivatedneuroscientists and engineers to search for alternatives to BP that use local learning rules that aremore compatible with biology.

Original: One alternative to BP is the inference learning algorithm (IL)
Revised: One alternative to BP is the inference learning algorithm (IL)

Revised: IL is the standardlearning algorithm used to train predictive coding (PC) models of biological neural circuits (e.g.[33, 14, 40, 4, 1, 8]).

Revised: Within neuroscience, PC models have grown in popularity, where some evenpropose PC may be a canonical biological circuit [16, 15, 4].

Original: IL is anenergy based algorithm [41, 47], which first minimizes an energy function, known as free energy,w.r.t.
Revised: IL works by first minimizing an energyfunction, known as free energy, w.r.t. neuron activities using the recurrent computations involved inpredictive coding.

Original: neuron activities, then at convergence updates weights to further minimize energy.
Revised: At convergence, IL updates weights to further minimize energy.

Original: Interest in
Revised: Unlike standard

Original: IL has grown in recent years because IL arguably better fits biological constraints than BP, whilematching BP’s performance on many tasks. For example, BP is sometimes said to compute weightupdates using information non-local to the synapse, whereas synapses in the brain depend on localinformation having to do with theactivity of pre-synaptic and post-synaptic neurons [10, 23]. IL, onthe other hand, uses local learning rules that are similar to Hebbian models of synaptic plasticity inthe brain (e.g., [46, 47]).
Revised: BP, IL uses local learning rules and recurrent circuits instead of a separate feedback stream to performcredit assignment. In addition to evidence the brain does predictive coding (and thus may do IL),recent empirical work has found evidence the brain may learn by first inferring target neural activitiesthrough recurrent processing then updating synapses to consolidate those neural activities, just like

Original: Another criticism is that error feedback propagated by BP does not alterfeed forward (FF) activities, which is hard to reconcile with the highly interconnected circuitry of FFand feedback signals in the brain [22, 25].
Revised: IL [39].

Original: Feedback used in IL, on the other hand, necessarily alters FF activity in order to minimize free energy. Further, IL haslong been used to train predictive coding(PC) models of circuits in the cortex (e.g. [36, 13, 43, 4, 1, 8]). PC and IL have even been proposedas canonical computations used throughout the brain [15, 14, 4].
Revised: Recently, IL has been used to train networks for machine learning tasks.

Original: More recently, IL has achievedcomparable accuracy to BP  on supervised learning tasks with small and medium scale data-sets (e.g,[46, 2, 39], see also below).
Revised: IL, for example, achieved comparable classification and recall accuracy to BP trained networks on supervised andauto-associative tasks (e.g, [44, 2, 37, 36], see also below).

Original: IL has also been used to train PC networks to perform auto-associativememory tasks, and  on some tests has achieved a higher recall accuracy than both modern Hopfieldnetworks [17, 35] and denoising auto-encoders trained with BP [40, 39]. IL’sbiological connections motivates further analysis into its optimization properties given thatsuch an analysis may lead to a better understanding of the principles behind synaptic plasticity andoptimization in the brain. Additionally, IL’s competitive performance with BP further motivates aninvestigation into the question of why IL is competitive with BP.
Revised: Despite IL’s wide use in biological models and success on machine learning tasks, a rigorous formalcharacterization of IL’s optimization properties and their differences and similarities with BP islacking.

Original: Similar questions, such as How does IL behave differently than BP? What advantages/disadvantages does IL have as compared to BP?can also aid in understanding which applications, if any, are better suited for IL rather than BP andvice versa. The optimization properties of IL and their relations to BP, however, are still not fullyunderstood.
Revised: Providing such a characterization could yield several useful contributions: 1) A basis for anovel theory of how optimization and credit assignment may work in the brain, 2) a formal basis forimproving, developing, and applying IL to machine learning tasks and neuromorphic hardware.

Original: This gap in our understanding is the main motivation for this paper.

Original: In this paper, we develop a novel theoretical framework for IL that describes its optimizationproperties and their similarities and differences from BP.
Revised: In this paper, we develop a novel theoretical framework for IL that describes its optimizationproperties and their similarities and differences from BP.

Original: More specifically, 1) we expand beyondcurrent literature by deriving a general form of the IL algorithm, which we call Generalized IL (G-IL).2)
Revised: More specifically, 1) we expand beyondcurrent literature by deriving a general form of the IL algorithm, which we call Generalized IL (G-IL).2)

Original: We demonstrate our main result, which is that G-IL closely approximates an optimization methodknown as implicit stochastic gradient descent (implicit SGD).
Revised: We demonstrate our main result, which is that G-IL closely approximates an optimization methodknown as implicit stochastic gradient descent (implicit SGD), where the approximation is closestin the case of mini-batch size 1.

Original: Implicit SGD is distinct from explicit 
Revised: Implicit SGD is distinct from explicit SGD (the standard form of

Original: SGD(the standard form of SGD used in machine learning and the sort executed by BP).
Revised: SGD used in machine learning and the sort executed by BP).

Original: 3) We providetheoretical guarantees concerning IL’s stability and a connection to Gauss-Newton optimization and
Revised: We provide theoretical guaranteesconcerning IL’s stability and a connection to Gauss-Newton optimization.

Original: BP.

Original: We identify a learning rule and variable settings needed in order for IL to equal implicit SGDand use this result to develop a novel implementation of IL called IL-prox.
Revised: We identify a learningrule and variable settings needed in order for IL to equal implicit SGD and use this result to developa novel implementation of IL called IL-prox.

Original: Finally, we presentextensive simulation results that support our theoretical findings, and for the first time show certainperformance advantages of IL over BP, such as better stability across learning rates and improvedconvergence speed with biologically realistic streaming data and small mini-batches.
Revised: Finally, we present extensive simulation results thatsupport our theoretical findings, and for the first time show certain performance advantages of IL over BP, such as better stability across learning rates and improved convergence speed with biologicallyrealistic online learning scenarios.

Original: These resultscollectively suggest that, not only does IL better fit biological constraints but  is mathematicallyjustified and has performance advantages over BP in more biologically realistic training scenarios.
Revised: These results collectively suggest that, not only does IL better fitbiological constraints, but it is also mathematically justified and has performance advantages over BPin more biologically realistic training scenarios.

Original:  Related Works
Revised:  Related Works

Original: Whittington and Bogacz [46] proved that parameter updates performed by the IL algorithm approachthose of BP as the global loss approaches zero and optimized activities approach feed-forwardactivities.
Revised: Whittington and Bogacz [44] proved that parameter updates performed by the IL algorithm approachthose of BP as the global loss approaches zero and optimized activities approach feed-forwardactivities.

Original: However, this same proof implies that non-zero IL updates essentially never equal thoseof BP.
Revised: However, this same proof implies that non-zero IL updates essentially never equal thoseof BP.

Original: Additionally, this analysis leaves open the question of how IL is able to minimize the loss ina stable manner early in training when the loss is large and IL updates diverge significantly fromthose of BP.
Revised: Additionally, this analysis leaves open the question of how IL is able to minimize the loss ina stable manner early in training when the loss is large and IL updates diverge significantly fromthose of BP.

Original: Other work has shown IL is formally related to variational expectation maximization[28, 24], which is a learning algorithm with convergence guarantees [30].
Revised: Other work has shown IL is formally related to variational expectation maximization[26, 23], which is a learning algorithm with convergence guarantees [28].

Original: This analysis still leavesopen the question of how IL relates to BP and the broader framework of gradient-based methods thatare the backbone of deep learning.
Revised: This analysis still leavesopen the question of how IL relates to BP and the broader framework of gradient-based methods thatare the backbone of deep learning.

Original: The focus of the current work is to develop insights into theseopen questions.
Revised: The focus of the current work is to develop insights into theseopen questions.

Original: More recently, several works have altered IL to more closely approximate SGD and
Revised: More recently, several works have altered IL to more closely approximate SGD and

Original: BP [29, 42].
Revised: BP [27, 38].

Original: These variants significantly change the original IL algorithm, and some the alterationsseem hard to reconcile with neurobiology [29, 42].
Revised: These variants significantly change the original IL algorithm, and some the alterationsseem hard to reconcile with neurobiology [27, 38].

Original: Alternatively, instead of altering IL to better fit
Revised: Alternatively, instead of altering IL to better fit

Original: SGD and BP, we show the standard implementation of IL already closely approximates a workingoptimization method known as implicit SGD, and we show the variable settings under which thestandard implementation of IL better approximates implicit SGD.
Revised: SGD and BP, we show the standard implementation of IL already closely approximates a workingoptimization method known as implicit SGD, and we show the variable settings under which thestandard implementation of IL better approximates implicit SGD.

Original: Algorithms similar to IL have been proposed.
Revised: Algorithms similar to IL have been proposed.

Original: For example, the alternating minimization algorithm [9],which was developed independently of the predictive coding and IL framework from neuroscience,updates weights to minimize local prediction errors similar to IL (see equation 3 below).
Revised: For example, the alternating minimization algorithm[10], which was developed independently of the predictive coding and IL framework from neuroscience, updates weights to minimize local prediction errors similar to IL (see equation 3 below).

Original: However,[9] do not connect their algorithm to implicit SGD.
Revised: However, [10] do not connect their algorithm to implicit SGD.

Original: A more recent variant of this algorithm by Qiaoet al.
Revised: A more recent variant of this algorithmby Qiao et al.

Original: [34] show some formal links to proximal operators, an optimization process related to implicit SGD [33] (see appendix A.1).
Revised: [32] show some formal links to proximal operators, an optimization process relatedto implicit SGD [31] (see appendix B.1).

Original: However, they do not interpret the weight updates as performing theproximal update or implicit SGD as we do here with IL.
Revised: However, they do not interpret the weight updates asperforming the proximal update or implicit SGD as we do here with IL.

Original: Proximal operators have also recently beenincorporated into learning algorithms for deep networks.
Revised: Proximal operators have alsorecently been incorporated into learning algorithms for deep networks.

Original: Frerix et al.
Revised: Frerix et al.

Original: [12] developed a BP, proximalalgorithm hybrid.
Revised: [13] developed a BP, proximal algorithm hybrid.

Original: Lau et al.
Revised: Lau et al.

Original: [19] developed an algorithm that combined and algorithm known asblock coordinate descent with the proximal operator.
Revised: [18] developed an algorithm that combined and algorithmknown as block coordinate descent with the proximal operator.

Original: Other works use proximal operators to reducenoise or perform other tasks, then BP is used to to update at convergence [26, 49].
Revised: Other works use proximal operators toreduce noise or perform other tasks, then BP is used to to update at convergence [24, 46].

Original: All of thesealgorithms, however, are distinct from IL, as they use BP or other non-local gradient information tocompute weight updates or local targets.
Revised: All of thesealgorithms, however, are distinct from IL, as they use BP or other non-local gradient information tocompute weight updates or local targets.

Original: Finally, IL has similarities to target propagation algorithms, in which local target activities arecomputed and used to compute local error gradients to update weights.
Revised: Finally, IL has similarities to target propagation algorithms, in which local target activities arecomputed and used to compute local error gradients to update weights.

Original: Target propagation [5] anddifference target propagation [21] utilize approximate Gauss-Newton updates on activities to createlocal targets [27, 6], while other algorithms compute targets using gradient updates on activities
Revised: Target propagation [5] anddifference target propagation [20] utilize approximate Gauss-Newton updates on activities to create local targets [25, 6], while other algorithms compute targets using gradient updates on activities[19, 30, 3, 17].

Original: These algorithms have important differences from IL, e.g., they use a different learningrules than IL, which we discuss further below.
Revised: These algorithms have important differences from IL, e.g., they use a different learningrules than IL, which we discuss further below.

Original: In sum, previous works have not developed the samemathematical interpretation of IL that we do here and have not developed the same mathematicaldescriptions of the differences between IL and BP.
Revised: In sum, previous works have not developed the samemathematical interpretation of IL that we do here and have not developed the same mathematicaldescriptions of the differences between IL and BP.

Original: Further, none of the works mentioned aboveproduced the same empirical findings we do in our simulations below.
Revised: Further, none of the works mentioned aboveproduced the same empirical findings we do in our simulations below.

Original:  Background and Notation
Revised:  Background and Notation

Original: 3.1 Notation

Original: [Table]
Revised: [Table]

Original: Notation describing a multi-layered FF network (MLP) is summarized in table 1.
Revised: Notation describing a multi-layered FF network (MLP) is summarized in table 1.

Original: To simplifynotation, we assume bias is stored in an extra column on each weight matrix W n and we assume a 1is concatenated to the end of the pre-synaptic activity vector ( h n or ˆ h n depending).
Revised: To simplifynotation, we assume bias is stored in an extra column on each weight matrix W n and we assume a 1is concatenated to the end of the pre-synaptic activity vector ( h n or ˆ h n depending).

Original: All the followingresults should hold with and without biases.
Revised: All the followingresults should hold with and without biases.

Original: 3.2 Generalized Backpropagation
Revised: 3.2 Generalized Backpropagation

Original: Consider a set of parameters θ ( b ) at training iteration b with global loss function L .
Revised: Consider a set of parameters θ ( b ) at training iteration b with global loss function L .

Original: An explicit SGDiteration subtracts the gradient of the loss from the parameters: θ ( b +1)
Revised: An explicit SGDiteration subtracts the gradient of the loss from the parameters: θ ( b +1)

Original: = θ ( b ) − α ∂L∂θ ( b ) , with stepsize α . BP [38] is a common algorithm for implementing SGD in deep networks.
Revised: = θ ( b ) − α ∂L∂θ ( b ) , with stepsize α . BP [35] is a common algorithm for implementing SGD in deep networks.

Original: Here, to allowfor easier comparison to IL, we consider an alternative implementation of SGD in deep networksthat involves computing local target activities at hidden and output layers, similar to [20], [3], [32].
Revised: Here, to allowfor easier comparison to IL, we consider an alternative implementation of SGD in deep networksthat involves computing local target activities at hidden and output layers, similar to [19], [3], [30].

Original: Specifically, the local target at layer n , ˆ h n , is computed as follows
Revised: Specifically, the local target at layer n , ˆ h n , is computed as follows

Original: [Equation]
Revised: [Equation]

Original: where local loss l n +1 = 12 ∥
Revised: where local loss l n +1 = 12 ∥

Original: ˆ h n +1 − h n +1 ∥ 

Revised: ˆ h n +1 − h n +1 ∥

Original: = 12 ∥
Revised: = 12 ∥

Original: e n +1 ∥ 2 .
Revised: e n +1 ∥ 2 .

Original: The target at the output layer ˆ h N =h N − ∂L∂h N .
Revised: The target at the output layer ˆ h N =h N − ∂L∂h N .

Original: Notice that e n +

Revised: Notice that e n +

Original: =
Revised: =

Original: ˆ h n +1 − h n +

Revised: ˆ h n +1 − h n +

Original: =
Revised: =

Original: h n +1 + ∂l n +2 ∂h n +1 − h n +

Revised: h n +1 + ∂l n +2 ∂h n +1 − h n +

Original: =
Revised: =

Original: ∂l n +2 ∂h n +1 = − ∂L ∂h n +1 .
Revised: ∂l n +2 ∂h n +1 = − ∂L ∂h n +1 .

Original: As noted in the above equation, this local target update is equivalent to subtracting the global lossgradient from the FF activity h n (for proof see appendix [3]).
Revised: As noted in the above equation, this local target update is equivalent to subtracting the global lossgradient from the FF activity h n (for proof see appendix [3]).

Original: To update weight W n one uses thegradient of the local loss:
Revised: To update weight W n one uses thegradient of the local loss:

Original: [Equation]
Revised: [Equation]

Original: This update uses only local information and is equivalent to subtracting the global loss gradient fromthe weights (see [3]).
Revised: This update uses only local information and is equivalent to subtracting the global loss gradient fromthe weights (see [3]).

Original: This local target formulation of SGD is similar to other target propagation (TP)[5, 21] algorithms, which use a similar learning rule but compute targets slightly differently.
Revised: This local target formulation of SGD is similar to other target propagation (TP)[5, 20] algorithms, which use a similar learning rule but compute targets slightly differently.

Original: Forexample, a well-known variant of TP is difference target propagation (DTP) [21], which computeslocal targets by approximating a Gauss-Newton update on activities rather than gradient updates[27, 6]:
Revised: Forexample, a well-known variant of TP is difference target propagation (DTP) [20], which computeslocal targets by approximating a Gauss-Newton update on activities rather than gradient updates[25, 6]:

Original: ˆ h n ≈ h n − J + N,n e N , where J + N,n is the pseudo-inverse of the Jacobian of the forward networkfrom layer n to N and e N =
Revised: ˆ h n ≈ h n − J + N,n e N , where J + N,n is the pseudo-inverse of the Jacobian of the forward networkfrom layer n to N and e N =

Original: y − h N .
Revised: y − h N .

Original: Gauss-Newton updates approximate (are within 90 ◦ ) of SGDupdates [27], and recent attempts to improve DTP generally do so by making DTP updates moresimilar to SGD updates (e.g. [27, 11]).
Revised: Gauss-Newton updates approximate (are within 90 ◦ ) of SGDupdates [25], and recent attempts to improve DTP generally do so by making DTP updates moresimilar to SGD updates (e.g. [25, 12]).

Original: Additionally, variants of BP algorithms like random feedbackalignment [22] and sign symmetric feedback alignment [48] approximate SGD.
Revised: Additionally, variants of BP algorithms like random feedbackalignment [21] and sign symmetric feedback alignment [45] approximate SGD.

Original: Since all of thesemethods approximate the (explicit) SGD update over weights, we consolidate them all under theheading of generalized BP (G-BP).
Revised: Since all of thesemethods approximate the (explicit) SGD update over weights, we consolidate them all under theheading of generalized BP (G-BP).

Original: The target-based formulation of G-BP is shown in algorithm 1.
Revised: The target-based formulation of G-BP is shown in algorithm 1.

Original: 3.3 Generalized Inference Learning
Revised: 3.3 Generalized Inference Learning

Original: We now present a general description of the IL algorithm, which we call Generalized Inference
Revised: We now present a general description of the IL algorithm, which we call Generalized Inference

Original: Learning (G-IL).
Revised: Learning (G-IL).

Original: A summary of the algorithm is presented in algorithm 2.
Revised: A summary of the algorithm is presented in algorithm 2.

Original: At each layer n of an MLP,
Revised: At each layer n of an MLP,

Original: G-IL stores two variables: ˆ h n and p n .
Revised: G-IL stores two variables: ˆ h n and p n .

Original: At initialization these variables are set to feed-forward activityvalues:
Revised: At initialization these variables are set to feed-forward activityvalues:

Original: ˆ h n = p n = h n .
Revised: ˆ h n = p n = h n .

Original: Then ˆ h n are altered over time to minimize the energy function F :
Revised: Then ˆ h n are altered over time to minimize the energy function F :

Original: [Equation]
Revised: [Equation]

Original: where L is the global loss,
Revised: where L is the global loss,

Original: 12 ∥
Revised: 12 ∥

Original: ˆ h n − p n ∥ 2 are local losses, γ decayn 12 ∥
Revised: ˆ h n − p n ∥ 2 are local losses, γ decayn 12 ∥

Original: ˆ h n ∥ 2 is an optional regularization term, and γ are positive scalar weighting terms.
Revised: ˆ h n ∥ 2 is an optional regularization term, and γ are positive scalar weighting terms.

Original: The prediction p n is computed asp n = W n − 1 f (ˆ h n − 1 ) .
Revised: The prediction p n is computed asp n = W n − 1 f (ˆ h n − 1 ) .

Original: (Note p n may also be computed as p n = f ( W n − 1 ˆ h n − 1 ) .
Revised: (Note p n may also be computed as p n = f ( W n − 1 ˆ h n − 1 ) .

Original: Our theoreticalresults use the first formulation, but we find little difference in performance between the two inpractice.
Revised: Our theoreticalresults use the first formulation, but we find little difference in performance between the two inpractice.

Original: See simulations below.)
Revised: See simulations below.)

Original: G-IL first minimizes F w.r.t.
Revised: G-IL first minimizes F w.r.t.

Original: to activities ˆ h (inference phase).
Revised: to activities ˆ h (inference phase).

Original: Whileoptimizing ˆ h , predictions also change since p n +1 = W n f (ˆ h n ) .
Revised: Whileoptimizing ˆ h , predictions also change since p n +1 = W n f (ˆ h n ) .

Original: After argmin ˆ h F is approximated,
Revised: After argmin ˆ h F is approximated,

Original: F is again minimized, now w.r.t.
Revised: F is again minimized, now w.r.t.

Original: the weight, i.e. argmin W F .
Revised: the weight, i.e. argmin W F .

Original: Typical IL algorithms use SGD toperform the inference phase and either the LMS rule (below) or Adam optimizers to update weights(e.g.
Revised: Typical IL algorithms use SGD toperform the inference phase and either the LMS rule (below) or Adam optimizers to update weights(e.g.

Original: G-IL, on the other hand, is general w.r.t.
Revised: G-IL, on the other hand, is general w.r.t.

Original: the optimization processes that approximateargmin W F and argmin ˆ h F .
Revised: the optimization processes that approximateargmin W F and argmin ˆ h F .

Original: We consider two weight updates in our analysis and simulations below.
Revised: We consider two weight updates in our analysis and simulations below.

Original: First, is a local error gradient update, which is equivalent to the least-mean squares (LMS) rule:
Revised: First, is a local error gradient update, which is equivalent to the least-mean squares (LMS) rule:

Original: [Equation]
Revised: [Equation]

Original: where e n +

Revised: where e n +

Original: =
Revised: =

Original: ˆ h n +1 − p n +1 and local loss l n +1 = 

Revised: ˆ h n +1 − p n +1 and local loss l n +1 =

Original: ∥
Revised: ∥

Original: e n +1 ∥ 2 .
Revised: e n +1 ∥ 2 .

Original: The LMS rule does not solveargmin W F but only approximates it.
Revised: The LMS rule does not solveargmin W F but only approximates it.

Original: The next rule, which is equivalent to the normalized leastmean squared rule (NLMS) [31] is
Revised: The next rule, which is equivalent to the normalized leastmean squared rule (NLMS) [29] is

Original: [Equation]
Revised: [Equation]

Original: The NLMS rule performs a gradient update but with an inverse squared ℓ 2 norm of the pre-synapticactivities as its learning rate instead of a hyper-parameter.
Revised: The NLMS rule performs a gradient update but with an inverse squared ℓ 2 norm of the pre-synapticactivities as its learning rate instead of a hyper-parameter.

Original: The NLMS rule is the minimum-normsolution to argmin W F in the case where we use mini-batch size 1 (see proposition A.1).
Revised: The NLMS rule is the minimum-normsolution to argmin W F in the case where we use mini-batch size 1 (see proposition B.1).

Original: The LMSrule closely approximates the NLMS update in the sense the two are proportional.
Revised: The LMSrule closely approximates the NLMS update in the sense the two are proportional.

Original: Comparing theserules to the update rule for G-BP (equation 2)
Revised: Comparing theserules to the update rule for G-BP (equation 2)

Original: we see the main difference is the pre-synaptic termused in the rule: BP uses FF activity h n , while IL uses optimized/target activity ˆ h n .
Revised: we see the main difference is the pre-synaptic termused in the rule: BP uses FF activity h n , while IL uses optimized/target activity ˆ h n .

Original: This differenceleads to significantly different stability properties between the algorithms, as we note below.
Revised: This differenceleads to significantly different stability properties between the algorithms, as we note below.

Original: Algorithm 1: Generalized BP begin //
Revised: Algorithm 1: Generalized BP begin //

Original: Feedforward Pass h 0 ← x ( b ) for n = 0 to N − 1 do h n +1 ← W n f ( h n ) end ˆ h N ← y ( b ) //
Revised: Feedforward Pass h 0 ← x ( b ) for n = 0 to N − 1 do h n +1 ← W n f ( h n ) end ˆ h N ← y ( b ) //

Original: Compute Local Targets for n = 1 to N do ˆ h n ≈ h n − ∂L ∂h n end //
Revised: Compute Local Targets for n = 1 to N do ˆ h n ≈ h n − ∂L ∂h n end //

Original: Update Weight Matrices
Revised: Update Weight Matrices

Original: Eqn.end
Revised: Eqn.end

Original: Algorithm 2: Generalized IL begin // Feedforward Pass
Revised: Algorithm 2: Generalized IL begin // Feedforward Pass

Original: ˆ h 0 ← x ( b ) for n = 0 to N − 1 do p n +1 , ˆ h n +1 ← W n f ( h n ) end //
Revised: ˆ h 0 ← x ( b ) for n = 0 to N − 1 do p n +1 , ˆ h n +1 ← W n f ( h n ) end //

Original: Compute Local Targets
Revised: Compute Local Targets

Original: [Equation]
Revised: [Equation]

Original: argmin
Revised: argmin

Original: [Equation]
Revised: [Equation]

Original: // Update Weight Matrices
Revised: // Update Weight Matrices

Original: [Equation]
Revised: [Equation]

Original: end
Revised: end

Original: W W
Revised: W W

Original: W 2 h 0 Initialize
Revised: W 2 h 0 Initialize

Original: W
Revised: W

Original: W W
Revised: W W

Original: W
Revised: W

Original: W 2 P 2 W
Revised: W 2 P 2 W

Original: Energy
Revised: Energy

Original: Minimization
Revised: Minimization

Original: Weight
Revised: Weight

Original: Update
Revised: Update

Original: W 0 *
Revised: W 0 *

Original: W 1 *
Revised: W 1 *

Original: W 2 *
Revised: W 2 *

Original: L(h 3 , y)
Revised: L(h 3 , y)

Original: y h
Revised: y h

Original: L(h 3 , y) L(h 3 , y)
Revised: L(h 3 , y) L(h 3 , y)

Original: L(h 3 , y)
Revised: L(h 3 , y)

Original:  Theoretical Results
Revised:  Theoretical Results

Revised: [Figure]

Original: 4.1 Implicit Gradient Descent
Revised: 4.1 Implicit Gradient Descent

Original: Let the set of MLP weight parameters θ ( b ) =
Revised: Let the set of MLP weight parameters θ ( b ) =

Original: [ W ( b ) 0 , ...
Revised: [ W ( b ) 0 , ...

Original: W ( b ) N − 1 ] , input data x ( b ) , and output target y ( b ) ,where b is the current training iteration.
Revised: W ( b ) N − 1 ] , input data x ( b ) , and output target y ( b ) ,where b is the current training iteration.

Original: The explicit SGD update uses the loss gradient produced bythe current parameters:
Revised: The explicit SGD update uses the loss gradient produced bythe current parameters:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where α is step size and L is the loss measure.
Revised: where α is step size and L is the loss measure.

Original: This update is explicit because the gradient canbe readily computed given ( θ ( b ) , x ( b ) , y ( b ) ) .
Revised: This update is explicit because the gradient canbe readily computed given ( θ ( b ) , x ( b ) , y ( b ) ) .

Original: The implicit SGD update uses the loss gradient of theparameters at the next training iteration:
Revised: The implicit SGD update uses the loss gradient of theparameters at the next training iteration:

Original: [Equation]
Revised: [Equation]

Original: This is an implicit update because θ ( b +1) shows up on both sides of the equation.
Revised: This is an implicit update because θ ( b +1) shows up on both sides of the equation.

Original: Unlike explicit
Revised: Unlike explicit

Original: SGD, θ ( b +1) cannot be readily computed using available quantities.
Revised: SGD, θ ( b +1) cannot be readily computed using available quantities.

Original: However, the implicit gradient isequivalent to the solution of the following optimization problem (see appendix equation 12):
Revised: However, the implicit gradient isequivalent to the solution of the following optimization problem (see appendix equation 13):

Original: [Equation]
Revised: [Equation]

Original: This update is equivalent to the proximal operator/update [33].
Revised: This update is equivalent to the proximal operator/update [31].

Original: This proximal update changesparameters θ ( b ) in a way that minimizes the loss L and the magnitude of the update, which helps keepθ ( b +1) in the proximity of θ ( b ) .
Revised: This proximal update changesparameters θ ( b ) in a way that minimizes the loss L and the magnitude of the update, which helps keepθ ( b +1) in the proximity of θ ( b ) .

Original: (For more details on the proximal update and its relation to implicit
Revised: (For more details on the proximal update and its relation to implicit

Original: SGD see appendix A.1).
Revised: SGD see appendix B.1).

Original: 4.2 Main Results
Revised: 4.2 Main Results

Original: We now present our main result, which is to show G-IL is equivalent to implicit SGD in certain limitsthat are approximated well in practice.
Revised: We now present our main result, which is to show G-IL is equivalent to implicit SGD in certain limitsthat are approximated well in practice.

Original: We focus on the case where single data-points (mini-batch size1) are presented each training iteration.
Revised: We focus on the case where single data-points (mini-batch size1) are presented each training iteration.

Original: This scenario is more biologically realistic than mini-batchingand is the case where IL best approximates implicit SGD.
Revised: This scenario is more biologically realistic than mini-batchingand is the case where IL best approximates implicit SGD.

Original: We discuss the case of mini-batching inappendix A.5.
Revised: We discuss the case of mini-batching inappendix B.5.

Original: We first define a kind of IL algorithm that minimizes the proximal loss (equation 8)w.r.t.
Revised: We first define a kind of IL algorithm that minimizes the proximal loss (equation 8)w.r.t.

Original: neuron activities and show G-IL approaches this algorithm in certain limits.
Revised: neuron activities and show G-IL approaches this algorithm in certain limits.

Original: Definition 4.1.
Revised: Definition 4.1.

Original: Proximal Inference Learning (IL-prox).
Revised: Proximal Inference Learning (IL-prox).

Original: An algorithm identical to G-IL (algorithm2), except activities are optimized according to argmin ˆ h prox =
Revised: An algorithm identical to G-IL (algorithm2), except activities are optimized according to argmin ˆ h prox =

Original: argmin
Revised: argmin

Original: ˆ h ( L ( θ ∗ ) +
Revised: ˆ h ( L ( θ ∗ ) +

Original: 12 α ∥ θ ∗ − θ ( b ) ∥ 2 )and the NLMS rule is used to update weights.
Revised: 12 α ∥ θ ∗ − θ ( b ) ∥ 2 )and the NLMS rule is used to update weights.

Original: Here b is the current training iteration and θ ∗ are the parameters updated with the NLMS rule.
Revised: Here b is the current training iteration and θ ∗ are the parameters updated with the NLMS rule.

Original: IL-proxis the same as G-IL except during the inference phase, it minimizes the proximal loss w.r.t.
Revised: IL-proxis the same as G-IL except during the inference phase, it minimizes the proximal loss w.r.t.

Original: activitiesˆ h instead of the energy F .
Revised: activitiesˆ h instead of the energy F .

Original: Intuitively, minimizing the proximal loss w.r.t.
Revised: Intuitively, minimizing the proximal loss w.r.t.

Original: activities will result in ˆ hthat yields a small loss after weights are updated and small weight update norms.
Revised: activities will result in ˆ hthat yields a small loss after weights are updated and small weight update norms.

Original: Weight updatenorm 12 ∥
Revised: Weight updatenorm 12 ∥

Original: θ ∗ − θ ( b ) ∥ 2 is known during the inference phase because the NLMS learning rule is usedand known explicitly:
Revised: θ ∗ − θ ( b ) ∥ 2 is known during the inference phase because the NLMS learning rule is usedand known explicitly:

Original: [Equation]
Revised: [Equation]

Original: where α n = ∥
Revised: where α n = ∥

Original: f
Revised: f

Original: (ˆ h n ) ∥ − 2 .
Revised: (ˆ h n ) ∥ − 2 .

Original: Thus, it can be minimized w.r.t.
Revised: Thus, it can be minimized w.r.t.

Original: ˆ h n .
Revised: ˆ h n .

Original: As we explain in the appendix (seelemma A.1 and proposition A.1), the loss L ( θ ∗ ) is also explicitly known during the inference phaseand can be optimized w.r.t.
Revised: As we explain in the appendix (seelemma B.1 and proposition B.1), the loss L ( θ ∗ ) is also explicitly known during the inference phaseand can be optimized w.r.t.

Original: ˆ h n .
Revised: ˆ h n .

Original: Now, let α be a ’global’ learning rate hyper-parameter, and α n be layer-wise learning rates used inthe actual weight update ∆ W n .
Revised: Now, let α be a ’global’ learning rate hyper-parameter, and α n be layer-wise learning rates used inthe actual weight update ∆ W n .

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Let α n = ∥
Revised: Let α n = ∥

Original: f
Revised: f

Original: (ˆ h n ) ∥ − 2 and assume mini-batch size 1.
Revised: (ˆ h n ) ∥ − 2 and assume mini-batch size 1.

Original: In the limit where γ decayn →∥
Revised: In the limit where γ decayn →∥

Original: e n +1 ∥ 2 (1 + − 2 α 6 n ) , γ N → α N − 1 α , and γ n → α − 1 n − 1 for all n < N , it is the case that argmin
Revised: e n +1 ∥ 2 (1 + − 2 α 6 n ) , γ N → α N − 1 α , and γ n → α − 1 n − 1 for all n < N , it is the case that argmin

Original: ˆ h F =argmin ˆ h L ( θ ∗ )
Revised: ˆ h F =argmin ˆ h L ( θ ∗ )

Original: +
Revised: +

Original: 12 α ∥ ∆ θ ∥ 2 .
Revised: 12 α ∥ ∆ θ ∥ 2 .

Original: Hence, in these limits G-IL is equivalent to IL-prox.
Revised: Hence, in these limits G-IL is equivalent to IL-prox.

Original: The proof can be found in supplementary materials theorem A.1.
Revised: The proof can be found in supplementary materials theorem B.1.

Original: This theorem states that whenusing the NLMS rule, G-IL is increasingly similar to IL-prox, as the γ weighting terms in the freeenergy (equation 3) approach the scalar values specified in the above theorem.
Revised: This theorem states that whenusing the NLMS rule, G-IL is increasingly similar to IL-prox, as the γ weighting terms in the freeenergy (equation 3) approach the scalar values specified in the above theorem.

Original: In these limits,argmin ˆ h prox =
Revised: In these limits,argmin ˆ h prox =

Original: argmin ˆ h F and G-IL is equivalent to IL-prox.
Revised: argmin ˆ h F and G-IL is equivalent to IL-prox.

Original: The intuitive explanation of thistheorem goes as follows: minimizing the proximal loss (equation 11) requires minimizing L w.r.t.θ ∗ while also minimizing the update norm 12 α ∥ θ ∗ − θ ( b ) ∥ 2 .
Revised: The intuitive explanation of thistheorem goes as follows: minimizing the proximal loss (equation 11) requires minimizing L w.r.t.θ ∗ while also minimizing the update norm 12 α ∥ θ ∗ − θ ( b ) ∥ 2 .

Original: The inference phase of IL minimizes
Revised: The inference phase of IL minimizes

Original: L w.r.t.
Revised: L w.r.t.

Original: θ ∗ by initializing output layer activity ˆ h N to FF output
Revised: θ ∗ by initializing output layer activity ˆ h N to FF output

Original: h
Revised: h

Original: N then shifting it toward globaltarget y .
Revised: N then shifting it toward globaltarget y .

Original: Upon a weight update this will yield a θ ∗ that produces a smaller loss L .
Revised: Upon a weight update this will yield a θ ∗ that produces a smaller loss L .

Original: See figure 4.1for visualization.
Revised: See figure 4.1for visualization.

Original: The magnitude of a weight update 12 α ∥ αe n +1 f
Revised: The magnitude of a weight update 12 α ∥ αe n +1 f

Original: (ˆ h n ) T ∥ 2 intuitively depends onthe magnitude of local prediction error e n +1 .
Revised: (ˆ h n ) T ∥ 2 intuitively depends onthe magnitude of local prediction error e n +1 .

Original: F is a sum of the magnitudes of errors (see equation3), so minimizing F minimizes the magnitudes of errors and consequently weight update norms12 α ∥ θ ∗ − θ ( b ) ∥ 2 .
Revised: F is a sum of the magnitudes of errors (see equation3), so minimizing F minimizes the magnitudes of errors and consequently weight update norms12 α ∥ θ ∗ − θ ( b ) ∥ 2 .

Original: Importantly, the γ settings under which theorem 4.1 holds can be computed exactlyin practice and are of reasonable magnitude that is easily approximated in practice (e.g., they do notapproach ∞ or 0 and typically approach positive scalars).
Revised: Importantly, the γ settings under which theorem 4.1 holds can be computed exactlyin practice and are of reasonable magnitude that is easily approximated in practice (e.g., they do notapproach ∞ or 0 and typically approach positive scalars).

Original: This is opposed to the limits under which
Revised: This is opposed to the limits under which

Original: IL approximates BP/explicit SGD, which is the limit where
Revised: IL approximates BP/explicit SGD, which is the limit where

Original: ˆ h n → h n and thus ∆ W n → 0 [46], alimit that does not typically obtain in practice.
Revised: ˆ h n → h n and thus ∆ W n → 0 [44], alimit that does not typically obtain in practice.

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Let θ ( b ) be a set of MLP parameters at training iteration b .
Revised: Let θ ( b ) be a set of MLP parameters at training iteration b .

Original: Let θ ( b +1) prox =
Revised: Let θ ( b +1) prox =

Original: argmin θ L ( θ ) + 12 α ∥ θ − θ ( b ) ∥ 2 .
Revised: argmin θ L ( θ ) + 12 α ∥ θ − θ ( b ) ∥ 2 .

Original: Let θ ( b +1)
Revised: Let θ ( b +1)

Original: IL − prox be the parameters updated by IL-prox (see def.4.1) and θ ( b +1)
Revised: IL − prox be the parameters updated by IL-prox (see def.4.1) and θ ( b +1)

Original: IL the parameters updated by G-IL under γ values in theorem 4.1.
Revised: IL the parameters updated by G-IL under γ values in theorem 4.1.

Original: Assume mini-batchsize 1.
Revised: Assume mini-batchsize 1.

Original: Under this assumption, it is the case θ ( b +1) prox =
Revised: Under this assumption, it is the case θ ( b +1) prox =

Original: θ ( b +1) IL − prox
Revised: θ ( b +1) IL − prox

Original: =
Revised: =

Original: θ ( b +1) IL .
Revised: θ ( b +1) IL .

Original: This theorem states that the IL-prox update is equivalent to the proximal update (equation 8), and thusimplicit SGD.
Revised: This theorem states that the IL-prox update is equivalent to the proximal update (equation 8), and thusimplicit SGD.

Original: It follows that, under the γ settings in theorem 4.1 the G-IL update is also equivalentto the proximal update/implicit gradient update.
Revised: It follows that, under the γ settings in theorem 4.1 the G-IL update is also equivalentto the proximal update/implicit gradient update.

Original: Further theoretical results are developed in the appendix.
Revised: Further theoretical results are developed in the appendix.

Original: A novel closed form description of ILtargets is developed B.5.
Revised: A novel closed form description of ILtargets is developed C.5.

Original: The closed form description shows local targets approximate a regularized
Revised: The closed form description shows local targets approximate a regularized

Original: Gauss-Newton update on FF activities (see section B).
Revised: Gauss-Newton update on FF activities (see section C).

Original: We call this closed form approximation IL-GN.
Revised: We call this closed form approximation IL-GN.

Original: We study the stability of IL-GN when using the LMS learning rule, and compare to an analogous BPnetwork, which uses Gauss-Newton updates (BP-GN).
Revised: We study the stability of IL-GN when using the LMS learning rule, and compare to an analogous BPnetwork, which uses Gauss-Newton updates (BP-GN).

Original: We show that IL-GN weight updates pushoutput layer activities down their loss gradients for any positive learning rate , i.e., for any positivelearning rate
Revised: We show that IL-GN weight updates pushoutput layer activities down their loss gradients for any positive learning rate , i.e., for any positivelearning rate

Original: ˆ h ( b ) N − ˆ h ( b +1) N =
Revised: ˆ h ( b ) N − ˆ h ( b +1) N =

Original: j ∂L ∂ ˆ h ( b ) N where j is a positive scalar (theorem C.1).
Revised: j ∂L ∂ ˆ h ( b ) N where j is a positive scalar (theorem D.1).

Original: We show the same is not true of BP-GN, which only has this property for a small range of learning rates (theorem C.2).
Revised: We show BP-GN only has this property for a small range of learning rates (theorem D.2).

Original: We develop a partial explanation of why IL is more stable than BP that is based on differences intheir learning rule (for details see section D).
Revised: We develop a partial explanationof why IL is more stable than BP that is based on differences in their learning rule (for details seesection E).

Original: These results make the important point that the way ILcomputes and uses local targets in its weight update significantly contributes to its stability, whilethe way G-BP computes and uses local targets in its weight update (which is distinct from IL, seesection 2) is largely responsible for its instability.
Revised: These results suggest that the differences in the way IL and BP compute and use localtargets in their weight updates can explain much of their differences in stability across learning rate.

Original: The IL learning rule and method of computing ˆ h isadvantageous over that of G-BP in this sense.
Revised: The IL learning rule and target computations are advantageous over that of G-BP in this sense.

Original:  Experiments
Revised:  Experiments

Original: CIFAR

Original: [Table]
Revised: [Table]

Original: Test Accuracy (mean [Table]
Revised: with mini-batch size 1 after 50,000 iterations (1 epoch) and with mini-batch size 64 with 77000 iterations.

Original: In this section, wetest and compare  BP based algorithms to several IL algorithms.
Revised: In this section, we compare the performance of BP based algorithms to ILalgorithms.

Original: BP algorithms computeand use explicit gradients to updateweights, while IL algorithms computeand use approximate implicit gradients to perform updates.
Revised: BP algorithms computeand use explicit gradients to updateweights, while IL algorithms computeand use approximate implicit gradients to perform updates.

Original: We test BPmodels that either perform simple gradient descent ( BP-SGD ) or use Adamoptimizers ( BP-Adam ).
Revised: We test BPmodels that either perform simple gradient descent ( BP-SGD ) or use Adamoptimizers ( BP-Adam ).

Original: IL-SGD isa simple implementation of IL that issimilar to implementations of [36, 46].
Revised: IL-SGD isa simple implementation of IL that issimilar to implementations of [33, 44].

Original: It uses the LMS rule (local error gradient update) to update weights andoptimizes ˆ h using SGD to near convergence (25 update steps).
Revised: It uses the LMS rule (local error gradient update) to update weights andoptimizes ˆ h using SGD to near convergence (25 update steps).

Original: IL-Proxis our novel algorithm that is basedon definition 4.1.
Revised: IL-Proxis our novel algorithm that is based ondefinition 4.1.

Original: IL-prox uses the NLMS rule to update weights.
Revised: IL-prox uses the NLMS rule to update weights.

Original: As theorem 4.1 shows, to minimize the proximal loss, the learning rate is only used to determine how much the output layer activities ˆ h N arepushed toward the target y .
Revised: Following theorem 4.1, the learningrate is only used to determine how much the output layer activities ˆ h N are pushed toward the targety .

Original: As α → ∞ , ˆ h N → y and as α → 0 , ˆ h N → h N .
Revised: As α → ∞ , ˆ h N → y and as α → 0 , ˆ h N → h N .

Original: IL-Prox optimizesˆ h using SGD to near convergence (25 update steps).
Revised: IL-Prox optimizes ˆ h using SGD to nearconvergence (25 update steps).

Original: IL-Prox Fast is the same as the IL-Proxalgorithm except it truncates the optimization of ˆ h to only 12 iterations.
Revised: IL-Prox Fast is the same as the IL-Prox algorithm except it truncatesthe optimization of ˆ h to only 12 iterations.

Original: IL-Adam computes theweight gradients using IL-SGD, then uses Adam optimizers to update weights.
Revised: IL-Adam computes the weight gradients using IL-SGD,then uses Adam optimizers to update weights.

Original: IL-prox Adamcomputes the normalized weight gradient using IL-prox and uses Adam optimizers to update weights.
Revised: IL-prox Adam computes the normalized weightgradient using IL-prox and uses Adam optimizers to update weights.

Original: All simulations average results over at least 5 seeds of each model type.
Revised: All simulations average resultsover at least 5 seeds of each model type.

Original: Best test scores from theseaverages are shown in tables.
Revised: Best test scores from these averages are shown in tables.

Original: All models use ReLU activations at hidden layers.
Revised: All models use ReLU activations at hidden layers.

Original: Softmax is used atoutput layer for classification, while sigmoid is used on the autoencoder task.
Revised: Softmax is used at output layer for classification,while sigmoid is used on the autoencoder task.

Original: [Table]
Revised: Mini-batch Training Previous work has shown that IL algorithms can achieve the same accuracy as BP on small scale data sets (e.g., MNIST and Fashion-MNIST), in the case where Adam optimizersand medium sized mini-batches (e.g, 64) are used ( [44, 2].

Original: Supervised and Self-Supervised Learning We begin by training these models on supervised classification tasks.
Revised: Here we train our BP and IL algorithmson CIFAR-10 with mini-batch size 64.

Original: Resultsfor CIFAR-10 are summarized in table 2.
Revised: Results are summarized in table 2.

Original: Consistent with previous results[46, 2], in the case where large mini-batches are used, IL algorithms achieve near equal accuracy as 
Revised: Consistent with previousresults, IL algorithms, including our novel IL-prox, achieve near equal accuracy as BP-SGD and [Table]

Original: BP-SGD and BP-Adam when using Adam optimizers.
Revised: BP-Adam when using Adam optimizers.

Original: But IL-SGD and IL-prox algorithms, whichdo not use Adam optimizers, do not perform as well as BP algorithms in this scenario.
Revised: However, unlike previous studies, we also test IL-SGD and IL-prox algorithms without Adam, and find they do not perform as well as BP algorithms in thisscenario.

Revised: IL algorithms without Adam tend to converge more quickly but to shallower local minima.

Original: However, inthe case where they are trained with  mini-batch size 1, IL algorithms improve accuracy significantlyfaster than BP-SGD for theentire first epoch(50,000 iterations) of training.
Revised: Adam, with its momentum and adaptive learning rate, may help prevent IL algorithms from gettingcaught in shallow local minima (see discussion). Online Learning We find IL algorithms tend to converge significantly quicker and to more similaraccuracies as BP when mini-batch size 1 is used. This is especially apparent when each data point isseen only once by the network, similar to online learning scenarios. IL algorithms improve accuracysignificantly faster during the first epoch of training on CIFAR-10 with mini-batch size 1 (table 2,figure 5).

Original: IL-SGD and IL-proxalgorithms perform similar to and even slightly better than BP-Adam, despite not using momentumor stored, parameter-wise adaptive learning rates like Adam.
Revised: IL-SGD and IL-prox algorithms perform similar to and even slightly better than BP-Adam,despite not using momentum or stored, parameter-wise adaptive learning rates like Adam.

Original: As far as we know, this is a novelresult.
Revised: As far aswe know, this is a novel result.

Original: Previous works have not tested IL in case where small mini-batches are used (e.g., [46, 2]). To further test this improvement in optimization speed, we also trained these models on MNISTand fashion-MNIST classification tasks (table 6), and we trained autoencoders on CIFAR-10 images(table 3), with mini-batch size 1.
Revised: Less significant speedups were also seen on MNIST and FashionMNIST classification tasks (see figure F.3). On autoencoder tasks, IL-prox algorithms, in particular,decreased loss significantly quicker than BP-SGD and even slightly quicker than BP-Adam (figure 5).

Original: Slight speed-ups in training were seen on MNIST data sets (seefigure E.1.1).
Revised: Learning Under Memory Constraints We also find that IL algorithms can converge to betteraccuracy than BP in certain memory constrained scenarios. We trained IL algorithms using on only10, 100, or 500 data points from each category on F-MNIST and CIFAR-10, with mini-batch size100.

Original: On the autoencoder task with CIFAR-10, we found again IL-prox algorithms improveloss as quickly and even slightly quicker than BP-Adam (see figure 2).
Revised: IL algorithms achieved better accuracies than BP-SGD in the 10 and 100 data point scenarios,while performances evened out in the 500 data point scenario (see appendix, table 8).

Original: Stability 
Revised: Stability Test Explicit SGD is well-known to be highly sensitive to learning rate.

Original: Test Implicit SGD is unconditionally stable, which, roughly, means that it is able to discountthe loss in a stable manner for nearly any positive learning rate [44].
Revised: Implicit SGD, on theother hand, is highly insensitive to learning rate and is unconditionally stable, i.e., it is able to discountthe loss in a stable manner for nearly any positive learning rate [42].

Original: We compare performance of
Revised: We compare performance of

Original: BP-SGD, IL-SGD, and IL-prox on MNIST (table 5) and CIFAR-10 (table 4) classification task acrossdifferent learning rates.
Revised: BP-SGD, IL-SGD, and IL-prox on MNIST (table 5) and CIFAR-10 (table 4) classification task acrossdifferent learning rates.

Original: As a control, we also test a hybrid of BP and IL-prox (BP-prox).
Revised: As a control, we train a hybrid of BP and IL-prox (BP-prox).

Original: BP-prox,like IL-prox, uses the learning rate only to adjust the target at the output layer and uses the NLMSrule to update weights.
Revised: BP-prox, like IL-prox, uses the learning rate only to adjust the target at the output layer and uses the NLMS rule toupdate weights.

Original: Unlike IL-prox, BP-prox computes local targets using the G-BP algorithmrather than the IL algorithm.
Revised: Unlike IL-prox, BP-prox computes local targets using the G-BP algorithm ratherthan the IL algorithm.

Original: This control helps to show the stability of IL-prox is not only due tonormalizing the weight updates and softly clamping the output layer.

Original: Rather, the way IL computesand uses local targets in its learning rule also contributes to stability.

Original: Results can be found in table 5and 4.
Revised: Results can be found in table 5 and 4.

Original: Blank entries are those with accuracies below 12% .
Revised: Blank entries are those with accuraciesbelow 12% .

Original: IL-SGD is more stable than BP-SGD,and IL-prox is more stable than BP-prox.
Revised: IL-SGD is more stable than BP-SGD, and IL-prox is more stable than BP-prox.

Original: These results provide evidence that the way IL computesand uses targets contributes to stability.
Revised: Thissuggests that the way IL computes and uses targets contributes to stability.

Original: Additionally, the prox algorithms are more stable than SGDalgorithms, showing that using the normalized gradient to update weights and α to adjust the outputlayer target rather than scale weight updates also contributes to stability.
Revised: Additionally, IL-proxand BP-prox algorithms are more stable than SGD algorithms, showing that using the normalizedgradient to update weights and α to adjust the output layer target (rather than scale weight updates)also contributes to stability.

Original: [Table]
Revised: [Table]

Original: Further Results We computed the update norms ∥ ∆ θ ∥ during training of each model on the CIFAR10 classification task. Average magnitudes and standard deviations are shown in appendix E. ILalgorithms had similar or smaller update magnitudes than BP-SGD on most of their training runs.
Revised: lr=

Original: This suggests IL algorithms do not minimize loss more quickly because their updates are greater inmagnitude than BP-SGD. One possibility is that IL updates take a more direct path to local minimathan the path of steepest descent.
Revised: [Table]

Original: This is consistent with other experiments in which we analyzedthe output layer of BP-SGD and IL-SGD.
Revised: lr=. lr= lr=2. lr= lr= [Table] − − − − −

Original: We measured how the FF output layer values h N changedafter updates.
Revised: [Table]

Original: IL updates tended to change the output layer values more and in a direction nearer theminimum norm (shortest) path to that target y than BP-SGD updates (figures E.1.3, E.1.3).
Revised: − − − − [Table]

Original:  Limitations
Revised:  Limitations

Original: Our theoretical results show that IL algorithms closely approximate implicit SGD in the case whereonly a single data-point (i.e., mini-batch size 1) is presented each training iteration.
Revised: Our theoretical results show that IL algorithms closely approximate implicit SGD in the case whereonly a single data-point (i.e., mini-batch size 1) is presented each training iteration.

Original: This approximation is looser in the case where large mini-batches are used (see appendix A.5).
Revised: This approximation is looser in the case where large mini-batches are used (see appendix B.5).

Original: This is no problemfrom a biological point of view, since the brain does not train with large mini-batches.
Revised: This is no problemfrom a biological point of view, since the brain does not train with large mini-batches.

Original: Additionally,when Adam optimizers are used IL still performs similarly to BP.
Revised: Additionally,when Adam optimizers are used IL still performs similarly to BP.

Original: However, it may still be desirable tofurther study the optimization properties of IL with mini-batches to improve performance on machinelearning tasks where mini-batches are typically used.
Revised: However, it may still be desirable tofurther study the optimization properties of IL with mini-batches to improve performance on machinelearning tasks where mini-batches are typically used.

Original:  Discussion
Revised:  Discussion

Original:  IL was originally used to train predictive coding models of cortical circuits [36].
Revised: In this paper, we found that IL closely approximates implicit SGD, which is distinct from explicit

Original: The successof subsequent predictive coding models lead some neuroscientists to propose the hypothesis thatpredictive coding and IL are canonical computations used throughout the brain (e.g., [13, 14, 4, 16]).
Revised: SGD implemented by BP.

Original: In this paper, we provided novel mathematical justification for IL by showing it closely approximatesa proper optimization method known as implicit SGD, which  is distinct from the  explicit SGDexecuted by BP. This finding, in conjunction with the theory IL and predictive coding are canonicalcomputations in the brain, suggests the interesting possibility that implicit SGD, andnot explicit SGD,is the central optimization method utilized by biological neural circuits.
Revised: This approximation holds best when mini-batch size one is used. Thistheoretical result, along with results that help to characterize how IL and BP behave differently,suggest a hypothesis, which as far was we know is novel: learning in the brain is more similar toimplicit gradient descent than explicit gradient descent. This hypothesis would imply the learningcharacteristics of the brain behave differently from BP and explicit SGD.

Original: We observed two performanceadvantages that fit with this possibility.
Revised: More empirical work willbe needed to test this theory.

Original: First, consistent with our theoretical interpretation, ILalgorithms are more stable across learning rates than BP. This property is compatible with the fact thatsynaptic plasticity in  the brain can fluctuate rapidly, e.g. due to neuromodulation, yet maintains stablelearning properties.
Revised: However, recent evidence that directly supports the idea the brain doessomething similar to IL [39] and evidence that performs predictive coding provides the idea withinitial plausibility. We also found that IL often converged more quickly than BP to similar, and sometimes better,performance in mini-batch size one and memory constrained scenarios. However, in large mini-batchscenarios without memory constraints and without the use of Adam optimizers, IL sometimes converged to worse accuracies than BP in classification tasks.

Original: The stability of IL is arguably its main advantage over BP, which is consistentwith claims thatthe main advantage of  implicit SGD  over standard/explicit SGD is its stability acrosslearning rates (e.g., see [44, 45]).
Revised: With thus use of Adam, IL’s performancemore closely matched that of BP. One possible explanation of this empirical finding may be related tothe fact that IL, and implicit SGD more generally, tend to take smaller steps over parameters and amore direct path toward local minima than BP (e.g., see figures F.6 and table 9). More direct paths tominima often lead to faster convergence.

Original: Second, we found IL minimized the loss more quickly than BP inthe more biologically realistic scenario where single data points, rather than large mini-batches, arepresented each training iteration. These simulations provide evidence that IL is more compatible withbiologically realistic learning scenarios than BP.
Revised: Smaller update magnitudes may act as a kind of built-inregularization that aids in difficult, noisy learning scenarios, such as small mini-batch and memoryconstrained scenarios. In cases where regularization is less advantageous (e.g., no memory constraintsand large mini-batches), IL’s small update magnitudes may provide less of an advantage, and its moredirect optimization paths may push parameters into shallow, nearby minima. Adam optimizers, withtheir momentum and adaptive learning rates, may help push parameters out of shallow minima.

Original: Collectively, our findings suggest IL is a promisingbasis for developing biologically constrained, high performing, local learning algorithms.  Broader Impacts Our work is largely theoretical so it does not suggest any direct, possibly negative, societal impacts,as far as we can see.
Revised: Morework will be needed to fully characterize and explain these behavioral differences, but we believe thework done here provides a good formal basis for doing so. These results further suggest IL may haveadvantages over BP in more biologically realistic machine learning tasks, such as online learningscenarios with autonomous and/or neuromorphic systems, where data may be noisy, sparse, andmini-batches are small/size of one.

Original: Checklist
Revised: Checklist

Original: For all authors
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See section
Revised: [Yes] See section

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See section
Revised: [Yes] See section??(d)

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: For all theorems,
Revised: For all theorems,

Original: propositions, and lemmas assumptions were stated in the statement.
Revised: propositions, and lemmas assumptions were stated in the statement.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] All proofs are in appendix
Revised: [Yes] All proofs are inappendix

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] This information
Revised: [Yes] This information

Original: is in supplementary material and in link to code
Revised: is in supplementary material and in link to code

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] This information is reported in supplementary material
Revised: [Yes] This information is reported in supplementary material

Original: (c) Did you report error bars (e
Revised: (c) Did you report error bars (e

Original: g., with respect to the random seed after running exper-
Revised: g., with respect to the random seed after running exper-

Original: iments multiple times)?
Revised: iments multiple times)?

Original: [Yes] Standard deviations over multiple seeds are shown intables and in plots using error bars.
Revised: [Yes] Standard deviations over multiple seeds are shown in tables and in plots using error bars.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g.,
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g.,

Original: type of GPUs, internal cluster, or cloud provider)?
Revised: type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] Is included in supplementary
Revised: [Yes] Is included in supplementary

Original: material
Revised: material

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N /A]
Revised: [N/A]