Original: A Theoretical Framework for Inference Learning
Revised: A Theoretical Framework for Inference Learning

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Backpropagation (BP) is the most successful and widely used algorithm in deeplearning.
Revised: Backpropagation (BP) is the most successful and widely used algorithm in deep learning.

Original: However, the computations required by BP are challenging to reconcilewith known neurobiology.
Revised: However, the computations required by BP are challenging to reconcile with known neurobiology.

Original: This difficulty has stimulated interest in more biologically plausible alternatives to BP.
Revised: This difficulty has stimulated interest in more biologically plausible alternatives to BP.

Original: One such algorithm is the inference learningalgorithm (IL).
Revised: One such algorithm is the inference learning algorithm (IL).

Original: IL has close connections to models of cortical circuits and hasachieved equal performance to BP on supervised and auto-associative tasks.
Revised: IL trains predictive coding models of neural circuits and has achieved equal performance to BP on supervised and auto-associative tasks.

Original: In contrast to BP, however, the mathematical foundations of IL are not well-understood.
Revised: In contrast to BP, however, the mathematical foundations of IL are not well-understood.

Original: Here, we develop a novel theoretical framework for IL.
Revised: Here, we develop a novel theoretical framework for IL.

Original: Our main result is that ILclosely approximates an optimization method known as implicit stochastic gradientdescent (implicit SGD), which is distinct from the explicit SGD implemented by BP.
Revised: Our main result is that IL closely approximates an optimization method known as implicit stochastic gradient descent (implicit SGD), which is distinct from the explicit SGD implemented by BP.

Original: Our results further show how the standard implementation of IL can be altered tobetter approximate implicit SGD.
Revised: Our results further show how the standard implementation of IL can be altered to better approximate implicit SGD.

Original: Our novel implementation considerably improvesthe stability of IL across learning rates, which is consistent with our theory, as a keyproperty of implicit SGD is its stability.
Revised: Our novel implementation considerably improves the stability of IL across learning rates, which is consistent with our theory, as a key property of implicit SGD is its stability.

Original: We provide extensive simulation resultsthat further support our theoretical interpretations and find IL achieves quickerconvergence when trained with mini-batch size one while performing competitivelywith BP for larger mini-batches when combined with Adam.
Revised: We provide extensive simulation results that further support our theoretical interpretations and find IL achieves quicker convergence when trained with mini-batch size one while performing competitively with BP for larger mini-batches when combined with Adam.

Original:  Introduction
Revised:  Introduction

Original: Backpropagation (BP) [34], the most successful and ubiquitously used learning algorithm in deeplearning, has long been criticized as being incompatible with known neurobiology [11, 22].
Revised: Backpropagation (BP) [34], the most successful and ubiquitously used learning algorithm in deep learning, has long been criticized as being incompatible with known neurobiology [11, 22].

Original: Forexample, unlike the highly interconnected, recurrent circuits in the brain, BP performs credit assignment through a separate feedback stream that does not alter feedforward (FF) signals [11, 22].
Revised: For example, unlike the highly interconnected, recurrent circuits in the brain, BP performs credit assignment through a separate feedback stream that does not alter feedforward (FF) signals [11, 22].

Original: Relatedly, standard implementations of BP are largely incompatible with energy efficient, neuromorphic hardware, due, in part, to BP’s non-local gradient computations.
Revised: Additionally, standard implementations of BP are largely incompatible with energy efficient, neuromorphic hardware, due, in part, to BP’s non-local gradient computations.

Original: These concerns have motivatedneuroscientists and engineers to search for alternatives to BP that use  local learning rules that aremore compatible with biology.
Revised: These concerns have motivated neuroscientists and engineers to search for alternatives to BP that use more biologically compatible, local learning rules.

Original: One alternative to BP is the inference learning algorithm (IL)
Revised: One alternative to BP is the inference learning algorithm (IL)

Original: IL is the standardlearning algorithm used to train predictive coding (PC) models of biological neural circuits (e.g.[33, 14, 40, 4, 1, 8]).
Revised: IL is the standard learning algorithm used to train predictive coding (PC) models of biological neural circuits (e.g.

Original: Within neuroscience, PC models have grown in popularity, where some evenpropose PC may be a canonical biological circuit [16, 15, 4].
Revised: Within neuroscience, PC models have grown in popularity, where some even propose PC may be a canonical biological circuit [16, 15, 4].

Original: IL works by first minimizing an energyfunction, known as free energy, w.r.t.
Revised: IL works by first minimizing an energy function, known as free energy, w.r.t.

Original: neuron activities using the recurrent computations involved inpredictive coding.
Revised: neuron activities using the recurrent computations involved in predictive coding.

Original: At convergence, IL updates weights to further minimize energy.
Revised: At convergence, IL updates weights to further minimize energy.

Original: Unlike standard
Revised: Unlike standard

Original: BP, IL uses local learning rules and recurrent circuits instead of a separate feedback stream to performcredit assignment.
Revised: BP, IL uses local learning rules and recurrent circuits instead of a separate feedback stream to perform credit assignment.

Original: In addition to evidence the brain does predictive coding (and thus may do IL),recent empirical work has found evidence the brain may learn by first inferring target neural activitiesthrough recurrent processing then updating synapses to consolidate those neural activities, just like
Revised: In addition to evidence the brain does predictive coding (and thus may do IL), recent empirical work has found evidence the brain may learn by first inferring target neural activities through recurrent processing then updating synapses to consolidate those neural activities, just like

Original: IL [39].
Revised: IL [39].

Original: Recently, IL has  been used to train  networks for machine learning tasks.
Revised: IL has also been used to train deep networks for machine learning tasks.

Original: IL, for example, achieved comparable classification and recall accuracy to BP trained networks on supervised andauto-associative tasks (e.g, [44, 2, 37, 36], see also below).
Revised: IL, for example, achieved comparable classification and recall accuracy to BP trained networks on supervised and auto-associative tasks (e.g, [44, 2, 37, 36], see also below).

Original: Despite IL’s wide use in biological models and success on machine learning tasks, a rigorous formalcharacterization of IL’s optimization properties and their differences and similarities with BP islacking.
Revised: Despite IL’s wide use in biological models and success on machine learning tasks, a rigorous formal characterization of IL’s optimization properties and their differences and similarities with BP is lacking.

Original: Providing such a characterization could yield several useful contributions: 1) A basis for anovel theory of how optimization and credit assignment may work in the brain, 2) a formal basis forimproving, developing, and applying IL to machine learning tasks and neuromorphic hardware.
Revised: Providing such a characterization could yield several useful contributions: 1) A basis for a novel theory of how optimization and credit assignment may work in the brain, 2) a formal basis for improving, developing, and applying IL to machine learning tasks and neuromorphic hardware.

Original: In this paper, we develop a novel theoretical framework for IL that describes its optimizationproperties and their similarities and differences from BP.
Revised: In this paper, we develop a novel theoretical framework for IL that describes its optimization properties and their similarities and differences from BP.

Original: More specifically, 1) we expand beyondcurrent literature by deriving a general form of the IL algorithm, which we call Generalized IL (G-IL).2)
Revised: More specifically, 1) we expand beyond current literature by deriving a general form of the IL algorithm, which we call Generalized IL (G-IL).

Original: We demonstrate our main result, which is that G-IL closely approximates an optimization methodknown as implicit stochastic gradient descent (implicit SGD), where the approximation is closestin the case of mini-batch size 1.
Revised: We demonstrate our main result, which is that G-IL closely approximates an optimization method known as implicit stochastic gradient descent (implicit SGD), where the approximation is closest in the case of mini-batch size 1.

Original: Implicit SGD is distinct from explicit SGD (the standard form of
Revised: Implicit SGD is distinct from explicit SGD (the standard form of

Original: SGD used in machine learning and the sort executed by BP).
Revised: SGD used in machine learning and the sort executed by BP).

Original: We provide theoretical guaranteesconcerning IL’s stability and a connection to Gauss-Newton optimization.
Revised: We provide theoretical guarantees concerning IL’s stability and a connection to Gauss-Newton optimization.

Original: We identify a learningrule and variable settings needed in order for IL to equal implicit SGD and use this result to developa novel implementation of IL called IL-prox.
Revised: We identify a learning rule and variable settings needed in order for IL to equal implicit SGD and use this result to develop a novel implementation of IL called IL-prox.

Original: Finally, we present extensive simulation results thatsupport our theoretical findings, and for the first time show certain performance advantages of IL over
Revised: Finally, we present extensive simulation results that support our theoretical findings, and for the first time show certain performance advantages of IL over

Original: BP, such as better stability across learning rates and improved convergence speed with biologicallyrealistic online learning scenarios.
Revised: BP, such as better stability across learning rates and improved convergence speed with biologically realistic online learning tasks.

Original: These results collectively suggest that, not only does IL better fitbiological constraints, but it is also mathematically justified and has performance advantages over BPin more biologically realistic training scenarios.
Revised: These results collectively suggest that, not only does IL better fit biological constraints, but it is also mathematically justified and has performance advantages over BP in more biologically realistic training scenarios.

Original:  Related Works
Revised:  Related Works

Original:  Whittington and Bogacz [44] proved that parameter updates performed by the IL algorithm approachthose of BP as the global loss approaches zero and optimized activities approach feed-forwardactivities.
Revised: Inference Learning and Backpropagation Whittington and Bogacz [44] proved that parameter updates performed by the IL algorithm approach those of BP as the global loss approaches zero and optimized activities approach feed-forward activities.

Original: However, this same proof implies that non-zero IL updates essentially never equal thoseof BP.
Revised: However, this same proof implies that non-zero IL updates essentially never equal those of BP.

Original: Additionally, this analysis leaves open the question of how IL is able to minimize the loss ina stable manner early in training when the loss is large and IL updates diverge significantly fromthose of BP.
Revised: Additionally, this analysis leaves open the question of how IL is able to minimize the loss in a stable manner early in training when the loss is large and IL updates diverge significantly from those of BP.

Original: Other work has shown IL is formally related to variational expectation maximization[26, 23], which is a learning algorithm with convergence guarantees [28].
Revised: Other work has shown IL is formally related to variational expectation maximization [26, 23], which is a learning algorithm with convergence guarantees [28].

Original: This analysis still leavesopen the question of how IL relates to BP and the broader framework of gradient-based methods thatare the backbone of deep learning.
Revised: This analysis still leaves open the question of how IL relates to BP and the broader framework of gradient-based methods that are the backbone of deep learning.

Original: The focus of the current work is to develop insights into theseopen questions.
Revised: The focus of the current work is to develop insights into these open questions.

Original: More recently, several works have altered IL to more closely approximate SGD and BP [27, 38].
Revised: More recently, several works have altered IL to more closely approximate SGD and BP [27, 38].

Original: These variants significantly change the original IL algorithm, and some  the alterationsseem hard to reconcile with neurobiology [27, 38].
Revised: These variants significantly change the original IL algorithm, and some of the alterations seem hard to reconcile with neurobiology [27, 38].

Original: Alternatively, instead of altering IL to better fit SGD and BP, we show the standard implementation of IL already closely approximates a workingoptimization method known as implicit SGD, and we show the variable settings under which thestandard implementation of IL better approximates implicit SGD.
Revised: Alternatively, instead of altering IL to better fit SGD and BP, we show the standard implementation of IL already closely approximates a working optimization method known as implicit SGD, and we identify the variable settings under which the standard implementation of IL better approximates implicit SGD.

Original:  Algorithms similar to IL have been proposed.
Revised: Related Algorithms Algorithms similar to IL have been proposed.

Original: For example, the alternating minimization algorithm[10], which was developed independently of the predictive coding and IL framework from neuroscience, updates weights to minimize local prediction errors similar to IL (see equation 3 below).
Revised: For example, the alternating minimization algorithm [10], which was developed independently of the predictive coding and IL framework from neuroscience, updates weights to minimize local prediction errors similar to IL (see equation 4 below).

Original: However, [10] do not connect their algorithm to implicit SGD.
Revised: However, [10] do not connect their algorithm to implicit SGD.

Original: A more recent variant of this algorithmby Qiao et al.
Revised: A more recent variant of this algorithm by Qiao et al.

Original: [32] show some formal links to proximal operators, an optimization process relatedto implicit SGD [31] (see appendix B.1).
Revised: [32] show some formal links to proximal operators, an optimization process related to implicit SGD [31] (see appendix B.1).

Original: However, they do not interpret the weight updates asperforming the proximal update or implicit SGD as we do here with IL.
Revised: However, they do not interpret the weight updates as performing the proximal update or implicit SGD as we do here with IL.

Original: Proximal operators have alsorecently been incorporated into learning algorithms for deep networks.
Revised: Proximal operators have also recently been incorporated into learning algorithms for deep networks.

Original: Frerix et al.
Revised: Frerix et al.

Original: [13] developed a BP, proximal algorithm hybrid.
Revised: [13] developed a BP, proximal algorithm hybrid.

Original: Lau et al.
Revised: Lau et al.

Original: [18] developed an algorithm that combinedand algorithmknown as block coordinate descent with the proximal operator.
Revised: [18] developed an algorithm that combined block coordinate descent with the proximal operator.

Original: Other works use proximal operators toreduce noise or perform other tasks, then BP is used to to update at convergence[24, 46].
Revised: Other works use proximal operators to reduce noise or perform other tasks, then BP is used to to update at convergence

Original: All of thesealgorithms, however, are distinct from IL, as they use BP or other non-local gradient information tocompute weight updates or local targets.
Revised: All of these algorithms, however, are distinct from IL, as they use BP or other non-local gradient information to compute weight updates or local targets.

Original:  Finally, IL has similarities to target propagation algorithms, in which local target activities arecomputed and used to compute local error gradients to update weights.
Revised: Target Propagation Finally, IL has similarities to target propagation algorithms, in which local target activities are computed and used to compute local error gradients to update weights.

Original: Target propagation [5] anddifference target propagation [20] utilize approximate Gauss-Newton updates on activities to create local targets [25, 6], while other algorithms compute targets using gradient updates on activities[19, 30, 3, 17].
Revised: Target propagation [5] and difference target propagation [20] utilize approximate Gauss-Newton updates on activities to create local targets [25, 6], while other algorithms compute targets using gradient updates on activities [19, 30, 3, 17].

Original: These algorithms have important differences from IL, e.g., they usea different learningrules than IL, which we discuss further below.
Revised: These algorithms have important differences from IL, e.g., they use different learning rules than IL, which we discuss further below.

Original: In sum, previous works have not developed the samemathematical interpretation of IL that we do here and have not developed the same mathematicaldescriptions of the differences between IL and BP.
Revised: In sum, previous works have not developed the same mathematical interpretation of IL that we do here and have not developed the same mathematical descriptions of the differences between IL and BP.

Original: Further, none of the works mentioned aboveproduced the same empirical findings we do in our simulations below.
Revised: Further, none of the works mentioned above produced the same empirical findings we do in our simulations below [Table]

Original:  Background and Notation
Revised:  Background 3.1 Notation

Original: [Table]
Revised: [Table]

Original: Notation describing a multi-layered FF network (MLP) is summarized in table 1.
Revised: Notation describing a multi-layered feed-forward (FF) network (MLP) is summarized in table 1.

Original: To simplifynotation, we assume  bias is stored in an extra column on each weight matrix W n and we assume a 1is concatenated to the end of the pre-synaptic activity vector ( h n or ˆ h n depending).
Revised: To simplify notation, we assume that the bias is stored in an extra column on each weight matrix W n and we assume a 1 is concatenated to the end of the pre-synaptic activity vector ( h n or ˆ h n depending).

Original: All the followingresults should hold with and without biases.
Revised: All the following results should hold with and without biases.

Original: 3.2 Generalized Backpropagation
Revised: 3.2 Generalized Backpropagation

Original: Consider a set of parameters θ ( b ) at training iteration b with global loss function L .
Revised: Consider a set of parameters θ ( b ) at training iteration b with global loss function L .

Original: An explicit SGDiteration subtracts the gradient of the loss from the parameters: θ ( b +1)
Revised: An explicit SGD iteration subtracts the gradient of the loss from the parameters: θ ( b +1)

Original: = θ ( b ) − α ∂L∂θ ( b ) , with stepsize α . BP [35] is a common algorithm for implementing SGD in deep networks.
Revised: = θ ( b ) − α ∂L∂θ ( b ) , with step size α . BP [35] is the common algorithm for implementing SGD in deep networks.

Original: Here, to allowfor easier comparison to IL, we consider an alternative implementation of SGD in deep networksthat involves computing local target activities at hidden and output layers, similar to [19], [3], [30].
Revised: Here, to allow for easier comparison to IL, we consider an alternative implementation of SGD in deep networks that involves computing local target activities at hidden and output layers, similar to [19], [3], [30].

Original: Specifically, the local target at layer n, ˆ h n , is computed as follows [Equation] where local loss l n +1 = 12 ∥
Revised: Under this regime, each weight matrix W n is updated using the gradient of local loss l n +1 = 12 ∥

Original: ˆ h n +1 − h n +1 ∥ 

Revised: ˆ h n +1 − h n +1 ∥

Original: = 12 ∥
Revised: = 12 ∥

Original: e n +1 ∥ 2 .
Revised: e n +1 ∥ 2 .

Revised: Here ˆ h n +1 is the local target, while h n +1 is the FF activity.

Original: The target at the output layer  ˆ h N =h N −∂L∂h N .
Revised: The target at the output layer is computed ˆ h N = h N −

Original: [-Notice that e n +
 =-]
Revised: [-Notice that e n +
 =-][Equation] N .

Original: ˆ h n +1 − h n [-+
-]
Revised: At hidden layer n [-+
-], the local target is computed

Original: =
Revised: [Equation]

Original:  h n+1 + ∂l n +2 ∂h n +1 − h n [-+
-]
Revised: It can be shown h n − ∂l n [-+
-]+

Original: =
Revised: [Equation]

Original: ∂l n +2 ∂h n+1 =  − ∂L ∂h n +1 .
Revised: n = h n − ∂L∂h n by first computing ˆ h N − 1 : [Equation]

Original: As noted in the above equation, this local target update is equivalent to subtracting the  global lossgradient from the FF activity h n (for proof see appendix [3]).
Revised: From this, it is trivial to recursively apply this same process to show the remaining hidden layer targets are also global loss gradient steps over h n . See appendix of [3] for similar proof.

Original: To update weight W n one uses thegradient of the local loss:
Revised: Updates to weights W n also use the gradient of the local loss such that ∆ W n = − α ∂l n +1 ∂W n .

Revised: If we use the result from the above derivation that − ∂l n ∂h n = − ∂L∂h n , we can see this update is equivalent to a global loss gradient step over the weights:

Original: [Equation]
Revised: [Equation]

Original: This update uses only local information and is equivalent to subtracting the global loss gradient fromthe weights (see [3]).

Original: This local target formulation of SGD is similar to other target propagation (TP)[5, 20] algorithms, which use a similar learning rule but compute targets slightly differently.
Revised: This local target formulation of SGD is similar to other target propagation (TP) [5, 20] algorithms, which use essentially the same learning rule but compute targets slightly differently.

Original: Forexample, a well-known variant of TP is difference target propagation (DTP) [20], which computeslocal targets by approximating a Gauss-Newton update on activities rather than gradient updates[25, 6]:
Revised: For example, a well-known variant of TP is difference target propagation (DTP) [20], which computes local targets by approximating a Gauss-Newton update on activities rather than gradient updates [25, 6]:

Original: ˆ h n ≈ h n − J + N,n e N , where J + N,n is the pseudo-inverse of the Jacobian of the forward networkfrom layer n to N and e N =
Revised: ˆ h n ≈ h n − J + N,n e N , where J + N,n is the pseudo-inverse of the Jacobian of the forward network from layer n to N and e N =

Original: y − h N .
Revised: y − h N .

Original: Gauss-Newton updates approximate (are within 90 ◦) of SGDupdates [25], and recent attempts to improve DTP generally do so by making DTP updates moresimilar to SGD updates (e.g. [25, 12]).
Revised: Gauss-Newton updates are within 90 ◦ of SGD updates [25], and recent attempts to improve DTP generally do so by making DTP updates more similar to SGD updates (e.g. [25, 12]).

Original: Additionally, variants of BP algorithms like random feedbackalignment [21] and sign symmetric feedback alignment [45] approximate SGD.
Revised: Additionally, variants of BP algorithms like random feedback alignment [21] and sign symmetric feedback alignment [45] approximate SGD.

Original: Since all of thesemethods approximate the (explicit) SGD update over weights, we consolidate them all under theheading of generalized BP (G-BP).
Revised: Since all of these methods approximate the (explicit) SGD update over weights and use similar learning rules, we consolidate them all under the heading of generalized BP (G-BP).

Original: The target-based formulation of G-BP is shown in algorithm 1.
Revised: The target-based formulation of G-BP is shown in algorithm 1.

Original: 3.3 Generalized Inference Learning
Revised: 3.3 Generalized Inference Learning

Original: We now present a general description of the IL algorithm, which we call Generalized Inference
Revised: We now present a general description of the IL algorithm, which we call Generalized Inference

Original: Learning (G-IL).
Revised: Learning (G-IL).

Original: A summary of the algorithm is presented in algorithm 2.
Revised: A summary of the algorithm is presented in algorithm 2.

Original: At each layer n of an MLP,
Revised: At each layer n of an MLP,

Original: G-IL stores two variables: ˆ h n and p n .
Revised: G-IL stores two variables: ˆ h n and p n .

Original: At initialization these variables are set to feed-forward activityvalues:
Revised: At initialization these variables are set to feed-forward activity values:

Original: ˆ h n = p n = h n .
Revised: ˆ h n = p n = h n .

Original: Then ˆ h n are altered over time to minimize the energy function F :
Revised: Then ˆ h n are altered over time to minimize the energy function F :

Original: [Equation]
Revised: [Equation]

Original: where L is the global loss,
Revised: where L is the global loss,

Original: 12 ∥
Revised: 12 ∥

Original: ˆ h n − p n ∥ 2 are local losses, γ decayn 12 ∥
Revised: ˆ h n − p n ∥ 2 are local losses, γ decayn 12 ∥

Original: ˆ h n ∥ 2 is an optional regularization term, and γ are positive scalar weighting terms.
Revised: ˆ h n ∥ 2 is an optional regularization term, and γ are positive scalar weighting terms.

Original: The prediction p n is computed asp n = W n − 1 f (ˆ h n − 1 ) .
Revised: The prediction p n is computed as p n = W n − 1 f (ˆ h n − 1 ) .

Original: (Note p n may also be computed as p n = f ( W n − 1 ˆ h n − 1 ) .
Revised: (Note p n may also be computed as p n = f ( W n − 1 ˆ h n − 1 ) .

Original: Our theoreticalresults use the first formulation, but we find little difference in performance between the two inpractice.
Revised: Our theoretical results use the first formulation, but we find little difference in performance between the two in practice.

Original: See simulations below.)
Revised: See simulations below.)

Original: G-IL first minimizes F w.r.t.
Revised: G-IL first minimizes F w.r.t.

Original: to activities ˆ h (inference phase).
Revised: to activities ˆ h (inference phase).

Original: Whileoptimizing ˆ h , predictions also change since p n +1 = W n f (ˆ h n ) .
Revised: While optimizing ˆ h , predictions also change since p n +1 = W n f (ˆ h n ) .

Original: After argmin ˆ h F is approximated,
Revised: After argmin ˆ h F is estimated,

Original: F is again minimized, now w.r.t.
Revised: F is again minimized, now w.r.t.

Original: the weight, i.e. argmin W F .
Revised: the weight, i.e. argmin W F .

Original: Typical IL algorithms use SGD toperform the inference phase and either the LMS rule (below) or Adam optimizers to update weights(e.g.
Revised: Typical IL algorithms use SGD to perform the inference phase with either the LMS rule (below) or Adam optimizers to update weights (e.g. [33, 44]).

Original: G-IL, on the other hand, is general w.r.t.
Revised: G-IL, on the other hand, is general w.r.t.

Original: the optimization processes that approximateargmin W F and argmin ˆ h F .
Revised: the optimization processes that approximate argmin W F and argmin ˆ h F .

Original: We consider two weight updates in our analysis and simulations below.
Revised: We consider two weight updates in our analysis and simulations below.

Original: First, is a local error gradient update, which is equivalent to the least-mean squares (LMS) rule:
Revised: First, is a local error gradient update, which is equivalent to the least-mean squares (LMS) rule:

Original: [Equation]
Revised: [Equation]

Original:  where e n [-+
-] =
Revised: (5) where e n [-+
-]+1 =

Original: ˆ h n +1 − p n +1 and local loss l n +1 = 

Revised: ˆ h n +1 − p n +1 and local loss l n +1 =

Original: ∥
Revised: ∥

Original: e n +1 ∥ 2 .
Revised: e n +1 ∥ 2 .

Original: The LMS rule does not solveargmin W F but only approximates it.
Revised: The LMS rule does not solve argmin W F but only approximates it.

Original: The next rule, which is equivalent to the normalized leastmean squared rule (NLMS) [29] is
Revised: The next rule, which is equivalent to the normalized least mean squared rule (NLMS) [29] is

Original: [Equation]
Revised: [Equation]

Original: The NLMS rule performs a gradient update but with an inverse squared ℓ 2 norm of the pre-synapticactivities as its learning rate instead of a hyper-parameter.
Revised: The NLMS rule performs a gradient update but with an inverse squared ℓ 2 norm of the pre-synaptic activities as its step size instead of a hyper-parameter.

Original: The NLMS rule is the minimum-normsolution to argmin W F in the case where we use mini-batch size 1 (see proposition B.1).
Revised: The NLMS rule is the minimum-norm solution to argmin W F in the case where we use mini-batch size 1 (see proposition B.1).

Original: The LMSrule closely approximates the NLMS update in the sense the two are proportional.
Revised: The LMS rule closely approximates the NLMS update in the sense the two are proportional.

Original: Comparing theserules to the update rule for G-BP (equation 2) we see the main difference is the pre-synaptic termused in the rule: BP uses FF activity h n , while IL uses optimized/target activity ˆ h n .
Revised: Comparing these rules to the update rule for G-BP (equation 3) we see the main difference is the pre-synaptic term used in the rule: BP uses FF activity h n , while IL uses optimized/target activity ˆ h n .

Original: This differenceleads to significantly different stability properties between the algorithms, as we note below.
Revised: This difference leads to significantly different stability properties between the algorithms, as we note below.

Original: Algorithm 1: Generalized BP begin //
Revised: begin //

Original: Feedforward Pass h 0 ← x ( b ) for n = 0 to N − 1 do h n +1 ← W n f ( h n ) end ˆ h N ← y ( b ) //
Revised: Feedforward Pass h 0 ← x ( b ) for n = 0 to N − 1 do h n +1 ← W n f ( h n ) end ˆ h N ← y ( b ) //

Original: Compute Local Targets for n = 1 to N do ˆ h n ≈ h n − ∂L ∂h n end //
Revised: Compute Local Targets for n = 1 to N do ˆ h n ≈ h n − ∂L∂h n end //

Original: Update Weight Matrices
Revised: Update Weight Matrices

Original: Eqn.end
Revised: Eqn.end

Original: Algorithm2: GeneralizedIL begin // Feedforward Pass
Revised: Algorithm Generalized begin // Feedforward Pass

Original: ˆ h 0 ← x ( b ) for n = 0 to N − 1 do p n +1 , ˆ h n +1 ← W n f ( h n ) end //
Revised: ˆ h 0 ← x ( b ) for n = 0 to N − 1 do p n +1 , ˆ h n +1 ← W n f ( h n ) end //

Original: Compute Local Targets [Equation]
Revised: Compute Local Targets (ˆ h 1 , ...

Original:  argmin[Equation]
Revised: ˆ h N ) ≈ argmin

Original:  // Update Weight Matrices [Equation] end W 
Revised: ˆ h F // Update Weight Matrices ( W 0 , ...

Original: W

Original: W 2 h 0 Initialize

Original: W

Original: W

Original: W

Original: W

Original: W 2 P 2 W

Original: Energy

Original: Minimization

Original: Weight

Original: Update

Original: W 0 * W 1 * W 2 * L(h 3 , y)
Revised: W N − 1 ) ≈ argmin W F , Eqn. 4, end

Original: y h

Original: L(h 3 , y)

Original: L(h 3 , y)

Original: L(h 3 , y)

Original:  Theoretical Results 
Revised:  Theoretical Results 4.1 Implicit Gradient Descent

Original: [Figure] 4.1 Implicit Gradient Descent

Original: Let the set of MLP weight parameters θ ( b ) =
Revised: Let the set of MLP weight parameters θ ( b ) =

Original: [ W ( b ) 0 , ...
Revised: [ W ( b ) 0 , ...

Original: W ( b ) N − 1 ] , input data x ( b ) , and output target y ( b ) ,where b is the current training iteration.
Revised: W ( b ) N − 1 ] , input data x ( b ) , and output target y ( b ) , where b is the current training iteration.

Original: The explicit SGD update uses the loss gradient produced bythe current parameters:[Equation]
Revised: The explicit SGD update uses the loss gradient produced by the current parameters:

Original: [Equation]
Revised: [Equation]

Original: where α is step size and L is the loss measure.
Revised: where α is step size and L is the loss measure.

Original: This update is explicit because the gradient canbe readily computed given ( θ ( b ) , x ( b ) , y ( b ) ) .
Revised: This update is explicit because the gradient can be readily computed given ( θ ( b ) , x ( b ) , y ( b ) ) .

Original: The implicit SGD update uses the loss gradient of theparameters at the next training iteration:
Revised: The implicit SGD update uses the loss gradient of the parameters at the next training iteration:

Original: [Equation]
Revised: [Equation]

Original: This is an implicit update because θ ( b +1) shows up on both sides of the equation.
Revised: This is an implicit update because θ ( b +1) shows up on both sides of the equation.

Original: Unlike explicit
Revised: Unlike explicit

Original: SGD, θ ( b +1) cannot be readily computed using available quantities.
Revised: SGD, θ ( b +1) cannot be readily computed using available quantities.

Original: However, the implicit gradient isequivalent to the solution of the following optimization problem (see appendix equation 13):
Revised: However, the implicit gradient is equivalent to the solution of the following optimization problem (see appendix equation 14):

Original: [Equation]
Revised: [Equation]

Original: This update is equivalent to the proximal operator/update [31].
Revised: This update is equivalent to the proximal operator/update [31].

Original: This proximal update changesparameters θ ( b ) in a way that minimizes the loss L and the magnitude of the update, which helps keepθ ( b +1) in the proximity of θ ( b ) .
Revised: This proximal update changes parameters θ ( b ) in a way that minimizes the loss L and the magnitude of the update, which helps keep θ ( b +1) in the proximity of θ ( b ) .

Original: (For more details on the proximal update and its relation to implicit
Revised: (For more details on the proximal update and its relation to implicit

Original: SGD see appendix B.1).
Revised: SGD see appendix B.1).

Original: 4.2 Main Results
Revised: 4.2 Main Results

Original: We now present our main result, which is to show G-IL is equivalent to implicit SGD in certain limitsthat are approximated well in practice.
Revised: We now present our main result, which is to show G-IL is equivalent to implicit SGD in certain limits that are approximated well in practice.

Original: We focus on the case where single data-points (mini-batch size1) are presented each training iteration.
Revised: We focus on the case where single data-points (mini-batch size 1) are presented each training iteration.

Original: This scenario is more biologically realistic than mini-batchingand is the case where IL best approximates implicit SGD.
Revised: This scenario is more biologically realistic than mini-batching and is the case where IL best approximates implicit SGD.

Original: We discuss the case of mini-batching inappendix B.5.
Revised: We discuss the case of mini-batching in appendix B.5.

Original: We first define a kind of IL algorithm that minimizes the proximal loss (equation 8)w.r.t.
Revised: We first define a kind of IL algorithm that minimizes the proximal loss (equation 9) w.r.t.

Original: neuron activities and show G-IL approaches this algorithm in certain limits.
Revised: neuron activities and show G-IL approaches this algorithm in certain limits.

Original: Definition 4.1.
Revised: Definition 4.1.

Original: Proximal Inference Learning (IL-prox).
Revised: Proximal Inference Learning (IL-prox).

Original: An algorithm identical to G-IL (algorithm2), except activities are optimized according to argmin ˆ h prox =
Revised: An algorithm identical to G-IL (algorithm 2), except activities are optimized according to argmin ˆ h prox =

Original: argmin
Revised: argmin

Original: ˆ h ( L ( θ ∗ ) +
Revised: ˆ h ( L ( θ ∗ ) +

Original: 12 α ∥ θ ∗ − θ ( b ) ∥ 2 )and the NLMS rule is used to update weights. 
Revised: 12 α ∥ θ ∗ − θ ( b ) ∥ 2 ) and the NLMS rule is used to update weights. W W W 2 h 0 Initialize

Original: Here b is the current training iteration and θ ∗ are the parameters updated with the NLMS rule.
Revised: W W W W

Revised: W 2 P 2 W

Revised: Energy

Revised: Minimization

Revised: Weight

Revised: Update

Revised: W 0 *

Revised: W 1 *

Revised: W 2 *

Revised: L(h 3 , y)

Revised: y h

Revised: L(h 3 , y)

Revised: L(h 3 , y)

Revised: L(h 3 , y)

Revised: Here b is the current training iteration and θ ∗ are the parameters updated with the NLMS rule.

Original: IL-proxis the same as G-IL except during the inference phase, it minimizes the proximal loss w.r.t.
Revised: IL-prox is the same as G-IL except during the inference phase, it minimizes the proximal loss w.r.t.

Original: activitiesˆ h instead of the energy F .
Revised: activities ˆ h instead of the energy F .

Original: Intuitively, minimizing the proximal loss w.r.t.
Revised: Intuitively, minimizing the proximal loss w.r.t.

Original: activities will result in ˆ hthat yields a small loss after weights are updated and small weight update norms.
Revised: activities will result in ˆ h that yields a small loss after weights are updated and small weight update norms.

Original: Weight updatenorm 12 ∥ θ ∗ − θ ( b ) ∥ 2 is known during the inference phase because the NLMS learning rule is usedand known explicitly:
Revised: Weight update norm 12 ∥ θ ∗ − θ ( b ) ∥ 2 is known during the inference phase because the NLMS learning rule is used and known explicitly:

Original: [Equation]
Revised: [Equation]

Original: where α n = ∥
Revised: where α n = ∥

Original: f
Revised: f

Original: (ˆ h n ) ∥ − 2 .
Revised: (ˆ h n ) ∥ − 2 .

Original: Thus, it can be minimized w.r.t.
Revised: Thus, it can be minimized w.r.t.

Original: ˆ h n .
Revised: ˆ h n .

Original: As we explain in the appendix (seelemma B.1 and proposition B.1), the loss L ( θ ∗ ) is also explicitly known during the inference phaseand can be optimized w.r.t.
Revised: As we explain in the appendix (see lemma B.1 and proposition B.1), the loss L ( θ ∗ ) is also explicitly known during the inference phase and can be optimized w.r.t.

Original: ˆ h n .
Revised: ˆ h n .

Original: Now, let α be a ’global’ learning rate hyper-parameter, and α n be layer-wise learning rates used inthe actual weight update ∆ W n .
Revised: Now, let α be a ’global’ learning rate hyper-parameter, and α n be layer-wise learning rates used in the actual weight update ∆ W n .

Original: Theorem 4.1.
Revised: Theorem 4.1.

Original: Let α n = ∥
Revised: Let α n = ∥

Original: f
Revised: f

Original: (ˆ h n ) ∥ − 2 and assume mini-batch size 1.
Revised: (ˆ h n ) ∥ − 2 and assume mini-batch size 1.

Original: In the limit where γ decayn →∥ e n +1 ∥ 2 (1+ − 2 α 6 n ) , γ N → α N − 1 α , and γ n → α − 1 n − 1 for all n < N , it is the case that argmin
Revised: In the limit where γ decay n → ∥ e n +1 ∥ 2 (1 − 2 α 6 n ) , γ N → α N − 1 α , and γ n → α − 1 n − 1 for all n < N , it is the case that argmin

Original: ˆ h F =argmin ˆ h L ( θ ∗ )
Revised: ˆ h F = argmin ˆ h L ( θ ∗ )

Original: +
Revised: +

Original: 12 α ∥ ∆ θ ∥ 2 .
Revised: 12 α ∥ ∆ θ ∥ 2 .

Original: Hence, in these limits G-IL is equivalent to IL-prox.
Revised: Hence, in these limits G-IL is equivalent to IL-prox.

Original: The proof can be found in supplementary materials theorem B.1.
Revised: The proof can be found in supplementary materials theorem B.1.

Original: This theorem states that whenusing the NLMS rule, G-IL is increasingly similar to IL-prox, as the γ weighting terms in the freeenergy (equation 3) approach the scalar values specified in the above theorem.
Revised: This theorem states that when using the NLMS rule, G-IL is increasingly similar to IL-prox, as the γ weighting terms in the free energy (equation 4) approach the scalar values specified in the above theorem.

Original: In these limits,argmin ˆ h prox =
Revised: In these limits, argmin ˆ h prox =

Original: argmin ˆ h F and G-IL is equivalent to IL-prox.
Revised: argmin ˆ h F and G-IL is equivalent to IL-prox.

Original: The intuitive explanation of thistheorem goes as follows: minimizing the proximal loss (equation 11) requires minimizing L w.r.t.θ ∗ while also minimizing the update norm 12 α ∥ θ ∗ − θ ( b ) ∥ 2 .
Revised: The intuitive explanation of this theorem goes as follows: minimizing the proximal loss (equation 9) requires minimizing L w.r.t. θ ∗ while also minimizing the update norm 12 α ∥ θ ∗ − θ ( b ) ∥ 2 .

Original: The inference phase of IL minimizes
Revised: The inference phase of IL minimizes

Original: L w.r.t.
Revised: L w.r.t.

Original: θ ∗ by initializing output layer activity ˆ h N to FF output
Revised: θ ∗ by initializing output layer activity ˆ h N to FF output

Original: h
Revised: h

Original: N then shifting it toward globaltarget y .
Revised: N then shifting it toward global target y .

Original: Upon a weight update this will yield a θ ∗ that produces a smaller loss L .
Revised: Upon a weight update this will yield a θ ∗ that produces a smaller loss L .

Original: See figure 4.1for visualization.
Revised: See figure 4. for visualization.

Original: The magnitude of a weight update12 α ∥ αe n +1 f
Revised: The magnitude of a weight update ∥ e n +1 f

Original: (ˆ h n ) T ∥ 2 intuitively depends onthe magnitude of local prediction error e n +1 .
Revised: (ˆ h n ) T ∥ 2 intuitively depends on the magnitude of local prediction error e n +1 .

Original: F is a sum of the magnitudes of errors (see equation3), so minimizing F minimizes the magnitudes of errors and consequently weight update norms12 α ∥ θ ∗ − θ ( b ) ∥ 2 .
Revised: F is a sum of the magnitudes of errors (see equation 4), so minimizing F minimizes the magnitudes of errors and consequently weight update norms 12 α ∥ θ ∗ − θ ( b ) ∥ 2 .

Original: Importantly, the γ settings under which theorem 4.1 holds can be computed exactlyin practice and are of reasonable magnitude that is easily approximated in practice (e.g., they do notapproach ∞ or 0 and typically approach positive scalars).
Revised: Importantly, the γ settings under which theorem 4.1 holds can be computed exactly in practice and are of reasonable magnitude that is easily approximated in practice (e.g., they do not approach ∞ or 0 and typically approach positive scalars).

Original: This is opposed to the limits under which
Revised: This is opposed to the limits under which

Original: IL approximates BP/explicit SGD, which is the limit where
Revised: IL approximates BP/explicit SGD, which is the limit where

Original: ˆ h n → h n and thus ∆ W n → 0 [44], alimit that does not typically obtain in practice.
Revised: ˆ h n → h n and thus ∆ W n → 0 [44], a condition that does not generally occur in practice.

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Let θ ( b ) be a set of MLP parameters at training iteration b .
Revised: Let θ ( b ) be a set of MLP parameters at training iteration b .

Original: Let θ ( b +1) prox =
Revised: Let θ ( b +1) prox =

Original: argmin θ L ( θ ) + 12 α ∥ θ − θ ( b ) ∥ 2 .
Revised: argmin θ L ( θ ) + 12 α ∥ θ − θ ( b ) ∥ 2 .

Original: Let θ ( b +1)
Revised: Let θ ( b +1)

Original: IL − prox be the parameters updated by IL-prox (see def.4.1) and θ ( b +1)
Revised: IL − prox be the parameters updated by IL-prox (see def. 4.1) and θ ( b +1)

Original: IL the parameters updated by G-IL under γ values in theorem 4.1.
Revised: IL the parameters updated by G-IL under γ values in theorem 4.1.

Original: Assume mini-batchsize 1.
Revised: Assume mini-batch size 1.

Original: Under this assumption, it is the case θ ( b +1) prox =
Revised: Under this assumption, it is the case θ ( b +1) prox =

Original: θ ( b +1) IL − prox
Revised: θ ( b +1) IL − prox

Original: =
Revised: =

Original: θ ( b +1) IL .
Revised: θ ( b +1) IL .

Original: This theorem states that the IL-prox update is equivalent to the proximal update (equation 8), and thusimplicit SGD.
Revised: This theorem states that the IL-prox update is equivalent to the proximal update (equation 9), and thus implicit SGD.

Original: It follows that, under the γ settings in theorem 4.1 the G-IL update is also equivalentto the proximal update/implicit gradient update.
Revised: It follows that, under the γ settings in theorem 4.1 the G-IL update is also equivalent to the proximal update/implicit gradient update.

Original: Further theoretical results are developed in the appendix.
Revised: Further theoretical results are developed in the appendix.

Original: A novel closed form description of ILtargets is developed  C.5.
Revised: A novel closed form description of IL targets is developed in C.5.

Original: The closed form description shows local targets approximate a regularized
Revised: The closed form description shows local targets approximate a regularized

Original: Gauss-Newton update on FF activities (see section C).
Revised: Gauss-Newton update on FF activities (see section C).

Original: We call this closed form approximation IL-GN.
Revised: We call this closed form approximation IL-GN.

Original: We study the stability of IL-GN when using the LMS learning rule, and compare to an analogous BPnetwork, which uses Gauss-Newton updates (BP-GN).
Revised: We study the stability of IL-GN when using the LMS learning rule, and compare to an analogous BP network, which uses Gauss-Newton updates (BP-GN).

Original: We show that IL-GN weight updates pushoutput layer activities down their loss gradients for any positive learning rate , i.e., for any positivelearning rate
Revised: We show that IL-GN weight updates push output layer activities down their loss gradients for any positive learning rate , i.e., for any positive learning rate

Original: ˆ h ( b ) N − ˆ h ( b +1) N =
Revised: ˆ h ( b ) N − ˆ h ( b +1) N =

Original: j ∂L ∂ ˆ h ( b ) N where j is a positive scalar (theorem D.1).
Revised: j ∂L ∂ ˆ h ( b ) N where j is a positive scalar (theorem D.1).

Original: We show BP-GN only has this property for a small range of learning rates (theorem D.2).
Revised: We show BP-GN only has this property for a small range of learning rates (theorem D.2).

Original: We develop a partial explanationof why IL is more stable than BP that is based on differences in their learning rule (for details seesection E).
Revised: We develop a partial explanation of why IL is more stable than BP that is based on differences in their learning rule (for details see section E).

Original: These results suggest that the differences in the way IL and BP compute and use localtargets in their weight updates can explain much of their differences in stability across learning rate.
Revised: These results suggest that the differences in the way IL and BP compute and use local targets in their weight updates can explain much of their differences in stability across learning rate.

Original: The IL learning rule and target computations are advantageous over that of G-BP in this sense.
Revised: The IL learning rule and target computations are advantageous over that of G-BP in this sense.

Original:  Experiments
Revised:  Experiments

Original: [Table]

Original: with mini-batch size 1 after 50,000 iterations (1 epoch) and with mini-batch size 64 with 77000 iterations.

Original: In this section, we compare the performance of BP based algorithms to ILalgorithms.
Revised: In this section, we compare the performance of BP based algorithms to IL algorithms.

Original: BP algorithms computeand use explicit gradients to updateweights, while IL algorithms computeand use approximate implicit gradients to perform updates.
Revised: BP algorithms compute and use explicit gradients to update weights, while IL algorithms compute and use approximate implicit gradients to perform updates.

Original: We test BPmodels that either perform simple gradient descent ( BP-SGD ) or use Adamoptimizers ( BP-Adam ).
Revised: We test BP models that either perform simple gradient descent ( BP-SGD ) or use Adam optimizers ( BP-Adam ).

Original: IL-SGD isa simple implementation of IL that issimilar to implementations of [33, 44].
Revised: IL-SGD is a simple implementation of IL that is similar to implementations of [33, 44].

Original: It uses the LMS rule (local error gradient update) to update weights andoptimizes ˆ h using SGD to near convergence (25 update steps).
Revised: It uses the LMS rule (local error gradient update) to update weights and optimizes ˆ h using SGD to near convergence (25 update steps).

Original: IL-Proxis our novel algorithm that is based ondefinition 4.1.
Revised: IL-Prox is our novel algorithm that is based on definition 4.1.

Original: IL-prox uses the NLMS rule to update weights.
Revised: IL-prox uses the NLMS rule to update weights.

Original: Following theorem 4.1, the learningrate is only used to determine how much the output layer activities ˆ h N are pushed toward the targety .
Revised: Following theorem 4.1, the learning rate is only used to determine how much the output layer activities ˆ h N are pushed toward the target y .

Original: As α → ∞ , ˆ h N → y and as α → 0 , ˆ h N → h N .
Revised: As α → ∞ , ˆ h N → y and as α → 0 , ˆ h N → h N .

Original: IL-Prox optimizes ˆ h using SGD to nearconvergence (25 update steps).
Revised: IL-Prox optimizes ˆ h using SGD to near convergence (25 update steps).

Original: IL-Prox Fast is the same as theIL-Prox algorithm
Revised: IL-Prox Fast is the same as the

Original:  except it truncatesthe optimization of ˆ h to only 12 iterations.
Revised: IL-Prox algorithm except it truncates the optimization of ˆ h to only 12 iterations.

Original: IL-Adam computes the weight gradients using IL-SGD,then uses Adam optimizers to update weights.
Revised: IL-Adam computes the weight gradients using IL-SGD, then uses Adam optimizers to update weights.

Original: IL-prox Adam computes the normalized weightgradient using IL-prox and uses Adam optimizers to update weights.
Revised: IL-prox Adam [Table]

Original: All simulations average resultsover at least 5 seeds of each model type.
Revised: All simulations average results over at least 5 seeds of each use used is used on

Original: Best test scores from these averages are shown in tables. All models use ReLU activations at hidden layers. Softmax is used at output layer for classification,while sigmoid is used on the autoencoder task.
Revised: [Table]

Original: Mini-batch Training Previous work has shown that IL algorithms can achieve the same accuracy as 
Revised: Mini-batch Training Previous work has shown that IL algorithms can achieve the same accuracy as BP on small scale data sets (e.g., MNIST and

Original: BP on small scale data sets (e.g., MNIST and Fashion-MNIST), in the case where Adam optimizersand medium sized mini-batches (e.g, 64) are used ( [44, 2].
Revised: Fashion-MNIST), in the case where Adam optimizers and medium sized mini-batches (e.g, 64) are used ( [44, 2].

Original: Here we train our BP and IL algorithmson CIFAR-10 with mini-batch size 64.
Revised: Here we train our BP and IL algorithms on CIFAR-10 with mini-batch size 64.

Original: Results are summarized in table 2.
Revised: Results are summarized in table 2.

Original: Consistent with previousresults, IL algorithms, including our novelIL-prox, achieve near equal accuracy as BP-SGD and [Table]
Revised: Consistent with previous results, IL algorithms, including our novel

Original:  BP-Adam when using Adam optimizers.
Revised: IL-prox, achieve near equal accuracy as BP-SGD and BP-Adam when using Adam optimizers.

Original: However, unlike previous studies, we also test IL-SGD and 
Revised: However, unlike previous studies, we also test ILSGD and IL-prox algorithms without

Original: IL-prox algorithms without Adam, and find they do not perform as well as BP algorithms in thisscenario.
Revised: Adam, and find they do not perform as well as BP algorithms in this scenario.

Original: IL algorithms without Adam tend to converge more quickly but to shallower local minima.
Revised: IL algorithms without Adam tend to converge more quickly but to shallower local minima.

Original: Adam, with its momentum and adaptive learning rate, may help prevent IL algorithms from gettingcaught in shallow local minima (see discussion).
Revised: Adam, with its momentum and adaptive learning rate, may help prevent IL algorithms from getting caught in shallow local minima (see discussion).

Original: Online Learning We find IL algorithms tend to converge significantly quicker and to more similaraccuracies as BP when mini-batch size 1 is used. This is especially apparent when each data point isseen only once by the network, similar to online learning scenarios.
Revised: Online Learning We find IL algorithms tend to converge significantly quicker and to more similar loss/accuracies as BP in a more biologically realist, online learning scenario, where a single data-point is presented each training iteration and each data point is seen only once.

Original: IL algorithms improve accuracysignificantly faster during the first epoch of training on CIFAR-10 with mini-batch size 1 (table 2,figure 5).
Revised: This is especially apparent with CIFAR-10, shown in tables 2 and 3 and figure 2.

Original: IL-SGD and IL-prox algorithms perform similar to and even slightly better than BP-Adam,despite not using momentum or stored, parameter-wise adaptive learning rates like Adam.
Revised: IL-SGD and IL-prox algorithms perform similar to and even slightly better than BP-Adam, despite not using momentum or stored, parameter-wise adaptive learning rates like Adam.

Original: As far aswe know, this is a novel result.
Revised: As far as we know, this is a novel result.

Original: Less significant speedups were also seen on MNIST and FashionMNIST classification tasks (see figure F.3).
Revised: Less significant speedups were also seen on MNIST and Fashion-MNIST classification tasks (see figure 5).

Original: On autoencoder tasks, IL-prox algorithms, in particular,decreased loss significantly quicker than BP-SGD and even slightly quicker than BP-Adam (figure 5).
Revised: On autoencoder tasks, IL-prox algorithms, in particular, decreased loss significantly quicker than BP-SGD and even slightly quicker than BP-Adam (figure 2).

Original: Learning Under Memory Constraints We also find that IL algorithms can converge to betteraccuracy than BP in certain memory constrained scenarios.
Revised: Learning Under Data Constraints We also find that IL algorithms can converge to better accuracy than BP in certain data constrained scenarios.

Original: We trained IL algorithms using on only10, 100, or 500 data points from each category on F-MNIST and CIFAR-10, with mini-batch size100.
Revised: We trained IL algorithms using on only 10, 100, ordata points from each category on F-MNIST and CIFAR-10, with mini-batch size 100. IL algorithms Stability Test: CIFAR-10 Test Accuracy Model lr=. lr=.

Original: IL algorithms achieved betteraccuracies than BP-SGD in the 10 and 100 data point scenarios,while performances evened out in the 500 data point scenario (see appendix, table 8).
Revised: lr= lr=2. lr= lr= [Table] achieved better

Original: Stability Test Explicit SGD is well-known to be highly sensitive to learning rate.
Revised: Stability Test Explicit SGD is well-known to be highly sensitive to learning rate.

Original: Implicit SGD, on theother hand, is highly insensitive to learning rate and is unconditionally stable, i.e., it is able to discountthe loss in a stable manner for nearly any positive learning rate [42].
Revised: Implicit SGD, on the other hand, is highly insensitive to learning rate and is unconditionally stable, i.e., it is able to discount the loss in a stable manner for nearly any positive learning rate [42].

Original: We compare performance of
Revised: We compare performance of

Original: BP-SGD, IL-SGD, and IL-prox on MNIST (table 5) and CIFAR-10 (table 4) classification task acrossdifferent learning rates.
Revised: BP-SGD, IL-SGD, and IL-prox on MNIST (table 5) and CIFAR-10 (table 4) classification task across different learning rates.

Original: As a control, we train a hybrid of BP and IL-prox (BP-prox).
Revised: As a control, we train a hybrid of BP and IL-prox (BP-prox).

Original: BP-prox, like
Revised: BP-prox, like

Original: IL-prox, uses the learning rate only to adjust the target at the output layer and uses the NLMS rule toupdate weights.
Revised: IL-prox, uses the learning rate only to adjust the target at the output layer and uses the NLMS rule to update weights.

Original: Unlike IL-prox, BP-prox computes local targets using the G-BP algorithm ratherthan the IL algorithm.
Revised: Unlike IL-prox, BP-prox computes local targets using the G-BP algorithm rather than the IL algorithm.

Original: Results can be found in table 5 and 4.
Revised: Results can be found in table 5 and 4.

Original: Blank entries are those with accuraciesbelow 12% .
Revised: Blank entries are those with accuracies below 12% .

Original: IL-SGD is more stable than BP-SGD, and IL-prox is more stable than BP-prox.
Revised: IL-SGD is more stable than BP-SGD, and IL-prox is more stable than BP-prox.

Original: Thissuggests that the way IL computes and uses targets contributes to stability.
Revised: This suggests that the way IL computes and uses targets contributes to stability.

Original: Additionally, IL-proxand BP-prox algorithms are more stable than SGD algorithms, showing that using the normalizedgradient to update weights and α to adjust the output layer target (rather than scale weight updates)also contributes to stability.[Table] lr= [Table]
Revised: Additionally, IL-prox and BP-prox algorithms are more stable than SGD algorithms, showing that using the normalized gradient to update weights and α to adjust the output layer target (rather than scale weight updates) also contributes to stability.

Original: lr=. lr= lr=2.

Original: lr= lr=

Original: [Table]

Original: − − − − −

Original: [Table]

Original: − − − −

Original: [Table]

Original:  Limitations
Revised:  Limitations

Original: Our theoretical results show that IL algorithms closely approximate implicit SGD in the case whereonly a single data-point (i.e., mini-batch size 1) is presented each training iteration.
Revised: Our theoretical results show that IL algorithms closely approximate implicit SGD in the case where only a single data-point (i.e., mini-batch size 1) is presented each training iteration.

Original: This approximation is looser in the case where large mini-batches are used (see appendix B.5).
Revised: This approximation is looser in the case where large mini-batches are used (see appendix B.5).

Original: This is no problemfrom a biological point of view, since the brain does not train with large mini-batches.
Revised: This is no problem from a biological point of view, since the brain does not train with large mini-batches.

Original: Additionally,when Adam optimizers are used IL still performs similarly to BP.
Revised: Additionally, when Adam optimizers are used IL still performs similarly to BP.

Original: However, it may still be desirable tofurther study the optimization properties of IL with mini-batches to improve performance on machinelearning tasks where mini-batches are typically used.
Revised: However, it may still be desirable to further study the optimization properties of IL with mini-batches to improve performance on machine learning tasks where mini-batches are typically used.

Original:  Discussion
Revised:  Discussion

Original: In this paper, we found that IL closely approximates implicit SGD, which is distinct from  explicit
Revised: In this paper, we found that IL closely approximates implicit SGD, which is distinct from the explicit

Original: SGD implemented by BP.
Revised: SGD performed by BP.

Original: This approximation holds best when mini-batch size one is used.

Original: Thistheoretical result, along with results that help to characterize how IL and BP behave differently,suggest a hypothesis, which as far was we know is novel: learning in the brain is more similar toimplicit gradient descent than explicit gradient descent.
Revised: This theoretical result, along with results that help to characterize how IL and BP behave differently, suggest a hypothesis, which as far was we know is novel: learning in the brain is more similar to implicit gradient descent than explicit gradient descent.

Original: This hypothesis would imply the learningcharacteristics of the brain behave differently from BP and explicit SGD.
Revised: This hypothesis would imply the learning characteristics of the brain behave differently from BP and explicit SGD, more generally.

Original: More empirical work willbe needed to test this theory.
Revised: More empirical work will be needed to test this theory.

Original: However, recent evidencethat directly supports the idea the brain doessomething similar to IL [39] and evidence that performs predictive coding provides  the idea withinitial plausibility.
Revised: However, recent evidence directly supports the idea the brain does something similar to IL [39]. Evidence that neural circuits perform predictive coding provides further, though more indirect, evidence for the idea.

Original: We also found that IL often converged more quickly than BPto similar, and sometimes better,performance in mini-batch size one and memory constrained scenarios.
Revised: We also found that IL often converged more quickly than BP and achieved similar test losses and accuracies in online and data constrained scenarios.

Original: However, in large mini-batchscenarios without memory constraints and without the use of Adam optimizers, IL sometimes converged to worse accuracies than BP in classification tasks.
Revised: However, in large mini-batch scenarios without data constraints and without the use of Adam optimizers, IL sometimes converged to worse accuracies than BP in classification tasks.

Original: With thus use of Adam, IL’s performancemore closely matched that of BP.
Revised: With the use of Adam, IL’s performance more closely matched that of BP.

Original: One possible explanation of this empirical finding may be related tothe fact that IL, and implicit SGD more generally, tend to take smaller steps over parameters and amore direct path toward local minima than BP (e.g., see figures F.6 and table 9).
Revised: One possible explanation of this empirical finding may be related to the fact that IL, and implicit SGD more generally, tend to take smaller steps over parameters and a more direct path toward local minima than BP (e.g., see figures 7 and table 9).

Original: More direct paths tominima often lead to faster convergence.
Revised: More direct paths to minima often lead to faster convergence.

Original: Smaller update magnitudes may  act as a kind of built-inregularization that aids in difficult, noisy learning scenarios, such as small mini-batch and memoryconstrained scenarios.
Revised: Smaller update magnitudes may also act as a kind of built-in regularization that aids in difficult, noisy learning scenarios, such as small mini-batch and data constrained scenarios.

Original: In cases where regularization is less advantageous (e.g., no memory constraintsand large mini-batches), IL’s small update magnitudes may provide less of an advantage, and its moredirect optimization paths may push parameters into shallow, nearby minima.
Revised: In cases where regularization is less advantageous (e.g., no data constraints and large mini-batches), IL’s small update magnitudes may provide less of an advantage, and its more direct optimization paths may push parameters into shallow, nearby minima.

Original: Adam optimizers, withtheir momentum and adaptive learning rates, may help push parameters out of shallow minima.
Revised: Adam optimizers, with their momentum and adaptive learning rates, may help push parameters out of shallow minima.

Original: Morework will be needed to fully characterize and explain these behavioral differences, but we believe thework done here provides a good formal basis for doing so.
Revised: More work will be needed to fully characterize and explain these behavioral differences, but we believe the work done here provides a good formal basis for doing so.

Original: These results further suggest IL may haveadvantages over BP in more biologically realistic machine learning tasks, such as  online learningscenarios with autonomous and/or neuromorphic systems, where  datamay be noisy, sparse, andmini-batches are small/size of one. Checklist For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
Revised: These results further suggest IL may have advantages over BP in more biologically realistic machine learning tasks, such as lifelong, online learning scenarios, where streams of data are received by the neural network, a single datapoint at a time, and the model must adapt quickly to changes in the environment and minimize the magnitude of weight updates to prevent catastrophic forgetting.

Original: [Yes]

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See section

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] See section??(d)

Original: Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: For all theorems,

Original: propositions, and lemmas assumptions were stated in the statement.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] All proofs are inappendix

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] This information is  in supplementary material and in link to code
Revised: This possibility is especially interesting given that, unlike BP, IL’s weight updates are both local in space and time and may thus be more compatible with energy efficient neuromorphic hardware, which is well suited for autonomous, embedded systems that run on batteries and receive data streams through sensors, similar to the brain.

Original:  (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised:  Broader Impacts

Original: [Yes] This information is reported in supplementary material (c) Did you report error bars (e g., with respect to the random seed after running exper- iments multiple times)?
Revised: This work is largely theoretical.

Original: [Yes] Standard deviations over multiple seeds are shown in

Original: tables and in plots using error bars.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Is included in supplementary material
Revised: As such, we do not see any potential negative societal impacts.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]

Original: (b) Did you mention the license of the assets?

Original: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original:  [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [N/A]
Revised:  Acknowledgements

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable? [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
Revised: This work was partially supported by the National Science Foundation (NSF) under grant 1652159.

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?

Original: [N/A]