Original: L EARNING TO SOLVE THE CREDIT ASSIGNMENT PROBLEM
Revised: L EARNING TO SOLVE THE CREDIT ASSIGNMENT PROBLEM

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Backpropagation is driving today’s artiﬁcial neural networks (ANNs).
Revised: Backpropagation is driving today’s artiﬁcial neural networks (ANNs).

Original: However, despite extensive research, it remains unclear if the brain implements this algorithm.
Revised: However, despite extensive research, it remains unclear if the brain implements this algorithm.

Original: Among neuroscientists, reinforcement learning (RL) algorithms are often seen as a realistic alternative: neurons can randomly introduce change, and use unspeciﬁc feedback signals to observe their effect on the cost and thus approximate their gradient.
Revised: Among neuroscientists, reinforcement learning (RL) algorithms are often seen as a realistic alternative: neurons can randomly introduce change, and use unspeciﬁc feedback signals to observe their effect on the cost and thus approximate their gradient.

Original: However, the convergence rate of such learning scales poorly with the number of involved neurons.
Revised: However, the convergence rate of such learning scales poorly with the number of involved neurons.

Original: Here we propose a hybrid learning approach.
Revised: Here we propose a hybrid learning approach.

Original: Each neuron uses an RL-type strategy to learn how to approximate the gradients that backpropagation would provide.
Revised: Each neuron uses an RL-type strategy to learn how to approximate the gradients that backpropagation would provide.

Original: We provide proof that our approach converges to the true gradient for certain classes of networks.
Revised: We provide proof that our approach converges to the true gradient for certain classes of networks.

Original: In both feedforward and convolutional networks, we empirically show that our approach learns to approximate the gradient, and can match  the performance of  gradient-based learning.
Revised: In both feedforward and convolutional networks, we empirically show that our approach learns to approximate the gradient, and can match or the performance of exact gradient-based learning.

Original: Learning feedback weights provides a biologically plausible mechanism of achieving good performance, without the need for precise, pre-speciﬁed learning rules.
Revised: Learning feedback weights provides a biologically plausible mechanism of achieving good performance, without the need for precise, pre-speciﬁed learning rules.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: It is unknown how the brain solves the credit assignment problem when learning: how does each neuron know its role in a positive (or negative) outcome, and thus know how to change its activity to perform better next time?
Revised: It is unknown how the brain solves the credit assignment problem when learning: how does each neuron know its role in a positive (or negative) outcome, and thus know how to change its activity to perform better next time?

Original: This is a challenge for models of learning in the brain.
Revised: This is a challenge for models of learning in the brain.

Original: Biologically plausible solutions to credit assignment include those based on reinforcement learning (RL) algorithms and reward-modulated STDP (Bouvier et al., 2016;
Revised: Biologically plausible solutions to credit assignment include those based on reinforcement learning (RL) algorithms and reward-modulated STDP (Bouvier et al., 2016;

Original: Fiete et al., 2007; Fiete & Seung, 2006; Legenstein et al., 2010; Miconi, 2017).
Revised: Fiete et al., 2007; Fiete & Seung, 2006; Legenstein et al., 2010; Miconi, 2017).

Original: In these approaches a globally distributed reward signal provides feedback to all neurons in a network.
Revised: In these approaches a globally distributed reward signal provides feedback to all neurons in a network.

Original: Essentially, changes in rewards from a baseline, or expected, level are correlated with noise in neural activity, allowing a stochastic approximation of the gradient to be computed.
Revised: Essentially, changes in rewards from a baseline, or expected, level are correlated with noise in neural activity, allowing a stochastic approximation of the gradient to be computed.

Original: However these methods have not been demonstrated to operate at scale.
Revised: However these methods have not been demonstrated to operate at scale.

Original: For instance, variance in the REINFORCE estimator (Williams, 1992) scales with the number of units in the network (Rezende et al., 2014).
Revised: For instance, variance in the REINFORCE estimator (Williams, 1992) scales with the number of units in the network (Rezende et al., 2014).

Original: This drives the hypothesis that learning in the brain must rely on additional structures beyond a global reward signal.
Revised: This drives the hypothesis that learning in the brain must rely on additional structures beyond a global reward signal.

Original: In artiﬁcial neural networks (ANNs), credit assignment is performed with gradient-based methods computed through backpropagation (Rumelhart et al., 1986).
Revised: In artiﬁcial neural networks (ANNs), credit assignment is performed with gradient-based methods computed through backpropagation (Rumelhart et al., 1986; Werbos, 1982; Linnainmaa, 1976).

Original: This is signiﬁcantly more efﬁcient than RL-based algorithms, with ANNs now matching or surpassing human-level performance in a number of domains (Mnih et al., 2015; Silver et al., 2017; LeCun et al., 2015;
Revised: This is signiﬁcantly more efﬁcient than RL-based algorithms, with ANNs now matching or surpassing human-level performance in a number of domains (Mnih et al., 2015; Silver et al., 2017; LeCun et al., 2015;

Original: He et al., 2015; Haenssle et al., 2018; Russakovsky et al., 2015).
Revised: He et al., 2015; Haenssle et al., 2018; Russakovsky et al., 2015).

Original: However there are well known problems with implementing backpropagation in biologically realistic neural networks.
Revised: However there are well known problems with implementing backpropagation in biologically realistic neural networks.

Original: One problem is known as weight transport: an exact implementation of backpropagation requires a feedback structure with the same weights as the feedforward network to communicate gradients.
Revised: One problem is known as weight transport (Grossberg, 1987): an exact implementation of backpropagation requires a feedback structure with the same weights as the feedforward network to communicate gradients.

Original: Such a symmetric feedback structure has not been observed in biological neural circuits.
Revised: Such a symmetric feedback structure has not been observed in biological neural circuits.

Original: Despite such issues, backpropagation is the only method known to solve supervised and reinforcement learning problems at scale.
Revised: Despite such issues, backpropagation is the only method known to solve supervised and reinforcement learning problems at scale.

Original: Thus modiﬁcations or approximations to backpropagation that are more plausible have been the focus of signiﬁcant recent attention (Scellier & Bengio, 2016; Lillicrap et al., 2016; Lee et al., 2015; Lansdell & Kording, 2018).
Revised: Thus modiﬁcations or approximations to backpropagation that are more plausible have been the focus of signiﬁcant recent attention (Scellier & Bengio, 2016; Lillicrap et al., 2016; Lee et al., 2015; Lansdell & Kording, 2018).

Original: These efforts do show some ways forward.
Revised: These efforts do show some ways forward.

Original: Synthetic gradients demonstrate that learning can be based on approximate gradients, and need not be temporally locked (Jaderberg et al., 2016; Czar- necki et al., 2017).
Revised: Synthetic gradients demonstrate that learning can be based on approximate gradients, and need not be temporally locked (Jaderberg et al., 2016; Czar- necki et al., 2017b).

Original: In small feedforward networks, somewhat surprisingly, ﬁxed random feedback matrices in fact sufﬁce for learning (Lillicrap et al., 2016) (a phenomenon known as feedback alignment).
Revised: In small feedforward networks, somewhat surprisingly, ﬁxed random feedback matrices in fact sufﬁce for learning (Lillicrap et al., 2016) (a phenomenon known as feedback alignment).

Original: But still issues remain: feedback alignment does not work in CNNs, very deep networks, or networks with tight bottleneck layers.
Revised: But still issues remain: feedback alignment does not work in CNNs, very deep networks, or networks with tight bottleneck layers.

Original: Regardless, synthetic gradients and feedback alignment show that rough approximations of a gradient signal can be used to learn; even relatively inefﬁcient methods of approximating the gradient may be good enough.
Revised: Regardless, these results show that rough approximations of a gradient signal can be used to learn; even relatively inefﬁcient methods of approximating the gradient may be good enough.

Original: On this basis, here we propose an RL algorithm to train a feedback system to enable learning.
Revised: On this basis, here we propose an RL algorithm to train a feedback system to enable learning.

Original: Recent work has explored similar ideas, but not with the explicit goal of approximating backpropagation (Miconi, 2017; Miconi et al., 2018; Song et al., 2017).
Revised: Recent work has explored similar ideas, but not with the explicit goal of approximating backpropagation (Miconi, 2017; Miconi et al., 2018; Song et al., 2017).

Original: RL-based methods like REINFORCE may be inefﬁcient when used as a base learner, but they may be sufﬁcient when used to train a system that itself instructs a base learner.
Revised: RL-based methods like REINFORCE may be inefﬁcient when used as a base learner, but they may be sufﬁcient when used to train a system that itself instructs a base learner.

Original: We propose to use REINFORCE-style perturbation approach to train feedback signals to approximate what would have been provided by backpropagation.
Revised: We propose to use REINFORCE-style perturbation approach to train feedback signals to approximate what would have been provided by backpropagation.

Original: This sort of two-learner system, where one network helps the other learn more efﬁciently, may in fact align well with cortical neuron physiology.
Revised: This sort of two-learner system, where one network helps the other learn more efﬁciently, may in fact align well with cortical neuron physiology.

Original: For instance, the dendritic trees of pyramidal neurons consist of an apical and basal component (Guergiuev et al., 2017; Kording & Konig, 2001).
Revised: For instance, the dendritic trees of pyramidal neurons consist of an apical and basal component.

Original: Such a setup has been shown to support supervised learning in feedforward networks (Guergiuev et al., 2017; Kording & Konig, 2001).
Revised: Such a setup has been shown to support supervised learning in feedforward networks (Guergiuev et al., 2017; Kording & Konig, 2001).

Original: Similarly, climbing ﬁbers and Purkinje cells may deﬁne a learner/teacher system in the cerebellum (Marr, 1969).
Revised: Similarly, climbing ﬁbers and Purkinje cells may deﬁne a learner/teacher system in the cerebellum (Marr, 1969).

Original: These components allow for independent integration of two different signals, and may thus provide a realistic solution to the credit assignment problem.
Revised: These components allow for independent integration of two different signals, and may thus provide a realistic solution to the credit assignment problem.

Original: Thus we implement a network that learns to use feedback signals trained with reinforcement learning via a global reward signal.
Revised: Thus we implement a network that learns to use feedback signals trained with reinforcement learning via a global reward signal.

Original: We mathematically analyze the model, and compare its capabilities to other methods for learning in ANNs (e.g. feedback alignment).
Revised: We mathematically analyze the model, and compare its capabilities to other methods for learning in ANNs.

Original: We prove consistency of the estimator in particular cases, extending the theory of synthetic gradient-type approaches (Jaderberg et al., 2016; Czarnecki et al., 2017).
Revised: We prove consistency of the estimator in particular cases, extending the theory of synthetic gradient-like approaches (Jaderberg et al., 2016; Czarnecki et al., 2017b; Werbos, 1992; Schmidhuber, 1990).

Original: We demonstrate that oursynthetic gradient model learns as well as regular backpropagation in small models, overcomes the limitations of feedback alignment on more complicated feedforward networks, and can be used in convolutional networks.
Revised: We demonstrate that our model learns as well as regular backpropagation in small models, overcomes the limitations of feedback alignment on more complicated feedforward networks, and can be used in convolutional networks.

Original: Thus our method illustrates a biologically realistic way by which the brain could perform gradient descent learning.
Revised: Thus, by combining local and global feedback signals, this method points to more plausible ways the brain could solve the credit assignment problem.

Original:  L EARNING FEEDBACK WEIGHTS THROUGH PERTURBATIONS
Revised:  L EARNING FEEDBACK WEIGHTS THROUGH PERTURBATIONS

Original: We use the following notation.
Revised: We use the following notation.

Original: Let x ∈ R
Revised: Let x ∈ R

Original: m represent an input vector.
Revised: m represent an input vector.

Original: Let an N hidden-layer network be given by ˆ y = f ( x ) ∈ R p .
Revised: Let an N hidden-layer network be given by ˆ y = f ( x ) ∈ R p .

Original: This is composed of a set of layer-wise summation and non-linear activations
Revised: This is composed of a set of layer-wise summation and non-linear activations

Original: [Equation]
Revised: [Equation]

Original: for hidden layer states h i ∈ R n i , non-linearity σ , weight matrices W i ∈ R n i × n i − 1 and denoting h 

Revised: for hidden layer states h i ∈ R n i , non-linearity σ , weight matrices W i ∈ R n i × n i − 1 and denoting h

Original: = x and h N +1 =
Revised: = x and h N +1 =

Original: ˆy .
Revised: ˆy .

Original: Some loss function L is deﬁned in terms of the network output: L ( y , ˆ y ( x )) .
Revised: Some loss function L is deﬁned in terms of the network output: L ( y , ˆ y ) .

Original: Let L denote the loss as a function of ( x , y ) :
Revised: Let L denote the loss as a function of ( x , y ) :

Original: L ( x , y )
Revised: L ( x , y )

Original: =
Revised: =

Original: L ( y , ˆ y ( x )) .
Revised: L ( y , f ( x )) .

Original: Let data ( x , y ) ∈ D be drawn from a distribution ρ .
Revised: Let data ( x , y ) ∈ D be drawn from a distribution ρ .

Original: Then we aim to minimize: E  [ L ( x , y )] .
Revised: We aim to minimize: E ρ [ L ( x , y )] .

Original: Backpropagation relies on the error signal e i , computed in a top-down fashion:
Revised: Backpropagation relies on the error signal e i , computed in a top-down fashion:

Original: [Equation]
Revised: [Equation]

Original: where ◦ denotes element-wise multiplication.
Revised: where ◦ denotes element-wise multiplication.

Original: 2. B ASIC SETUP
Revised: 2. B ASIC SETUP

Original: Let the loss gradient term be denoted as
Revised: Let the loss gradient term be denoted as

Original: [Equation]
Revised: [Equation]

Original: In this work we replace λ i with an approximation with its own parameters to be learned (known as a synthetic gradient (Jaderberg et al., 2016; Czarnecki et al., 2017), or error critic (Werbos, 1992)):
Revised: In this work we replace λ i with an approximation with its own parameters to be learned (known as a synthetic gradient, or conspiring network, (Jaderberg et al., 2016; Czarnecki et al., 2017b), or error critic (Werbos, 1992)):

Original: [Equation]
Revised: [Equation]

Original: x y + + x
Revised: x y + + x

Original: B
Revised: B

Original: B 1 x  x y
Revised: B 1 x  x y

Original: W
Revised: W

Original: Whhhho o
Revised: Whhhho o

Original: [Figure]
Revised: [Figure]

Original: for parameters θ .
Revised: for parameters θ .

Original: Note that we must distinguish the true loss gradients from their synthetic estimates.
Revised: Note that we must distinguish the true loss gradients from their synthetic estimates.

Original: Let ˜ e i be loss gradients computed by backpropagating the synthetic gradients
Revised: Let ˜ e i be loss gradients computed by backpropagating the synthetic gradients

Original: [Equation]
Revised: [Equation]

Original: For the ﬁnal layer the synthetic gradient matches the true gradient: e N +1 =
Revised: For the ﬁnal layer the synthetic gradient matches the true gradient: e N +1 =

Original: ˜e N +1 .
Revised: ˜e N +1 .

Original: This setup can accommodate both top-down and bottom-up information, and encompasses a number of published models (Jaderberg et al., 2016; Czarnecki et al., 2017;
Revised: This setup can accommodate both top-down and bottom-up information, and encompasses a number of published models (Jaderberg et al., 2016; Czarnecki et al., 2017b;

Original: Lillicrap et al., 2016; Nøkland, 2016;
Revised: Lillicrap et al., 2016; Nøkland, 2016;

Original: Liao et al., 2016; Xiao et al., 2018).
Revised: Liao et al., 2016; Xiao et al., 2018).

Original: 2. S TOCHASTIC NETWORKS AND GRADIENT DESCENT
Revised: 2. S TOCHASTIC NETWORKS AND GRADIENT DESCENT

Original: To learn a synthetic gradient we utilze the stochasticity inherent to biological neural networks.
Revised: To learn a synthetic gradient we utilze the stochasticity inherent to biological neural networks.

Original: A number of biologically plausible learning rules exploit random perturbations in neural activity (Xie & Seung, 2004; Seung, 2003; Fiete & Seung, 2006; Fiete et al., 2007; Song et al., 2017).
Revised: A number of biologically plausible learning rules exploit random perturbations in neural activity (Xie & Seung, 2004; Seung, 2003; Fiete & Seung, 2006; Fiete et al., 2007; Song et al., 2017).

Original: Here, at each time each unit produces a noisy response:
Revised: Here, at each time each unit produces a noisy response:

Original: [Equation]
Revised: [Equation]

Original: for independent Gaussian noise ξ i ∼ ν
Revised: for independent Gaussian noise ξ i ∼ ν

Original: = N (0 , I ) and standard deviation c h > 0 .
Revised: = N (0 , I ) and standard deviation c h > 0 .

Original: This generates a noisy loss ˜
Revised: This generates a noisy loss ˜

Original: L ( x , y , ξ ) and a baseline loss L ( x , y ) =
Revised: L ( x , y , ξ ) and a baseline loss L ( x , y ) =

Original: ˜ L ( x , y , 0) .
Revised: ˜ L ( x , y , 0) .

Original: We will use the noisy response to estimate gradients that then allow us to optimize the baseline L – the gradients used for weight updates are computed using the deterministic baseline.
Revised: We will use the noisy response to estimate gradients that then allow us to optimize the baseline L – the gradients used for weight updates are computed using the deterministic baseline.

Original: 2. S YNTHETIC GRADIENTS VIA PERTURBATION
Revised: 2. S YNTHETIC GRADIENTS VIA PERTURBATION

Original: For Gaussian white noise, the well-known REINFORCE algorithm (Williams, 1992) coincides with the node perturbation method (Fiete & Seung, 2006; Fiete et al., 2007).
Revised: For Gaussian white noise, the well-known REINFORCE algorithm (Williams, 1992) coincides with the node perturbation method (Fiete & Seung, 2006; Fiete et al., 2007).

Original: Node perturbation works by linearizing the loss:
Revised: Node perturbation works by linearizing the loss:

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: with expectation taken over the noise distribution ν ( ξ ) .
Revised: with expectation taken over the noise distribution ν ( ξ ) .

Original: This provides an estimator of the loss gradient
Revised: This provides an estimator of the loss gradient

Original: [Equation]
Revised: [Equation]

Original: The approximation(1) is made more precise in Theorem 1.
Revised: This approximation is made more precise in Theorem 1 (Supplementary material).

Original: 2. T RAINING A FEEDBACK NETWORK
Revised: 2. T RAINING A FEEDBACK NETWORK

Original: There are many possible sensible choices of g ( · ) .
Revised: There are many possible sensible choices of g ( · ) .

Original: For example, taking g as simply a function of each layer’s activations: λ i =
Revised: For example, taking g as simply a function of each layer’s activations: λ i =

Original: g ( h i ) is in fact sufﬁcient parameterization to express the true gradient function (Jaderberg et al., 2016).
Revised: g ( h i ) is in fact sufﬁcient parameterization to express the true gradient function (Jaderberg et al., 2016).

Original: We may expect, however, that the gradient estimation problem be simpler if each layer is provided with some error information obtained from the loss function and propagated in a top-down fashion.
Revised: We may expect, however, that the gradient estimation problem be simpler if each layer is provided with some error information obtained from the loss function and propagated in a top-down fashion.

Original: Symmetric feedback weights may not be biologically plausible, and random ﬁxed weights may only solve certain problems of limited size or complexity (Lillicrap et al., 2016).
Revised: Symmetric feedback weights may not be biologically plausible, and random ﬁxed weights may only solve certain problems of limited size or complexity (Lillicrap et al., 2016).

Original: However, a system that can learn to appropriate feedback weights B may be able to align the feedforward and feedback weights as much as is needed to successfully learn.
Revised: However, a system that can learn to appropriate feedback weights B may be able to align the feedforward and feedback weights as much as is needed to successfully learn.

Original: We investigate various choices of g ( h i , ˜
Revised: We investigate various choices of g ( h i , ˜

Original: e i +1 ; B i +1 ) outlined in the applications below.
Revised: e i +1 ; B i +1 ) outlined in the applications below.

Original: Parameters
Revised: Parameters

Original: B i +1 are estimated by solving the least squares problem: 
Revised: B i +1 are estimated by solving the least squares problem: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Unless otherwise noted this was solved by gradient-descent, updating parameters once with each minibatch.
Revised: Unless otherwise noted this was solved by gradient-descent, updating parameters once with each minibatch.

Original: Refer to the supplementary material for additional experimental descriptions and parameters.
Revised: Refer to the supplementary material for additional experimental descriptions and parameters.

Original:  T HEORETICAL RESULTS
Revised:  T HEORETICAL RESULTS

Original: We can prove the estimator (3) is consistent asc h → 0 in some particular cases. First consider consistency of the ‘feedback-alignment’ estimator: g FA ( h i , ˜ e i +1 ; B i +1 ) =
Revised: We can prove the estimator (3) is consistent as the noise

Original: B i +1 ˜ e i +1 . To prove consistency we must show the expectation of the Taylor series approximation (1) is well behaved. That is, we must show the expected remainder term of the expansion:
Revised: variance c h → 0 , in some particular cases.

Original: [Equation] is ﬁnite. This requires some additional assumptions on the problem. We prove the result under the following assumptions:
Revised: We state the results informally here, and give the exact details in the supplementary materials.

Original: • A1: the noise ξ is subgaussian, • A2: the loss function L ( x , y ) is analytic on D , • A3: the error matrices ˜e n ( ˜e n ) T are full rank, for 1 ≤ n ≤ N + 1 , • A4: the mean of the remainder and error terms is bounded:

Original: [Equation]

Original: for 1 ≤ n ≤ N .

Original: Under these assumptions convergence follows from consistency of the least squares estimator for linear models.

Original: Consider ﬁrst convergence of the ﬁnal layer feedback matrix, B N +1 .
Revised: Consider ﬁrst convergence of the ﬁnal layer feedback matrix, B N +1 .

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume A1-4.

Original:  For g FA ( h i , ˜
Revised: (Informal) For g FA ( h i , ˜

Original: e i +1 ; B i +1 )
Revised: e i +1 ; B i +1 )

Original: =
Revised: =

Original: B i +1 ˜
Revised: B i +1 ˜

Original: e i [Equation] 1 , then the least squares estimator
Revised: e i +1 , then the least squares estimator

Original: [Equation]
Revised: [Equation]

Original: solves (3) and converges to the true feedback matrix, in the sense that: [Equation]
Revised: solves (3) and converges to the true feedback matrix, in the sense that: lim c h → 0 plim T →∞ ˆ B N +1 =

Original:  where plim indicates convergence in probability.
Revised: W N +1 , where plim indicates convergence in probability.

Original: Theorem 1 thus establishes convergence of B in a shallow (1 hidden layer) non-linear network, provided the activation function and loss function are smooth.
Revised: Theorem 1 thus establishes convergence of B in a shallow (1 hidden layer) non-linear network.

Original: In a deep, linear network we can also use Theorem 1 to establish convergence over the rest of the layers of the network.
Revised: In a deep, linear network we can also use Theorem 1 to establish convergence over the rest of the layers.

Original: Theorem 2.
Revised: Theorem 2.

Original: Assume A1-4. For g FA ( h i , ˜
Revised: (Informal) For g FA ( h i , ˜

Original: e i +1 ; B i +1 )
Revised: e i +1 ; B i +1 )

Original: =
Revised: =

Original: B i +1 ˜
Revised: B i +1 ˜

Original: e i +1 and σ ( x ) = x , the least squares estimator
Revised: e i +1 and σ ( x ) = x , the least squares estimator

Original: [Equation]
Revised: [Equation]

Original: solves (3) and converges to the true feedback matrix, in the sense that: 
Revised: solves (3) and converges to the true feedback matrix, in the sense that: lim c h → 0 plim T →∞ ˆ B i

Original: [Equation]
Revised: [Equation]

Original:  Given these results we can establish consistency for the ‘direct feedback alignment’ (DFA;
Revised: W i , [Equation] Given these results we can establish consistency for the ‘direct feedback alignment’ (DFA;

Original: Nøkland (2016)) estimator: g DFA ( h i , ˜
Revised: Nøkland (2016)) estimator: g DFA ( h i , ˜

Original: e N +1 ; B i +1 ) =
Revised: e N +1 ; B i +1 ) =

Original: [Equation]

Original:  B i +1 ) T ˜
Revised: ( B i +1 ) T ˜

Original: e N +1 .
Revised: e N +1 .

Original: Theorem 1 applies trivially since for the ﬁnal layer, the two approximations have the same form: g FA ( h N , ˜ e N +1 ; θ N )
Revised: Theorem 1 applies trivially since for the ﬁnal layer, the two approximations have the same form: g FA ( h N , ˜ e N +1 ; θ N )

Original: =
Revised: =

Original: g DFA ( h N , ˜ e N +1 ; θ N ) .
Revised: g DFA ( h N , ˜ e N +1 ; θ N ) .

Original: Theorem 2 can be easily extended according to the following:
Revised: Theorem 2 can be easily extended according to the following:

Original: Corollary 1.
Revised: Corollary 1.

Original: Assume A1-4.
Revised: (Informal)

Original: For g DFA ( h i , ˜ e N +1 ; B i +1 )
Revised: For g DFA ( h i , ˜ e N +1 ; B i +1 )

Original: =
Revised: =

Original: B i +1 ˜
Revised: B i +1 ˜

Original: e N +1 and σ ( x ) = x , the least squares estimator
Revised: e N +1 and σ ( x ) = x , the least squares estimator

Original: [Equation]
Revised: [Equation]

Original:  solves (3) and converges to the true feedback matrix, in the sense that: 
Revised: (6) solves (3) and converges to the true feedback matrix, in the sense that: lim c h → 0 plim T →∞ ˆ B i = (cid:81) ij = N +1 W j , 1 ≤ i ≤ N + 1 .

Original: [Equation]
Revised: [Figure]

Original: +

Original: W j , 1 ≤ n ≤ N + 1 .

Original: Proofs and a discussion of the assumptions are provided in the supplementary material.

Original: Thus for a non-linear shallow network or a deep linear network, for both g FA and g DFA , we have the result that, for sufﬁciently small c h , if we ﬁx the network weights W and train B through node perturbation then we converge to W .
Revised: Thus for a non-linear shallow network or a deep linear network, for both g FA and g DFA , we have the result that, for sufﬁciently small c h , if we ﬁx the network weights W and train B through node perturbation then we converge to W .

Original: Validation that the method learns to approximate W , for ﬁxed
Revised: Validation that the method learns to approximate W , for ﬁxed

Original: W , is provided in the supplementary material.
Revised: W , is provided in the supplementary material.

Original: In practice, we update B and W simultaneously.
Revised: In practice, we update B and W simultaneously.

Original: Some convergence theory is established for this case in (Jaderberg et al., 2016; Czarnecki et al., 2017).
Revised: Some convergence theory is established for this case in (Jaderberg et al., 2016; Czarnecki et al., 2017b) [Figure]

Original:  A PPLICATIONS 4. F ULLY CONNECTED NETWORKS SOLVING MNIST
Revised:  A PPLICATIONS 4. F ULLY CONNECTED NETWORKS SOLVING MNIST

Original: First we investigate g ( h i , ˜ e i +1 ; B i +1 ) =
Revised: First we investigate g ( h i , ˜e i +1 ; B i +1 ) =

Original: ( B i +1 )
Revised: ( B i +1 )

Original: T ˜ e i +1 , which describes a non-symmetric feedback network (Figure 1).
Revised: T ˜e i +1 , which describes a non-symmetric feedback network (Figure 1).

Original: To demonstrate the method can be used to solve simple supervised learning problems we use node perturbation with a four-layer network and MSE loss to solve MNIST (Figure 2).
Revised: To demonstrate the method can be used to solve simple supervised learning problems we use node perturbation with a four-layer network and MSE loss to solve MNIST (Figure 2).

Original: Updates to W i are made using the synthetic gradients 
Revised: Updates to W i are made using the synthetic gradients ∆ W i =

Original: [Equation]
Revised: η ˜ e i h

Original:  for learning rate η .
Revised: i − 1 , for learning rate η .

Original: The feedback network needs to co-adapt with the feedforward network in order to continue to provide a useful error signal.
Revised: The feedback network needs to co-adapt with the feedforward network in order to continue to provide a useful error signal.

Original: We observed that the system is able to adjust to provide a close correspondence between the feedforward and feedback matrices in both layers of the network (Figure 2A).
Revised: We observed that the system is able to adjust to provide a close correspondence between the feedforward and feedback matrices in both layers of the network (Figure 2A).

Original: We observed that the relative error between B i and W i is lower than what is observed for feedback alignment, suggesting that this co-adaptation of both W i and B i is indeed beneﬁcial.
Revised: The relative error between B i and W i is lower than what is observed for feedback alignment, suggesting that this co-adaptation of both W i and B i is indeed beneﬁcial.

Original: The relative error depends on the amount of noise used in node perturbation – lower variance doesn’t necessarily imply the lowest error between W and B , suggesting there is an optimal noise level that balances bias in the estimate and the ability to co-adapt to the changing feedforward weights.
Revised: The relative error depends on the amount of noise used in node perturbation – lower variance doesn’t necessarily imply the lowest error between W and B , suggesting there is an optimal noise level that balances bias in the estimate and the ability to co-adapt to the changing feedforward weights.

Original: Consistent with the low relative error in both layers, we observe that the alignment (the angle between the estimated gradient and the true gradient – proportional to e T WB T ˜ e ) is low in each layer – much lower for node perturbation than for feedback alignment, again suggesting that the method is much better at communicating error signals between layers (Figure 2B).
Revised: Consistent with the low relative error in both layers, we observe that the alignment (the angle between the estimated gradient and the true gradient – proportional to e T WB T ˜ e ) is low in each layer – much lower for node perturbation than for feedback alignment, again suggesting that the method is much better at communicating error signals between layers (Figure 2B).

Original: In fact, recent studies[Figure] [Figure] have shown that sign congruence of the feedforward and feedback matrices is all that is required to achieve good performance (Liao et al., 2016; Xiao et al., 2018).
Revised: In fact, recent studies have shown that sign congruence of the feedforward and feedback matrices is all that is required to achieve good performance (Liao et al., 2016; Xiao et al., 2018).

Original: Here the sign congruence is also higher in node perturbation, again depending somewhat the variance.
Revised: Here the sign congruence is also higher in node perturbation, again depending somewhat the variance.

Original: The amount of congruence is comparable between layers (Figure 2C).
Revised: The amount of congruence is comparable between layers (Figure 2C).

Original: Finally, the learning performance of node perturbation is comparable to backpropagation (Figure 2D) – achieving close to 3% test error. It is better than feedback alignment in this case.
Revised: Finally, the learning performance of node perturbation is comparable to backpropagation (Figure 2D), and better than feedback alignment in this case, though not by much.

Original: The same learning rate was used for all experiments here, and was not optimized individually for each method.

Original: Thus this result is not indicative of the superior performance of one method over the other – all methods do converge, and each likely could be optimized to converge faster.

Original: These results instead highlight the qualitative differences between the methods. They suggest  node perturbation for learning feedback weights can be used  in deep networks. 
Revised: These results instead highlight the qualitative differences between the methods, and suggest that node perturbation for learning feedback weights can be used to approximate gradients in deep networks. [Figure]

Original: 4. A UTO - ENCODING MNIST
Revised: 4. A UTO - ENCODING MNIST

Original: The above results demonstrate node perturbation provides error signals closely aligned with the true gradients.
Revised: The above results demonstrate node perturbation provides error signals closely aligned with the true gradients.

Original: However, performance-wise they do not demonstrate any clear advantage over feedback alignment or backpropagation in this small network.
Revised: However, performance-wise they do not demonstrate any clear advantage over feedback alignment or backpropagation.

Original: A known shortcoming of feedback alignment is in very deep networks and in autoencoding networks with tight bottleneck layers (Lillicrap et al., 2016).
Revised: A known shortcoming of feedback alignment is in very deep networks and in autoencoding networks with tight bottleneck layers (Lillicrap et al., 2016).

Original: To see if node perturbation has the same shortcoming, we test performance of a g ( h i , ˜ e i +1 ; B i +1 ) =
Revised: To see if node perturbation has the same shortcoming, we test performance of a g ( h i , ˜e i +1 ; B i +1 ) =

Original: ( B i +1 )
Revised: ( B i +1 )

Original: T ˜ e i +1 synthetic gradient on a simple auto-encoding network with MNIST input data (size 784-200-2-200-784).
Revised: T ˜e i +1 model on a simple auto-encoding network with MNIST input data (size 784-200-2200-784).

Original: In this more challenging case we also compare the method to the ‘matching’ learning rule (Rombouts et al., 2015; Martinolli et al., 2018), in which updates to B match updates to W, and also a denoising autoencoder (DAE) (Vincent et al., 2008).
Revised: In this more challenging case we also compare the method to the ‘matching’ learning rule (Rombouts et al., 2015; Martinolli et al., 2018), in which updates to B match updates to W and weight decay is added, a denoising autoencoder (DAE) (Vincent et al., 2008), and the ADAM (Kingma & Ba, 2015) optimizer (with backprop gradients).

Original: As expected, feedback alignment performs poorly, while node perturbation performs better than backpropagationand comparable ADAM (Kingma & Ba, 2015) (Figure 3A).
Revised: As expected, feedback alignment performs poorly, while node perturbation performs better than backpropagation (Figure 3A).

Original: In fact ADAM begins to overﬁt in training, while node perturbation does not.

Original: The increased performance relative to backpropagation is surprising.
Revised: The increased performance relative to backpropagation may seem surprising.

Original: It may be a similar effect to that speculated to explain feedback alignment – the method strikes the right balance between providing a useful gradient signal to learn, and constraining the updates to be sufﬁciently aligned with B , acting as a type of regularization (Lillicrap et al., 2016).

Original: The addition of noise in our methodmay also produce similar behavior to a DAE, in which noise encourages learning of more robust latent factors (Alain & Bengio, 2015).
Revised: A possible reason is the addition of noise in our method encourages learning of more robust latent factors (Alain & Bengio, 2015).

Original: And, indeed, the DAE  improves the loss over vanilla backpropagation (Figure 3A).
Revised: The DAE also improves the loss over vanilla backpropagation (Figure 3A).

Original: In line with these ideas, the latent space learnt by node perturbation shows a more uniform separation between the digits, compared to the networks trained by backpropagation and ADAM.
Revised: And, in line with these ideas, the latent space learnt by node perturbation shows a more uniform separation between the digits, compared to the networks trained by backpropagation.

Original: Feedback alignment, in contrast, does not learn to separate digits in the bottleneck layer at all (Figure 3B), resulting in scrambled output (Figure 3C).
Revised: Feedback alignment, in contrast, does not learn to separate digits in the bottleneck layer at all (Figure 3B), resulting in scrambled output (Figure 3C).

Original: The matched learning rule performs similarly to backpropagation.
Revised: The matched learning rule performs similarly to backpropagation.

Revised: These possible explanations are investigated more below.

Original: These results show that node perturbation is able to successfully communicate error signals through thin layers of a network as needed.
Revised: Regardless, these results show that node perturbation is able to successfully communicate error signals through thin layers of a network as needed.

Original: 4. C ONVOLUTIONAL NEURAL NETWORKS SOLVING CIFAR
Revised: 4. C ONVOLUTIONAL NEURAL NETWORKS SOLVING CIFAR

Original: Convolutional networks are another known shortcoming of feedback alignment.
Revised: Convolutional networks are another known shortcoming of feedback alignment.

Original: Here we test the method on a convolutional neural network (CNN) solving CIFAR (Krizhevsky, 2009).
Revised: Here we test the method on a convolutional neural network (CNN) solving CIFAR (Krizhevsky, 2009).

Original: Refer to the supplementary material for architecture and parameter details.
Revised: Refer to the supplementary material for architecture and parameter details.

Original: For this network we learn feedback weights direct from the output layer to each earlier layer: g ( h i , ˜
Revised: For this network we learn feedback weights direct from the output layer to each earlier layer: g ( h i , ˜

Original: e i +1 ; B i +1 ) =
Revised: e i +1 ; B i +1 ) =

Original: ( B i +1 ) T ˜
Revised: ( B i +1 ) T ˜

Original: e N + (similar to ‘direct feedback alignment’ (Nøkland, 2016)).
Revised: e N +1 (similar to ‘direct feedback alignment’ (Nøkland, 2016)).

Original: Here this was solved by gradient-descent.
Revised: Here this was solved by gradient-descent.

Original: On CIFAR10 we obtain a test accuracy of 75%.
Revised: On CIFAR10 we obtain a test accuracy of 75%.

Original: When compared with ﬁxed feedback weights and backpropagation, we see it is advantageous to learn feedback weights on CIFAR10 and marginally advantageous on CIFAR100 (Table 1).
Revised: When compared with ﬁxed feedback weights and backpropagation, we see it is advantageous to learn feedback weights on CIFAR10 and marginally advantageous on CIFAR100 (Table 1).

Original: This shows the method can be used in a CNN, and can solve challenging computer vision problems without weight transport.
Revised: This shows the method can be used in a CNN, and can solve challenging computer vision problems without weight transport.

Revised: 4. W HAT IS HELPING , NOISY ACTIVATIONS OR APPROXIMATING THE GRADIENT ?

Revised: To solve the credit assignment problem, our method utilizes two well-explored strategies in deep learning: adding noise (generally used to regularize (Bengio et al., 2013; Gulcehre et al., 2016;

Revised: Neelakantan et al., 2015; Bishop, 1995)), and approximating the true gradients (Jaderberg et al.,

Original: [Table]
Revised: [Table]

Revised: To determine which of these features are responsible for the improvement in performance over ﬁxed weights, in the autoencoding and CIFAR10 cases, we study the performance while varying where noise is added to the models (Table 2).

Revised: Noise can be added to the activations (BP and FA w. noise, Table 2), or to the inputs, as in a denoising autoencoder (DAE, Table 2).

Revised: Or, noise can be used only in obtaining an estimator of the true gradients (as in our method; NP, Table 2).

Revised: For comparison, a noiseless version of our method must instead assume access to the true gradients, and use this to learn feedback weights (i.e. synthetic gradients (Jaderberg et al., 2016); SG, Table 2).

Revised: these models is tested the autoencoding

Original: D ISCUSSION
Revised: CIFAR10 tasks, allowing us to better understand the performance of the perturbation [Table] In the autoencoding task, both noise (either in the inputs or the activations) and using an approximator to the gradient improve performance (Table 2, left). Noise beneﬁts performance for both SGD optimization and ADAM (Kingma & Ba, 2015).

Revised: In fact in this task, the combination of both of these factors (i.e. our method) results in better performance over either alone.

Revised: Yet, the addition of noise to the activations does not help feedback alignment.

Revised: This suggests that our method is indeed learning useful approximations of the error signals, and is not merely improving due to the addition of noise to the system.

Revised: In the CIFAR10 task (Table 2, right), the addition of noise to the activations has minimal effect on performance, while having access to the true gradients (SG) does result in improved performance over ﬁxed feedback weights.

Revised: Thus in these tasks noise does not always help, but using a less-based gradient estimator does, and noisy activations are one way of obtaining an unbiased gradient estimator.

Revised: Our method also is the best performing method that does not require either weight transport or access to the true gradients as a supervisory signal.

Revised:  D ISCUSSION

Original: Here we implement a perturbation-based synthetic gradient method to train neural networks.
Revised: Here we implement a perturbation-based synthetic gradient method to train neural networks.

Original: We show that this hybrid approach can be used in both fully connected and convolutional networks.
Revised: We show that this hybrid approach can be used in both fully connected and convolutional networks.

Original: By removing the symmetric feedforward, feedback weight requirement imposed by backpropagation this approach is a step towards more biologically-plausible deep learning.
Revised: By removing the symmetric feedforward/feedback weight requirement imposed by backpropagation, this approach is a step towards more biologically-plausible deep learning.

Original: In contrast tomany perturbation-based methods, this hybrid approach can solve large-scale problems.
Revised: By reaching comparable performance to backpropagation on MNIST, the method is able to solve larger problems than perturbation-only methods (Xie & Seung, 2004; Fiete et al., 2007; Werfel et al., 2005). By working in cases that feedback alignment fails, the method can provide learning without weight transport in a more diverse set of network architectures.

Original: We thus believe this approach can provide powerful and  biologically plausible learning algorithms.
Revised: We thus believe the idea of integrating both local and global feedback signals is a promising direction towards biologically plausible learning algorithms.

Revised: Of course, the method does not solve all issues with implementing gradient-based learning in a biologically plausible manner.

Revised: For instance, in the current implementation, the forward and the backwards passes are locked.

Original: The main drawback is that the method does not reach state-of-the-art performance on more interesting datasets like CIFAR (let alone something like ImageNet).
Revised: Here we just focus on the weight transport problem. A current drawback is that the method does not reach state-of-the-art performance on more challenging datasets like CIFAR.

Original: We focused on demonstrating that it is advantageous to learn feedback weights, when compared with ﬁxed weights, and successfully did so in a number of cases.
Revised: We focused on demonstrating that it is advantageous to learn feedback weights, when compared with ﬁxed weights, and successfully did so in a number of cases.

Original: However, we did not use any additional data augmentation and regularization tricks often employed to reach state-of-the-art performance.
Revised: However, we did not use any additional data augmentation and regularization methods often employed to reach state-of-theart performance.

Original: Thus fully characterizing the performance of this method remains important future work.
Revised: Thus fully characterizing the performance of this method remains important future work.

Original: But the method does has a number of computational advantages.
Revised: However the method does has a number of computational advantages.

Original: First, without weight transport the method has better data-movement performance (Crafton et al., 2019; Akrout et al., 2019), meaning it may be more efﬁciently implemented than backpropagation on specialized hardware.
Revised: First, without weight transport the method has better data-movement performance (Crafton et al., 2019; Akrout et al., 2019), meaning it may be more efﬁciently implemented than backpropagation on specialized hardware.

Original: Second, by relying on random perturbations to measure gradients, the method does not rely on the environment to provide gradients.
Revised: Second, by relying on random perturbations to measure gradients, the method does not rely on the environment to provide gradients (compared with e.g. Czarnecki et al.

Original: It works in cases where gradients cannot be backpropagated because our environment does not readily allow us to do so, which is common in reinforcement learning settings.
Revised: (2017a)

Original: Further, the addition of noise to different parts of a neural network is common either as a form of data augmentation, regularization, or to improve exploration during training (Bengio et al., 2013; Gulcehre et al., 2016; Neelakantan et al., 2015; Bishop, 1995). And, as mentioned above, noise may confer robustness to learnt latent spaces in autoencoding applications, similar to the motivation behind denoising autoencoders (Vincent et al., 2008).
Revised: ; Jaderberg et al.

Original: In fact, our theoretical results are somewhat similar to that of Alain & Bengio (2015), who demonstrate that a denoising autoencoder converges to the unperturbed solution as Gaussian noise goes to zero.
Revised: Our theoretical results are somewhat similar to that of Alain & Bengio (2015), who demonstrate that a denoising autoencoder converges to the unperturbed solution as Gaussian noise goes to zero.

Original: Our results apply to subgaussian noise more generally.
Revised: However our results apply to subgaussian noise more generally.

Original: Exploring the correspondence between our model and other generative neural networks is the subject of future work.

Original: While previous research has provided some insight and theory for how feedback alignment works (Lillicrap et al., 2016; Ororbia et al., 2018; Moskovitz et al., 2018; Bartunov et al., 2018; Baldi et al., 2018) the effect remains somewhat mysterious, and not applicable in some network architectures.
Revised: While previous research has provided some insight and theory for how feedback alignment works (Lillicrap et al., 2016; Ororbia et al., 2018; Moskovitz et al., 2018; Bartunov et al., 2018; Baldi et al., 2018) the effect remains somewhat mysterious, and not applicable in some network architectures.

Original: Recent studies have shown that some of these weaknesses can be addressed by instead imposing sign congruent feedforward and feedback matrices (Xiao et al., 2018).
Revised: Recent studies have shown that some of these weaknesses can be addressed by instead imposing sign congruent feedforward and feedback matrices (Xiao et al., 2018).

Original: Yet what mechanism may produce congruence in biological networks is unknown.
Revised: Yet what mechanism may produce congruence in biological networks is unknown.

Original: Here we show that the shortcomings of feedback alignment can be addressed in another way: the system can learn to adjust weights as needed to provide a useful error signal.
Revised: Here we show that the shortcomings of feedback alignment can be addressed in another way: the system can learn to adjust weights as needed to provide a useful error signal.

Original: Our work is closely related to Akrout et al.
Revised: Our work is closely related to Akrout et al.

Original: (2019), which also uses perturbations to learn feedback weights.
Revised: (2019), which also uses perturbations to learn feedback weights.

Original: However our approach does not divide learning into two phases, and training of the feedback weights does not occur in a layer-wise fashion.
Revised: However our approach does not divide learning into two phases, and training of the feedback weights does not occur in a layer-wise fashion, assuming only one layer is noisy at a time, which is a strong assumption.

Revised: Here instead we focus on combining global and local learning signals.

Original: Here we tested our method in an idealized setting.
Revised: Here we tested our method in an idealized setting.

Original: However the method is consistent with neurobiology in two important ways.
Revised: However the method is consistent with neurobiology in two important ways.

Original: First, it involves separate learning of feedforward and feedback weights.
Revised: First, it involves separate learning of feedforward and feedback weights.

Original: This is possible in cortical networks, where complex feedback connections exist between layers (Laceﬁeld et al., 2019; Richards & Lillicrap, 2019) and pyramidal cells have apical and basal compartments that allow for separate integration of feedback and feedforward signals (Guerguiev et al., 2017;
Revised: This is possible in cortical networks, where complex feedback connections exist between layers (Laceﬁeld et al., 2019; Richards & Lillicrap, 2019) and pyramidal cells have apical and basal compartments that allow for separate integration of feedback and feedforward signals (Guerguiev et al., 2017;

Original: K ¨ ording & K ¨ onig, 2001).
Revised: K ¨ ording & K ¨ onig, 2001).

Original: A recent ﬁnding that apical dendrites receive reward information is particularly interesting (Laceﬁeld et al., 2019).
Revised: A recent ﬁnding that apical dendrites receive reward information is particularly interesting (Laceﬁeld et al., 2019).

Original: Models like Guerguiev et al.
Revised: Models like Guerguiev et al.

Original: (2017) show how the ideas in this paper may be implemented inrealistic models of spiking neural networks.
Revised: (2017) show how the ideas in this paper may be implemented in spiking neural networks.

Original: We believe such models can be augmented with a perturbation-based rule like ours to provide a better learning system.
Revised: We believe such models can be augmented with a perturbation-based rule like ours to provide a better learning system.

Original: The second feature is that perturbations are used to learn the feedback weights.
Revised: The second feature is that perturbations are used to learn the feedback weights.

Original: How can a neuron measure these perturbations?
Revised: How can a neuron measure these perturbations?

Original: There are many plausible mechanisms (Seung, 2003; Xie & Seung, 2004; Fiete & Seung, 2006; Fiete et al., 2007).
Revised: There are many plausible mechanisms (Seung, 2003; Xie & Seung, 2004; Fiete & Seung, 2006; Fiete et al., 2007).

Original: For instance, birdsong learning uses empiric synapses from area LMAN (Fiete et al., 2007), others proposed it is approximated (Legenstein et al., 2010; Hoerzer et al., 2014), or neurons could use a learning rule that does not require knowing the noise (Lansdell & Kording, 2018).
Revised: For instance, birdsong learning uses empiric synapses from area LMAN (Fiete et al., 2007), others proposed it is approximated (Legenstein et al., 2010; Hoerzer et al., 2014), or neurons could use a learning rule that does not require knowing the noise (Lansdell & Kording, 2018).

Original: Further, our model involves the subtraction of a baseline loss to reduce the variance of the estimator.
Revised: Further, our model involves the subtraction of a baseline loss to reduce the variance of the estimator.

Original: This does not affect the expected value of the estimator – technically the baseline could be removed or replaced with a approximation (Legenstein et al., 2010; Loewenstein & Seung, 2006).
Revised: This does not affect the expected value of the estimator – technically the baseline could be removed or replaced with an approximation (Legenstein et al., 2010; Loewenstein & Seung, 2006).

Original: Thus both separation of feedforward and feedback systems and perturbation-based estimators can be implemented by neurons.
Revised: Thus both separation of feedforward and feedback systems and perturbation-based estimators can be implemented by neurons.

Revised: As RL-based methods do not scale by themselves, and exact gradient signals are infeasible, the brain may well use a feedback system trained through reinforcement signals to usefully approximate gradients.

Original: There is a large space of plausible learning rules that can learn to use feedback signals in order to more efﬁciently learn. These promise to inform both models of learning in the brain and learning algorithms in artiﬁcial networks.
Revised: There is a large space of plausible learning rules that can learn to use feedback signals in order to more efﬁciently learn, and these promise to inform both models of learning in the brain and learning algorithms in artiﬁcial networks.

Original: Here we take an early step in this direction.
Revised: Here we take an early step in this direction.

Original: A P ROOFS
Revised: A P ROOFS

Original: We review the key components of the model.
Revised: We review the key components of the model.

Original: Data ( x , y ) ∈ D are drawn from a distribution ρ .
Revised: Data ( x , y ) ∈ D are drawn from a distribution ρ .

Original: The loss function is linearized:
Revised: The loss function is linearized:

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: with expectation taken over the noise distribution ν ( ξ ) .
Revised: with expectation taken over the noise distribution ν ( ξ ) .

Original: This suggests a good estimator of the loss gradient is
Revised: This suggests a good estimator of the loss gradient is

Original: [Equation]
Revised: [Equation]

Original: Let ˜ e i be the error signal computed by backpropagating the synthetic gradients:
Revised: Let ˜ e i be the error signal computed by backpropagating the synthetic gradients:

Original: [Equation]
Revised: [Equation]

Original: Then parameters B i +1 are estimated by solving the least squares problem:
Revised: Then parameters B i +1 are estimated by solving the least squares problem:

Original: [Equation]
Revised: [Equation]

Revised: Note that the matrix-vector form of backpropagation given here is setup so that we can think of each term as either a vector for a single input, or as matrices corresponding to a set of T inputs.

Original: Under what conditions can we show that ˆ B i +

Revised: Here we focus on the question, under what conditions can we show that ˆ B i +

Original: → W i +1 (with enough data)?
Revised: → W i +1 , as T → ∞ ?

Original: One way to ﬁnd an answer is to deﬁne the synthetic gradient in terms of the system without noise added.
Revised: One way to ﬁnd an answer is to deﬁne the synthetic gradient in terms of the system without noise added.

Original: Then B T ˜ e is deterministic with respect to x , y and, assuming ˜ L has a convergent power series around ξ = 0 , we can write
Revised: Then B T ˜ e is deterministic with respect to x , y and, assuming ˜ L has a convergent power series around ξ = 0 , we can write

Original: [Equation]
Revised: [Equation]

Original: Taken together these suggest we can prove ˆ B
Revised: Taken together these suggest we can prove ˆ B

Original: i +

Revised: i +

Original: → W i +1 in the same way we prove consistency of the linear least squares estimator.
Revised: → W i +1 in the same way we prove consistency of the linear least squares estimator.

Original: For this to work we must show the expectation of the Taylor series approximation (1) is well behaved.
Revised: For this to work we must show the expectation of the Taylor series approximation (1) is well behaved.

Original: That is, we must show the expected remainder term of the expansion:
Revised: That is, we must show the expected remainder term of the expansion:

Original: [Equation]
Revised: [Equation]

Original: is ﬁnite and goes to zero as c h → 0 .
Revised: is ﬁnite and goes to zero as c h → 0 .

Original: This requires some additional assumptions on the problem.
Revised: This requires some additional assumptions on the problem.

Original: We make the following assumptions:
Revised: We make the following assumptions:

Original: • A1: the noise ξ is subgaussian, • A2: the loss function L ( x , y ) is analytic on D , • A3: the error matrices ˜e n ( ˜e n ) T are full rank, for 1 ≤ n ≤ N + 1 ,  • A4: the mean of the remainder and error terms is bounded: 
Revised: • A1: the noise ξ is subgaussian, • A2: the loss function L ( x , y ) is analytic on D , • A3: the error matrices ˜e i ( ˜e i ) T are full rank, for 1 ≤ i ≤ N + 1 , with probability 1, • A4: the mean of the remainder and error terms is bounded: E

Original: [Equation]
Revised: [Equation]

Original:  for 1 ≤ n ≤ N .
Revised: E i ( c h )( ˜e i +1 ) T (cid:3) < ∞ , for 1 ≤ i ≤ N .

Original: Consider ﬁrst convergence of the ﬁnal layer feedback matrix, B N +1 .
Revised: Consider ﬁrst convergence of the ﬁnal layer feedback matrix, B N +1 .

Original: In the ﬁnal layer it is true that e N +1 =
Revised: In the ﬁnal layer it is true that e N +1 =

Original: ˜
Revised: ˜

Original: e N +1 .
Revised: e N +1 .

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume A1-4.
Revised: Assume A1-4.

Original: For g FA ( h i , ˜
Revised: For g FA ( h i , ˜

Original: e i +1 ; B i +1 )
Revised: e i +1 ; B i +1 )

Original: =
Revised: =

Original: B i +1 ˜
Revised: B i +1 ˜

Original: e i +1 , then the least squares estimator
Revised: e i +1 , then the least squares estimator

Original: [Equation]
Revised: [Equation]

Original: solves (3) and converges to the true feedback matrix, in the sense that:
Revised: solves (3) and converges to the true feedback matrix, in the sense that:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let L ( m ) ij :=
Revised: Let L ( m ) ij :=

Original: ∂ m L ∂h imj .
Revised: ∂ m L ∂h imj .

Original: We ﬁrst show that, under A1-2, the conditional expectation of the estimator (2) converges to the gradient L (1) Nj as c h → 0 .
Revised: We ﬁrst show that, under A1-2, the conditional expectation of the estimator (2) converges to the gradient L (1) Nj as c h → 0 .

Original: For each ˆ λ Nj , by A2, we have the following series expanded around ξ = 0 :
Revised: For each ˆ λ Nj , by A2, we have the following series expanded around ξ = 0 :

Original: [Equation]
Revised: [Equation]

Original: Taking a conditional expectation gives:
Revised: Taking a conditional expectation gives:

Original: [Equation]
Revised: [Equation]

Original: We must show the remainder term
Revised: We must show the remainder term

Original: [Equation]
Revised: [Equation]

Original: goes to zero as c h → 0 .
Revised: goes to zero as c h → 0 .

Original: This is true provided each moment E (( ξ Nj ) m | x , y ) is sufﬁciently wellbehaved.
Revised: This is true provided each moment E (( ξ Nj ) m | x , y ) is sufﬁciently wellbehaved.

Original: Using Jensen’s inequality and the triangle inequality in the ﬁrst line, we have that
Revised: Using Jensen’s inequality and the triangle inequality in the ﬁrst line, we have that

Original: [Equation]
Revised: [Equation]

Original: [monotone convergence]
Revised: [monotone convergence]

Revised: [Equation]

Revised: [subgaussian]

Original: [Equation]
Revised: [Equation]

Original: With this in place, we have that the problem (9) is close to a linear least squares problem, since
Revised: With this in place, we have that the problem (9) is close to a linear least squares problem, since

Original: [Equation]
Revised: [Equation]

Original: (12) with residual η N = ˆ λ N − E (ˆ λ N | x , y ) .
Revised: [Equation] with residual η N = ˆ λ N − E (ˆ λ N | x , y ) .

Original: The residual satisﬁes
Revised: The residual satisﬁes

Original: [Equation]
Revised: [Equation]

Original: This follows since e N +1 is deﬁned in relation to the baseline loss, not the stochastic loss, meaning it is measurable with respect to ( x , y ) and can be moved into the conditional expectation.
Revised: This follows since e N +1 is deﬁned in relation to the baseline loss, not the stochastic loss, meaning it is measurable with respect to ( x , y ) and can be moved into the conditional expectation.

Original: From (12) and A3, we have that the least squares estimator (10) satisﬁes
Revised: From (12) and A3, we have that the least squares estimator (10) satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Thus, using the continuous mapping theorem
Revised: Thus, using the continuous mapping theorem

Original: [Equation]
Revised: [Equation]

Original: [A4 and Eq.
Revised: [A4 and Eq.

Original: [Equation]
Revised: [Equation]

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: We can use Theorem 1 to establish convergence over the rest of the layers of the network when the activation function is the identity.
Revised: We can use Theorem 1 to establish convergence over the rest of the layers of the network when the activation function is the identity [Equation]

Original: Theorem 2.
Revised: Theorem 2.

Original: Assume A1-4.
Revised: Assume A1-4.

Original: For g FA ( h i , ˜ e 
Revised: For g FA ( h i , ˜ e i

Original: [Equation]
Revised: [Equation]

Original:  ; B i +1 )
Revised: 1 ; B i +1 )

Original: =
Revised: =

Original: B i +1 ˜
Revised: B i +1 ˜

Original: e i +1 and σ ( x ) = x , the least squares estimator
Revised: e i +1 and σ ( x ) = x , the least squares estimator

Original: [Equation]
Revised: [Equation]

Original: solves (9) and converges to the true feedback matrix, in the sense that:
Revised: solves (9) and converges to the true feedback matrix, in the sense that:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: assuming this limit exists.
Revised: assuming this limit exists.

Original: From Theorem 1 the top layer estimate ˆ B N +1 converges in probability to ˜ W N +1 ( c ) .
Revised: From Theorem 1 the top layer estimate ˆ B N +1 converges in probability to ˜ W N +1 ( c ) .

Original: We can then use induction to establish that ˆ B j in the remaining layers also converges in probability to ˜ W j ( c ) .
Revised: We can then use induction to establish that ˆ B j in the remaining layers also converges in probability to ˜ W j ( c ) .

Original: That is, assume that ˆ B j converge in probability to ˜ W j ( c ) in higher layers N +1 ≥ j > n .
Revised: That is, assume that ˆ B j converge in probability to ˜ W j ( c ) in higher layers N +1 ≥ j > i .

Original: Then we must establish that ˆ B n also converges in probability.
Revised: Then we must establish that ˆ B i also converges in probability.

Original: To proceed it is useful to also deﬁne
Revised: To proceed it is useful to also deﬁne

Original: [Equation]
Revised: [Equation]

Original: as the error signal backpropagated through the converged (but biased) weight matrices ˜ W ( c ) .
Revised: as the error signal backpropagated through the converged (but biased) weight matrices ˜ W ( c ) .

Original: Again it is true that ˜˜ e N +1 =
Revised: Again it is true that ˜˜ e N +1 =

Original: e N +1 .
Revised: e N +1 .

Original: As in Theorem 1, the least squares estimator has the form:
Revised: As in Theorem 1, the least squares estimator has the form:

Original: [Equation]
Revised: [Equation]

Original: Thus, again by the continuous mapping theorem:
Revised: Thus, again by the continuous mapping theorem:

Original: [Equation]
Revised: [Equation]

Original: In this case continuity again allows us to separate convergence of each term in the product:
Revised: In this case continuity again allows us to separate convergence of each term in the product:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: using the weak law of large numbers in the ﬁrst term, and the induction assumption for the remaining terms.
Revised: using the weak law of large numbers in the ﬁrst term, and the induction assumption for the remaining terms.

Original: In the same way
Revised: In the same way

Original: [Equation]
Revised: [Equation]

Original: Note that the induction assumption also implies lim c → 0 ˜˜ e n ( c )=
Revised: Note that the induction assumption also implies lim c → 0 ˜˜ e i ( c )

Original:  e n .
Revised: = e i .

Original: Thus, putting it together, by
Revised: Thus, putting it together, by

Original: [Equation]
Revised: [Equation]

Original: , A4 and the same reasoning as in Theorem 1 we have the result:
Revised: , A4 and the same reasoning as in Theorem 1 we have the result:

Original: [Equation]
Revised: [Equation]

Original: Corollary 1.
Revised: Corollary 1.

Original: Assume A1-4.
Revised: Assume A1-4.

Original: For g DFA ( h i , ˜ e N +1 ; B i +1 )
Revised: For g DFA ( h i , ˜ e N +1 ; B i +1 )

Original: =
Revised: =

Original: B i +1 ˜
Revised: B i +1 ˜

Original: e N +1 and σ ( x ) = x , the least squares estimator
Revised: e N +1 and σ ( x ) = x , the least squares estimator

Original: [Equation]
Revised: [Equation]

Original: solves (3) and converges to the true feedback matrix, in the sense that:
Revised: solves (3) and converges to the true feedback matrix, in the sense that:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For a deep linear network notice that the node perturbation estimator can be expressed as:
Revised: For a deep linear network notice that the node perturbation estimator can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: (17) where the ﬁrst term represents the true gradient, given by the simple linear backpropagation, the second and third terms are the remainder and a noise term, as in Theorem 1.
Revised: (17) where the ﬁrst term represents the true gradient, given by the simple linear backpropagation, the second and third terms are the remainder and a noise term, as in Theorem 1.

Original: Deﬁne
Revised: Deﬁne

Original: [Equation]
Revised: [Equation]

Original: Then following the same reasoning as the proof of Theorem
Revised: Then following the same reasoning as the proof of Theorem

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Then we have:
Revised: Then we have:

Original: [Equation]
Revised: [Equation]

Original: A. D ISCUSSION OF ASSUMPTIONS
Revised: A. D ISCUSSION OF ASSUMPTIONS

Original: It is worth making the following points on each of the assumptions:
Revised: It is worth making the following points on each of the assumptions:

Original: • A1.
Revised: • A1.

Original: In the paper we assume ξ is Gaussian.
Revised: In the paper we assume ξ is Gaussian.

Original: Here we prove the more general result of convergence for any subgaussian random variable.
Revised: Here we prove the more general result of convergence for any subgaussian random variable.

Original: • A2.
Revised: • A2.

Original: In practice this may be a fairly restrictive assumption, since it precludes using relu nonlinearities.
Revised: In practice this may be a fairly restrictive assumption, since it precludes using relu nonlinearities.

Original: Other common choices, such as hyperbolic tangent and sigmoid non-linearities with an analytic cost function do satisfy this assumption, however.
Revised: Other common choices, such as hyperbolic tangent and sigmoid non-linearities with an analytic cost function do satisfy this assumption, however.

Original: • A3.
Revised: • A3.

Original: It is hard to establish general conditions under which ˜ e n (˜ e n ) T will be full rank.
Revised: It is hard to establish general conditions under which ˜ e i (˜ e i ) T will be full rank.

Original: While it may be a reasonable assumption in some cases.
Revised: While it may be a reasonable assumption in some cases.

Revised: [Figure]

Original: Extensions of Theorem 2 to a non-linear network may be possible.
Revised: Extensions of Theorem 2 to a non-linear network may be possible.

Original: However, the method of proof used here is not immediately applicable because the continuous mapping theorem can not be applied in such a straightforward fashion as in Equation (15).
Revised: However, the method of proof used here is not immediately applicable because the continuous mapping theorem can not be applied in such a straightforward fashion as in Equation (15).

Original: In the non-linear case the resulting sums over all observations are neither independent or identically distributed, which makes applying any law of large numbers complicated.[Figure]
Revised: In the non-linear case the resulting sums over all observations are neither independent or identically distributed, which makes applying any law of large numbers complicated.

Original: B V ALIDATION WITH FIXED W
Revised: B V ALIDATION WITH FIXED W

Original: We demonstrate the method’s convergence in a small non-linear network solving MNIST for different noise levels, c h , and layer widths (Figure 4).
Revised: We demonstrate the method’s convergence in a small non-linear network solving MNIST for different noise levels, c h , and layer widths (Figure 4).

Original: As basic validation of the method, in this experiment the feedback matrices are updated while the feedforward weights W i are held ﬁxed.
Revised: As basic validation of the method, in this experiment the feedback matrices are updated while the feedforward weights W i are held ﬁxed.

Original: We should expect the feedback matrices B i to converge to the feedforward matrices W i .
Revised: We should expect the feedback matrices B i to converge to the feedforward matrices W i .

Original: Here different noise variance does results equally accurate estimators (Figure 4A).
Revised: Here different noise variance does results equally accurate estimators (Figure 4A).

Original: The estimator correctly estimates the true feedback matrix W 2 to a relative error of 0.8%.
Revised: The estimator correctly estimates the true feedback matrix W 2 to a relative error of 0.8%.

Original: The convergence is layer dependent, with the second hidden layer matrix, W 2 , being accurately estimated, and the convergence of the ﬁrst hidden layer matrix, W 1 , being less accurately estimated.
Revised: The convergence is layer dependent, with the second hidden layer matrix, W 2 , being accurately estimated, and the convergence of the ﬁrst hidden layer matrix, W 1 , being less accurately estimated.

Original: Despite this, the angles between the estimated gradient and the true gradient (proportional to e T WB T ˜ e ) are very close to zero for both layers (Figure 4B) (less than 90 degrees corresponds to a descent direction).
Revised: Despite this, the angles between the estimated gradient and the true gradient (proportional to e T WB T ˜ e ) are very close to zero for both layers (Figure 4B) (less than 90 degrees corresponds to a descent direction).

Original: Thus the estimated gradients strongly align with true gradients in both layers.
Revised: Thus the estimated gradients strongly align with true gradients in both layers.

Original: Recent studies have shown that sign congruence of the feedforward and feedback matrices is all that is required to achieve good performance Liao et al.
Revised: Recent studies have shown that sign congruence of the feedforward and feedback matrices is all that is required to achieve good performance Liao et al.

Original: (2016); Xiao et al.
Revised: ; Xiao et al.

Original: Here signiﬁcant sign congruence is achieved in both layers (Figure 4C), despite the matrices themselves being quite different in the ﬁrst layer.
Revised: Here signiﬁcant sign congruence is achieved in both layers (Figure 4C), despite the matrices themselves being quite different in the ﬁrst layer.

Original: The number of neurons has an effect on both the relative error in each layer and the extent of alignment between true and synthetic gradient (Figure 4D,E).
Revised: The number of neurons has an effect on both the relative error in each layer and the extent of alignment between true and synthetic gradient (Figure 4D,E).

Original: The method provides useful error signals for a variety of sized networks, and can provide useful error information to layers through a deep network.
Revised: The method provides useful error signals for a variety of sized networks, and can provide useful error information to layers through a deep network.

Original: C E XPERIMENT DETAILS
Revised: C E XPERIMENT DETAILS

Original: Details of each task andparameters are provided here.
Revised: Details of each task and

Original: All code is implemented in TensorFlow.

Original: C. F IGURE
Revised: C. F IGURE

Original: Networks are 784-50-20-10 with an MSE loss function.
Revised: Networks are 784-50-20-10 with an MSE loss function.

Original: A sigmoid non-linearity is used.
Revised: A sigmoid non-linearity is used.

Original: A batch size of 32 is used.
Revised: A batch size of 32 is used.

Original: B is updated using synthetic gradient updates with learning rate η = 0 . 
Revised: B is updated using synthetic gradient updates with learning rate η = 0 . W is updated with learning rate 0 .

Original: 0005 , W is updated with learning rate
Revised: 0004 , standard deviation of noise is 0.01. Same step size is used for feedback alignment, backpropagation and node perturbation. An initial warm-up period ofiterations is used, in which the feedforward weights are frozen but the feedback weights are adjusted. C. F IGURE

Original: Network has dimensions 784-200-2-200-784.
Revised: Network has dimensions 784-200-2-200-784.

Original: Activation functions are, in order: tanh, identity, tanh, relu.
Revised: Activation functions are, in order: tanh, identity, tanh, relu.

Original: MNIST input data with MSE reconstruction loss is used.
Revised: MNIST input data with MSE reconstruction loss is used.

Original: A batch size of 32 was used.
Revised: A batch size of 32 was used.

Original: In this case stochastic gradient descent was used to update B .
Revised: In this case stochastic gradient descent was used to update B .

Original: Values for W step size, noise variance and B step size were found by random hyperparameter search for each method.
Revised: Values for W step size, noise variance and B step size were found by random hyperparameter search for each method.

Original: The denoising autoencoder used Gaussian noise with zero mean and standard deviation σ = 0 .
Revised: The denoising autoencoder used Gaussian noise with zero mean and standard deviation σ = 0 .

Original: 3 added to the input training data.
Revised: 3 added to the input training data.

Original:  F 
Revised: C. F IGURE

Original: Networks are 784-50-20-10 (noise variance) or 784-N-50-10 (number of neurons) solving MNIST with an MSE loss function.
Revised: Networks are 784-50-20-10 (noise variance) or 784-N-50-10 (number of neurons) solving MNIST with an MSE loss function.

Original: A sigmoid non-linearity is used.
Revised: A sigmoid non-linearity is used.

Original: A batch size of 32 is used.
Revised: A batch size of 32 is used.

Original: Here
Revised: Here

Original: W is ﬁxed, and B is updated according to an online ridge regression least-squares solution.
Revised: W is ﬁxed, and B is updated according to an online ridge regression least-squares solution.

Original: This was used becase it converges faster than the gradient-descent based optimization used for learning
Revised: This was used becase it converges faster than the gradient-descent based optimization used for learning

Original: B throughout the rest of the text, so is a better test of consistency.
Revised: B throughout the rest of the text, so is a better test of consistency.

Original: A regularization parameter of γ = 0 .
Revised: A regularization parameter of γ = 0 .

Original: 1 was used for the ridge regression.
Revised: 1 was used for the ridge regression.

Original: That is, for each update, B i was set to the exact solution of the following:
Revised: That is, for each update, B i was set to the exact solution of the following:

Original: [Equation]
Revised: [Equation]

Original: C. CNN ARCHITECTURE AND IMPLEMENTATION
Revised: C. CNN ARCHITECTURE AND IMPLEMENTATION

Original: Code and CNN architecture are based on the direct feedback alignment implementation of Crafton et al.
Revised: Code and CNN architecture are based on the direct feedback alignment implementation of Crafton et al.

Original: Speciﬁcally, for both CIFAR10 and CIFAR100, the CNN has the architecture
Revised: Speciﬁcally, for both CIFAR10 and CIFAR100, the CNN has the architecture

Original: Conv(3x3, 1x1, 32), MaxPool(3x3, 2x2), Conv(5x5, 1x1, 128), MaxPool(3x3, 2x2), Conv(5x5, 1x1, 256), MaxPool(3x3, 2x2), FC 2048, FC 2048, Softmax(10).
Revised: Conv(3x3, 1x1, 32), MaxPool(3x3, 2x2), Conv(5x5, 1x1, 128), MaxPool(3x3, 2x2), Conv(5x5, 1x1, 256), MaxPool(3x3, 2x2), FC 2048, FC 2048, Softmax(10).

Original: Hyperparameters (learning rate, feedback learning rate, and perturbation noise level) were found through random search.
Revised: Hyperparameters (learning rate, feedback learning rate, and perturbation noise level) were found through random search.

Original: All other parameters are the same as Crafton et al.
Revised: All other parameters are the same as Crafton et al.

Original: In particular, ADAM optimizer was used, and dropout with probability 0.5 was used.
Revised: In particular, ADAM optimizer was used, and dropout with probability 0.5 was used.

Revised: C. N OISE ABLATION STUDY

Revised: The methods listed in Table 2 are implemented as follows.

Revised: For the autoencoding task: Through hyperparameter search, a noise standard deviation of c ∗ h = 0 .

Revised: 02 was found to give optimal performance for our method.

Revised: For BP(SGD), BP(ADAM), FA, the ‘noise’ results in the Table are obtained by adding zero-mean Gaussian noise to the activations with the same standard deviation, c ∗ h .

Revised: For the

Revised: DAE, a noise standard deviation of c i = 0 .

Revised: 3 was added to the inputs of the network.

Revised: Implementation of the synthetic gradient method here takes the same form as our method: g ( h , e , y ; B ) =

Revised: B e (this contrasts with the form used in Jaderberg et al.

Revised: (2016): g ( h , e , y ; B, c ) =

Revised: B T h + c ).

Revised: But the matrices B are trained by providing true gradients λ , instead of noisy estimators based on node perturbation.

Revised: This is not biologically plausible, but provides a useful baseline to determine the source of good performance.

Revised: The other co-adapting baseline we investigate is the ‘matching’ rule (similar to (Akrout et al., 2019; Rombouts et al., 2015; Martinolli et al., 2018)): the updates to B match those of W , and weight decay is used to drive the feedforward and feedback matrices to be similar.

Revised: For the CIFAR10 results, our hyperparameter search identiﬁed a noise standard deviation of c h = 0 .

Revised: 067 to be optimal.

Revised: This was added to the activations .

Revised: The synthetic gradients took the same form as above.