Theories of Error Back-Propagation in the Brain
Authors: James C.R. Whittington, Rafal Bogacz
Journal: Trends in Cognitive Sciences (2019)

The error back-propagation algorithm can be approximated in networks of neurons, in which plasticity only depends on the activity of presynaptic and postsynaptic neurons. These biologically plausible deep learning models include both feedforward and feedback connections, allowing the errors made by the network to propagate through the layers. The learning rules in different biologically plausible models can be implemented with different types of spike-time-dependent plasticity. The dynamics and plasticity of the models can be described within a common framework of energy minimisation. This review article summarises recently proposed theories on how neural circuits in the brain could approximate the error back-propagation algorithm used by artificial neural networks. Computational models implementing these theories achieve learning as efficient as artificial neural networks, but they use simple synaptic plasticity rules based on activity of presynaptic and postsynaptic neurons. The models have similarities, such as including both feedforward and feedback connections, allowing information about error to propagate throughout the network. Furthermore, they incorporate experimental evidence on neural connectivity, responses, and plasticity. These models provide insights on how brain networks might be organised such that modification of synaptic weights on multiple levels of cortical hierarchy leads to improved performance on tasks. This review article summarises recently proposed theories on how neural circuits in the brain could approximate the error back-propagation algorithm used by artificial neural networks. Computational models implementing these theories achieve learning as efficient as artificial neural networks, but they use simple synaptic plasticity rules based on activity of presynaptic and postsynaptic neurons. The models have similarities, such as including both feedforward and feedback connections, allowing information about error to propagate throughout the network. Furthermore, they incorporate experimental evidence on neural connectivity, responses, and plasticity. These models provide insights on how brain networks might be organised such that modification of synaptic weights on multiple levels of cortical hierarchy leads to improved performance on tasks. In the past few years, computer programs using deep learning (see Glossary) have achieved impressive results in complex cognitive tasks that were previously only in the reach of humans. These tasks include processing of natural images and language [1LeCun Y. et al.Deep learning.Nature. 2015; 521: 436-444Crossref PubMed Scopus (42113) Google Scholar], or playing arcade and board games [2Mnih V. et al.Human-level control through deep reinforcement learning.Nature. 2015; 518: 529-533Crossref PubMed Scopus (13741) Google Scholar, 3Silver D. et al.Mastering the game of Go with deep neural networks and tree search.Nature. 2016; 529: 484-489Crossref PubMed Scopus (8554) Google Scholar]. Since these recent deep learning applications use extended versions of classic artificial neural networks [4Rumelhart D.E. et al.Learning representations by back-propagating errors.Nature. 1986; 323: 533-536Crossref Scopus (15380) Google Scholar], their success has inspired studies comparing information processing in artificial neural networks and the brain. It has been demonstrated that when artificial neural networks learn to perform tasks such as image classification or navigation, the neurons in their layers develop representations similar to those seen in brain areas involved in these tasks, such as receptive fields across the visual hierarchy or grid cells in the entorhinal cortex [5Banino A. et al.Vector-based navigation using grid-like representations in artificial agents.Nature. 2018; 557: 429-433Crossref PubMed Scopus (289) Google Scholar, 6Whittington, J.C.R. et al. (2018) Generalisation of structural knowledge in the hippocampal-entorhinal system. In 31st Conference on Neural Information Processing Systems (NIPS 2018), MontrealGoogle Scholar, 7Yamins D.L. DiCarlo J.J. Using goal-driven deep learning models to understand sensory cortex.Nat. Neurosci. 2016; 19: 356-365Crossref PubMed Scopus (650) Google Scholar]. This suggests that the brain may use analogous algorithms. Furthermore, thanks to current computational advances, artificial neural networks can now provide useful insights on how complex cognitive functions are achieved in the brain [8Bowers J.S. Parallel distributed processing theory in the age of deep networks.Trends Cogn. Sci. 2017; 21: 950-961Abstract Full Text Full Text PDF PubMed Scopus (22) Google Scholar]. A key question that remains open is how the brain could implement the error back-propagation algorithm used in artificial neural networks. This algorithm describes how the weights of synaptic connections should be modified during learning, and its attractiveness, in part, comes from prescribing weight changes that reduce errors made by the network, according to a theoretical analysis. Although artificial neural networks were originally inspired by the brain, the modification of their synaptic connections, or weights, during learning appears biologically unrealistic [9Crick F. The recent excitement about neural networks.Nature. 1989; 337: 129-132Crossref PubMed Scopus (353) Google Scholar, 10Grossberg S. Competitive learning: from interactive activation to adaptive resonance.Cogn. Sci. 1987; 11: 23-63Crossref Google Scholar]. Nevertheless, recent models have demonstrated that learning as efficient as in artificial neural networks can be achieved in distributed networks of neurons using only simple plasticity rules [11Bengio Y. et al.STDP-Compatible approximation of backpropagation in an energy-based model.Neural Comput. 2017; 29: 555-577Crossref PubMed Scopus (47) Google Scholar, 12Guerguiev J. et al.Towards deep learning with segregated dendrites.eLife. 2017; 6e22901Crossref PubMed Scopus (173) Google Scholar, 13Sacramento, J. et al. (2018) Dendritic cortical microcircuits approximate the backpropagation algorithm. In 31st Conference on Neural Information Processing Systems (NIPS 2018), MontrealGoogle Scholar, 14Whittington J.C.R. Bogacz R. An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity.Neural Comput. 2017; 29: 1229-1262Crossref PubMed Scopus (91) Google Scholar]. These theoretic studies are important because they overrule the dogma, generally accepted for the past 30 years, that the error back-propagation algorithm is too complicated for the brain to implement [9Crick F. The recent excitement about neural networks.Nature. 1989; 337: 129-132Crossref PubMed Scopus (353) Google Scholar, 10Grossberg S. Competitive learning: from interactive activation to adaptive resonance.Cogn. Sci. 1987; 11: 23-63Crossref Google Scholar]. Before discussing this new generation of models in detail, we first provide a brief overview of how the back-propagation algorithm is used to train artificial neural networks and discuss why it was considered biologically implausible. To effectively learn from feedback, the synaptic connections often need to be appropriately adjusted in multiple hierarchical areas simultaneously. For example, when a child learns to name letters, the incorrect pronunciation may be a combined result of incorrect synaptic connections in speech, associative, and visual areas. When a multi-layer artificial neural network makes an error, the error back-propagation algorithm appropriately assigns credit to individual synapses throughout all levels of hierarchy and prescribes which synapses need to be modified and by how much. How is the back-propagation algorithm used to train artificial neural networks? The algorithm is trained on a set of examples, each consisting of an input pattern and a target pattern. For each such pair, the network first generates its prediction based on the input pattern and then the synaptic weights are modified to minimise the difference between the target and the predicted pattern. To determine the appropriate modification, an error term is computed for each neuron throughout the network. This describes how the activity of the neuron should change to reduce the discrepancy between the predicted and target pattern (Box 1). Each weight is modified by an amount determined by the product between the activity of the neuron it projects from and the error term of the neuron it projects to.Box 1Artificial Neural NetworksA conventional artificial neural network consists of layers of neurons, with each neuron within a layer receiving a weighted input from the neurons in the previous layer (Figure IA). The input layer is first set to be the input pattern and then a prediction is made by propagating the activity through the layers, according to Equation 1.1, where xl is a vector denoting neurons in layer l and Wl−1 is a matrix of synaptic weights from layer l − 1 to layer l. An activation function f is applied to each neuron to allow for nonlinear computations.During learning, the synaptic connections are modified to minimise a cost function quantifying the discrepancy between the predicted and target patterns (typically defined as in Equation 1.2). In particular, the weights are modified in the direction of steepest decrease (or gradient) of the cost function (Figure ID). Such modification is described in Equation 1.3, where δl+1 is a vector of error terms associated with neurons xl+1. The error terms for the last layer L are defined in Equation 1.4 as the difference between the target activity t and the predicted activity. Thus, the error of an output neuron is positive if its target activity is higher than the predicted activity. For the earlier layers, the errors are computed according to Equation 1.5 as a sum of the errors of neurons in the layer above weighted by the strengths of their connections (and further scaled by the derivative of the activation function; in Equation 1.5 · denotes element-wise multiplication). For example, an error of a hidden unit is positive if it sends excitatory projections to output units with high error terms, so increasing the activity of such a hidden neuron would reduce the error on the output. Once the errors are computed, each weight is changed according to Equation 1.3 in proportion to the product of the error term associated with a postsynaptic neuron and the activity of a presynaptic neuron. A conventional artificial neural network consists of layers of neurons, with each neuron within a layer receiving a weighted input from the neurons in the previous layer (Figure IA). The input layer is first set to be the input pattern and then a prediction is made by propagating the activity through the layers, according to Equation 1.1, where xl is a vector denoting neurons in layer l and Wl−1 is a matrix of synaptic weights from layer l − 1 to layer l. An activation function f is applied to each neuron to allow for nonlinear computations. During learning, the synaptic connections are modified to minimise a cost function quantifying the discrepancy between the predicted and target patterns (typically defined as in Equation 1.2). In particular, the weights are modified in the direction of steepest decrease (or gradient) of the cost function (Figure ID). Such modification is described in Equation 1.3, where δl+1 is a vector of error terms associated with neurons xl+1. The error terms for the last layer L are defined in Equation 1.4 as the difference between the target activity t and the predicted activity. Thus, the error of an output neuron is positive if its target activity is higher than the predicted activity. For the earlier layers, the errors are computed according to Equation 1.5 as a sum of the errors of neurons in the layer above weighted by the strengths of their connections (and further scaled by the derivative of the activation function; in Equation 1.5 · denotes element-wise multiplication). For example, an error of a hidden unit is positive if it sends excitatory projections to output units with high error terms, so increasing the activity of such a hidden neuron would reduce the error on the output. Once the errors are computed, each weight is changed according to Equation 1.3 in proportion to the product of the error term associated with a postsynaptic neuron and the activity of a presynaptic neuron. Although the described procedure is used to train artificial neural networks, analogous steps may take place during learning in the brain. For example, in the case of the child naming letters mentioned above, the input pattern corresponds to an image of a letter. After seeing an image, the child makes a guess at the name (predicted pattern) via a neural network between visual and speech areas. On supervision by his or her parent of the correct pronunciation (target pattern), synaptic weights along the processing stream are modified so that it is more likely that the correct sound will be produced when seeing that image again. Although the algorithmic process described above appears simple enough, there are a few problems with implementing it in biology. Below, we briefly discuss three key issues. Conventional artificial neural networks are only defined to compute information in a forward direction, with the back-propagating errors computed separately by an external algorithm. Without local error representation, each synaptic weight update depends on the activity and computations of all downstream neurons. Since biological synapses change their connection strength based solely on local signals (e.g., the activity of the neurons they connect), it appears unclear how the synaptic plasticity afforded by the back-propagation algorithm could be achieved in the brain. Historically, this is a major criticism; thus it is a main focus of our review article. In artificial neural networks, the errors are back-propagated using the same weights as those when propagating information forward during prediction. This weight symmetry suggests that identical connections should exist in both directions between connected neurons. Although bidirectional connections are significantly more common in cortical networks than expected by chance, they are not always present [15Song S. et al.Highly nonrandom features of synaptic connectivity in local cortical circuits.PLoS Biol. 2005; 3: 507-519Google Scholar]. Furthermore, even if bidirectional connections were always present, the backwards and forwards weights would still have to correctly align themselves. Artificial neural networks use artificial neurons that send a continuous output (corresponding to a firing rate of biological neurons), whereas real neurons use spikes. Generalising the back-propagation algorithm to neurons using discrete spikes is not trivial, because it is unclear how to compute the derivate term found in the back-propagation algorithm (Box 1). Away from the back-propagation algorithm, the description of computations inside neurons in artificial neural networks is also simplified as a linear summation of inputs. Each of the above-mentioned issues has been investigated by multiple studies. The lack of local error representation has been addressed by early theories by proposing that the errors associated with individual neurons are not computed, but instead the synaptic plasticity is driven by a global error signal carried by neuromodulators [16Mazzoni P. et al.A more biologically plausible learning rule for neural networks.Proc. Natl. Acad. Sci. U. S. A. 1991; 88: 4433-4437Crossref PubMed Scopus (138) Google Scholar, 17Williams R.J. Simple statistical gradient-following algorithms for connectionist reinforcement learning.Mach. Learn. 1992; 8: 229-256Crossref Google Scholar, 18Unnikrishnan K.P. Venugopal K.P. Alopex: a correlation-based learning algorithm for feedforward and recurrent neural networks.Neural Comput. 1994; 6: 469-490Crossref Google Scholar, 19Seung H.S. Learning in spiking neural networks by reinforcement of stochastic synaptic transmission.Neuron. 2003; 40: 1063-1073Abstract Full Text Full Text PDF PubMed Scopus (238) Google Scholar]. However, it has been demonstrated that learning in such models is slow and does not scale with network size [20Werfel J. et al.Learning curves for stochastic gradient descent in linear feedforward networks.Neural Comput. 2005; 17: 2699-2718Crossref PubMed Scopus (47) Google Scholar]. More promisingly, in the past few years, several models have been proposed that do represent errors locally and thus more closely approximate the back-propagation algorithm. These models perform similarly to artificial neural networks on standard benchmark tasks (e.g., handwritten digit classification) [12Guerguiev J. et al.Towards deep learning with segregated dendrites.eLife. 2017; 6e22901Crossref PubMed Scopus (173) Google Scholar, 13Sacramento, J. et al. (2018) Dendritic cortical microcircuits approximate the backpropagation algorithm. In 31st Conference on Neural Information Processing Systems (NIPS 2018), MontrealGoogle Scholar, 14Whittington J.C.R. Bogacz R. An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity.Neural Comput. 2017; 29: 1229-1262Crossref PubMed Scopus (91) Google Scholar, 21Lillicrap T.P. et al.Random synaptic feedback weights support error backpropagation for deep learning.Nat. Commun. 2016; 713276Crossref PubMed Scopus (336) Google Scholar, 22Scellier B. Bengio Y. Equilibrium propagation: bridging the gap between energy-based models and backpropagation.Front. Comput. Neurosci. 2017; 11: 24Crossref PubMed Scopus (146) Google Scholar], and we summarise several of them in more detail in the following sections. The criticism of weight symmetry has been addressed by demonstrating that even if the errors in artificial neural networks are back-propagated by random connections, good performance in classification tasks can still be achieved [21Lillicrap T.P. et al.Random synaptic feedback weights support error backpropagation for deep learning.Nat. Commun. 2016; 713276Crossref PubMed Scopus (336) Google Scholar, 23Zenke F. Ganguli S. SuperSpike: supervised learning in multilayer spiking neural networks.Neural Comput. 2018; 30: 1514-1541Crossref PubMed Scopus (209) Google Scholar, 24Mostafa, H. et al. (2017) Deep supervised learning using local errors. arXiv preprint arXiv:1711.06756Google Scholar, 25Scellier, B. et al. (2018) Generalization of equilibrium propagation to vector field dynamics. arXiv 1808.04873Google Scholar, 26Liao, Q. et al. (2016) How important is weight symmetry in backpropagation? In AAAI Conference on Artificial Intelligence, pp. 1837–1844, AAAIGoogle Scholar, 27Baldi P. Sadowski P. A theory of local learning, the learning channel, and the optimality of backpropagation.Neural Netw. 2016; 83: 51-74Crossref PubMed Scopus (39) Google Scholar]. This being said, there is still some concern regarding this issue [28Bartunov, S. et al. (2018) Assessing the scalability of biologically-motivated deep learning algorithms and architectures. In 31st Conference on Neural Information Processing Systems (NIPS 2018), MontrealGoogle Scholar]. With regard to the biological realism of neurons, it has been shown that the back-propagation algorithm can be generalised to neurons producing spikes [29Sporea I. Grüning A. Supervised learning in multilayer spiking neural networks.Neural Comput. 2013; 25: 473-509Crossref PubMed Scopus (97) Google Scholar] and that problems with calculating derivatives using spikes can be overcome [23Zenke F. Ganguli S. SuperSpike: supervised learning in multilayer spiking neural networks.Neural Comput. 2018; 30: 1514-1541Crossref PubMed Scopus (209) Google Scholar]. Furthermore, it has been proposed that when more biologically realistic neurons are considered, they themselves may approximate a small artificial neural network in their dendritic structures [30Schiess M. et al.Somato-dendritic synaptic plasticity and error-backpropagation in active dendrites.PLoS Comput. Biol. 2016; 12e1004638Crossref PubMed Scopus (43) Google Scholar]. There is a diversity of ideas on how the back-propagation algorithm may be approximated in the brain [31Balduzzi, D. et al. (2015) Kickback cuts backprop's red-tape: biologically plausible credit assignment in neural networks. In AAAI Conference on Artificial Intelligence, pp. 485–491, AAAIGoogle Scholar, 32Krotov, D. and Hopfield, J. (2018) Unsupervised learning by competing hidden units. arXiv preprint arXiv:1806.10181Google Scholar, 33Kuśmierz Ł. et al.Learning with three factors: modulating Hebbian plasticity with errors.Curr. Opin. Neurobiol. 2017; 46: 170-177Crossref PubMed Scopus (52) Google Scholar, 34Marblestone A.H. et al.Toward an integration of deep learning and neuroscience.Front. Comput. Neurosci. 2016; 10: 94Crossref PubMed Scopus (316) Google Scholar, 35Bengio, Y. (2014) How auto-encoders could provide credit assignment in deep networks via target propagation. arXiv preprint arXiv:1407.7906Google Scholar, 36Lee, D.-H. et al. (2015) Difference target propagation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 498–515, SpringerGoogle Scholar]; however, we review the principles behind a set of related models [11Bengio Y. et al.STDP-Compatible approximation of backpropagation in an energy-based model.Neural Comput. 2017; 29: 555-577Crossref PubMed Scopus (47) Google Scholar, 13Sacramento, J. et al. (2018) Dendritic cortical microcircuits approximate the backpropagation algorithm. In 31st Conference on Neural Information Processing Systems (NIPS 2018), MontrealGoogle Scholar, 14Whittington J.C.R. Bogacz R. An approximation of the error backpropagation algorithm in a predictive coding network with local Hebbian synaptic plasticity.Neural Comput. 2017; 29: 1229-1262Crossref PubMed Scopus (91) Google Scholar, 37O'Reilly R.C. Biologically plausible error-driven learning using local activation differences: the generalized recirculation algorithm.Neural Comput. 1996; 8: 895-938Crossref Scopus (211) Google Scholar] that have substantial connections with biological data while closely paralleling the back-propagation algorithm. These models operate with minimal external control, as they can compute the errors associated with individual neurons through the dynamics of the networks. Thus, synaptic weight modifications depend only on the activity of presynaptic and postsynaptic neurons. Furthermore, these models incorporate important features of brain biology, such as spike time-dependent plasticity, patterns of neural activity during learning, and properties of pyramidal neurons and cortical microcircuits. We emphasise that these models rely on fundamentally similar principles. In particular, the models include both feedforward and feedback connections, thereby allowing information about the errors made by the network to propagate throughout the network without requiring an external program to compute the errors. Furthermore, these dynamics, as well as the synaptic plasticity, can be described within a common framework of energy minimisation. We divide the reviewed models in two classes differing in how the errors are represented, and we summarise them in the following sections. This class of model encodes errors in the differences in neural activity across time. The first model in this class is the contrastive learning model [37O'Reilly R.C. Biologically plausible error-driven learning using local activation differences: the generalized recirculation algorithm.Neural Comput. 1996; 8: 895-938Crossref Scopus (211) Google Scholar]. It relies on an observation that weight changes proportional to an error (difference between predicted and target pattern) can be decomposed into two separate updates: one update based on activity without the target present and the other update with the target pattern provided to the output neurons [38Ackley D.H. et al.A learning algorithm for Boltzmann machines.Cogn. Sci. 1985; 9: 147-169Crossref Google Scholar] (Box 2). Thus, the error back-propagation algorithm can be approximated in a network in which the weights are modified twice: during prediction according to anti-Hebbian plasticity and then according to Hebbian plasticity once the target is provided and the network converges to an equilibrium (after the target activity has propagated to earlier layers via feedback connections) [37O'Reilly R.C. Biologically plausible error-driven learning using local activation differences: the generalized recirculation algorithm.Neural Comput. 1996; 8: 895-938Crossref Scopus (211) Google Scholar]. The role of the first modification is to 'unlearn' the existing association between input and prediction, while the role of the second modification is to learn the new association between input and target.Box 2Temporal-Error ModelsTemporal-error models describe learning in networks with recurrent feedback connections to the hidden nodes (Figure IA). The rate of change of activity of a given node is proportional to the summed inputs from adjacent layers, along with a decay term proportional to the current level of activity (Figure IB). As the network is now recurrent, it is no longer possible to write a simple equation describing how the activity depends on other nodes (such as Equation 1.1 in Box 1); instead, the dynamics of neurons is described by the differential Equation 2.1 [72Pineda F.J. Generalization of back-propagation to recurrent neural networks.Phys. Rev. Lett. 1987; 59: 2229-2232Crossref PubMed Scopus (594) Google Scholar], where x˙l denotes the rate of change over time of xl (all equations in this figure ignore nonlinearities for brevity).In the contrastive learning model, the weight modifications based on errors are decomposed into two separate changes occurring at different times. To understand learning in this model, it is easiest to consider how the weights connecting to the output layer are modified. Substituting Equation 1.4 into Equation 1.3, we see in Equation 2.2 that the weight modification required by the back-propagation algorithm can be decomposed into two terms. The first term corresponds to anti-Hebbian plasticity that should take place when the output activity is predicted based on the input propagated through the network. The second term corresponds to Hebbian plasticity that should take place when the output layer is set to the target pattern. O'Reilly [37O'Reilly R.C. Biologically plausible error-driven learning using local activation differences: the generalized recirculation algorithm.Neural Comput. 1996; 8: 895-938Crossref Scopus (211) Google Scholar] demonstrated that in the presence of backward connections, the information about the target pattern propagates to earlier layers, and an analogous sequence of weight modifications in the hidden layers also approximates a version of the back-propagation algorithm for recurrent networks [72Pineda F.J. Generalization of back-propagation to recurrent neural networks.Phys. Rev. Lett. 1987; 59: 2229-2232Crossref PubMed Scopus (594) Google Scholar].In the continuous update model, the output nodes are gradually changed from the predicted pattern (x3|¬t) towards the target values (t), as shown for a sample neuron in Figure ID. Thus, the temporal derivative of output activity (x˙3) is proportional to (t −x3|¬t), that is, to the error on the output (defined in Equation 1.4). Hence, the weight modification required by back-propagation is simply equal to the product of presynaptic activity and the rate of change of the postsynaptic activity (Equation 2.3). Temporal-error models describe learning in networks with recurrent feedback connections to the hidden nodes (Figure IA). The rate of change of activity of a given node is proportional to the summed inputs from adjacent layers, along with a decay term proportional to the current level of activity (Figure IB). As the network is now recurrent, it is no longer possible to write a simple equation describing how the activity depends on other nodes (such as Equation 1.1 in Box 1); instead, the dynamics of neurons is described by the differential Equation 2.1 [72Pineda F.J. Generalization of back-propagation to recurrent neural networks.Phys. Rev. Lett. 1987; 59: 2229-2232Crossref PubMed Scopus (594) Google Scholar], where x˙l denotes the rate of change over time of xl (all equations in this figure ignore nonlinearities for brevity). In the contrastive learning model, the weight modifications based on errors are decomposed into two separate changes occurring at different times. To understand learning in this model, it is easiest to consider how the weights connecting to the output layer are modified. Substituting Equation 1.4 into Equation 1.3, we see in Equation 2.2 that the weight modification required by the back-propagation algorithm can be decomposed into two terms. The first term corresponds to anti-Hebbian plasticity that should take place when the output activity is predicted based on the input propagated through the network. The second term corresponds to Hebbian plasticity that should take place when the output layer is set to the target pattern. O'Reilly [37O'Reilly R.C. Biologically plausible error-driven learning using local activation differences: the generalized recirculation algorithm.Neural Comput. 1996; 8: 895-938Crossref Scopus (211) Google Scholar] demonstrated that in the presence of backward connections, the information about the target pattern propagates to earlier layers, and an analogous sequence of weight modifications in the hidden layers also approximates a version of the back-propagation algorithm for recurrent networks [72Pineda F.J. Generalization of back-propagation to recurrent neural networks.Phys. Rev. Lett. 1987; 59: 2229-2232Crossref PubMed Scopus (594) Google Scholar]. In the continuous update model, the o