Original: P REDICTIVE C ODING A PPROXIMATES B ACKPROP ALONG A RBITRARY C OMPUTATION G RAPHS
Revised: P REDICTIVE C ODING A PPROXIMATES B ACKPROP ALONG A RBITRARY C OMPUTATION G RAPHS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Backpropagation of error (backprop) is a powerful algorithm for training machine learning architectures through end-to-end differentiation.
Revised: Backpropagation of error (backprop) is a powerful algorithm for training machine learning architectures through end-to-end differentiation.

Original: Recently it has been shown that backprop in multilayer-perceptrons (MLPs) can be approximated using predictive coding, a biologically-plausible process theory of cortical computation which relies solely on local and Hebbian updates.
Revised: Recently it has been shown that backprop in multilayer-perceptrons (MLPs) can be approximated using predictive coding, a biologically-plausible process theory of cortical computation which relies solely on local and Hebbian updates.

Original: The power of backprop, however, lies not in its instantiation in MLPs, but rather in the concept of automatic differentiation which allows for the optimisation of any differentiable program expressed as a computation graph.
Revised: The power of backprop, however, lies not in its instantiation in MLPs, but rather in the concept of automatic differentiation which allows for the optimisation of any differentiable program expressed as a computation graph.

Original: Here, we demonstrate that predictive coding converges asymptotically (and in practice rapidly) to exact backprop gradients on arbitrary computation graphs using only local learning rules.
Revised: Here, we demonstrate that predictive coding converges asymptotically (and in practice rapidly) to exact backprop gradients on arbitrary computation graphs using only local learning rules.

Original: We apply this result to develop a straightforward strategy to translate core machine learning architectures into their predictive coding equivalents.
Revised: We apply this result to develop a straightforward strategy to translate core machine learning architectures into their predictive coding equivalents.

Original: We construct predictive coding CNNs, RNNs, and the more complex LSTMs, which include a non-layer-like branching internal graph structure and multiplicative interactions.
Revised: We construct predictive coding CNNs, RNNs, and the more complex LSTMs, which include a non-layer-like branching internal graph structure and multiplicative interactions.

Original: Our models perform equivalently to backprop on challenging machine learning benchmarks, while utilising only local and (mostly) Hebbian plasticity.
Revised: Our models perform equivalently to backprop on challenging machine learning benchmarks, while utilising only local and (mostly) Hebbian plasticity.

Original: Our method raises the potential that standard machine learning algorithms could in principle be directly implemented in neural circuitry, and may also contribute to the development of completely distributed neuromorphic architectures.
Revised: Our method raises the potential that standard machine learning algorithms could in principle be directly implemented in neural circuitry, and may also contribute to the development of completely distributed neuromorphic architectures.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning has seen stunning successes in the last decade in computer vision (Krizhevsky et al., 2012; Szegedy et al., 2015), natural language processing and translation (Vaswani et al., 2017;
Revised: Deep learning has seen stunning successes in the last decade in computer vision (Krizhevsky et al., 2012; Szegedy et al., 2015), natural language processing and translation (Vaswani et al., 2017;

Original: Radford et al., 2019; Kaplan et al., 2020), and computer game playing (Mnih et al., 2015; Silver et al., 2017; Schrittwieser et al., 2019; Vinyals et al., 2019).
Revised: Radford et al., 2019; Kaplan et al., 2020), and computer game playing (Mnih et al., 2015; Silver et al., 2017; Schrittwieser et al., 2019; Vinyals et al., 2019).

Original: While there is a great variety of architectures and models, they are all trained by gradient descent using gradients computed by automatic differentiation (AD).
Revised: While there is a great variety of architectures and models, they are all trained by gradient descent using gradients computed by automatic differentiation (AD).

Original: The key insight of AD is that it sufﬁces to deﬁne a forward model which maps inputs to predictions according to some parameters.
Revised: The key insight of AD is that it sufﬁces to deﬁne a forward model which maps inputs to predictions according to some parameters.

Original: Then, using the chain rule of calculus, it is possible, as long as every operation of the forward model is differentiable, to differentiate back through the computation graph of the model so as to compute the sensitivity of every parameter in the model to the error at the output, and thus adjust every single parameter to best minimize the total loss.
Revised: Then, using the chain rule of calculus, it is possible, as long as every operation of the forward model is differentiable, to differentiate back through the computation graph of the model so as to compute the sensitivity of every parameter in the model to the error at the output, and thus adjust every single parameter to best minimize the total loss.

Original: Early models were typically simple artiﬁcial neural networks where the computation graph is simply a composition of matrix multiplications and elementwise nonlinearities, and for which the implementation of automatic differentation has become known as ‘backpropagation’ (or ’backprop’).
Revised: Early models were typically simple artiﬁcial neural networks where the computation graph is simply a composition of matrix multiplications and elementwise nonlinearities, and for which the implementation of automatic differentation has become known as ‘backpropagation’ (or ’backprop’).

Original: However, automatic differentiation allows for substantially more complicated graphs to be differentiated through, up to, and including, arbitrary programs (Griewank et al., 1989;
Revised: However, automatic differentiation allows for substantially more complicated graphs to be differentiated through, up to, and including, arbitrary programs (Griewank et al., 1989;

Original: Baydin et al., 2017; Paszke et al., 2017; Revels et al., 2016; Innes et al., 2019; Werbos, 1982; Rumelhart and Zipser, 1985; Linnainmaa, 1970).
Revised: Baydin et al., 2017; Paszke et al., 2017; Revels et al., 2016; Innes et al., 2019; Werbos, 1982; Rumelhart and Zipser, 1985; Linnainmaa, 1970).

Original: In recent years this has enabled the differentiation through differential equation solvers (Chen et al., 2018; Tzen and Raginsky, 2019; Rackauckas et al., 2019), physics engines (Degrave et al., 2019; Heiden et al., 2019), raytracers (Pal, 2019), and planning algorithms (Amos and Yarats, 2019; Okada et al., 2017).
Revised: In recent years this has enabled the differentiation through differential equation solvers (Chen et al., 2018; Tzen and Raginsky, 2019; Rackauckas et al., 2019), physics engines (Degrave et al., 2019; Heiden et al., 2019), raytracers (Pal, 2019), and planning algorithms (Amos and Yarats, 2019; Okada et al., 2017).

Original: These advances allow the straightforward training of models which intrinsically embody complex processes and which can encode signiﬁcantly more prior knowledge and structure about a given problem domain than previously possible.
Revised: These advances allow the straightforward training of models which intrinsically embody complex processes and which can encode signiﬁcantly more prior knowledge and structure about a given problem domain than previously possible.

Original: Modern deep learning has also been closely intertwined with neuroscience (Hassabis et al., 2017;
Revised: Modern deep learning has also been closely intertwined with neuroscience (Hassabis et al., 2017;

Original: Hawkins and Blakeslee, 2007; Richards et al., 2019).
Revised: Hawkins and Blakeslee, 2007; Richards et al., 2019).

Original: The backpropagation algorithm itself arose as a technique for training multi-layer perceptrons – simple hierarchical models of neurons inspired by the brain (Werbos, 1982).
Revised: The backpropagation algorithm itself arose as a technique for training multi-layer perceptrons – simple hierarchical models of neurons inspired by the brain (Werbos, 1982).

Original: Despite this origin, and its empirical successes, a consensus has emerged that the brain cannot directly implement backprop, since to do so would require biologically implausible connection rules (Crick, 1989).
Revised: Despite this origin, and its empirical successes, a consensus has emerged that the brain cannot directly implement backprop, since to do so would require biologically implausible connection rules (Crick, 1989).

Original: There are two principal problems.
Revised: There are two principal problems.

Original: Firstly, backprop in the brain appears to require non-local information (since the activity of any speciﬁc neuron affects all subsequent neurons down to the ﬁnal output neuron).
Revised: Firstly, backprop in the brain appears to require non-local information (since the activity of any speciﬁc neuron affects all subsequent neurons down to the ﬁnal output neuron).

Original: It is difﬁcult to see how this information could be transmitted ’backwards’ throughout the brain with the required ﬁdelity without precise connectivity constraints.
Revised: It is difﬁcult to see how this information could be transmitted ’backwards’ throughout the brain with the required ﬁdelity without precise connectivity constraints.

Original: The second problem – the ‘weight transport problem’ is that backprop through MLP style networks requires identical forward and backwards weights.
Revised: The second problem – the ‘weight transport problem’ is that backprop through MLP style networks requires identical forward and backwards weights.

Original: In recent years, however, a succession of models have been introduced which claim to implement backprop in MLP-style models using only biologically plausible connectivity schemes, and Hebbian learning rules (Liao et al., 2016;
Revised: In recent years, however, a succession of models have been introduced which claim to implement backprop in MLP-style models using only biologically plausible connectivity schemes, and Hebbian learning rules (Liao et al., 2016;

Original: Guerguiev et al., 2017; Sacramento et al., 2018; Bengio and Fischer, 2015; Bengio et al., 2017; Ororbia et al., 2020; Whittington and Bogacz, 2019).
Revised: Guerguiev et al., 2017; Sacramento et al., 2018; Bengio and Fischer, 2015; Bengio et al., 2017; Ororbia et al., 2020; Whittington and Bogacz, 2019).

Original: Of particular signiﬁcance is Whittington and Bogacz (2017) who show that predictive coding networks – a type of biologically plausible network which learn through a hierarchical process of prediction error minimization – are mathematically equivalent to backprop in MLP models.
Revised: Of particular signiﬁcance is Whittington and Bogacz (2017) who show that predictive coding networks – a type of biologically plausible network which learn through a hierarchical process of prediction error minimization – are mathematically equivalent to backprop in MLP models.

Original: In this paper we extend this work, showing that predictive coding can not only approximate backprop in MLPs, but can approximate automatic differentiation along arbitrary computation graphs.
Revised: In this paper we extend this work, showing that predictive coding can not only approximate backprop in MLPs, but can approximate automatic differentiation along arbitrary computation graphs.

Original: This means that in theory there exist  biologically plausible algorithms for differentiating through arbitrary programs, utilizing only local connectivity.
Revised: This means that in theory there exist potentially biologically plausible algorithms for differentiating through arbitrary programs, utilizing only local connectivity.

Original: Moreover, in a class of models which we call parameter-linear, which includes many current machine learning models, the required update rules are Hebbian, raising the possibility that a wide range of current machine learning architectures may be faithfully implemented in the brain, or in neuromorphic hardware.
Revised: Moreover, in a class of models which we call parameter-linear, which includes many current machine learning models, the required update rules are Hebbian, raising the possibility that a wide range of current machine learning architectures may be faithfully implemented in the brain, or in neuromorphic hardware.

Original: In this paper we provide two main contributions.
Revised: In this paper we provide two main contributions.

Original: (i)
Revised: (i)

Original: We show that predictive coding converges to automatic differentiation across arbitrary computation graphs.
Revised: We show that predictive coding converges to automatic differentiation across arbitrary computation graphs.

Original: (ii)
Revised: (ii)

Original: We showcase this result by implementing three core machine learning architectures (CNNs, RNNs, and LSTMs) in a predictive coding framework which utilises only local learning rules and mostly Hebbian plasticity.
Revised: We showcase this result by implementing three core machine learning architectures (CNNs, RNNs, and LSTMs) in a predictive coding framework which utilises only local learning rules and mostly Hebbian plasticity.

Original:  P REDICTIVE C ODING ON A RBITRARY C OMPUTATION G RAPHS
Revised:  P REDICTIVE C ODING ON A RBITRARY C OMPUTATION G RAPHS

Original: [Equation]
Revised: [Equation]

Original: Predictions, and prediction errors are updated in parallel using only local information.
Revised: T

Original:  Predictive coding is an inﬂuential theory of cortical function in theoretical and computational neuroscience.
Revised: [Figure] Predictive coding is an inﬂuential theory of cortical function in theoretical and computational neuroscience.

Original: Central to the theory is the idea that the core function of the brain is to minimize prediction errors between what is expected to happen and what actually happens.
Revised: Central to the theory is the idea that the core function of the brain is to minimize prediction errors between what is expected to happen and what actually happens.

Original: Predictive coding views the brain as composed of multiple hierarchical layers which predict the activities of the layers below.
Revised: Predictive coding views the brain as composed of multiple hierarchical layers which predict the activities of the layers below.

Original: Unpredicted activity is registered as prediction error which is then transmitted upwards for a higher layer to process.
Revised: Unpredicted activity is registered as prediction error which is then transmitted upwards for a higher layer to process.

Original: Over time, synaptic connections are adjusted so that the system improves at minimizing prediction error.
Revised: Over time, synaptic connections are adjusted so that the system improves at minimizing prediction error.

Original: Predictive coding possesses a wealth of empirical support (Friston, 2003; 2005; Bogacz, 2017; Whittington and Bogacz, 2019) and offers a single mechanism that accounts for diverse perceptual phenomena such as repetition-suppression (Auksztulewicz and Friston, 2016), endstopping (Rao and Ballard, 1999), bistable perception (Hohwy et al., 2008; Weilnhammer et al., 2017) and illusory motions (Lotter et al., 2016; Watanabe et al., 2018), and even attentional modulation of neural activity (Feldman and Friston, 2010; Kanai et al., 2015).
Revised: Predictive coding possesses a wealth of empirical support (Friston, 2003; 2005; Bogacz, 2017; Whittington and Bogacz, 2019) and offers a single mechanism that accounts for diverse perceptual phenomena such as repetition-suppression (Auksztulewicz and Friston, 2016), endstopping (Rao and Ballard, 1999), bistable perception (Hohwy et al., 2008; Weilnhammer et al., 2017) and illusory motions (Lotter et al., 2016; Watanabe et al., 2018), and even attentional modulation of neural activity (Feldman and Friston, 2010; Kanai et al., 2015).

Original: Moreover, the central role of top-down predictions is consistent with the ubiquity, and importance of, top-down diffuse connections between cortical areas.
Revised: Moreover, the central role of top-down predictions is consistent with the ubiquity, and importance of, top-down diffuse connections between cortical areas.

Original: Predictive coding is consistent with many known aspects of neurophysiology, and has been translated into biologically plausible process theories which deﬁne candidate cortical microcircuits which can implement the algorithm.
Revised: Predictive coding is consistent with many known aspects of neurophysiology, and has been translated into biologically plausible process theories which deﬁne candidate cortical microcircuits which can implement the algorithm.

Original: (Spratling, 2008; Bastos et al., 2012;
Revised: (Spratling, 2008; Bastos et al., 2012;

Original: Kanai et al., 2015; Shipp, 2016).
Revised: Kanai et al., 2015; Shipp, 2016).

Original: In previous work, predictive coding has always been conceptualised as operating on hierarchies of layers (Bogacz, 2017; Whittington and Bogacz, 2017).
Revised: In previous work, predictive coding has always been conceptualised as operating on hierarchies of layers (Bogacz, 2017; Whittington and Bogacz, 2017).

Original: Here we present a generalized form of predictive coding applied to arbitrary computation graphs.
Revised: Here we present a generalized form of predictive coding applied to arbitrary computation graphs.

Original: A computation graph G = { E , V } is a directed acyclic graph (DAG) which can represent the computational ﬂow of essentially any program or computable function as a composition of elementary functions.
Revised: A computation graph G = { E , V } is a directed acyclic graph (DAG) which can represent the computational ﬂow of essentially any program or computable function as a composition of elementary functions.

Original: Each edge e i ∈
Revised: Each edge e i ∈

Original: E of the graph corresponds to an intermediate step – the application of an elementary function – while each vertex v i ∈ V is an intermediate variable computed by applying the functions of the edges to the values of their originating vertices.
Revised: E of the graph corresponds to an intermediate step – the application of an elementary function – while each vertex v i ∈ V is an intermediate variable computed by applying the functions of the edges to the values of their originating vertices.

Original: In this paper, v i denotes the vector of activations within a layer and we denote the set of all vertices as { v i } .
Revised: In this paper, v i denotes the vector of activations within a layer and we denote the set of all vertices as { v i } .

Original: Effectively, computation ﬂows ’forward’ from parent nodes to all their children through the edge functions until the leaf nodes give the ﬁnal output of the program as a whole (see Figure 1 and 2 for an example).
Revised: Effectively, computation ﬂows ’forward’ from parent nodes to all their children through the edge functions until the leaf nodes give the ﬁnal output of the program as a whole (see Figure 1 and 2 for an example).

Original: Given a target T and a loss function L = g ( T, v out ) , the graph’s output can be evaluated and, and if every edge function is differentiable, automatic differentiation can be performed on the computation graph.
Revised: Given a target T and a loss function L = g ( T, v out ) , the graph’s output can be evaluated and, and if every edge function is differentiable, automatic differentiation can be performed on the computation graph.

Original: Predictive coding can be derived elegantly as a variational inference algorithm under a hierarchical
Revised: Predictive coding can be derived elegantly as a variational inference algorithm under a hierarchical

Original: Gaussian generative model (Friston, 2005; Buckley et al., 2017).
Revised: Gaussian generative model (Friston, 2005; Buckley et al., 2017).

Original: We extend this approach to arbitrary computation graphs in a supervised setting by deﬁning the inference problem to be solved as that of inferring the vertex value v i of each node in the graph given ﬁxed start nodes v 0 (the data), and end nodes v N (the targets).
Revised: We extend this approach to arbitrary computation graphs in a supervised setting by deﬁning the inference problem to be solved as that of inferring the vertex value v i of each node in the graph given ﬁxed start nodes v 0 (the data), and end nodes v N (the targets).

Original: We deﬁne a generative model which parametrises the value of each vertex given the feedforward prediction of its parents, p ( { v i } ) = 
Revised: We deﬁne a generative model which parametrises the value of each vertex given the feedforward prediction of its parents, p ( { v i } ) = p ( v 0 . . . v N ) =

Original: (cid:81)
Revised: (cid:81)

Original: Ni p ( v i |P
Revised: Ni p ( v i |P

Original: ( v i ))  , and a factorised, variational posterior Q ( { v i } ) = (cid:81)
Revised: ( v i )) 1 , and a factorised, variational posterior Q ( { v i }| v 0 , v N ) = Q ( v 1 . . . v N − 1 | v 0 , v N ) = (cid:81)

Revised: Ni Q ( v i

Original: Ni Q ( v i ) ,  where P ( x ) denotes the set of parents and C ( x ) denotes the set of children of a given node x. From this, we can deﬁne a suitable objective functional, the variational free-energy F (VFE), which acts as an upper bound on the divergence between the true and variational posteriors.
Revised: |P ( v i ) , C ( v i )) , where P ( v i ) denotes the set of parents and C ( v i ) denotes the set of children of a given node v i . From this, we can deﬁne a suitable objective functional, the variational free-energy F (VFE), which acts as an upper bound on the divergence between the true and variational posteriors.

Original: [Equation]
Revised: [Equation]

Revised: v N − 1 | v 0 , v N )]

Revised: ≈

Revised: N (cid:88) i = (cid:15)

Revised: Ti (cid:15) i (1)

Original: Under Gaussian assumptions for the generative model p ( { [Equation]
Revised: Under Gaussian assumptions for the generative model p ( { v i } ) = (cid:81)

Original: = [Equation] and the variational posterior Q ( { v i } ) 
Revised: Ni N ( v i ; ˆ v i , Σ i ) , and the variational posterior Q ( { v i } ) =

Original: = (cid:81)
Revised: (cid:81)

Original: Ni N ( v i ) , where the ‘predictions’ ˆ v i =
Revised: Ni N ( v i ) , where the ‘predictions’ ˆ v i =

Original: f
Revised: f

Original: ( P ( v i ); θ i ) are deﬁned asas the feedforward value of the vertex produced by running the graph forwards, and all the precisions, or inverse variances, Σ − 1 i are ﬁxed at the identity, we can write F as simply a sum of prediction errors (see Appendix D or (Friston, 2003; Bogacz, 2017; Buckley et al., 2017) for full derivations), with the prediction errors deﬁned as (cid:15) i =v
Revised: ( P ( v i ); θ i ) are deﬁned as the feedforward value of the vertex produced by running the graph forward, and all the precisions, or inverse variances, Σ − 1 i are ﬁxed at the identity, we can write F as simply a sum of prediction errors (see Appendix D or (Friston, 2003; Bogacz, 2017; Buckley et al., 2017) for full derivations), with the prediction errors deﬁned as (cid:15) i =

Original:  i − ˆ v i .
Revised: v i − ˆ v i .

Original: These prediction errors play a core role in the framework and, in the biological process theories (Friston, 2005; Bastos et al., 2012), are generally considered to be represented by a distinct population of ‘error units’.
Revised: These prediction errors play a core role in the framework and, in the biological process theories (Friston, 2005; Bastos et al., 2012), are generally considered to be represented by a distinct population of ‘error units’.

Original: Since F is an upper bound on the divergence between true and approximate posteriors, by minimizing F , we reduce this divergence, thus improving the quality of the variational posterior and approximating exact Bayesian inference.
Revised: Since F is an upper bound on the divergence between true and approximate posteriors, by minimizing F , we reduce this divergence, thus improving the quality of the variational posterior and approximating exact Bayesian inference.

Original: Predictive coding minimizes F by employing the Cauchy method of steepest descent to set the dynamics of the vertex variables v i as a gradient descent directly on F (Bogacz, 2017).
Revised: Predictive coding minimizes F by employing the Cauchy method of steepest descent to set the dynamics of the vertex variables v i as a gradient descent directly on F (Bogacz, 2017).

Original: [Equation]
Revised: [Equation]

Original: The dynamics of the parameters of the edge functions θ such that ˆ v
Revised: The dynamics of the parameters of the edge functions θ such that ˆ v

Original: i =
Revised: i =

Original: f ( P ( v i ); θ ) , can also be derived as a gradient descent on F .
Revised: f ( P ( v i ); θ ) , can also be derived as a gradient descent on F .

Original: Importantly these dynamics require only information (the current vertex value, prediction error, and prediction errors of child vertices) locally available at the vertex.
Revised: Importantly these dynamics require only information (the current vertex value, prediction error, and prediction errors of child vertices) locally available at the vertex.

Original: [Equation]
Revised: [Equation]

Original: To run generalized predictive coding in practice on a given computation graph G = { E , V } , we augment the graph with error units (cid:15) ∈ E to obtain an augumented computation graph ˜ G =
Revised: To run generalized predictive coding in practice on a given computation graph G = { E , V } , we augment the graph with error units (cid:15) ∈ E to obtain an augumented computation graph ˜ G =

Original: { E , V , E} .
Revised: { E , V , E} .

Original: The predictive coding algorithm then operates in two phases – a feedforward sweep and a backwards iteration phase.
Revised: The predictive coding algorithm then operates in two phases – a feedforward sweep and a backwards iteration phase.

Original: In the feedforward sweep, the augmented computation graph is run forwards to obtain the set of predictions { ˆ v i } , and prediction errors { (cid:15) i } = { v i − ˆ v i } for every vertex.
Revised: In the feedforward sweep, the augmented computation graph is run forward to obtain the set of predictions { ˆ v i } , and prediction errors { (cid:15) i } = { v i − ˆ v i } for every vertex.

Original: Following
Revised: Following

Original: Whittington and Bogacz (2017), to achieve exact equivalence with the backprop gradients computed on the original computation graph, we initialize v i
Revised: Whittington and Bogacz (2017), to achieve exact equivalence with the backprop gradients computed on the original computation graph, we initialize v i

Original: =
Revised: =

Original: ˆ v i in the initial feedforward sweep so that the output error computed by the predictive coding network and the original graph are identical.
Revised: ˆ v i in the initial feedforward sweep so that the output error computed by the predictive coding network and the original graph are identical.

Original: In the backwards iteration phase, the vertex activities { v i } and prediction errors { (cid:15) i } are updated with Equation 2 for all vertices in parallel until the vertex values converge to a minimum of F .
Revised: In the backwards iteration phase, the vertex activities { v i } and prediction errors { (cid:15) i } are updated with Equation 2 for all vertices in parallel until the vertex values converge to a minimum of F .

Original: After convergence the parameters are updated according to Equation 3.
Revised: After convergence the parameters are updated according to Equation 3.

Original: Note we also assume, following
Revised: Note we also assume, following

Original: Whittington and Bogacz (2017),that the predictions at each layer are ﬁxed at the values assigned during the feedforward pass throughout the optimisation of the v s. We call this the ﬁxed-prediction assumption .
Revised: Whittington and Bogacz (2017), that the predictions at each layer are ﬁxed at the values assigned during the feedforward pass throughout the optimisation of the v s. We call this the ﬁxed-prediction assumption .

Original: In effect, by removing the coupling between the vertex activities of the parents and the prediction at the child, this assumption separates the global optimisation problem into a local one for each vertex.
Revised: In effect, by removing the coupling between the vertex activities of the parents and the prediction at the child, this assumption separates the global optimisation problem into a local one for each vertex.

Original: We implement these dynamics with a simple forward Euler integration scheme so that the update rule for the vertices became v t
Revised: We implement these dynamics with a simple forward Euler integration scheme so that the update rule for the vertices became v t

Original: +1 i ← [Equation] where η is the step-size parameter.
Revised: +1 i ← v ti − η d F dv ti where η is the step-size parameter.

Original: Importantly, if the edge function linearly combines the activities and the parameters followed by an elementwise nonlinearity – a condition which we call ‘parameter-linear’ – then both the update rule for the vertices (Equation 2) and the parameters (Equation 3) become Hebbian.
Revised: Importantly, if the edge function linearly combines the activities and the parameters followed by an elementwise nonlinearity – a condition which we call ‘parameter-linear’ – then both the update rule for the vertices (Equation 2) and the parameters (Equation 3) become Hebbian.

Original: Speciﬁcally, the update rules for the vertices and weights become ∂ ˆ v j ∂v i = f (cid:48)
Revised: Speciﬁcally, the update rules for the vertices and weights become ∂ ˆ v j ∂v i = f (cid:48)

Original: ( θ i v i ) θ
Revised: ( θ i v i ) θ

Original: Ti and ∂ ˆ v j ∂θ i = f (cid:48)
Revised: Ti and ∂ ˆ v j ∂θ i = f (cid:48)

Original: ( θ i v i ) v Ti , respectively.
Revised: ( θ i v i ) v Ti , respectively.

Original: 2. A PPROXIMATION TO B ACKPROP
Revised: 2. A PPROXIMATION TO B ACKPROP

Original: Here we show that at the equilibrium of the dynamics, the prediction errors (cid:15) ∗ i converge to the correct backpropagated gradients
Revised: Here we show that at the equilibrium of the dynamics, the prediction errors (cid:15) ∗ i converge to the correct backpropagated gradients

Original: ∂L∂v i , and consequently the parameter updates (Equation 3) become precisely those of a backprop trained network.
Revised: ∂L∂v i , and consequently the parameter updates (Equation 3) become precisely those of a backprop trained network.

Original: Standard backprop works by computing the gradient of a vertex as the sum of the gradients of the child vertices.
Revised: Standard backprop works by computing the gradient of a vertex as the sum of the gradients of the child vertices.

Original: Beginning with the gradient of the output vertex ∂L∂v L , it recursively computes the gradients of vertices deeper in the graph by the chain rule: 
Revised: Beginning with the gradient of the output vertex ∂L ∂v L , it recursively computes the gradients of vertices deeper in the graph by the chain rule: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In comparison, in our predictive coding framework, at the equilibrium point ( dv i dt = 0 ) the prediction errors (cid:15) ∗ i become,
Revised: In comparison, in our predictive coding framework, at the equilibrium point ( dv i dt = 0 ) the prediction errors (cid:15) ∗ i become,

Original: [Equation]
Revised: [Equation]

Original: Importantly, this means that the equilibrium value of the prediction error at a given vertex (Equation 5) satisﬁes the same recursive structure as the chain rule of backprop (Equation 4).
Revised: Importantly, this means that the equilibrium value of the prediction error at a given vertex (Equation 5) satisﬁes the same recursive structure as the chain rule of backprop (Equation 4).

Original: Since this relationship is recursive, all that is needed for the prediction errors throughout the graph to converge to the backpropagated derivatives is for the prediction errors at the ﬁnal layer to be equal to the output gradient: (cid:15) ∗ L =∂L
Revised: Since this relationship is recursive, all that is needed for the prediction errors throughout the graph to converge to the backpropagated derivatives is for the prediction errors at the ﬁnal layer to be equal to the output gradient: (cid:15) ∗ L =

Original: ∂ ˆ v L .
Revised: ∂L∂ ˆ v L .

Original: To see this explicitly, consider a mean-squared-error loss functionat the output layer L = 12 ( T − ˆ v L ) 2with T as a vector of targets, and deﬁning (cid:15) L = T − ˆ v L .
Revised: To see this explicitly, consider a mean-squared-error loss function 2 .

Original: We then consider the equilibrium value of the prediction error unit at a penultimate vertex (cid:15) L − 1 . By Equation 5, we can see that at equilibrium, [Equation]
Revised: at the

Original: Algorithm 
Revised: Algorithm 1:

Original: Data: Dataset D =
Revised: Data: Dataset D =

Original: { X , L } , Augmented Computation Graph ˜ G =
Revised: { X , L } , Augmented Computation Graph ˜ G =

Original: { E , V , E} , inference learning rate η v , weight learning rate η θ begin /*
Revised: { E , V , E} , inference learning rate η v , weight learning rate η θ begin /*

Original: For each minibatch in the dataset */ for ( x, L ) ∈ D do /*
Revised: For each minibatch in the dataset */ for ( x, L ) ∈ D do /*

Original: Fix start of graph to inputs */ ˆ v 0 ← x /* Forward pass to compute predictions */ for ˆ v i ∈ V do ˆ v
Revised: Fix start of graph to inputs */ ˆ v 0 ← x /* Forward pass to compute predictions */ for ˆ v i ∈ V do ˆ v

Original: i ← f ( {P (ˆ v i ); θ ) /
Revised: i ← f ( {P (ˆ v i ); θ ) /

Original: * Compute output error */ (cid:15)
Revised: * Compute output error */ (cid:15)

Original: L ← L − ˆ v L / * Begin backwards iteration phase of the descent on the free energy */ while not converged do for ( v i , (cid:15)
Revised: L ← L − ˆ v L / * Begin backwards iteration phase of the descent on the free energy */ while not converged do for ( v i , (cid:15)

Original: i ) ∈
Revised: i ) ∈

Original: ˜
Revised: ˜

Original: G do /*
Revised: G do /*

Original: Compute prediction errors */ (cid:15) i ← v i − ˆ v i /
Revised: Compute prediction errors */ (cid:15) i ← v i − ˆ v i /

Original: * Update the vertex values */ v t
Revised: * Update the vertex values */ v t

Original: +1 i ← v ti
Revised: +1 i ← v ti

Original: + η v d F dv ti /* Update weights at equilibrium */ for θ i
Revised: + η v d F dv ti /* Update weights at equilibrium */ for θ i

Original: ∈
Revised: ∈

Original: E do θ t +

Revised: E do θ t +

Original: i ← 
Revised: i ← θ ti + η θ d F dθ ti output layer L = 12 ( T − ˆ v L ) 2 with T as a vector of targets, and deﬁning (cid:15)

Original: [Equation]
Revised: L = T − ˆ v L .

Original: +
Revised: We then consider the equilibrium value of the prediction error unit at a penultimate vertex (cid:15) L − 1 . By Equation 5, we can see that at equilibrium,

Original: [Equation]
Revised: [Equation]

Original: dθ ti since, ( T − ˆ v L ) =
Revised: since, ( T − ˆ v L ) =

Original: ∂L∂ ˆ v L , we can then write,
Revised: ∂L∂ ˆ v L , we can then write,

Original: [Equation]
Revised: [Equation]

Original: Thus the prediction errors of the penultimate nodes converge to the correct backpropagated gradient.
Revised: Thus the prediction errors of the penultimate nodes converge to the correct backpropagated gradient.

Original: Furthermore, recursing through the graph from children to parents allows the correct gradients to be computed 1 .
Revised: Furthermore, recursing through the graph from children to parents allows the correct gradients to be computed 3 .

Original: Thus, by induction, we have shown that the ﬁxed points of the prediction errors of the global optimization correspond exactly to the backpropagated gradients.
Revised: Thus, by induction, we have shown that the ﬁxed points of the prediction errors of the global optimization correspond exactly to the backpropagated gradients.

Original: Intuitively, if we imagine the computation-graph as a chain and the error as ’tension’ in the chain, backprop loads all the tension at the end (the output) and then systematically propagates it backwards.
Revised: Intuitively, if we imagine the computation-graph as a chain and the error as ’tension’ in the chain, backprop loads all the tension at the end (the output) and then systematically propagates it backwards.

Original: Predictive coding, however, spreads the tension throughout the entire chain until it reaches an equilibrium where the amount of tension at each link is precisely the backpropagated gradient.
Revised: Predictive coding, however, spreads the tension throughout the entire chain until it reaches an equilibrium where the amount of tension at each link is precisely the backpropagated gradient.

Revised: The full algorithm for training the predictive coding network is explicitly set out in Algorithm 1.

Revised: Inference is just a forward pass through the network, and is identical to the corresponding ANN.

Revised: the generative model can be amended to simply set the output distribution to that distribution.

Revised: If not, then there is no fully consistent generative model (although all nodes except the output remain Gaussian), but the algorithm will still work in practice.

Original:  By a similar argument, it is apparent that the dynamics of the parameters θ i as a gradient descent on
Revised: See (Figure 6) in Appendix A for results for CNNs trained with a crossentropy loss. Some subtlety is needed here since v L − 1 may have many children which each contribute to the loss. However, these different paths sum together at the node v L − 1 , thus propagating the correct gradient backwards. By a similar argument, it is apparent that the dynamics of the parameters θ i as a gradient descent on

Original: F also exactly match the backpropagated parameter gradients.
Revised: F also exactly match the backpropagated parameter gradients.

Original: [Equation]
Revised: [Equation]

Original: Which follows from the fact that (cid:15) ∗ i =
Revised: Which follows from the fact that (cid:15) ∗ i =

Original: dLd ˆ v i and that d(cid:15) ∗ i dθ =
Revised: dLd ˆ v i and that d(cid:15) ∗ i dθ =

Original: d ˆ v
Revised: d ˆ v

Original: i dθ i .
Revised: i dθ i .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: A number of recent works have tried to provide biologically plausible approximations to backprop.
Revised: A number of recent works have tried to provide biologically plausible approximations to backprop.

Original: The requirement of symmetry between the forwards and backwards weights has been questioned byθ *(⋅,⋅) ⋅ tan(⋅) +(⋅,⋅) ( ⋅ )sin(⋅) μ 1 = θ * vμ 2 = ( ̂ v 1 ) μ 3 =tan( ̂ v 2 ) μ 4 = vμ 5 =sin( ̂ v 4 ) μ L = ̂ v 3 + ̂ v̂ v̂ v̂ v̂ v̂ vϵϵϵϵ 5 ϵ T ϵ L ϵ L dμ L d ̂ vϵ L dμ L d ̂ vϵ 5 dμ 5 d ̂ vϵ 3 dμ 3 d ̂ v 2 ϵ 2 dμ
Revised: The requirement of symmetry between the forwards and backwards weights has been questioned by

Original: 2 d ̂ v 1 ϵ 1 dμ 1 dθ ϵ 1 dμ 1 dvϵ 4 dμ 4 dvv

Original: Lillicrap et al.
Revised: Lillicrap et al.

Original: (2016) who show that random ﬁxed feedback weights sufﬁce for effective learning.
Revised: (2016) who show that random ﬁxed feedback weights sufﬁce for effective learning.

Original: Recent additional work has shown that learning the backwards weights also helps (Amit, 2019;
Revised: Recent additional work has shown that learning the backwards weights also helps (Amit, 2019;

Original: Akrout et al., 2019).
Revised: Akrout et al., 2019).

Original: Several schemes have also been proposed to approximate backprop using only local learning rules and/or Hebbian connectivity.
Revised: Several schemes have also been proposed to approximate backprop using only local learning rules and/or Hebbian connectivity.

Original: These include target-prop (Lee et al., 2015) which approximate the backward gradients with trained inverse functions, but which fails to asymptotically compute the exact backprop gradients, and contrastive Hebbian (Seung, 2003; Scellier and Bengio, 2017; Scellier et al., 2018) approaches which do exactly approximate backprop, but which require two separate learning phases and the storing of information across successive phases.
Revised: These include target-prop (Lee et al., 2015) which approximate the backward gradients with trained inverse functions, but which fails to asymptotically compute the exact backprop gradients, and contrastive Hebbian (Seung, 2003; Scellier and Bengio, 2017; Scellier et al., 2018) approaches which do exactly approximate backprop, but which require two separate learning phases and the storing of information across successive phases.

Original: There are also dendritic error theories (Guerguiev et al., 2017; Sacramento et al., 2018) which are computationally similar to predictive coding (Whittington and Bogacz, 2019; Lillicrap et al., 2020).
Revised: There are also dendritic error theories (Guerguiev et al., 2017; Sacramento et al., 2018) which are computationally similar to predictive coding (Whittington and Bogacz, 2019; Lillicrap et al., 2020).

Original: Whittington and Bogacz (2017) showed that predictive coding can approximate backprop in MLP models, and demonstrated comparable performance on MNIST.
Revised: Whittington and Bogacz (2017) showed that predictive coding can approximate backprop in MLP models, and demonstrated comparable performance on MNIST.

Original: We advance upon this work by extending the proof to arbitrary computation graphs, enabling the design of predictive coding variants of a range of standard machine learning architectures, which we show perform comparably to backprop on considerably more difﬁcult tasks than MNIST.
Revised: We advance upon this work by extending the proof to arbitrary computation graphs, enabling the design of predictive coding variants of a range of standard machine learning architectures, which we show perform comparably to backprop on considerably more difﬁcult tasks than MNIST.

Original: Our algorithm evinces asymptotic (and in practice rapid) convergence to the exact backprop gradients, does not require separate learning phases, and utilises only local information and largely Hebbian plasticity.
Revised: Our algorithm evinces asymptotic (and in practice rapid) convergence to the exact backprop gradients, does not require separate learning phases, and utilises only local information and largely Hebbian plasticity.

Original:  R ESULTS 4. N UMERICAL R ESULTS
Revised:  R ESULTS 4. N UMERICAL R ESULTS

Original: To demonstrate the correctness of our derivation and empirical convergence to the true gradients, we present a numerical test in the simple scalar case, where we use predictive coding to derive the gradients of an arbitrary, highly nonlinear test function v L = tan( √ θv 0 ) + sin( v 2 0 ) where θ is an arbitrary parameter.
Revised: To demonstrate the correctness of our derivation and empirical convergence to the true gradients, we present a numerical test in the simple scalar case, where we use predictive coding to derive the gradients of an arbitrary, highly nonlinear test function v L = tan( √ θv 0 ) + sin( v 20 ) where θ is an arbitrary parameter.

Original: For our tests, we set v 0 to 

Revised: For our tests, we set v 0 to

Original: and θ to 2.
Revised: and θ to 2.

Original: The computation graph for this function is presented in Figure 2.
Revised: The computation graph for this function is presented in Figure 2.

Original: Although simple, this is a good test of predictive coding because the function is highly nonlinear, and its computation graph does not follow a simple layer structure but includes some branching.
Revised: Although simple, this is a good test of predictive coding because the function is highly nonlinear, and its computation graph does not follow a simple layer structure but includes some branching.

Original: An arbitrary target of T = 3 was set at the output and the gradient of the loss
Revised: An arbitrary target of T = 3 was set at the output and the gradient of the loss

Original: L = ( v L − T ) 2 with respect to the input v 0 was computed by predictive coding.
Revised: L = ( v L − T ) 2 with respect to the input v 0 was computed by predictive coding.

Original: We show (Figure 2) that the predictive coding optimisation rapidly converges to the exact numerical gradients computed by automatic differentiation, and that moreover this optimization is very robust and can handle even exceptionally high learning rates (up to 0.5) without divergence.
Revised: We show (Figure 2) that the predictive coding optimisation rapidly converges to the exact numerical gradients computed by automatic differentiation, and that moreover this optimization is very robust and can handle even exceptionally high learning rates (up to 0.5) without divergence.

Original: In summary, we have shown and numerically veriﬁed that at the equilibrium point of the global free-energy F on an arbitrary computation graph, the error units exactly equal the backpropagated gradients, and that this descent requires only local connectivity, does not require a separate phases or a sequential backwards sweep, and in the case of parameter-linear functions, requires only Hebbian plasticity.
Revised: In summary, we have shown and numerically veriﬁed that at the equilibrium point of the global free-energy F on an arbitrary computation graph, the error units exactly equal the backpropagated gradients, and that this descent requires only local connectivity, does not require a separate phases or a sequential backwards sweep, and in the case of parameter-linear functions, requires only Hebbian plasticity.

Original: Our results provide a straightforward recipe for the direct implementation of predictive coding algorithms to approximate certain computation graphs, such as those found in common machine learning algorithms, in a  biologically plausible manner.
Revised: Our results provide a straightforward recipe for the direct implementation of predictive coding algorithms to approximate certain computation graphs, such as those found in common machine learning algorithms, in a potentially biologically plausible manner.

Original: Next, we showcase this capability by developing predictive coding variants of core machine learning architectures - convolutional neural networks (CNNs) recurrent neural networks (RNNs) and LSTMs (Hochreiter and Schmidhuber, 1997), and show performance comparable with backprop on tasks substantially more challenging than MNIST. 
Revised: Next, we showcase this capability by developing predictive coding variants of core machine learning architecturesconvolutional neural networks (CNNs) recurrent neural networks (RNNs) and LSTMs (Hochreiter and Schmidhuber, 1997), and show performance comparable with backprop on tasks substantially more challenging than MNIST. θ *(⋅,⋅) ⋅ tan(⋅) +(⋅,⋅) ( ⋅ )sin(⋅) μ 1 = θ * vμ 2 = ( ̂ v 1 ) μ 3 =tan( ̂ v 2 ) μ 4 = vμ 5 =sin( ̂ v 4 ) μ L =

Original: Due to the need to iterate the v s until convergence, the predictive coding network had roughly a 100x greater computational cost than the backprop network
Revised: ̂ v 3 + ̂ v̂ v̂ v̂ v̂ v̂ vϵϵϵϵ 5 ϵ T ϵ L ϵ L dμ L d ̂ vϵ L dμ L d ̂ vϵ 5 dμ 5 d ̂ vϵ 3 dμ 3 d ̂ v 2 ϵ 2 dμ 2 d ̂ v 1 ϵ 1 dμ 1 dθ ϵ 1 dμ 1 dvϵ 4 dμ 4 dvv 4. P REDICTIVE

Revised: [Figure]

Revised: ODING

Original: 4. P REDICTIVE C ODING CNN, RNN, AND LSTM
Revised: [Figure] RNN [Figure] AND LSTM

Revised: Due to the need to iterate the v s until convergence, the predictive coding network had roughly a 100x greater computational cost than the backprop network.

Original: First, we constructed predictive coding CNN models (see Appendix B for full implementation details).
Revised: First, we constructed predictive coding CNN models (see Appendix B for full implementation details).

Original: In the predictive coding CNN, each ﬁlter kernel was augmented with ’error maps’ which measured the difference between the forward convolutional predictions and the backwards messages.
Revised: In the predictive coding CNN, each ﬁlter kernel was augmented with ’error maps’ which measured the difference between the forward convolutional predictions and the backwards messages.

Original: Our
Revised: Our

Original: CNN was composed of a convolutional layer, followed by a max-pooling layer, then two further convolutional layers followed by 3 fully-connected layers.
Revised: CNN was composed of a convolutional layer, followed by a max-pooling layer, then two further convolutional layers followed by 3 fully-connected layers.

Original: We compared our predictive coding CNN to a backprop-trained CNN with the exact same architecture and hyperparameters.
Revised: We compared our predictive coding CNN to a backprop-trained CNN with the exact same architecture and hyperparameters.

Original: We tested our models on three image classiﬁcation datasets signiﬁcantly more challenging than MNIST –SVHN,
Revised: We tested our models on three image classiﬁcation datasets signiﬁcantly more challenging than MNIST –

Original:  CIFAR10, and CIFAR100.
Revised: SVHN, CIFAR10, and CIFAR100.

Original: SVHN is a digit recognition task like MNIST, but has more naturalistic backgrounds, is in colour with continuously varying inputs and contains distractor digits.
Revised: SVHN is a digit recognition task like MNIST, but has more naturalistic backgrounds, is in colour with continuously varying inputs and contains distractor digits.

Original: CIFAR and CIFAR100 are large image datasets composed of RGB 32x32 images.
Revised: CIFAR10 and CIFAR100 are large image datasets composed of RGB 32x32 images.

Original: CIFAR10 has 10 classes of image, while CIFAR100 is substantially more challenging with 100 possible classes.
Revised: CIFAR10 has 10 classes of image, while CIFAR100 is substantially more challenging with 100 possible classes.

Original: In general, performance was identical between the predictive coding and backprop CNNs and comparable to the standard performance of basic CNN models on these datasets, Moreover, the predictive coding gradient remained close to the true numerical gradient throughout training.
Revised: In general (Figure 3), performance was identical between the predictive coding and backprop CNNs and comparable to the standard performance of basic CNN models on these datasets, Moreover, the predictive coding gradient remained close to the true numerical gradient throughout training.

Original: We also constructed predictive coding RNN and LSTM models, thus demonstrating the ability of predictive coding to scale to non-parameter-linear, branching, computation graphs.
Revised: We also constructed predictive coding RNN and LSTM models, thus demonstrating the ability of predictive coding to scale to non-parameter-linear, branching, computation graphs.

Original: The RNN was trained on a character-level name classiﬁcation task, while the LSTM was trained on a next-character prediction task on the full works of Shakespeare.
Revised: The RNN was trained on a character-level name classiﬁcation task, while the LSTM was trained on a next-character prediction task on the full works of Shakespeare.

Original: Full implementation details can be found in Appendices B and C. LSTMs and RNNs are recurrent networks which are trained through backpropagation through time (BPTT).
Revised: Full implementation details can be found in Appendices B and C. LSTMs and RNNs are recurrent networks which are trained through backpropagation through time (BPTT).

Original: BPTT simply unrolls the network through time and backpropagates through the unrolled graph.
Revised: BPTT simply unrolls the network through time and backpropagates through the unrolled graph.

Original: Analogously we trained the predictive coding RNN and LSTM by applying predictive coding to the unrolled computation graph.
Revised: Analogously we trained the predictive coding RNN and LSTM by applying predictive coding to the unrolled computation graph.

Original: The depth of the unrolled graph depends heavily on the sequence length, and in our tasks using a sequence length of 100 we still found that predictive coding evinced rapid convergence to the correct numerical gradient,  thus showing that the algorithm is scalable even to very deep computation graphs.
Revised: The depth of the unrolled graph depends heavily on the sequence length, and in our tasks using a sequence length of 100 we still found that predictive coding evinced rapid convergence to the correct numerical gradient, and that the performance was approximately identical to the equivalent backprop-trained networks (Figure 3), thus showing that the algorithm is scalable even to very deep computation graphs.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We have shown that predictive coding provides a local and often biologically plausible approximation to backprop on arbitrary, deep, and branching computation graphs.
Revised: We have shown that predictive coding provides a local and potentially biologically plausible approximation to backprop on arbitrary, deep, and branching computation graphs.

Original: Moreover, convergence to the exact backprop gradients is rapid and robust, even in extremely deep graphs such as the unrolled LSTM.
Revised: Moreover, convergence to the exact backprop gradients is rapid and robust, even in extremely deep graphs such as the unrolled LSTM.

Original: Our algorithm is fully parallelizable, does not require separate phases, and can produce equivalent performance to backprop in core machine-learning architectures.
Revised: Our algorithm is fully parallelizable, does not require separate phases, and can produce equivalent performance to backprop in core machine-learning architectures.

Original: These results broaden the horizon of local approximations to backprop by demonstrating that they can be implemented on arbitrary computation graphs, not only simple MLP architectures.
Revised: These results broaden the horizon of local approximations to backprop by demonstrating that they can be implemented on arbitrary computation graphs, not only simple MLP architectures.

Original: Our work prescribes a straightforward recipe for backpropagating through any computation graph with predictive coding using only local learning rules.
Revised: Our work prescribes a straightforward recipe for backpropagating through any computation graph with predictive coding using only local learning rules.

Original: In the future, this process could potentially be made fully automatic and translated onto neuromorphic hardware.
Revised: In the future, this process could potentially be made fully automatic and translated onto neuromorphic hardware.

Original: Our results also raise the possibility that the brain may implement machine-learning type architectures much more directly than often considered.
Revised: Our results also raise the possibility that the brain may implement machine-learning type architectures much more directly than often considered.

Original: Many lines of work suggest a close correspondence between the representations and activations of CNNs and activity in higher visual areas (Yamins et al., 2014;
Revised: Many lines of work suggest a close correspondence between the representations and activations of CNNs and activity in higher visual areas (Yamins et al., 2014;

Original: Tacchetti et al., 2017; Eickenberg et al., 2017; KhalighRazavi and Kriegeskorte, 2014; Lindsay, 2020), for instance, and this similarity may be found to extend to other machine learning architectures.
Revised: Tacchetti et al., 2017; Eickenberg et al., 2017; Khaligh-Razavi and Kriegeskorte, 2014; Lindsay, 2020), for instance, and this similarity may be found to extend to other machine learning architectures.

Revised: It is important to note that predictive coding, as advanced here, still retains some biologically implausible features.

Revised: Although using only local and Hebbian updates, the predictive coding algorithm still requires identical forward and backwards weights, as well as mandating a very precise oneto-one connectivity structure between value neurons v i and error neurons (cid:15) i .

Revised: However, recent work (Millidge et al., 2020) has begun to show that these implausibilities can be relaxed using learnable backwards weights instead of requiring weight symmetry, and allowing for learnable dense connectivity between value and error neurons, without harm to performance in simple MLP settings.

Revised: An additional limitation to the biological plausibility of our method is the ﬁxed-prediction assumption, which requires that the feedforward pass values be somehow stored during the backwards iteration phase.

Original:  Although we have implemented three core machine learning architectures as predictive coding networks, we have nevertheless focused on relatively small and straightforward networks and thus both our backprop and predictive coding networks perform below the state of the art on the presented tasks.
Revised: In biological neurons this could potentially be implemented by utilizing synaptic mechanisms for maintaining information over short periods, such as eligibility traces, or alternatively through synchronised phase locking (Buzsaki, 2006). Alternatively, it is important to note that this ﬁxed-prediction assumption is only required for exact convergence to backprop, and predictive coding networks have been shown to be able to attain strong discriminative classiﬁcation performance without it (Whittington and Bogacz, 2017). s Although we have implemented three core machine learning architectures as predictive coding networks, we have nevertheless focused on relatively small and straightforward networks and thus both our backprop and predictive coding networks perform below the state of the art on the presented tasks.

Original: This is primarily because our focus was on demonstrating the theoretical convergence between the two algorithms.
Revised: This is primarily because our focus was on demonstrating the theoretical convergence between the two algorithms.

Original: Nevertheless, we believe that due to the generality of our theoretical results, ’scaling up’ the existing architectures to implement performance-matched predictive coding versions of more advanced machine learning architectures such as resnets (He et al., 2016), GANs (Goodfellow et al., 2014), and transformers (Vaswani et al., 2017) should be relatively straightforward.
Revised: Nevertheless, we believe that due to the generality of our theoretical results, ’scaling up’ the existing architectures to implement performance-matched predictive coding versions of more advanced machine learning architectures such as resnets (He et al., 2016), GANs (Goodfellow et al., 2014), and transformers (Vaswani et al., 2017) should be relatively straightforward.

Original: In terms of computational cost, one inference iteration in the predictive coding network is about as costly as a backprop backwards pass.
Revised: In terms of computational cost, one inference iteration in the predictive coding network is about as costly as a backprop backwards pass.

Original: Thus, due to using 100-200 iterations for full convergence, our algorithm is substantially more expensive than backprop which limits the scalability of our method.
Revised: Thus, due to using 100-200 iterations for full convergence, our algorithm is substantially more expensive than backprop which limits the scalability of our method.

Original: However, this serial cost is misleading when talking about highly parallel neural architectures.
Revised: However, this serial cost is misleading when talking about highly parallel neural architectures.

Original: In the brain, neurons cannot wait for a sequential forward and backward sweep.
Revised: In the brain, neurons cannot wait for a sequential forward and backward sweep.

Original: By phrasing our algorithm as a global descent, our algorithm is fully parallel across layers.
Revised: By phrasing our algorithm as a global descent, our algorithm is fully parallel across layers.

Original: There is no waiting and no phases to be coordinated.
Revised: There is no waiting and no phases to be coordinated.

Original: Each neuron need only respond to its local driving inputs and downwards error signals.
Revised: Each neuron need only respond to its local driving inputs and downwards error signals.

Original: We believe that this local and parallelizable property of our algorithm may engender the possibility of substantially more efﬁcient implementations on neuromorphic hardware (Furber et al., 2014; Merolla et al., 2014; Davies et al., 2018), which may ameliorate much of the computational overhead compared to backprop.
Revised: We believe that this local and parallelizable property of our algorithm may engender the possibility of substantially more efﬁcient implementations on neuromorphic hardware (Furber et al., 2014; Merolla et al., 2014; Davies et al., 2018), which may ameliorate much of the computational overhead compared to backprop.

Original: Future work could also examine whether our method is more capable than backprop of handling the continuously varying inputs the brain is presented with in practice, rather than the artiﬁcial paradigm of being presented with a series of i.i.d.
Revised: Future work could also examine whether our method is more capable than backprop of handling the continuously varying inputs the brain is presented with in practice, rather than the artiﬁcial paradigm of being presented with a series of i.i.d.

Original: datapoints.
Revised: datapoints.

Original: Our work also reveals a close connection between backprop and inference.
Revised: Our work also reveals a close connection between backprop and inference.

Original: Namely, the recursive computation of gradients is effectively a by-product of a variational-inference algorithm which infers the values of the vertices of the computation graph under a hierarchical Gaussian generative model.
Revised: Namely, the recursive computation of gradients is effectively a by-product of a variational-inference algorithm which infers the values of the vertices of the computation graph under a hierarchical Gaussian generative model.

Original: While the deep connections between stochastic gradient descent and inference in terms of Kalman ﬁltering (Ruck et al., 1992; Ollivier, 2019) or MCMC sampling methods (Chen et al., 2014; Mandt et al., 2017) is known, the relation between recursive gradient computation itself and variational inference is underexplored except in the case of a single layer (Amari, 1995).
Revised: While the deep connections between stochastic gradient descent and inference in terms of Kalman ﬁltering (Ruck et al., 1992; Ollivier, 2019) or MCMC sampling methods (Chen et al., 2014; Mandt et al., 2017) is known, the relation between recursive gradient computation itself and variational inference is underexplored except in the case of a single layer (Amari, 1995).

Original: Our method can provide a principled generalisation of backprop through the inverse-variance Σ − 1 parameters of the Gaussian generative model.
Revised: Our method can provide a principled generalisation of backprop through the inverse-variance Σ − 1 parameters of the Gaussian generative model.

Original: These parameters weight the relative contribution of different factors to the overall gradient by their uncertainty, thus naturally handling the case of backprop with differentially noisy inputs.
Revised: These parameters weight the relative contribution of different factors to the overall gradient by their uncertainty, thus naturally handling the case of backprop with differentially noisy inputs.

Original: Moreover, the Σ − 1 parameters can be learnt as a gradient descent on
Revised: Moreover, the Σ − 1 parameters can be learnt as a gradient descent on

Original: F :
Revised: F :

Original: [Equation]
Revised: [Equation]

Original: i [Equation] − Σ − 1 i . This speciﬁc generalisation is afforded by the Gaussian form of the generative model, however, and other generative models may yield novel optimisation algorithms able to quantify and handle uncertainties throughout the entire computational graph.
Revised: This speciﬁc generalisation is afforded by the Gaussian form of the generative model, however, and other generative models may yield novel optimisation algorithms able to quantify and handle uncertainties throughout the entire computational graph.

Original: A PPENDIX A: P REDICTIVE C ODING CNN I MPLEMENTATION D ETAILS
Revised: A PPENDIX A: P REDICTIVE C ODING CNN I MPLEMENTATION D ETAILS

Original: The key concept in a CNN is that of an image convolution, where a small weight matrix is ’slid’ (or convolved) across an image to produce an output image.
Revised: The key concept in a CNN is that of an image convolution, where a small weight matrix is ’slid’ (or convolved) across an image to produce an output image.

Original: Each patch of the output image only depends on a relatively small patch of the input image.
Revised: Each patch of the output image only depends on a relatively small patch of the input image.

Original: Moreover, the weights of the ﬁlter stay the same during the convolution, so each pixel of the output image is generated using the same weights.
Revised: Moreover, the weights of the ﬁlter stay the same during the convolution so each pixel of the output image is generated using the same weights.

Original: The weight sharing implicit in the convolution operation enforces translational invariance, since different image patches are all processed with the same weights
Revised: The weight sharing implicit in the convolution operation enforces translational invariance, since different image patches are all processed with the same weights.

Original: . The forward equations of a convolutional layer for a speciﬁc output pixel
Revised: The forward equations of a convolutional layer for a speciﬁc output pixel

Original: [Equation]
Revised: [Equation]

Original: Where v i,j is the ( i, j ) th element of the output, x i,j is the element of the input image and θ k,l is an weight element of a feature map.
Revised: Where v i,j is the ( i, j ) th element of the output, x i,j is the element of the input image and θ k,l is an weight element of a feature map.

Original: To setup a predictive coding CNN, we augment each intermediate x
Revised: To setup a predictive coding CNN, we augment each intermediate x

Original: i
Revised: i

Original: and v i with error units (cid:15) i of the same dimension as the output of the convolutional layer.
Revised: and v i with error units (cid:15) i of the same dimension as the output of the convolutional layer.

Original: Predictions ˆ v are projected forward using the forward equations.
Revised: Predictions ˆ v are projected forward using the forward equations.

Original: Prediction errors also need to be transmitted backwards for the architecture to work.
Revised: Prediction errors also need to be transmitted backwards for the architecture to work.

Original: To achieve this we must have that prediction errors are transmitted upwards by a ’backwards convolution’.
Revised: To achieve this we must have that prediction errors are transmitted upwards by a ’backwards convolution’.

Original: We thus deﬁne the backwards prediction errors ˆ (cid:15) j as follows:
Revised: We thus deﬁne the backwards prediction errors ˆ (cid:15) j as follows:

Original: [Equation]
Revised: [Equation]

Original: Where ˜ (cid:15) is an error map zero-padded to ensure the correct convolutional output size.
Revised: Where ˜ (cid:15) is an error map zero-padded to ensure the correct convolutional output size.

Original: Inference in the predictive coding network then proceeds by updating the intermediate values of each layer as follows:
Revised: Inference in the predictive coding network then proceeds by updating the intermediate values of each layer as follows:

Original: [Equation]
Revised: [Equation]

Original: Since the CNN is also parameter-linear, weights can be updated using the simple Hebbian rule of the multiplication of the pre and post synaptic potentials.
Revised: Since the CNN is also parameter-linear, weights can be updated using the simple Hebbian rule of the multiplication of the pre and post synaptic potentials.

Original: [Equation]
Revised: [Equation]

Original: There is an additional biological implausibility here due to the weight sharing of the CNN.
Revised: There is an additional biological implausibility here due to the weight sharing of the CNN.

Original: Since the same weights are copied for each position on the image, the weight updates have contributions from all aspects of the image simultaneously which violates the locality condition.
Revised: Since the same weights are copied for each position on the image, the weight updates have contributions from all aspects of the image simultaneously which violates the locality condition.

Original: A simple ﬁx for this, which makes the network scheme plausible is to simply give each position on the image a ﬁlter with separate weights, thus removing the weight sharing implicit in the CNN.
Revised: A simple ﬁx for this, which makes the network scheme plausible is to simply give each position on the image a ﬁlter with separate weights, thus removing the weight sharing implicit in the CNN.

Original: In effect this gives each patch of pixels a local receptive ﬁeld with its own set of weights.
Revised: In effect this gives each patch of pixels a local receptive ﬁeld with its own set of weights.

Original: The performance and scalability of such a locally connected predictive coding architecture would be an interesting avenue for future work, as this architecture has substantial homologies with the structure of the visual cortex.
Revised: The performance and scalability of such a locally connected predictive coding architecture would be an interesting avenue for future work, as this architecture has substantial homologies with the structure of the visual cortex.

Original: In our experiments we used a relatively simple CNN architecture consisting of one convolutional layer of kernel size 5, and a ﬁlter bank of 6 ﬁlters.
Revised: In our experiments we used a relatively simple CNN architecture consisting of one convolutional layer of kernel size 5, and a ﬁlter bank of 6 ﬁlters.

Original: This was followed by a max-pooling layer with a (2,2) kernel and a further convolutional layer with a (5,5) kernel and ﬁlter bank of 16 ﬁlters.
Revised: This was followed by a max-pooling layer with a (2,2) kernel and a further convolutional layer with a (5,5) kernel and ﬁlter bank of 16 ﬁlters.

Original: This was then followed by three fully connected layers of 200, 150, and 10 (or 100 for CIFAR100) output units.
Revised: This was then followed by three fully connected layers of 200, 150, and 10 (or 100 for CIFAR100) output units.

Revised: Each convolutional and fully connected layer used the relu activation function, except the output layer which was linear.

Original: Although this architecture is far smaller than state of the art for convolutional networks, the primary point of our paper was to demonstrate the equivalence of predictive coding and backprop.
Revised: Although this architecture is far smaller than state of the art for convolutional networks, the primary point of our paper was to demonstrate the equivalence of predictive coding and backprop.

Original: Further work could investigate scaling up predictive coding to more state-of-the-art architectures.
Revised: Further work could investigate scaling up predictive coding to more state-of-the-art architectures.

Original: Our datasets consisted of 32x32 RGB images.
Revised: Our datasets consisted of 32x32 RGB images.

Original: We normalised the values of all pixels of each image to lie between 0 and 1, but otherwise performed no other image preprocessing.
Revised: We normalised the values of all pixels of each image to lie between 0 and 1, but otherwise performed no other image preprocessing.

Original: We did not use data augmentation of any kind.
Revised: We did not use data augmentation of any kind.

Original: We set the weight learning rate for the predictive coding and backprop networks 0.0001.
Revised: We set the weight learning rate for the predictive coding and backprop networks 0.0001.

Original: A minibatch size of 64 was used.
Revised: A minibatch size of 64 was used.

Original: These parameters were chosen without any detailed hyperparameter search and so are likely suboptimal.
Revised: These parameters were chosen without any detailed hyperparameter search and so are likely suboptimal.

Original: The magnitude of the gradient updates was clamped to lie between -50 and 50 in all of our models.
Revised: The magnitude of the gradient updates was clamped to lie between -50 and 50 in all of our models.

Original: This was done to prevent divergences, as occasionally occurred in the LSTM networks, likely due to exploding gradients.
Revised: This was done to prevent divergences, as occasionally occurred in the LSTM networks, likely due to exploding gradients.

Original: The predictive coding scheme converged to the exact backprop gradients very precisely withininference iterations using an inference learning rate of 0.1.
Revised: The predictive coding scheme converged to the exact backprop gradients very precisely withininference iterations using an inference learning rate of 0.1.

Original: This gives the predictive coding CNN approximately a 100x computational overhead compared to backprop.
Revised: This gives the predictive coding CNN approximately a 100x computational overhead compared to backprop.

Original: The divergence between the true and approximate gradients remained approximately constant throughout training, as shown by Figure 5, which shows the mean divergence for each layer of the CNN over the course of an example training run on the CIFAR10 dataset.
Revised: The divergence between the true and approximate gradients remained approximately constant throughout training, as shown by Figure 5, which shows the mean divergence for each layer of the CNN over the course of an example training run on the CIFAR10 dataset.

Original: The training loss of the predictive coding and backprop networks for SVHN, CIFAR10 and CIFAR100 are presented in Figure 4.
Revised: The training loss of the predictive coding and backprop networks for SVHN, CIFAR10 and CIFAR100 are presented in Figure 4.

Revised: While the experiments in the main paper all used the mean-squared-error loss function, it is also possible to use alternative loss functions.

Revised: Here (Figure 6) we show performance of the CNN on

Revised: Cifar, Cifar100 and SVHN datasets is also identical to backprop when trained with a multi-class cross-entropy loss L = (cid:80) i T i ln v Li .

Revised: In this case the output layer used a softmax function as its nonlinearity, to ensure that the logits passed to the cross-entropy loss were valid probabilities.

Original:  A PPENDIX B: P REDICTIVE C ODING RNN
Revised: The cross-entropy loss is also straightforward to ﬁt into the predictive coding framework since the gradient with respect to the pre-activations of the output is also just the negative prediction error ∂L∂v L = T − v L , although the softmax function itself may be challenging to implement neurally since it is non-local as its’ normalisation coefﬁcient requires of the exponentiated activities of all neurons in a layer. Nevertheless, this demonstrates that predictive coding can approximate backprop for any given loss function, not simply mean-square-error. A PPENDIX B: P REDICTIVE C ODING RNN

Original: The computation graph on RNNs is relatively straightforward.
Revised: The computation graph on RNNs is relatively straightforward.

Original: We consider only a single layer RNN here although the architecture can be straightforwardly extended to hierarchically stacked RNNs.
Revised: We consider only a single layer RNN here although the architecture can be straightforwardly extended to hierarchically stacked RNNs.

Original: An
Revised: An

Original: RNN is similar to a feedforward network except that it possesses an additional hidden state h which is maintained and updated over time as a function of both the current input x and the previous hidden  state.
Revised: RNN is similar to a feedforward network except that it possesses an additional hidden state h which is maintained and updated over time as a function of both the current input x and the previous hidden [Figure] state.

Original: The output of the network y is a function of h .
Revised: The output of the network y is a function of h .

Original: By considering the RNN at a single timestep we obtain the following equations.
Revised: By considering the RNN at a single timestep we obtain the following equations.

Original: [Equation]
Revised: [Equation]

Original: Where f and g are elementwise nonlinear activation functions.
Revised: Where f and g are elementwise nonlinear activation functions.

Original: And θ h , θ x , θ y are weight matrices for each speciﬁc input.
Revised: And θ h , θ x , θ y are weight matrices for each speciﬁc input.

Original: To predict a sequence the RNN simply rolls forward the above equations to generate new predictions and hidden states at each timestep.
Revised: To predict a sequence the RNN simply rolls forward the above equations to generate new predictions and hidden states at each timestep.

Original: [Figure]

Original: RNNs are typically trained through an algorithm called backpropagation through time (BPTT) which essentially just unrolls the RNN into a single feedforward computation graph and then performs backpropagation through this unrolled graph.
Revised: RNNs are typically trained through an algorithm called backpropagation through time (BPTT) which essentially just unrolls the RNN into a single feedforward computation graph and then performs backpropagation through this unrolled graph.

Original: To train the RNN using predictive coding we take the same approach and simply apply predictive coding to the unrolled graph.
Revised: To train the RNN using predictive coding we take the same approach and simply apply predictive coding to the unrolled graph.

Original: It is important to note that this is an additional aspect of biological implausibility that we do not address in this paper.
Revised: It is important to note that this is an additional aspect of biological implausibility that we do not address in this paper.

Original: BPTT requires updates to proceed backwards through time from the end of the sequence to the beginning.
Revised: BPTT requires updates to proceed backwards through time from the end of the sequence to the beginning.

Original: Ignoring any biological implausibility with the rules themselves, this updating sequence is clearly not biologically plausible as naively it requires maintaining the entire sequence of predictions and prediction errors perfectly in memory until the end of the sequence, and waiting until the sequence ends before making any updates.
Revised: Ignoring any biological implausibility with the rules themselves, this updating sequence is clearly not biologically plausible as naively it requires maintaining the entire sequence of predictions and prediction errors perfectly in memory until the end of the sequence, and waiting until the sequence ends before making any updates.

Original: There is a small literature on trying to produce biologically plausible, or forward-looking approximations to BPTT which does not require updates to be propagated back through time (Williams and Zipser, 1989; Lillicrap and Santoro, 2019;
Revised: There is a small literature on trying to produce biologically plausible, or forward-looking approximations to BPTT which does not require updates to be propagated back through time (Williams and Zipser, 1989; Lillicrap and Santoro, 2019;

Original: Steil, 2004; Ollivier et al., 2015; Tallec and Ollivier, 2017).
Revised: Steil, 2004; Ollivier et al., 2015; Tallec and Ollivier, 2017).

Original: While this is a fascinating area, we do not address it in this paper.
Revised: While this is a fascinating area, we do not address it in this paper.

Original: We are solely concerned with the fact that predictive coding approximates backpropagation on feedforward computation graphs for which the unrolled RNN graph is a sufﬁcient substrate.
Revised: We are solely concerned with the fact that predictive coding approximates backpropagation on feedforward computation graphs for which the unrolled RNN graph is a sufﬁcient substrate.

Original: To learn a predictive coding RNN, we ﬁrst augment each of the variables h t and y t of the original graph with additional error units (cid:15) h t and (cid:15) y t .
Revised: To learn a predictive coding RNN, we ﬁrst augment each of the variables h t and y t of the original graph with additional error units (cid:15) h t and (cid:15) y t .

Original: Predictions ˆ y t , ˆ h t are generated according to the feedforward rules (16).
Revised: Predictions ˆ y t , ˆ h t are generated according to the feedforward rules (16).

Original: A sequence of true labels { T 1 ...T T } is then presented to the network, and then inference proceeds by recursively applying the following rules backwards through time until convergence.
Revised: A sequence of true labels { T 1 ...T T } is then presented to the network, and then inference proceeds by recursively applying the following rules backwards through time until convergence.

Original: [Equation]
Revised: [Equation]

Original: Upon convergence the weights are updated according to the following rules.
Revised: Upon convergence the weights are updated according to the following rules.

Original: [Equation]
Revised: [Equation]

Original: Since the RNN feedforward updates are parameter-linear, these rules are Hebbian, only requiring the multiplication of pre and post-synaptic potentials.
Revised: Since the RNN feedforward updates are parameter-linear, these rules are Hebbian, only requiring the multiplication of pre and post-synaptic potentials.

Original: This means that the predictive coding updates proposed here are biologically plausible and could in theory be implemented in the brain.
Revised: This means that the predictive coding updates proposed here are biologically plausible and could in theory be implemented in the brain.

Original: The only biological implausibility remains the BPTT learning scheme.
Revised: The only biological implausibility remains the BPTT learning scheme.

Original: Our RNN was trained on a simple character-level name-origin dataset which can be found here: https://download.pytorch.org/tutorial/data.zip .
Revised: Our RNN was trained on a simple character-level name-origin dataset which can be found here: https://download.pytorch.org/tutorial/data.zip .

Original: The RNN was presented with sequences of characters representing names and had to predict the national origin of the name – French, Spanish, Russian, etc.
Revised: The RNN was presented with sequences of characters representing names and had to predict the national origin of the name – French, Spanish, Russian, etc.

Original: The characters were presented to the network as one-hot-encoded vectors without any embedding.
Revised: The characters were presented to the network as one-hot-encoded vectors without any embedding.

Original: The output categories were also presented as a one-hot vector.
Revised: The output categories were also presented as a one-hot vector.

Original: The RNN has a hidden size ofunits.
Revised: The RNN has a hidden size ofunits.

Original: A tanh nonlinearity was used between hidden states and the output layer was linear.
Revised: A tanh nonlinearity was used between hidden states and the output layer was linear.

Original: The network was trained on randomly selected name-category pairs from the dataset.
Revised: The network was trained on randomly selected name-category pairs from the dataset.

Original: The training loss for the predictive coding and backprop RNNs, averaged over 5 seeds is presented below (Figure 6).
Revised: The training loss for the predictive coding and backprop RNNs, averaged over 5 seeds is presented below (Figure 7).

Original: A PPENDIX C: P REDICTIVE C ODING LSTM I MPLEMENTATION D ETAILS
Revised: A PPENDIX C: P REDICTIVE C ODING LSTM I MPLEMENTATION D ETAILS

Original: Unlike the other two models, the LSTM possesses a complex and branching internal computation graph, and is thus a good opportunity to make explicit the predictive coding ’recipe’ for approximating [Figure] x [Figure]
Revised: Unlike the other two models, the LSTM possesses a complex and branching internal computation graph, and is thus a good opportunity to make explicit the predictive coding ’recipe’ for approximating backprop on arbitrary computation graphs.

Original: h [Figure] σ × c t θ inp θ c σ
Revised: The computation graph for a single LSTM cell is shown (with backprop updates) in Figure 8.

Original: [Figure]
Revised: Prediction for the LSTM occurs by simply rolling forward a copy of the LSTM cell for each timestep.

Original: × + σ θ o × tanh c t [Figure] h t 
Revised: The LSTM cell receives its hidden state h t and cell state c t from the previous timestep.

Original: [Figure]
Revised: During training we compute derivatives on the unrolled computation graph and receive backwards derivatives (or prediction errors) from the LSTM cell at time t + 1 .

Original: σ θ y y t − T v [Figure] x [Figure]
Revised: ⊕ θ f x t h t σ × c t θ inp θ c σ tanh × + σ θ o × tanh

Original: [Figure]
Revised: [Figure]

Original: backprop on arbitrary computation graphs. The computation graph for a single LSTM cell is shown (with backprop updates) in Figure 7. Prediction for the LSTM occurs by simply rolling forward a copy of the LSTM cell for each timestep. The LSTM cell receives its hidden state ht and cell state c t from the previous timestep.
Revised: h

Original: During training we compute derivatives on the unrolled computation graph and receive backwards derivatives (or prediction errors) from the LSTM cell at time t + 1 .
Revised: [Figure] σ θ y y t − T v 1 = h t ⊕ x t v

Original:  The equations that specify the computation graph of the LSTM cell are as follows.
Revised: [Figure] [Figure] The equations that specify the computation graph of the LSTM cell are as follows.

Original: [Equation]
Revised: [Equation]

Original: The recipe to convert this computation graph into a predictive coding algorithm is straightforward.
Revised: The recipe to convert this computation graph into a predictive coding algorithm is straightforward.

Original: We ﬁrst rewire the connectivity so that the predictions are set to the forward functions of their parents.
Revised: We ﬁrst rewire the connectivity so that the predictions are set to the forward functions of their parents.

Original: We then compute the errors between the vertices and the predictions.
Revised: ⊕ θ f x t h t σ × c t θ inp θ c σ tanh × + σ θ [Figure] × [Figure]

Revised: h

Original: [Equation]
Revised: [Figure]

Original: (cid:15)
Revised: σ θ y v 1 = h t ⊕ x t v 2 = σ ( θ f v 1 )

Original: [Equation]
Revised: [Figure]

Original: (cid:15)
Revised: We then compute the errors between the vertices and the predictions.

Original: [Equation]
Revised: [Equation]

Original:  − {+ˆ v(cid:15) 
+}
Revised: 10 ) (cid:15) 1 = v 1 − ˆ v(cid:15) 2 = v 2 − {+ˆ v(cid:15) 
+}

Original: [Equation] (cid:15) 4 = v [Equation]
Revised: = v 3 − ˆ v(cid:15) 4 = v 4 − ˆ v(cid:15)

Original: − ˆ v(cid:15) 5 = v 5 − ˆ v(cid:15) 6 = v 6 − ˆ v(cid:15) 7 = v 7 − ˆ v(cid:15) 8 = v 8 − ˆ v(cid:15) 9 = v 9 − ˆ v(cid:15) 10 = v 10 − ˆ v
Revised: 5 = v 5 − ˆ v(cid:15) 6 = v 6 − ˆ v(cid:15) 7 = v 7 − ˆ v(cid:15) 8 = v 8 − ˆ v(cid:15) 9 = v 9 − ˆ v(cid:15) 10 = v 10 − ˆ v

Original: During inference, the inputs h t , x t and the output y t are ﬁxed.
Revised: During inference, the inputs h t , x t and the output y t are ﬁxed.

Original: The vertices and then the prediction errors are updated according to Equation 2.
Revised: The vertices and then the prediction errors are updated according to Equation 2.

Original: This recipe is straightforward and can easily be extended to other more complex machine learning architectures.
Revised: This recipe is straightforward and can easily be extended to other more complex machine learning architectures.

Original: The full augmented computation graph, including the vertex update rules, is presented in Figure 8.
Revised: The full augmented computation graph, including the vertex update rules, is presented in Figure 9.

Original: Empirically, we observed rapid convergence to the exact backprop gradients even in the case of very deep computation graphs (as is an unrolled LSTM with a sequence length of 100).
Revised: Empirically, we observed rapid convergence to the exact backprop gradients even in the case of very deep computation graphs (as is an unrolled LSTM with a sequence length of 100).

Original: Although convergence was slower than was the case for CNNs or lesser sequence lengths, it was still straightforward to achieve convergence to the exact numerical gradients with sufﬁcient iterations.
Revised: Although convergence was slower than was the case for CNNs or lesser sequence lengths, it was still straightforward to achieve convergence to the exact numerical gradients with sufﬁcient iterations.

Original: Below we plot the mean divergence between the predictive coding and true numerical gradients as a function of sequence length (and hence depth of graph) for a ﬁxed computational budget ofiterations with an inference learning rate of 0.05.
Revised: Below we plot the mean divergence between the predictive coding and true numerical gradients as a function of sequence length (and hence depth of graph) for a ﬁxed computational budget ofiterations with an inference learning rate of 0.05.

Original: As can be seen, the divergence increases roughly 
Revised: As can be seen, the divergence increases roughly linearly with sequence length.

Original: ⊕ θ f x t h t σ × c t θ inp θ c σ tanh × + σ θ o × tanh c [Figure] h
Revised: Importantly, even with long sequences, the divergence is not especially large, and can be decreased further by increasing the computational budget.

Original: [Figure] σ θ y v 1 = h t ⊕ x t v 2 =
Revised: As the increase is linear, we believe that predictive coding approaches should be scalable even for backpropagating through very deep and complex graphs.

Original: σ ( θ f v 1 ) [Figure] [Figure] linearly with sequence length.
Revised: We also plot the number of iterations required to reach a given convergence threshold (here taken to be 0.005) as a function of sequence length (Figure 11).

Original: Importantly, even withlong sequences, the divergence is not especially large, and can be decreased further by increasing the computational budget. As the increase is linear, we believe that predictive coding approaches should be scalable even for backpropagating through very deepand complex graphs.
Revised: We see that the number of iterations required increases sublinearly with the sequence length, and likely asymptotes at about 300 iterations. Although this is a lot of iterations, the sublinear convergence nevertheless shows that the method can scale to even extremely deep graphs.

Original: Our architecture consisted of a single LSTM layer (more complex architectures would consist of multiple stacked LSTM layers).
Revised: Our architecture consisted of a single LSTM layer (more complex architectures would consist of multiple stacked LSTM layers).

Original: The LSTM was trained on a next-character character-level prediction task.
Revised: The LSTM was trained on a next-character character-level prediction task.

Original: The dataset was the full works of Shakespeare, downloadable from Tensorﬂow.
Revised: The dataset was the full works of Shakespeare, downloadable from Tensorﬂow.

Original: The text was shufﬂed and split into sequences of 50 characters, which were fed to the LSTM one character at a time.
Revised: The text was shufﬂed and split into sequences of 50 characters, which were fed to the LSTM one character at a time.

Original: The LSTM was trained then to predict the next character, so as to ultimately be able to generate text.
Revised: The LSTM was trained then to predict the next character, so as to ultimately be able to generate text.

Original: The characters were presented as one-hot-encoded vectors.
Revised: The characters were presented as one-hot-encoded vectors.

Original: The LSTM had a hidden size and a cell-size of 1056 units.
Revised: The LSTM had a hidden size and a cell-size of 1056 units.

Original: A minibatch size of 64 was used and a weight learning rate of 0.0001 was used for both predictive coding and backprop networks.
Revised: A minibatch size of 64 was used and a weight learning rate of 0.0001 was used for both predictive coding and backprop networks.

Original: To achieve sufﬁcient numerical convergence to the correct gradient, we used 200 variational iterations with an inference learning rate of 0.1.
Revised: To achieve sufﬁcient numerical convergence to the correct gradient, we used 200 variational iterations with an inference learning rate of 0.1.

Original: This rendered the predictive LSTM approximately 200x as costly as the backprop LSTM to run.
Revised: This rendered the predictive LSTM approximately 200x as costly as the backprop LSTM to run.

Original: A graph of the LSTM training loss for both predictive coding and backprop LSTMs, averaged over 5 random seeds, can be found below (Figure 10).
Revised: A graph of the LSTM training loss for both predictive coding and backprop LSTMs, averaged over 5 random seeds, can be found below (Figure 12).

Original: A PPENDIX D: D ERIVATION OF THE F REE E NERGY F UNCTIONAL
Revised: A PPENDIX D: D ERIVATION OF THE F REE E NERGY F UNCTIONAL

Original: Here we derive in detail the form of the free-energy functional used in sections 2 and 4.
Revised: Here we derive in detail the form of the free-energy functional used in sections 2 and 4.

Original: We also expand upon the assumptions required and the precise form of the generative model and variational density.
Revised: We also expand upon the assumptions required and the precise form of the generative model and variational density.

Original: Much of this material is presented with considerably more detail in Buckley et al.
Revised: Much of this material is presented with considerably more detail in Buckley et al.

Original: (2017), and more approachably in Bogacz (2017).
Revised: (2017), and more approachably in Bogacz (2017).

Original: Given an arbitrary computation graph with vertices { y i } , which we treat as random variables.
Revised: Given an arbitrary computation graph with vertices { y i } , which we treat as random variables.

Original: Here we treat explicitly an important fact that we glossed over for notational convenience in the introduction.
Revised: Here we treat explicitly an important fact that we glossed over for notational convenience in the introduction.

Original: The v i s which are optimized in the free-energy functional are technically the mean parameters of the variational density Q ( y i ; v i , σ i ) – i.e. they represent the mean (variational) belief of the value of the vertex.
Revised: The v i s which are optimized in the free-energy functional are technically the mean parameters of the variational density Q ( y i ; v i , σ i ) – i.e. they represent the mean (variational) belief of the value of the vertex.

Original: The vertex values in the model, which we here denote as { y i } , are technically separate.
Revised: The vertex values in the model, which we here denote as { y i } , are technically separate.

Original: However, due to our Gaussian assumptions, and the expectation under the variational density, in effect we end up replacing the y i with the v i and optimizing the v
Revised: However, due to our Gaussian assumptions, and the expectation under the variational density, in effect we end up replacing the y i with the v i and optimizing the v

Original: i s, so in the interests of space and notational simplicity we began as if the v i s were variables in the generative model, but they are not.
Revised: i s, so in the interests of space and notational simplicity we began as if the v i s were variables in the generative model, but they are not.

Original: They are parameters of the variational distribution.
Revised: They are parameters of the variational distribution.

Original: Given an input y 0 and a target y N (the multiple input and/or output case is a straightforward generalization).
Revised: Given an input y 0 and a target y N (the multiple input and/or output case is a straightforward generalization).

Original: We wish to infer the posterior p ( y 1: N − 1 | y 0 , y N ) .
Revised: We wish to infer the posterior p ( y 1: N − 1 | y 0 , y N ) .

Original: We approximate this intractable posterior with variational inference.
Revised: We approximate this intractable posterior with variational inference.

Original: Variational inference proceeds by deﬁning an approximate posterior
Revised: Variational inference proceeds by deﬁning an approximate posterior

Original: Q ( y 1: N − 1 ; φ ) with some arbitrary parameters φ .
Revised: Q ( y 1: N − 1 ; φ ) with some arbitrary parameters φ .

Original: We then wish to minimize the KL divergence between the true and approximate posterior.
Revised: We then wish to minimize the KL divergence between the true and approximate posterior.

Original: [Equation]
Revised: [Equation]

Original: Although this KL is itself intractable, since it includes the intractable posterior, we can derive a tractable bound on this KL called the variational free-energy.
Revised: Although this KL is itself intractable, since it includes the intractable posterior, we can derive a tractable bound on this KL called the variational free-energy.

Original: [Equation]
Revised: [Equation]

Original: We deﬁne the negative free-energy −F = KL [ Q ( y 1: N − 1) (cid:107) p ( y 1: N − 1 , y 0 , y N )] which is a lower bound on the divergence between the true and approximate posteriors.
Revised: We deﬁne the negative free-energy −F = KL [ Q ( y 1: N − 1) (cid:107) p ( y 1: N − 1 , y 0 , y N )] which is a lower bound on the divergence between the true and approximate posteriors.

Original: By thus maximizing the negative free-energy (which is identical to the ELBO (Beal et al., 2003; Blei et al., 2017)), or equivalently minimizing the free-energy, we decrease this divergence and make the variational distribution a better approximation to the true posterior.
Revised: By thus maximizing the negative free-energy (which is identical to the ELBO (Beal et al., 2003; Blei et al., 2017)), or equivalently minimizing the free-energy, we decrease this divergence and make the variational distribution a better approximation to the true posterior.

Original: To proceed further, it is necessary to deﬁne an explicit form of the generative model p ( y 0 , y 1: N − 1 , y N ) and the approximate posterior Q ( y 1: N − 1 ; φ ) .
Revised: To proceed further, it is necessary to deﬁne an explicit form of the generative model p ( y 0 , y 1: N − 1 , y N ) and the approximate posterior Q ( y 1: N − 1 ; φ ) .

Original: In predictive coding, we deﬁne a hierarchical Gaussian generative model which mirrors the exact structure of the computation graph
Revised: In predictive coding, we deﬁne a hierarchical Gaussian generative model which mirrors the exact structure of the computation graph

Original: [Equation]
Revised: [Equation]

Original: Where essentially each vertex
Revised: Where essentially each vertex

Original: y i is a Gaussian with a mean which is a function of the prediction of all the parents of the vertex, and the parameters of their edge-functions.
Revised: y i is a Gaussian with a mean which is a function of the prediction of all the parents of the vertex, and the parameters of their edge-functions.

Original: ¯ y 0 is effectively an ”input-prior” which is set to 0 throughout and ignored.
Revised: ¯ y 0 is effectively an ”input-prior” which is set to 0 throughout and ignored.

Original: The output vertices y N =
Revised: The output vertices y N =

Original: T are set to the target T .
Revised: T are set to the target T .

Original: We also deﬁne the variational density to be Gaussian with mean v 1: N − 1 and variance σ 1: N − 1 , but under a mean ﬁeld approximation, so that the approximation at each node is independent of all others (note the variational variance is denoted σ while the variance of the generative model is denoted Σ .
Revised: We also deﬁne the variational density to be Gaussian with mean v 1: N − 1 and variance σ 1: N − 1 , but under a mean ﬁeld approximation, so that the approximation at each node is independent of all others (note the variational variance is denoted σ while the variance of the generative model is denoted Σ .

Original: The lower-case σ is not used to denote a scalar variable – both variances can be multivariate – but to distinguish between variational and generative variances)
Revised: The lower-case σ is not used to denote a scalar variable – both variances can be multivariate – but to distinguish between variational and generative variances)

Original: [Equation]
Revised: [Equation]

Original: We now can express the free-energy functional concretely.
Revised: We now can express the free-energy functional concretely.

Original: First we decompose it as the sum of an energy and an entropy
Revised: First we decompose it as the sum of an energy and an entropy

Original: [Equation]
Revised: [Equation]

Original: Then, taking the entropy term ﬁrst, we can express it concretely in terms of normal distributions
Revised: Then, taking the entropy term ﬁrst, we can express it concretely in terms of normal distributions.

Original: [Equation]
Revised: [Equation]

Original: The entropy of a multivariate gaussian has a simple analytical form depending only on the variance.
Revised: The entropy of a multivariate gaussian has a simple analytical form depending only on the variance.

Original: Next we turn to the energy term, which is more complex.
Revised: Next we turn to the energy term, which is more complex.

Original: To derive a clean analytical result, we must make a further assumption, the Laplace approximation, which requires the variational density to be tightly peaked around the mean so the only non-negligible contribution to the expectation is from regions around the mean.
Revised: To derive a clean analytical result, we must make a further assumption, the Laplace approximation, which requires the variational density to be tightly peaked around the mean so the only non-negligible contribution to the expectation is from regions around the mean.

Original: This means that we can successfully approximate the approximate posterior with a second-order Taylor expansion around the mean.
Revised: This means that we can successfully approximate the approximate posterior with a second-order Taylor expansion around the mean.

Original: From the ﬁrst line onwards we ignore the ln p ( y 0 ) and ln p ( y N |P
Revised: From the ﬁrst line onwards we ignore the ln p ( y 0 ) and ln p ( y N |P

Original: ( y N )) which lie outside the expectation.
Revised: ( y N )) which lie outside the expectation.

Original: [Equation]
Revised: [Equation]

Original: Where the second term in the Taylor expansion evaluates to 0 since E Q [ y i − v i ]
Revised: Where the second term in the Taylor expansion evaluates to 0 since E Q [ y i − v i ]

Original: = ( v i − v i )
Revised: = ( v i − v i )

Original: = 

Revised: =

Original: and the third term contains the expression for the variance E Q [( y i − v i ) 2 ]
Revised: and the third term contains the expression for the variance E Q [( y i − v i ) 2 ]

Original: = σ i .
Revised: = σ i .

Original: We can then write out the full Laplace-encoded free-energy as:
Revised: We can then write out the full Laplace-encoded free-energy as:

Original: [Equation]
Revised: [Equation]

Original: We wish to minimize F with respect to the variational parameters v i and σ i .
Revised: We wish to minimize F with respect to the variational parameters v i and σ i .

Original: There is in fact a closedform expression for the optimal variational variance which can be obtained simply by differentiating and setting the derivative to 0.
Revised: There is in fact a closedform expression for the optimal variational variance which can be obtained simply by differentiating and setting the derivative to 0.

Original: [Equation]
Revised: [Equation]

Original: Because of this analytical result for the variational variance, we do not need to consider it further in the optimisation problem, and only consider minimizing the variational means v i .
Revised: Because of this analytical result for the variational variance, we do not need to consider it further in the optimisation problem, and only consider minimizing the variational means v i .

Original: This renders all the terms in the free-energy except the ln p ( v i |P
Revised: This renders all the terms in the free-energy except the ln p ( v i |P

Original: ( y i ))
Revised: ( y i ))

Original: terms constant with respect to the variational parameters.
Revised: terms constant with respect to the variational parameters.

Original: This allows us to write:
Revised: This allows us to write:

Original: [Equation]
Revised: [Equation]

Original: as presented in section 2.
Revised: as presented in section 2.

Original: The ﬁrst term ln p ( y N |P
Revised: The ﬁrst term ln p ( y N |P

Original: ( y N )) is effectively the loss at the output ( y N = T ) so becomes an additional prediction error ln p ( y N |P ( y N ))
Revised: ( y N )) is effectively the loss at the output ( y N = T ) so becomes an additional prediction error ln p ( y N |P ( y N ))

Original: ∝ ( T − ˆ v N ) T Σ − 1 N ( T − ˆ v N ) which can be absorbed into the sum over other prediction errors.
Revised: ∝ ( T − ˆ v N ) T Σ − 1 N ( T − ˆ v N ) which can be absorbed into the sum over other prediction errors.

Original: Crucially, although the variational variances have an analytical form, the variances of the generative model (the precisions Σ i ) do not and can be optimised directly to improve the log model-evidence.
Revised: Crucially, although the variational variances have an analytical form, the variances of the generative model (the precisions Σ i ) do not and can be optimised directly to improve the log model-evidence.

Original: These precisions allow for a kind of ’uncertainty-aware’ backprop.
Revised: These precisions allow for a kind of ’uncertainty-aware’ backprop.

Original: D ERIVATION OF V ARIATIONAL U PDATE R ULES AND F IXED POINTS
Revised: D ERIVATION OF V ARIATIONAL U PDATE R ULES AND F IXED POINTS

Original: Here, starting from Equation 10, we show how to obtain the variational update rule for the v
Revised: Here, starting from Equation 10, we show how to obtain the variational update rule for the v

Original: i ’s (Equation 2), and the ﬁxed point equations (Equation 5) (Friston, 2008; 2005; Bogacz, 2017).
Revised: i ’s (Equation 2), and the ﬁxed point equations (Equation 5) (Friston, 2008; 2005; Bogacz, 2017).

Original: We ﬁrst reduce the free-energy to a sum of prediction errors.
Revised: We ﬁrst reduce the free-energy to a sum of prediction errors.

Original: [Equation]
Revised: [Equation]

Original: Where (cid:15) i
Revised: Where (cid:15) i

Original: =
Revised: =

Original: v
Revised: v

Original: i − f
Revised: i − f

Original: ( P ( v 1 ) , and we have utilized the assumption made in section 2 that Σ − 1 =
Revised: ( P ( v 1 )) , and we have utilized the assumption made in section 2 that Σ − 1 =

Original: I .
Revised: I .

Original: By setting all precisions to the identity, we are implicitly assuming that all datapoints and vertices of the computational graph have equal variance.
Revised: By setting all precisions to the identity, we are implicitly assuming that all datapoints and vertices of the computational graph have equal variance.

Original: Next we assume that the dynamics of each vertex v i follow a gradient descent on the free-energy.
Revised: Next we assume that the dynamics of each vertex v i follow a gradient descent on the free-energy.

Original: [Equation]
Revised: [Equation]

Original: Where we have used the fact that ∂(cid:15) i ∂v i = 1 and ∂(cid:15) j ∂v j = − ∂ ˆ v j ∂v i .
Revised: Where we have used the fact that ∂(cid:15) i ∂v i = 1 and ∂(cid:15) j ∂v j = − ∂ ˆ v j ∂v i .

Original: To obtain the ﬁxed point of the dynamics, we simply solve for dv i dt = 0 . 
Revised: To obtain the ﬁxed point of the dynamics, we simply solve for dv i dt = 0 . [Equation] i

Original: [Equation]
Revised: [Equation]

Revised: ∂v

Revised: [Equation]

Revised: =

Revised: [Equation]

Original: {+⇒ 
+}
Revised: {+⇒ 
+}

Revised: = (cid:15) i − (cid:88)

Original: j
Revised: j

Original: [Equation]
Revised: [Equation]

Original: ∂v ∗ i
Revised: v [Equation]

Revised: (cid:15)

Revised: j

Original:  Similarly, since (cid:15) ∗ i = v ∗ i − ˆ v ∗ i then v ∗ i = (cid:15) ∗ i
Revised: ∂ ˆ v j ∂v i ⇒ (cid:15) ∗ i = (cid:88) j ∈C ( v i ) (cid:15) j ∂ ˆ v ∗ j ∂v ∗ i Similarly, since (cid:15) ∗ i = v ∗ i − ˆ v ∗ i then v ∗ i = (cid:15) ∗ i

Original: + ˆ v ∗ i .
Revised: + ˆ v ∗ i .

Original: So:
Revised: So:

Original: [Equation]
Revised: [Table]