Predictive Coding beyond Gaussian Distributions
Anonymous Author(s)
AffiliationAddress email
Abstract
A large amount of recent research has the far-reaching goal of finding training methods for deep neural networks that can serve as alternatives to backpropagation (BP).
A prominent example is predictive coding (PC), which is a neuroscience-inspiredmethod that performs inference on hierarchical Gaussian generative models.
Thesemethods, however, fail to keep up with modern neural networks, as they are unableto replicate the dynamics of complex layers and activation functions.
In this work,we solve this problem by generalizing PC to arbitrary probability distributions,enabling the training of architectures, such as transformers, that are hard to approximate with only Gaussian assumptions.
We perform three experimental analyses.
First, we study the gap between our method and the standard formulation of PCon multiple toy examples.
Second, we test the reconstruction quality on variational autoencoders, where our method reaches the same reconstruction qualityas BP.
Third, we show that our method allows us to train transformer networksand achieve performance comparable with BP on conditional language models.
More broadly, this method allows neuroscience-inspired learning to be applied tomultiple domains, since the internal distributions can be flexibly adapted to thedata, tasks, and architectures used.
Introduction
The last decade has seen an explosion of machine learning research fueled by the collection of anunprecedented amount of data and the development of models that can make use of it.
Startingwith AlexNet
[Krizhevsky et al., 2012], deep neural networks trained with backpropagation (BP)[Rumelhart et al., 1986] have been established as the best-performing models in many fields [Silveret al., 2016,
He et al., 2016, Brown et al., 2020, Ramesh et al., 2021, Devlin et al., 2018].
Despitereaching human-level performance in several tasks [Silver et al., 2016, Vinyals et al., 2017, 2019],we are still far from artificial general intelligence.
The trend has been to constantly increase thenumber of parameters in such networks, from millions [Devlin et al., 2018] to hundreds of billions[Brown et al., 2020].
The limitations and drawbacks given by the large size of modern architectureshave motivated research that looks for alternative methods to train them.
The direction of researchthat has inspired this work is that of neuroscience-inspired alternatives to BP, which promise toovercome these drawbacks, due to both interesting properties of their credit assignment, such asplasticity [Hebb, 1949], and their biological hardware [Kendall et al., 2020].
These methods havetwo main advantages relative to standard deep learning models trained with BP.
First, it is much morefeasible to train them on analog and neuromorphic chips [Kendall et al., 2020].
Second, the resultingcomputational models are extremely flexible in both network layout design and querying techniques[Salvatori et al., 2022a, Millidge et al., 2021, Salvatori et al., 2021, Papadimitriou et al., 2020].
Thesetwo properties could play a crucial role in overcoming the limitations of BP-based learning towardsartificial general intelligence.
Predictive coding (PC) is one of the most influential theories of information processing in thebrain, initially proposed to explain a large amount of brain behaviours [Mumford, 1992, Friston and
Kiebel, 2009], and now also a topic of research in machine learning, thanks to the computationalmodel proposed by Rao and Ballard [Rao and Ballard, 1999].
This method has in fact been used insupervised and unsupervised tasks [Ororbia and Kifer, 2020, Whittington and Bogacz, 2017, Hanet al., 2018], with an important theoretical result: the original formulation is able to approximate theweight update of BP [Whittington and Bogacz, 2017, Millidge et al., 2020], and to exactly replicateit when introducing small variations [Salvatori et al., 2022b, Song et al., 2020].
These results areimportant, as they draw a strong connection with the aforementioned results obtained by BP in thelast decade.
PC, however, also presents several interesting properties that make it different from BP:it has an energy-based formulation that allows the design of powerful associative memory models[Salvatori et al., 2021] and to train graphs of any topology [Salvatori et al., 2022a].
PC can also be studied from an information theory perspective, as it is an hierarchical generativemodel
[Rao and Ballard, 1999].
This is a strength of the model, as it has allowed such models toachieve competitive performance to standard models trained with BP on several generative tasks.
These results, however, are all obtained on simple models: sequential architectures with element-wiseactivations and quadratic energy functions.
Deep learning, however, has progressed far from those inrecent years, and, therefore, it is necessary to evaluate the performance of PC on more up-to-dateand complex architectures to obtain a complete comparison of PC against BP.
In this paper, we seethat the strict Gaussian assumption is limiting when dealing with more complex architectures such astransformers, preventing PC from reaching the performance obtained by BP.
Note that this limitationis not unique to PC, but it is shared among all neuroscience-inspired methods: for example, to ourknowledge, none of these framework has been successfully used to train language models to date.
In this work, we address this problem by generalizing PC to arbitrary distributions.
This allows us toboth use Gaussian distributions when allowed, but also to deal with intractable ones by approximatingthem using a sampling scheme.
The resulting framework is coherent with the PC theory, as itenables the definition of layer-wise energy functions that represent prediction errors [Rao and Ballard,1999, Whittington and Bogacz, 2017].
In standard PC networks, the error is given by the differencebetween the expected and actual input of a layer; here, it is defined as the KL-divergence betweenthe expected and actual distributions.
We show that these formulations are equivalent when using
Gaussian distributions, meaning that our proposed framework is a generalization of standard PC.
Theresults of this paper are briefly summarized as follows:
• We generalize PC beyond the assumption of a Gaussian generative model.
This lets us
define prediction errors as ‘distances’ between arbitrary distributions at each hierarchical
layer of a PC network, and to derive a novel free-energy objective to train such networks.
• We empirically show that standard PC is ineffective in training models with complex
structure and activation functions, as they cannot be approximated by a Gaussian generative
model.
Our proposed method, instead, significantly outperforms it, reaching a competitiveperformance with BP in training both variational autoencoders [Kingma and Welling, 2014]and transformers [Vaswani et al., 2017].
This further bridges the gap in performance betweenstate-of-the-art deep learning methods and neuroscience-inspired learning.
The rest of this paper is structured as follows.
Section 2 introducing the notation used throughout thepaper and describes the probabilistic interpretation of PC.
In Section 3, we propose how to generalizethe definition of PC beyond Gaussian generative models.
In Section 4, we evaluate the performanceof our proposed method in comparison to standard PC and BP.
Finally, in Sections 5 and 6, we discussrelated work and provide a conclusion, respectively.
Predictive Coding: A Probabilistic Interpretation
In this work, we focus on multi-layer networks organised in a sequence of L layers.
These networksdefine a model M θ that learns the relationship between the input d and the target o by updatingits weight parameters θ =
( θ 1 , . .
, θ L ) .
Networks trained with BP have a forward step, wherean output
ˆ o is computed from an input d , and a backward step, where the error given by a lossfunction L = loss ( o, ˆ o ) is propagated back to the network.
The nodes x l of each layer storethe activation µ l , which is computed as a function of the previous layer x l − 1 , and the networkparameters θ : µ l = f l ( x l − 1 , θ l ) , where f l is an activation function.
In PC, the activation µ l is [Figure] not stored directly inside the nodes x l .
Instead, a separate value ϕ l is optimized by minimizing anenergy function F given the activation µ l .
Therefore, a model M BP trained via BP consists of 93 exclusively the parameters θ , while a PC network requires the two sets of parameters, θ and ϕ , whereϕ = ( ϕ 0 , . . .
, ϕ L ) represents the neural activities of different layers (Fig. 1).
2.1 PC as Variational Inference
PC as a learning algorithm can be mathematically interpreted as a variational inference problem.
It ispossible to consider learning as an intractable Bayesian inference process that can be approximatedwith a tractable optimization problem [Friston, 2003, 2005, 2008].
Under this assumption, the neuralactivities ϕ i in a PC network represent probability distributions of specific layers.
In particular, Fristonbased his theory of PC on Gaussian generative models.
A detailed review is provided in Millidge et al.[2021].
Formally, the model can be explicated in probabilistic terms as a generative model, describedby the joint probablility p ( o, x )
=
p ( o | x ) p ( x ) , where o is the observed data and x the latent state tobe inferred.
In particular, we want to compute the true posterior p ( x | o ) .
However, computing it by
Bayes rule as p ( x | o ) =
p ( o, x ) /p
( o ) requires the normalizing factor p ( o )
= (cid:82) p ( x, o ) dx , which is,for all but trivial problems, intractable.
Variational inference aims to approximate the intractableposterior with a family of distributions q ϕ ( x | o ) , where the parameters ϕ have to be learnt.
This isgenerally done via gradient descent on the KL divergence [Kullback and Leibler, 1951] between theapproximated and true posterior.
The goal is to compute
[Equation]
by minimizing an upper bound on the divergence, called the variational free energy F :
[Equation]
The PC framework assumes a Gaussian form for the generative model p ( o, x )
= p ( o | x ) p ( x ) =
N ( o ; f ( x, θ ) , Σ 2 ) N ( x, µ, Σ 1 ) , where Σ 2 , Σ 1 , and µ are prior parameters that can optionally belearnt.
Using as variational posterior the Dirac-delta distribution 1 q ϕ ( x | o )
=
δ ( x − ϕ ) , we get that
[Equation]
where the entropy of q is 0.
This scheme can be applied to deep neural networks, where x doesnot represent an homogeneous latent space (e.g., a single layer), but is, instead, organised in ahierarchical structure, defined by the multiple layers of a PC network with widths w 1 , ..., w l andnodes x 0 , x 1 , ..., x L .
Under this premise, the generative model is as follow:
[Equation]
(4) where µ l = f l ( x l − 1 , θ l ) , and x L corresponds to the observation layer and is set to x L = o duringtraining.
The parameters Σ l are prior diagonal covariance matrices, 2 , which can be optionally learnt, and µ 0 is an arbitrary prior that can be set to some given data d .
This is equivalent to the training of asupervised network with data point d and label o .
The energy becomes:
[Equation]
where k is a constant, ϵ l = ( ϕ l − µ l ) , and q ϕ ( x 0: L | d, o )
= (cid:81)
Ll {+=
+} δ ( x l − ϕ l ) (implying that x l = ϕ l ).
In this equation, the total energy is given by the sum of the energies
E l of every layer, where
E l := − ln p ( ϕ l | µ l ) .
By assuming identity covariance matrices (i.e., Σ l = I ), the energy becomes thesum of quadratic errors, introduced by Rao and Ballard [1999]:
[Equation]
In most cases, however, the generative model is not known.
Instead, we parameterize with a setof parameters θ : p ( x 0 , . . .
, x L ; θ ) , which we learn alongside the posterior.
This can be done viaexpectation maximization (EM)
[Dempster et al., 1977], where both the parameters θ and ϕ arelearned.
This is achieved using gradient descent to minimize F .
In detail, the training process of a
PC network given a labelled point ( d, o ) is divided into two phases: inference and weight update.
Inboth phases, the input layer prior is set to µ 0 =
d , while the output layer nodes are fixed to ϕ L
= o .
During the inference phase, the weight parameters are fixed, and the node values ϕ are continuouslyupdated via gradient descent to minimize the energy F .
This process either runs until convergence,or for a fixed number of steps T .
When the inference phase ends, a weight update is performed asfollows: the node values ϕ are fixed, and the weights are updated once via gradient descent on thesame energy function F .
Generalization to Arbitrary Distributions
In this section, we go beyond the strict Gaussian assumption of PC.
The probabilistic interpretationdescribed above relies on a specific structure, where the information signal that flows through thelayers is subsequently interpreted as distribution parameters and as evaluation points.
We use the·
D and · S superscripts to highlight that a value is interpreted as a distribution or as a single sample,respectively.
In particular, the value ϕ l of the nodes x l is used to compute the distribution parameterµ D l +1 =
f l ( ϕ S l , θ l +1 ) and to evaluate the probability p ( ϕ S l | µ D l ) =
N ( ϕ S l ; µ D l , Σ l ) ).
This model maybe expressive enough in a generic fully connected PC network that assumes a Gaussian form for thegenerative model, like the one used by Salvatori et al.
[2021, 2022a].
However, when consideringcomplex architectures, it could become a limiting factor.
For example, a layer employing a softmaxactivation cannot be easily approximated through a multi-variate Gaussian, since the softmax functionintroduces a strong dependency between the nodes (i.e., their values have to sum to 1 ).
Under
Gaussian assumptions, it may be reasonable to interpret the values ϕ l both as parameters ϕ D l and asevaluation points ϕ S l , since ϕ S l corresponds to the maximum likelihood estimation of the Gaussiandistribution N ( ϕ D l , Σ l ) , and thus ϕ S l = ϕ D l .
It is not clear, however, how this would translate under ageneric categorical distribution obtained via softmax .
As we will show in the next sections, its usecan hinder the ability of a network to learn using PC.
In order to generalize and extend the canonical formulation of PC, we do the following: insteadof directly maximizing the likelihoods p l := p ( ϕ S l | µ D l ) , we assume that this process is happeningrepeatedly between each pair of consecutive layers for a single optimization step.
We consider thenodes x l as storing the parameters ϕ
D l of an arbitrary distribution X l ( ϕ D l ) , 3 we draw N samplepoints from it, s ( i ) l ∼ X l ( ϕ D l ) , i ∈ { 1 , . . .
, N } , and we minimize each individual likelihoodp ( i )
l = p ( s ( i ) l | µ D l ) .
Furthermore, we remove the Gaussian assumption and consider µ D l = f l ( ϕ Dl − 1 , θ l )to be a parametrization of a generic distribution (cid:98) X l ( µ D l ) .
By doing so, the node values ϕ l areinterpreted exclusively as distribution parameters: both the energy E l and the activation µ D l are afunction of ϕ D l .
It derives that the variational free energy F is a function of the expected values givenby the likelihoods p ( s ( i ) l | µ D l ) , i ∈ { 1 , . . .
, N } , for each layer l ∈ { 0 , . . .
, L } .
The energy of each [Figure] layer can be defined as:
[Equation] [Equation]
A detailed derivation of the above equation is presented in the supplementary material.
Knowing thatthe cross-entropy between two distributions H ( a, b )
=
D KL [ a || b ] + H ( a ) and that H ( a ) ≥ 0 , the 166 total energy of the network can be optimized by minimizing
[Equation]
This follows, as the entropy of X l ( ϕ D l ) can be assumed as being a constant depending on the trainingdata.
Figure 2 highlights the difference between the original and new PC formulation.
This definitionof the variational free energy F KL does not rely on the Gaussian generative assumption, and (aslong as the KL divergence between X l and (cid:98) X l can be efficiently computed) there is no limit on the 171 kind of distribution that can be used.
Throughout our experiments, we assumed that the distributions
X l and (cid:98)
X l belong to the same family, but different families can be used in different layers.
In thesupplementary material, we show how F KL is equivalent to F when assuming a Gaussian generative 174 model.
We also analyze the learning dynamics defined by applying the EM algorithm to Eq.
Experiments
We compare the results of ANNs trained with BP and PC on various tasks.
The main goal is to make
PC competitive to BP for complex deep neural architectures.
Only recently, PC has begun to beapplied to train ANNs and has shown a similar performance in a restricted set of tasks, but a greaterversatility compared to BP.
We show that our extended probabilistic formulation can be applied tomore complex architectures.
We refer to each specific version of PC by specifying its energy function.
4.1 Classification
We trained an ANN on classification tasks for both the MNIST [Deng, 2012] and CIFAR10[Krizhevsky et al.]
datasets.
We evaluated the final accuracy of the model when training with
BP compared to PC, as well as the training efficiency, measured as improvements over epochs.
Setup: We defined three variations of a fully connected network with L = 3 hidden layers andwidth w = 512 :
• M 1 uses tanh as activation function for the hidden and final layers.
The mean squared error
(MSE) is the loss function used for BP and to compute the test loss of both PC and BP.
• M 2 uses the softmax activation function for the final layer.
Consequently, the loss function
used is cross-entropy (CE).
This architecture corresponds to the one normally used inclassification tasks.
• M 3 is a copy of M 2 where the activation function of the second hidden layer is replaced
with softmax .
CE is again the loss function used.
M
A cc u r a c y
M
M    0 .0 .0 .1 .
L o ss
CIFAR10 classiﬁcation
BP PC F
PC F KL
Accuracy Loss
A cc u r a c y 20     20    0 .0 .0 .0 .1 .3 .0 .1 .3 .
L o ss
MNIST classiﬁcation
PC F KL noticeably outperforms PC F , reaching performances comparable with BP.
This is true especially for M 2 , which reflects the most commonly used architecture among the three.
The x axis represents the number of epochs.
Note that the experiments performed with a softmax activation in a hidden layer are merely presentedwith the goal of empirically validating the effectiveness of our theory, as networks of this kind havenever been in practical tasks.
Effectively, M 2 represents the only widely used architecture.
Weused a weight learning rate of β θ = 0 . 0001 for both PC and BP.
For PC, we used T = 32 ϕ -steps.
We assumed identity covariance matrices for the Gaussian distributions of the generative model.
Consequently, the energies F and F KL differ only for the function used for a softmax -activated layer.
For F KL , that is
[Equation]
(9) where X l and (cid:98) X l are discrete distributions.
Therefore, in the model M 1 , PC F KL and PC F are 202 algorithmically equivalent (see the supplementary material).
More details about the hyperparameterswe used are given in the supplementary material.
Results: The results are plotted in Fig.
They show that neural networks trained with BP and
PC F KL perform similarly in both M 1 and M 2 .
The original formulation of PC, on the other hand, 206 performs much worse in all the considered experiments.
The first column of plots shows that PCand BP perform similarly when trained with fully Gaussian assumptions.
In the experiment on M 2 ,the standard definition of PC is performing well, even though it is significantly outperformed by
BP and PC F KL .
When training on M 3 , however, the performances are poor.
This again does nothappen when training using PC F KL , which obtains performances comparable to those of BP.
Overall,this clearly indicates that PC F is not suitable to train neural networks that do not rely uniquely on
Gaussian generative models.
Instead, we experienced a solid improvement when using the F KLenergy function.
It is enough to reach the same accuracy-over-epochs ratio achieved by BP whenusing M 2 .
Under more uncommon architectural choices, such as M 3 , the ratio is slightly worse in 215 favour of BP, but still decisively better than PC F .
The difference is particularly noticeable in thefirst epochs.
We believe it may be due to a not ideal initialization of the weights for the PC network,which is currently using the default initialization designed for BP networks [He et al., 2015].
Furtherresearch in this direction could improve the training performance.
4.2 Variational Autoencoders
Varitational AutoEncoders (VAEs) are an architecture that, in its standard formulation [Kingma and
Welling, 2014], relies on explicit distributions that differ from Gaussian distributions with fixedcovariance matrix.
In fact, the bottleneck layer b of a VAE is required to model the distributionsof both the mean and the variance of the latent posterior given a sample d , p ( u b , Σ b | d ) . However,both PC F and PC (cid:101) F are not suitable for that, as they require each layer to represent exclusivelythe Gaussian mean u l . The optionally learnable parameter Σ l does not depend on the particularinput sample d . Our proposed method, however, overcomes this limitation, as shown in Figure 4.
E n c ode r
E n c ode r D e c ode r D e c ode r
[Figure]
Overall PC and BP perform similarly.
By considering the bottleneck layer b as storing the distribution parameters ϕ D b = ( u b , Σ b ) , we canlearn the full posterior distribution N d ( u b , Σ b ) .
In this case, µ D b = ( (cid:98) u b , (cid:98) Σ b ) =
f b ( ϕ D b − 1 , θ b ) .
We thenemploy the reparametrization trick [Kingma and Welling, 2014] by sampling some Gaussian noise¯ ϵ ∼ N (0 , 1) to compute µ b +1 =
f b +1 ( u b + ¯ ϵDiag (Σ 1 / 2 b ) , θ b +1 ) , which is fed to the next layer.
Setup: We trained multiple VAEs on the MNIST dataset, comparing the PC and BP trainingalgorithms.
Our architectures employed exclusively fully connected layers, with Hardtanh activationfunction for the hidden layers and sigmoid for the output layer.
The bottleneck layer has a totalof w b = 32 (= 16 + 16) latent units.
When training with PC, we assumed Gaussian distributionswith identity covariance matrix for all but the bottleneck layer, for which we model both the meanu b and the diagonal covariance matrix Σ b .
We used T = 32 , but we sampled, for each data point,a single ¯ ϵ at t = 0 .
We use the same weight learning rate β θ = 0 . 0001 for both BP and PC.
Allhyperparameters and implementation details are described in the appendix.
Results: We observed similar results on a wide range of hyperparameters.
In Figure 5, we report theperformance of both training methods on two different architectures.
The final test loss is overallsimilar, with neither method being decisively better than the other.
The learning curves are alsocomparable, despite PC being generally faster than BP in the first training epochs.
By reconstructingthe maximum likelihood estimation of some data points, we can observe how all models produce verysimilar images.
We also performed an analysis of the latent space produced by the encoders and didnot detect any significant difference between the two training modes.
Figure 6 reports the results.
Wesampled the latent posterior distribution by encoding a data point d and decoding multiple data pointsd ′ obtained by sampling from N d ( µ b , Σ b ) .
To perform a latent traversal, we encoded two differentdata points, d 1 and d 2 , and reconstructed the maximum likelihood estimation of the vectors obtainedby interpolating the two latent embeddings.
[Figure]
4.3 Transformer Language Models
We train transformer conditional language models based on the BERT architecture [Devlin et al.
The conditional language model objective is enforced by modifying the self-attention mechanism with a triangular mask, so that no position can attend to any later position in the sequence.
Setup: We use the 1B Word Benchmark dataset [Chelba et al., 2013], from which we randomlysample 200,000 training and 10,000 dev instances.
We choose to restrict the model’s input lengthto 32 input tokens from a vocabulary of 8001 tokens generated by the BPE-based SentencePiecetokenizer [Kudo and Richardson, 2018].
We use one transformer block with one head and a hiddensize of 128 throughout the model.
For the PC networks, we assume Gaussian distributions withidentity covariance matrix for all but the layers that employ a softmax activation function (i.e., theattention layers [Vaswani et al., 2017]).
In the latter case, we assume a categorical distribution for thegenerative model.
Consequently, the energy function for those layers is the one defined in Eq.
More implementation details and the hyperparameters are given in the supplementary material.
Results:
For each model, we compare the three training methods: BP , PC F , PC F KL .
We found itbeneficial to run multiple weight updates for a single training step when using PC F KL , but not for
PC F where it led to instability.
We run a hyperprameter search for each training method, select thebest models, compare their training curves, their test performance, and show qualitative examplesof model predictions.
Figure 7 shows that PC F KL significantly outperforms PC F in terms of testperplexity, while having a more stable training curve.
We believe that this is because of the softmaxin both the attention mechanism and the output layer of the transformer model.
The performanceof PC F KL is close to that of BP and both training curves look stable.
In practical terms, thelanguage models trained by BP and PC F KL do not differ significantly on the test set.
In somecases, the predictions given by PC F KL are closer to the ground-truth, e.g., for “Yet the bank and itsexecutives are still ready to support specific Democratic [candidates]”, PC F KL predicts “leaders”and “candidates” as top-2 choices.
All models show failure in commonsense reasoning, e.g., for “I’vebeen dreaming about this since I was a [child]” they fail to assign “child” with > 1% probability,which shows the limitations of small language models.
More examples are given in the appendix.
Related Work
In the last years, an active research direction that lies at the intersection of machine learning andcognitive science focuses on finding training algorithms for deep neural networks that have a degreeof biological plausibility while obtaining good results on machine learning benchmarks.
The mostpopular ones are PC [Rao and Ballard, 1999, Whittington and Bogacz, 2017], equilibrium propagation[Scellier and Bengio, 2017, Scellier et al., 2018, Scellier and Bengio, 2019], and target propagation[Lee et al., 2015, Meulemans et al., 2020, Ernoult et al., 2022].
These methods share multiplesimilarities, both theoretically and in terms of performance.
The first two methods, PC and equilibriumpropagation, are able to approximate the weight update of BP when provided with a label that isclose in distance to the neural activities of the last layer
[Whittington and Bogacz, 2017, Scellierand Bengio, 2019].
Target propagation fails to have this property, but instead has been shown to
[Figure]
[Figure]
approximate Gauss-Newton optimization [Meulemans et al., 2020].
However, PC possesses manyunique properties that these methods lack.
PC networks can in fact be used to produce efficientassociative memory models [Salvatori et al., 2021], have an update mechanism that produces betterlearning properties than BP under specific conditions [Song et al., 2022], and allow training on graphsof any topology [Salvatori et al., 2022a].
Furthermore, they have achieved good results in classification[Han et al., 2018] and generation tasks [Ororbia and Kifer, 2020].
For a recent survey on these aspects,see [Millidge et al., 2022].
To conclude, we are not aware of any neuroscience-inspired learningmethod before this work that is able to generalize to complex tasks such as language modeling.
Progress in this direction is promising, as one of the main limitations of modern architectures isthat they are extremely computationally expensive to be trained, with large-scale models sometimesrequiring hundreds of GPUs for several weeks
[Brown et al., 2020].
On the other hand, significantbreakthroughs on neuromorphic and analog hardware have recently been achieved [Strukov et al.
,2008, Sebastian et al., 2020], which can exploit the aforementioned properties of neuroscienceinspired learning methods, as shown in [Kendall et al., 2020], where the authors simulated the trainingof a multilayer network on an analog chip in an end-to-end fashion.
There has been a lot of research done towards bridging the gap in performance between state-of-the-artdeep learning methods and neuroscience-inspired learning.
Both fields can benefit from each otherby drawing inspiration from each other’s techniques.
In neuroscience, understanding how the brainlearns to associate different areas (e.g., visual and motor cortices) to successfully drive behaviour isof fundamental importance [Petreanu et al., 2012, Manita et al., 2015, Makino and Komiyama, 2015,
Poort et al., 2015, Pakan et al., 2016, Zmarz and Keller, 2016, Attinger et al., 2017].
However, howto correctly modify synapses to achieve this has puzzled neuroscientists for decades.
This is oftenreferred to as the synaptic credit assignment problem [Rumelhart et al., 1986, Sutton and Barto, 1998,
Roelfsema and van Ooyen, 2005, Bengio, 2014, Lee et al., 2015, Roelfsema and Holtmaat, 2018], forwhich the BP algorithm provides an elegant solution.
Conclusion
The main motivation behind this work was to make PC competitive to BP for complex deep neuralarchitectures.
The tasks in this work are among the most popular and important in the field: imagegeneration and language modeling.
In the first case, we trained a variational autoencoder.
This modelis fully Gaussian, but the bottleneck requires explicitly computable variances.
While variations of
PC with trainable variances are already defined in the literature [Millidge et al., 2021], they do notallow dependencies between the variance and the input.
Rather, they act as a regulariser within thenetwork.
Consequently, they have not been used as a sampling scheme in a specific layer of a PCnetwork.
In the second case, we trained a transformer model, intractable before by PC networks,due to the presence of attention (and hence softmax ), and showed results comparable to those of BP.
Future work includes applying this method to other complex deep learning architectures, with thefar-reaching goal of scaling PC to large-scale machine learning tasks, and hence further closing thegap with BP-based learning.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[No] The code andthe data are proprietary.
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Supplementary material
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Supplementary material
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a)
If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[N/A](c)
Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A](b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]