S OFT H EBB : B AYESIAN INFERENCE IN UNSUPERVISED H EBBIAN SOFT WINNER - TAKE - ALL NETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
State-of-the-art artiﬁcial neural networks (ANNs) require labelled data or feedback between layers, are often biologically implausible, and are vulnerable to adversarial attacks that humans are not susceptible to.
On the other hand, Hebbian learning in winner-take-all (WTA) networks, is unsupervised, feed-forward, and biologically plausible.
However, a modern objective optimization theory for WTA networks has been missing, except under very limiting assumptions.
Here we derive formally such a theory, based on biologically plausible but generic ANN elements.
Through
Hebbian learning, network parameters maintain a Bayesian generative model of the data.
There is no supervisory loss function, but the network does minimize cross-entropy between its activations and the input distribution.
The key is a "soft" WTA where there is no absolute "hard" winner neuron, and a speciﬁc type of Hebbian-like plasticity of weights and biases.
We conﬁrm our theory in practice, where, in handwritten digit (MNIST) recognition, our Hebbian algorithm,
SoftHebb, minimizes cross-entropy without having access to it, and outperforms the more frequently used, hard-WTA-based method.
Strikingly, it even outperforms supervised end-to-end backpropagation, under certain conditions.
Speciﬁcally, in a two-layered network, SoftHebb outperforms backpropagation when the training dataset is only presented once, when the testing data is noisy, and under gradientbased adversarial attacks.
Notably, adversarial attacks that confuse SoftHebb are also confusing to the human eye.
Finally, the model can generate interpolations of objects from its input distribution.
All in all, SoftHebb extends Hebbian WTA theory with modern machine learning tools, thus making these networks relevant to pertinent issues in deep learning.
I NTRODUCTION
State-of-the-art artiﬁcial neural networks (ANNs) achieve impressive results in a variety of machine intelligence tasks (Sejnowski, 2020).
However, they largely rely on mechanisms that diverge from the original inspiration from biological neural networks (Bengio et al., 2015; Illing et al., 2019).
As a result, only a small part of this proliﬁc ﬁeld also contributes to computational neuroscience.
In fact, this biological implausibility is also an important issue for machine intelligence.
For their impressive performance, ANNs trade off other desired properties, which are present in biological systems.
For example, ANN training often demands very large and labelled datasets.
When labels are unavailable, self-supervised learning schemes exist, where supervisory error signals generated by the network itself are exploited and backpropagated from the output towards the input to update the network’s parameters (Goodfellow et al., 2014;
Devlin et al., 2018; Chen et al., 2020).
However, this global propagation of signals in deep networks introduces another limitation.
Namely, it prevents the implementation of efﬁcient distributed computing hardware that would be based on only local signals from neighbouring physical nodes in the network, and is in contrast to the local synaptic plasticity rules that are believed to govern biological learning.
Several pieces of work have been addressing parts of the biological implausibility and drawbacks of backpropagation in ANNs (Bengio et al., 2015;
Lillicrap et al., 2016; Guerguiev et al., 2017; Pfeiffer & Pfeil, 2018; Illing et al., 2019;
Pogodin & Latham, 2020; Millidge et al., 2020; Pogodin et al., 2021).
Recently, an approximation to backpropagation that is mostly Hebbian, i.e. relies on mostly pre- and post-synaptic activity of each synapse, has been achieved by reducing the global error requirements to 1-bit information (Pogodin & Latham, 2020).
Two schemes that further localize the signal that is required for a weight update are Equilibrium Propagation (Scellier & Bengio, 2017) and Predictive Coding (Millidge et al., 2020).
Both methods approximate backpropagation through Hebbian-like learning, by delegating the global aspect of the computation, from a global error signal, to a global convergence of the network state to an equilibrium.
This equilibrium is reached through several iterative steps of feed-forward and feed-back communication throughout the network, before the ultimate weight update by one training example.
The biological plausibility and hardware-efﬁciency of this added iterative and heavily feedback-dependent process are open questions that begin to be addressed (Ernoult et al., 2020).
Moreover, learning through backpropagation, and presumably also its approximations, has another indication of biological implausibility, which also signiﬁcantly limits ANN applicability.
Namely, it produces networks that are confused by small adversarial perturbations of the input, which are imperceptible by humans.
It has recently been proposed that a defence strategy of "deﬂection" of adversarial attacks may be the ultimate solution to that problem (Qin et al., 2020).
Through this strategy, to cause confusion in the network’s inferred class, the adversary is forced to generate such a changed input that really belongs to the distribution of a different input class.
Intuitively, but also strictly by deﬁnition, this deﬂection is achieved if a human assigns to the perturbed input the same label that the network does.
Deﬂection of adversarial attacks in ANNs has been demonstrated by an elaborate scheme that is based on detecting the attacks (Qin et al., 2020).
However, the human ability to deﬂect adversarial perturbations likely does not rely on detecting them, but rather on effectively ignoring them, making the deﬂecting type of robustness an emergent property of biological computation rather than a defence mechanism.
The biological principles that underlie this property of robustness are unclear, but it might emerge from the distinct algorithms that govern learning in the brain.
Therefore, what is missing is a biologically plausible model that can learn from fewer data-points, without labels, through local plasticity, and without feedback from distant layers.
This model could then be tested for emergent adversarial robustness.
A good candidate category of biological networks and learning algorithms is that of competitive learning.
Neurons that compete for their activation through lateral inhibition are a common connectivity pattern in the superﬁcial layers of the cerebral cortex (Douglas & Martin, 2004; Binzegger et al., 2004).
This pattern is described as winner-take-all (WTA), because competition suppresses activity of weakly activated neurons, and emphasizes strong ones.
Combined with Hebbian-like plasticity rules, WTA connectivity gives rise to competitivelearning algorithms.
These networks and learning schemes have been long studied (Von der Malsburg, 1973) and a large literature based on simulations and analyses describes their functional properties.
A WTA neuronal layer, depending on its speciﬁcs, can restore missing input signals (Rutishauser et al., 2011; Diehl & Cook, 2016), perform decision making i.e. winner selection (Hahnloser et al., 1999; Maass, 2000; Rutishauser et al., 2011) generate oscillations such as those that underlie brain rhythms (Cannon et al., 2014).
Perhaps more importantly, its neurons can learn to become selective to different input patterns, such as orientation of visual bars in models of the primary visual cortex (Von der Malsburg, 1973), MNIST handwritten digits (Nessler et al., 2013; Diehl & Cook, 2015;
Krotov & Hopﬁeld, 2019), CIFAR10 objects (Krotov & Hopﬁeld, 2019), spatiotemporal spiking patterns (Nessler et al., 2013), and can adapt dynamically to model changing objects (Moraitis et al., 2020).
The WTA model is indeed biologically plausible, Hebbian plasticity is local, and learning is input-driven, relying on only feed-forward communication of neurons – properties that seem to address several of the limitations of ANNs.
However, the model’s applicability is limited to simple tasks, because, the theoretical literature related to Hebbian WTA remains surprisingly limiting, despite its long history, and the strong and productive community interest (Sanger, 1989; Földiák & Fdilr, 1989; Földiak, 1990; Linsker, 1992; Olshausen & Field, 1996; Bell & Sejnowski, 1995; Olshausen & Field, 1997; Lee et al., 1999; Nessler et al., 2013; Pehlevan & Chklovskii, 2014; Hu et al., 2014;
Pehlevan & Chklovskii, 2015; Pehlevan et al., 2017; Isomura & Toyoizumi, 2018). Nessler et al.
(2009; 2013) described a very related theory but for a model that is largely incompatible with ANNs and thus impractical.
It uses spiking and stochastic neurons, input has to be discretized, and each input feature must be encoded through multiple binary neurons.
Moreover, it was only proven for neurons with an exponential activation function.
It remains therefore unclear which speciﬁc plasticity rule and structure could optimize an ANN WTA for Bayesian inference.
It is also unclear how to minimize a common loss function such as cross-entropy despite unsupervised learning, and how a WTA could represent varying families of probability distributions.
In summary, on the theoretical side, an algorithm that is simultaneously normative, based on WTA networks and Hebbian unsupervised plasticity, performs Bayesian inference, and, importantly, is composed of conventional, i.e. non-spiking, ANN elements and is rigorously linked to modern ANN tools such as cross-entropy loss, would be an important advance but has been missing.
On the practical side, evidence that Hebbian WTA networks could be useful for presently pertinent issues of modern ANNs such as adversarial robustness, generation of synthetic images, or faster learning, has remained limited.
Here we aim to ﬁll these gaps.
Recently, when WTA networks were studied in a theoretical framework compatible with conventional machine learning (ML), but in the context of short-term as opposed to long-term Hebbian plasticity, it resulted in surprising practical advantages over supervised ANNs (Moraitis et al., 2020).
A similar theoretical approach could also reveal unknown advantages of long-term Hebbian plasticity in WTA networks.
In addition, it could provide insights into how a WTA microcircuit could participate in larger-scale computation by deeper cortical or artiﬁcial networks.
Here we construct "SoftHebb", a biologically plausible WTA model that is based on standard ratebased neurons as in ANNs, can accommodate various activation functions, and learns without labels, using local plasticity and only feed-forward communication, i.e. the properties we seek in an ANN.
Importantly, it is equipped with a simple normalization of the layer’s activations, and an optional temperature-scaling mechanism (Hinton et al., 2015), producing a soft WTA instead of selecting a single "hard" winner neuron.
This allows us to prove formally that a SoftHebb layer is a generative mixture model that objectively minimizes its Kullback-Leibler (KL) divergence from the input distribution through Bayesian inference, thus providing a new formal ML-theoretic perspective of these networks.
We complement our main results, which are theoretical, with experiments that are small-scale but produce intriguing results.
As a generative model, SoftHebb has a broader scope than classiﬁcation, but we test it in simulations on the tasks of recognizing MNIST handwritten digits and
Fashion-MNIST fashion products.
First, we conﬁrm that SoftHebb is more accurate than a hard-WTA model.
Second, we validate that it minimizes a loss function (cross-entropy) even though it has no access to it or to labels during learning.
In addition, likely owing to its Bayesian and generative properties, the unsupervised WTA model outperforms a supervised two-layer perceptron in several aspects: learning speed and accuracy in the ﬁrst presentation of the training dataset, robustness to noisy data, and increased robustness to one of the strongest white-box adversarial attacks, i.e. projected gradient descent (PGD) (Madry et al., 2017), and without any explicit defence.
Interestingly, the SoftHebb model also exhibits inherent properties of deﬂection (Qin et al., 2020) of the adversarial attacks, and generates object interpolations.
T HEORY
Deﬁnition 2.1 ( The input assumptions).
Each observation j x ∈ R n is generated by a hidden "cause" j C from a ﬁnite set of K possible such causes: j C ∈ { C k , ∀ k ≤ K ∈ N } .
Therefore, the data is generated by a mixture of the probability distributions attributed to each of the K classes
C k :
[Equation]
In addition, the dimensions of x , x i are conditionally independent from each other, i.e. p ( x )
= (cid:81)
ni
=1 p ( x i ) .
The number K of the true causes or classes of the data is assumed to be known.
The term "cause" is used here in the sense of causal inference.
It is important to emphasize that the true cause of each input is hidden, i.e. not known.
In the case of a labelled dataset, labels may correspond to causes, and the labels are deleted before presenting the training data to the model.
We choose a mixture model that corresponds to the data assumptions but is also interpretable in neural terms (Paragraph 2.4):
Deﬁnition 2.2 ( The generative probabilistic mixture model).
We consider a mixture model distribution q : q ( x ) =
(cid:80)
Kk =1 q ( x | C k ) Q ( C k ) , approximating the data distribution p .
We choose speciﬁcally a mixture of exponentials and we parametrize Q ( C k ; w 0 k ) also as an exponential, speciﬁcally:
[Equation]
In addition, the parameter vectors are subject to the normalization constraints: || w k ||
= 1 , ∀ k , and (cid:80)
[-=
-]Kk =1 e w 0 k
= 1 .
The model we have chosen is a reasonable choice because it factorizes similarly to the data of
Deﬁnition 2.1:
[Equation]
where u k = w k · x || w k
||·||
x
|| , i.e. the cosine similarity of the two vectors.
A similar probabilistic model was used in related previous theoretical work (Nessler et al., 2009; 2013; Moraitis et al., 2020), but for different data assumptions, and with certain further constraints to the model.
Namely, (Nessler et al., 2009; 2013) considered data that was binary, and created by a population code, while the model was stochastic.
These works provide the foundation of our derivation, but here we consider the more generic scenario where data are continuous-valued and input directly into the model, which is deterministic and, as we will show, more compatible with standard ANNs.
In (Moraitis et al., 2020), data had particular short-term temporal dependencies, whereas here we consider the distinct case of independent and identically distributed (i.i.d.)
input samples.
The Bayes-optimal parameters of a model mixture of exponentials can be found analytically as functions of the input distribution’s parameters, and the model is equivalent to a soft winner-take-all neural network (Moraitis et al., 2020).
After describing this, we will prove here that Hebbian plasticity of synapses combined with local plasticity of the neuronal biases sets the parameters to their optimal values.
Theorem 2.3 ( The optimal parameters of the model).
The parameters that minimize the KL divergence of such a mixture model from the data are, for every k ,
[Equation]
where opt w k = c · µ p k ( x ) , c ∈ R + , µ p k ( x ) is the mean of the distribution p k , and p k : = p ( x | C k ) .
In other words, the optimal parameter vector of each component k in this mixture is proportional to the mean of the corresponding component of the input distribution, i.e. it is a centroid of the component.
In addition, the optimal parameter of the model’s prior Q ( C k ) is the logarithm of the corresponding component’s prior probability.
The Theorem’s proof was provided in the supplementary material of (Moraitis et al., 2020), but for completeness we also provide it in our Appendix.
These centroids and priors of the input’s component distributions, as well as the method of their estimation, however, are different for different input assumptions, and we will derive a learning rule that provably sets the parameters to their Maximum Likelihood Estimate for the inputs addressed here.
The learning rule is a Hebbian type of synaptic plasticity combined with a plasticity for neuronal biases.
Before providing the rule and the related proof, we will describe how our mixture model is equivalent to a
WTA neural network.
2. E QUIVALENCE OF THE PROBABILISTIC MODEL TO A WTA NEURAL NETWORK
The cosine similarity between the input vector and each centroid’s parameters underpins the model (Eq. 4).
This similarity is precisely computed by a linear neuron that receives normalized inputs x ∗ :
= x || x || , and that normalizes its vector of synaptic weights: w ∗ k :=
w || w || .
Speciﬁcally, the neuron’s summed weighted input u k = w ∗ k
· x ∗ then determines the cosine similarity of an input sample to the weight vector, thus computing the likelihood function of each component of the input mixture (Eq. 2).
It should be noted that even though u k depends on the weights of all input synapses, the weight values of other synapses do not need to be known to each updated synapse.
Therefore, in the SoftHebb plasticity rule that we will present (Eq.
8), the term u k is a local, postsynaptic variable that does not undermine the locality of the plasticity.
The bias term of each neuron can store the parameter w 0 k of the prior Q ( C k ; w 0 k ) .
Based on these, it can also be shown that a set of K such neurons can actually compute the Bayesian posterior, if the neurons are connected in a conﬁguration that implements softmax.
Softmax has a biologically-plausible implementation through lateral inhibition (divisive normalization) between neurons (Nessler et al., 2009; 2013; Moraitis et al., 2020).
Speciﬁcally, based on the model of Deﬁnition 2.2, the posterior probability is
[Equation]
But in the neural description, u k + w 0 k is the activation of the k -th linear neuron.
That is, Eq.shows that the result of Bayesian inference of the hidden cause from the input Q ( C k | x ) is found by a softmax operation on the linear neural activations.
In this equivalence, we will be using y k :
=
Q ( C k | x ; w ) to symbolize the softmax output of the k -th neuron, i.e. the output after the WTA operation, interchangeably with Q ( C k | x ) .
It can be seen in Eq.
7 that the probabilistic model has one more, alternative, but equivalent neural interpretation.
Speciﬁcally, Q ( C k | x ) can be described as the output of a neuron with exponential activation function (numerator in Eq.
7) that is normalized by its layer’s total output (denominator).
This is equally accurate, and more directly analogous to the biological description (Nessler et al., 2009; 2013; Moraitis et al., 2020).
This shows that the exponential activation of each individual neuron k directly equals the k -th exponential component distribution of the generative mixture model (Eq. 4).
Therefore the softmax-conﬁgured linear neurons, or the equivalent normalized exponential neurons, fully implement the generative model of Deﬁnition 2.2, and also infer the Bayesian posterior probability given an input and the model parameters.
However, the problem of calculating the model’s parameters from data samples is a difﬁcult one, if the input distribution’s parameters are unknown.
In the next sections we will show that this neural network can ﬁnd these optimal parameters through Bayesian inference, in an unsupervised and on-line manner, based on only local Hebbian plasticity.
2. A H EBBIAN RULE THAT OPTIMIZES THE WEIGHTS
Several Hebbian-like rules exist and have been combined with WTA networks.
For example, in the case of stochastic binary neurons and binary population-coded inputs, it has been shown that weight updates with an exponential weight-dependence ﬁnd the optimal weights (Nessler et al., 2009; 2013).
Oja’s rule is another candidate (Oja, 1982).
An individual linear neuron equipped with this learning rule ﬁnds the ﬁrst principal component of the input data (Oja, 1982).
A variation of Oja’s rule combined with hard-WTA networks and additional mechanisms has achieved good experimental results performance on classiﬁcation tasks (Krotov & Hopﬁeld, 2019), but lacks the theoretical underpinning that we aim for.
Here we propose a Hebbian-like rule for which we will show it optimizes the soft WTA’s generative model.
The rule is similar to Oja’s rule, but considers, for each neuron k , both its linear weighted summation of the inputs u k , and its nonlinear output of the WTA y k :
[Equation]
where w ik is the synaptic weight from the i -th input to the k -th neuron, and η is the learning rate hyperparameter.
As can be seen, all involved variables are local to the synapse, i.e. only indices i and k are relevant.
No signals from distant layers, from non-perisynaptic neurons, or from other synapses are involved.
By solving the equation E [∆ w ik ] = 0 where E [] is the expected value over the input distribution, we can show that, with this rule, there exists a stable equilibrium value of the weights, and this equilibrium value is an optimal value according to Theorem 2.3:
Theorem 2.5.
The equilibrium weights of the SoftHebb synaptic plasticity rule are
[Equation]
The proof is provided in the Appendix.
Therefore, our update rule (Eq.
8) optimizes the weights of the neural network.
2. L OCAL LEARNING OF NEURONAL BIASES AS B AYESIAN PRIORS
For the complete optimization of the model, the neuronal biases w 0 k must also be optimized to satisfy
Eq.
5, i.e. to optimize the Bayesian prior belief for the probability distribution over the K input causes.
We deﬁne the following rate-based rule inspired from the spike-based bias rule of (Nessler et al., 2013):
[Equation]
With the same technique we used for Theorem 2.5, we also provide proof in the Appendix that the equilibrium of the bias with this rule matches the optimal value opt w 0 k = ln P ( C k ) of Theorem 2.3:
Theorem 2.6.
The equilibrium biases of the SoftHebb bias learning rule are
[Equation]
2. A LTERNATE ACTIVATION FUNCTIONS
The model of Deﬁnition 2.2 uses for each component p ( x | C k ) an exponential probability distribution with a base of Euler’s e, equivalent to a model using similarly exponential neurons (Subsection 2.4). [Figure] A B
D
[Figure] C [Figure] Depending on the task, different probability distribution shapes, i.e. different neuronal activation functions, may be better models.
This is compatible with our theory (see Appendix B).
Firstly, the base of the exponential activation function can be chosen differently, resulting in a softmax function with a different base, such that Eq.
7 becomes more generally Q ( C k | x ) =
[Figure]
This is equivalent to Temperature Scaling (Hinton et al., 2015), a mechanism that also maintains the probabilistic interpretation of the output.
It can also be implemented by a normalized layer of exponential neurons, and are compatible with our theoretical derivations and the optimization by the plasticity rule of Eq. 8.
Moreover, we show in the Appendix that soft WTA models can be constructed by rectiﬁed linear units (ReLU) or in general by neurons with any non-negative monotonically increasing activation function, and their weights are also optimized by the same plasticity rule.
2. P OST - HOC CROSS - ENTROPY
It is not obvious how to measure the loss of the WTA network during the learning process, since the ground truth for causes C may not match the labels, or even their number.
For example, in reality, the cause C of each MNIST example in the sense implied by causal inference is not the digit cause itself, but a combination of a single digit cause D , which is the MNIST label, with one of many handwriting styles S .
In the Appendix C, we derive a method for measuring the loss minimization during training in spite of this mismatch.
The method’s steps can be summarised as (a) unsupervised training of
SoftHebb, then (b) training a supervised classiﬁer on top, and ﬁnally (c1) repeating the training of
SoftHebb with the same initial weights and ordering of the training inputs, while (c2) measuring the trained classiﬁer’s loss. In this way, we can observe the cross-entropy loss H labels of SoftHebb while it is being minimized, and infer that H causes is also minimized (Eq. 72).
We call this the post-hoc cross-entropy method, and we have used it in our experiments (Section 3.2 and Fig.
1 C and D) to evaluate the learning process in a theoretically sound manner.
E XPERIMENTS 3. MNIST ACCURACY VS HARD WTA
We implemented the theoretical SoftHebb model in simulations and tested it in the task of learning to classify MNIST handwritten digits.
The network received the MNIST frames normalized by their
Euclidean norm, and the plasticity rule we derived updated its weights and biases in an unsupervised manner.
We used K = 2000 neurons.
First we trained the network for 100 epochs, i.e. randomly ordered presentations of the 60000 training digits.
In our validation testing we found that softmax with a base of 1000 (see Section 2.7) performed best.
The learning rate η of Eq. 8 decreased linearly from 0.03 to 0 throughout training.
Each training experiment we will describe was repeated ﬁve times with varying random initializations and input order.
We will report the mean and standard deviation of accuracies.
Inference of the input labels by the WTA network of 2000 neurons was performed in two different ways.
The ﬁrst approach is single-layer, where after training the network we assigned a label to each of the 2000 neurons, in a standard approach that is used in unsupervised clustering.
Namely, for each neuron, we found the label of the training set that makes it win the WTA competition most often.
In this single-layer approach, this is the only time when labels were used, and at no point were weights updated using labels.
The second approach was two-layer and based on supervised training of a perceptron classiﬁer on top of the WTA layer.
The classiﬁer layer was trained with the Adam optimizer and cross-entropy loss for 60 epochs, while the previously-trained
WTA parameters were frozen.
SoftHebb achieved an accuracy of (96 . 18 ± 0 .
06)% and (96 . 94 ± 0 .
02)% in its 1- and 2-layer form respectively.
To conﬁrm the strength of the soft WTA approach combined with training the priors through biases, which makes the network Bayesian, we also trained the weights of a network with a hard-WTA setup, i.e. where the strongest-activated neuron’s output y k is 1, and the other neurons are suppressed to 0, for each input.
We found that an initial learning rate of 0.05 was best for the hardWTA network.
The SoftHebb model outperformed the hard WTA (Fig. 1A).
However, SoftHebb’s accuracy is signiﬁcantly lower than a multi-layer perceptron (MLP) with one hidden layer of also 2000 neurons that is trained end-to-end exhaustively.
The MLP achieves a (98 .
33 ± 0 .
06)% accuracy (not shown in the ﬁgure).
This is expected, due to end-to-end training, supervision, and the MLP being a discriminative model as opposed to a generative model merely applied to a classiﬁcation task, as SoftHebb is.
If the Bayesian and generative aspects that follow from our theory were not required, several additional mechanisms exist to enhance the discriminative power of WTA networks (Krotov & Hopﬁeld, 2019), and even a random projection layer instead of a trained WTA performs well (Illing et al., 2019).
The generative approach however has its own advantages even for a discriminative task, and we will show some of these here.
3. C ROSS - ENTROPY MINIMIZATION AND SINGLE - EPOCH ADVANTAGE OVER BACKPROPAGATION
First, we show as a validation of the theory that the SoftHebb model minimizes cross-entropy H causesQ of its activations from its input’s causes, even though no explicit loss is provided.
According to our post-hoc cross-entropy method (Section 2.8), as a proxy we observed the minimization of H labelsQ during the ﬁrst epoch of on-line Hebbian learning.
The loss on the training inputs as they appear (running loss), as well as on the whole testing dataset can be seen in Fig.
1C and D respectively (blue curves).
The method allows us to observe the discriminative aspect of the generative model, as it is optimized.
After this one epoch, the accuracy of the 1-layer form of the SoftHebb model is (95 . 44 ± 0 .
The 2-layer form is again obtained by training a supervised classiﬁer output layer for 60 epochs, and its accuracy is (96 . 21 ± 0 .
15)% (Fig.
1B, blue bars).
We then also train for a single epoch a 2-layer MLP with a hidden-layer of 2000 neurons, with backpropagation of stochastic gradient descent (SGD) and cross-entropy loss.
We found, through grid search, the optimal minibatch size and learning rate of the MLP (4 and 0.2 respectively).
The MLP achieves an accuracy of (95 . 39 ± 0 .
45)% (Fig.
1B, orange bar), if we exclude one run of the experiment which only achieved an accuracy of 86.92%.
Surprisingly, it does not surpass SoftHebb, not even in its 1-layer form.
In addition, the cross-entropy of the SoftHebb model is visibly minimized faster than through SGD (orange curves of Fig.
1C & D).
It is possible that SoftHebb’s advantage in terms of loss and accuracy is a side-effect of pre-training the second layer when obtaining SoftHebb’s post-hoc cross-entropy, or of that layer’s 60-epoch training.
To test this possibility, we similarly obtained a trained version of the MLP’s output layer alone, and then trained its ﬁrst layer with backpropagation and the second layer frozen.
Meanwhile, we recorded its loss, thus obtaining its own version of post-hoc cross-entropy (1C & D, yellow curve).
SoftHebb still showed an advantage in terms of loss minimization speed, and its 2-layer form’s accuracy is still not surpassed (1B, blue & yellow bars), despite the fully unsupervised and local learning in the core of the network.
Moreover, the ﬁgure shows that the minimization of the loss on the general test set by SoftHebb is smoother than the running loss, while SGD’s test-set loss is inﬂuenced by the speciﬁcs of the individual training examples.
This may indicate stronger generalization by the SoftHebb model, emerging from its Bayesian and generative nature.
If this is true, SoftHebb may be more robust to input perturbations.
3. R OBUSTNESS TO NOISE AND ADVERSARIAL ATTACKS - G ENERATIVE ADVERSARIAL PROPERTIES
Indeed, we tested the trained SoftHebb and MLP models for robustness, and found that SoftHebb is signiﬁcantly more robust than the backprop-trained MLP, both to added Gaussian noise and to
PGD adversarial attacks (see Fig. 2).
PGD (Madry et al., 2017) produces perturbations in a direction that maximizes the loss of each targeted network, and in size controlled by a parameter (cid:15) .
Strikingly,
SoftHebb has a visible tendency to deﬂect the attacks, i.e. its confusing examples actually belong [Figure] to a perceptually different class (Fig.
2B and 3).
This effectively nulliﬁes the attack and was previously shown in elaborate state-of-the-art adversarial-defence models (Qin et al., 2020).
The pair of the adversarial attacker with the generative SoftHebb model essentially composes a generative adversarial network (GAN), even though the term is usually reserved for pairs trained in tandem (Goodfellow et al., 2014;
Creswell et al., 2018).
As a result, the model could inherit certain properties of GANs.
It can be seen that it is able to generate interpolations between input classes (Fig. 3).
The parameter (cid:15) of the adversarial attack can control the balance between the interpolated objects.
Similar functionality has existed in the realm of GANs (Radford et al., 2015), autoencoders (Berthelot et al., 2018), and other deep neural networks (Bojanowski et al., 2017), but was not known for simple biologically-plausible models.
3. G ENERALIZABILITY OF THE ALGORITHM TO OTHER DATASETS : F ASHION -MNIST
Finally, we trained SoftHebb on a more difﬁcult dataset, namely Fashion-MNIST (Xiao et al., 2017) which contains grey-scale images of clothing products.
A supervised MLP of the same size achieved a test accuracy of (88 .
7 ± 0 .
34)% on this task.
We used the exact same SoftHebb model and hyperparameters that we used on MNIST, without any adjustment for the changed dataset.
Still, the model achieved an accuracy of (75 .
14 ± 0 .
In addition, with very small adversarial perturbations, the MLP drops to an accuracy lower than the SoftHebb model despite our generic training, while SoftHebb’s robustness is reconﬁrmed (dashed lines in Fig. 2) as are its generative interpolations (Fig.
3B).
D ISCUSSION
We have described SoftHebb, a biologically plausible model that is unsupervised, local, and requires no error or other feedback from upper layers.
The model consists of elements fully compatible with conventional ANNs.
We have shown the importance of soft competition in rate-based WTA networks, and derived formally the plasticity that optimizes the network through Bayesian computations and learns a generative model of the input distribution.
We also developed a method for quantifying its unsupervised discriminative loss in a theoretically sound manner.
Our experiments are small, but they conﬁrm our optimization theory, and show that SoftHebb has signiﬁcant strengths that emerge from its unsupervised, generative, and Bayesian nature.
It is intriguing that, through biological plausibility, emerge properties commonly associated with biological intelligence, such as speed of learning, robustness to noise and adversarial attacks, and deﬂection of the attacks.
In particular, its ability to learn better than even supervised networks when training time is limited is interesting for resource-limited neuromorphic applications.
Its robustness to noise and adversarial attacks is [Figure] impressive, considering that it is intrinsic and was not instilled by specialized defences.
SoftHebb tends to not merely be robust to attacks, but actually deﬂect them.
We also showed that these networks can generate image interpolations in the latent space.
However, the model is quite limited compared to state of-the-art ML, if classiﬁcation accuracy, exhaustive training, and unperturbed inputs are the benchmark.
To address this, its potential integration into multilayer networks should be explored using our new tools.
This could also provide insights into the role of WTA microcircuits in larger networks in cortex.
A ﬁrst approach to such multilayer networks could be by stacking individuallytrained layers.
The remaining piece to successfully integrate SoftHebb in deep networks is to make the learned representation in each layer distributed, in spite of a WTA approach.
SoftHebb is indeed more distributed than a hard WTA, and further-distributed features may be achieved by localized receptive ﬁelds (Pogodin et al., 2021) similar to area V1 of cortex (Hubel & Wiesel, 1962).
R EPRODUCIBILITY S TATEMENT
Some of our main contributions are theoretical and their proof is fully reproducible by following the rigorous derivations in the main text and in the appendices.
Our experimental results are based on standard machine-learning techniques, while we also describe the hyperparameters and experimental protocols.
Moreover, we provide Python code with speciﬁc instructions to reproduce the main experiments.
A PPENDIX A T HEOREM P ROOFS
Proof of Theorem 2.3 .
The parameters of model q are optimal w = opt w if they minimize the model’s Kullback-Leibler divergence with the data distribution p .
D KL ( p ( x ) || q ( x ; w )) .
Because p k := p ( x | C k ) is independent from p l , and q k :=
q ( x | C k ; w k ) is independent from w l for every l (cid:54)
=
k , we can ﬁnd the set of parameters that minimize the KL divergence of the mixtures, by minimizing the KL divergence of each component k : min D KL ( p k || q k ) , ∀ k, and simultaneously setting
[Equation]
From Eq.
3 and this last condition, Eq.
5 of the Theorem is proven:
[Equation]
Further,
[Equation]
where we used for Eq.
13 the fact that (cid:82) x p k ln p k d x is a constant because it is determined by the environment’s data and not by the model’s parametrization on w .
Eq.
14 follows from the deﬁnition of q k .
The result in Eq.
15 is the mean value of the cosine similarity u k .
Due to the symmetry of the cosine similarity, it follows that
[Equation]
Enforcement of the requirement for normalization of the vector leads to the unique solution
[Equation]
Proof of Theorem 2.5 .
We will ﬁnd the equilibrium point of the SoftHebb plasticity rule, i.e. the weight w ik that implies E [∆ w ( SoftHebb )
ik ]
= 0 .
We will expand this expected value based on the plasticity rule itself, and on the probability distribution of the input x .
[Equation]
Based on this, we will now show that
[Equation]
Using the premise 21, we can take the following steps, where steps 3, 4, and 6 are the main ones, and steps 1, 2, and 5, as well as Theorem A.1 support those.
The cosine similarity function u k ( x ) = xw k , as determined by the total weighted input to the neuron k , and appropriately normalized, deﬁnes a probability distribution centred symmetrically around the vector w k , i
e. µ u k ( x ) = w k and u k ( µ u k − x ) =
u k ( µ u k + x ) .
w k , as premised, is equal to the normalized µ p k ( x ) , i.e. the mean of the distribution p k
( x ) =
p ( x | C k ) , therefore: µ u k ( x ) = µ p k ( x ) .
The soft-WTA version of the neuronal transformation
, i.e. the softmax version of the model’s inference is
[Equation]
But because of the premise 21 that the parameters of the model u k are set to their optimal value, it follows that exp( u k ( x )) =
p k ( x ) and exp( w 0 k ) =
P ( C k ) , ∀ k (see also Theorem 2.6), therefore
[Equation]
Eq.
20 involves twice the function y k
[Equation]
x
[Equation]
p
[Equation]
x
[Equation]
=
[Equation]
p
[Equation]
x ) P
[Equation]
C l ) .
Using the above two points, we will now show that this is approximately equal to y k
( x ) p k ( x ) P ( C k ) .
(a) We assume that the "support" O pk of the component p k ( x ) , i.e. the region where p k is not negligible, is not fully overlapping with that of other components p l .
In addition, O pk is narrow relative to the input space ∀ k , because, ﬁrst, the cosine similarity u k (
x
) diminishes fast
from its maximum at arg max u k = w k in case of non-trivial dimensionality of the input space, and, second, p k ( x ) = exp ( u k ( x )) applies a further exponential decrease
Therefore, the overlap O pk ∩ O pl is small, or none, ∀ l (cid:54)
=
k .
If O yk is the "support" of y k , then this is even narrower than O pk , due to the softmax.
As a result, the overlap O yk ∩ O pl of y k and p l is even smaller than the overlap O pk ∩ O pl of p k and p l , ∀ l (cid:54) = k .
(b) Because of the numerator in Eq.
24, the overlap O yk ∩ O pk of y k and p k is large.
Based on these two points, the overlaps O yk ∩ O pl can be neglected for l (cid:54) =
k , and it follows that
[Equation]
Therefore, we can write Eq.
20 as
[Equation]
Next, we aim to show that the integrals (cid:82) x x i
y k ( x ) p k ( x ) d x and (cid:82) x x y k ( x ) p k ( x ) d x involved in that Equation equal the mean value of x i and x respectively according to the distribution p k .
To show this, we observe that the integrals are indeed mean values of x i and x according to a probability distribution, and speciﬁcally the distribution y k p k .
We will ﬁrst show that the distribution p k is symmetric around its mean µ p k
( x ) .
Then we will show that y k ( x ) is also symmetric around the same mean.
Then we will use the fact that the product of two such symmetric distributions with common mean, such as y k p k , is a distribution with the same mean, a fact that we will prove in Theorem A.1.
Because the cosine similarity function u k ( x ) is symmetric around the mean value µ u k ( x )
= w k : u k ( µ u k − x ) =
u k ( µ u k + x ) , it follows that p k ( µ u k − x ) =
exp( u k ( µ u k − x ))
=
exp( u k ( µ u k + x ))
=
p k ( µ u k + x ) .
Therefore, p k ( x ) = exp ( u k ( x )) does have the sought property of symmetry, around µ p k
( x ) .
In point 1 of this list we have also shown that µ u k ( x ) = µ p k ( x ) , thus p k is symmetric around its own mean µ p k
( x ) .
The reason why the softmax output y k is symmetric around the same mean as p k consists in the following arguments:
(a) The numerator p k of the y k softmax in Eq.
24 is symmetric around µ p k ( x ) , as was shown in the preceding point.
(b) The denominator of Eq.
24, i.e. (cid:80) Kl =1 exp ( p l ( x ))
P ( C l ) is also symmetric around µ p k
( x ) in O pk , where O pk is the "support" of the p k distribution, i.e. the region where the numerator p k is not negligible.
This
is
because:
i. We assume that the data is distributed on the unit hypersphere of the input space according to p ( x ) without a bias.
Therefore the total contribution of components (cid:80) l (cid:54) =
k p l ( x ) P ( C l ) to p ( x ) in that neighbourhood
O pk is approximately symmetric around µ p k
( x ) .
ii.
(cid:80) l (cid:54) =
k p l ( x ) P ( C l ) is not only approximately symmetric, but also its remaining asymmetry has a negligible contribution to p ( x ) in O pk , because p ( x ) in O pk is mostly determined by p k ( x ) .
This is true, because as we showed in point 3a, O pk ∩ O pl is a narrow overlap ∀ l (cid:54)
=
k . iii.
p k is also symmetric, therefore, the total sum (cid:80) Kl =1 exp ( p l ( x ))
P ( C l ) is symmetric around µ
p k
( x ) .
(c)
The inverse fraction 1 f of a symmetric function f is also symmetric around the same mean, therefore the inverse of the denominator
[Equation]
is also symmetric around the mean value µ p k ( x ) .
(d) The normalized product of two distributions that are symmetric around the same mean is a probability distribution with the same mean.
We prove this formally in Theorem A.1 and in its Proof at the end of the present
Appendix A.
Therefore y k = p k 1 p ( x ) is indeed symmetric around µ y k
( x ) =
µ u k ( x ) =
w k = µ p k ( x ) .
In summary,
[Equation]
and, due to the symmetry,
[Equation]
Because the means of y k ( x ) and of p k ( x ) are both equal to µ p k ( x ) , and because both distributions are symmetric around that mean, the probability distribution y k
( x ) p k ( x ) P ( C k ) /I
k , where I k is the normalization factor, also has a mean µ y k p k ( x ) equal to µ p k
( x ) :
[Equation]
We prove this formally in Theorem A.1 and in its Proof at the end of the present Appendix
A.
Therefore, the ﬁrst component of the sum in Eq.
26 is
[Equation]
and, similarly, the second component is
[Equation]
From the above conclusions about the two components of the sum in Eq.
26, it follows that
[Equation]
Therefore, it is indeed true that
[Equation]
Thus, the optimal weights of the model opt w ∗ k = µ pk ( x ) || µ pk ( x ) || ∀
k are equilibrium weights of the
SoftHebb plasticity rule and network.
However, it is not yet clear that the weights that are normalized to a unit vector are those that the rule converges to, and that other norms of the vector are unstable.
We will now give an intuition, and then prove that this is the case.
The multiplicative factor u k is common between our rule and Oja’s rule (Oja, 1982).
The effect of this factor is known to normalize the weight vector of each neuron to a length of one (Oja, 1982), as also shown in similar rules with this multiplicative factor (Krotov & Hopﬁeld, 2019).
We prove that this is the effect of the factor also in the SoftHebb rule, separately in Theorem A.2 and its Proof, provided at the end of the present Appendix A.
This proves Theorem 2.5, and satisﬁes the optimality condition derived in Theorem 2.3.
Proof of Theorem 2.6 .
Similarly to the Proof of Theorem 2.5, we ﬁnd the equilibrium parameter w 0 k of the SoftHebb plasticity rule.
[Equation]
Therefore,
[Equation]
which proves Theorem 2.6 and shows the SoftHebb plasticity rule of the neuronal bias ﬁnds the optimal parameter of the Bayesian generative model as deﬁned by Eq.
5 of Theorem 2.3.
Theorem A.1.
Given two probability density functions (PDF) y
( x ) and p ( x ) that are both centred symmetrically around the same mean value µ , their product y ( x ) p ( x ) , normalized appropriately, is a
PDF with the same mean, i.e.
[Equation]
Proof of Theorem A.
[Equation]
[Equation] [Equation]
Therefore,
[Equation]
where I = (cid:82) + ∞
−∞ y ( x ) p ( x ) dx .
Theorem A.2.
The equilibrium weights of the SoftHebb synaptic plasticity rule of Eq.
8 are implicitly normalized by the rule to a vector of length 1.
Proof of Theorem A.2 .
Using a technique similar to (Krotov & Hopﬁeld, 2019), we write the SoftHebb plasticity rule as a differential equation
[Equation]
The derivative of the norm of the weight vector is
[Equation]
Replacing d w k dt in this equation with the SoftHebb rule of Eq.
61, it is
[Equation]
This differential equation shows that the derivative of the norm of the weight vector increases if || w k || < 1 and decreases if || w k || > 1 , such that the weight vector tends a sphere of radius 1 , which proves the Theorem.
A PPENDIX B D ETAILS TO Alternate activation functions (S ECTION 2.7)
Theorem 2.3, which concerns the synaptic plasticity rule in Eq. 8, was proven for the model of
Deﬁnition 2.2, which uses a mixture of natural exponential component distributions, i.e. with base e
(Eq. 4):
[Equation] [Equation]
This implied an equivalence to a WTA neural network with natural exponential activation functions (Section 2.4).
However, it is simple to show that these results can be extended to other model probability distributions, and thus other neuronal activations.
Firstly, in the simplest of the alternatives, the base of the exponential function can be chosen differently.
In that case, the posterior probabilities that are produced by the model’s Bayesian inference, i.e. the network outputs, Q ( C k | x ; w ) =
y k ( x ; w ) are given by a softmax with a different base.
If the base of the exponential is b , then
[Equation]
It is obvious in the Proof of Theorem 2.3 in Appendix A that the same proof also applies to the changed base, if we use the appropriate logarithm for describing KL divergence.
Therefore, the optimal parameter vector does not change, and the SoftHebb plasticity rule also applies to the
SoftHebb model with a different exponential base.
This change of the base in the softmax bears similarities to the change of its exponent, in a technique that is called Temperature Scaling and has been proven useful in classiﬁcation (Hinton et al., 2015).
Secondly, the more conventional type of Temperature Scaling, i.e. that which scales exponent, is also possible in our model, while maintaining a Bayesian probabilistic interpretation of the outputs, a neural interpretation of the model, and the optimality of the plasticity rule.
In this case, the model becomes
[Equation]
The Proof of Theorem 2.3 in Appendix A also applies in this case, with a change in Eq. 14, but resulting in the same solution.
Therefore, the SoftHebb synaptic plasticity rule is applicable in this case too.
The solution for the neuronal biases, i.e. the parameters of the prior in the Theorem (Eq. 5), also remains the same, but with a factor of T : opt
w 0 k = T
ln P ( C k ) .
Finally, and most generally, the model can be generalized to use any non-negative and monotonically increasing function h ( x ) for the component distributions, i.e. for the activation function of the neurons, assuming h ( x ) is appropriately normalized to be interpretable as a probability density function.
In this case the model becomes
[Equation]
Note that there is a change in the parametrization of the priors into a multiplicative bias w 0 , compared to the additive bias in the previous versions above.
This change is necessary in this general case, because not all functions have the property e a + b = e a · e b that we used in the exponential case.
We can show that the optimal weight parameters remain the same as in the previous case of an exponential activation function, also for this more general case of activation h .
It can be seen in the
Proof of Theorem 2.3, that for a more general function h ( x ) than the exponential, Eq. 14 would instead become:
[Equation]
where g ( x ) =
ln h ( x ) .
We have assumed that h is an increasing function, therefore g is also increasing.
The cosine similarity is symmetrically decreasing as a function of x around w k .
Therefore, the function g (cid:48)
( x ) =
g (cos( w k , x )) also decreases symmetrically around w k .
Thus, the mean of that function g (cid:48) under the probability distribution p k is maximum when µ p k = w k .
As a result, Eq.implies that in this more general model too, the optimal weight vector is opt w k
=
c · µ p k ( x ) , c ∈ R , and, consequently, it is also optimized by the same SoftHebb plasticity rule.
The implication of this is that the SoftHebb WTA neural network can use activation functions such as rectiﬁed linear units (ReLU), or other non-negative and increasing activations, such as rectiﬁed polynomials (Krotov & Hopﬁeld, 2019) etc., and maintain its generative properties, its Bayesian computation, and the theoretical optimality of the plasticity rule.
A more complex derivation of the optimal weight vector for alternative activation functions, which was speciﬁc to ReLU only, and did not also derive the associated long-term plasticity rule for our problem category (Deﬁnition 2.1), was provided by Moraitis et al.
A PPENDIX C C ROSS - ENTROPY AND TRUE CAUSES , AS OPPOSED TO LABELS
It is important to note that, in labelled datasets, the labels that have been assigned by a human supervisor may not correspond exactly to the true causes that generate the data, which SoftHebb infers.
For example, consider MNIST.
The 10 labels indicating the 10 decimal digits do not correspond exactly to the true cause of each example image.
In reality, the cause C of each MNIST example in the sense implied by causal inference is not the digit cause itself, but a combination of a single digit cause D , which is the MNIST label, with one of many handwriting styles S .
That is, the probabilistic model is such that in the Eq.
P ( x ) = (cid:80) k P ( x | C k ) P
( C k ) of Deﬁnition 2.1, the cause C of each sample is dual, i.e. there exists a digit D d ( d ∈ [0 , 9]) and a style S s such that
[Equation]
and P
[Equation]
This is important for our unsupervised model.
To illustrate this point, a network with K competing neurons trained on MNIST may learn not to specialize to K digits D , but rather to K handwriting styles S of one digit D d , or in general K combinations of digits with styles – combinations, which are the true causes C that generate the data.
This leads in this case to a mismatch between the labels
D , and the true causes C of the data.
Therefore, given the labels and not the causes, it is not obvious which number K should be chosen for the number of neurons.
Practically speaking, K can be chosen using common heuristics from cluster analysis.
It is also not obvious how to measure the loss of the WTA network during the learning process, since the ground truth for causes C is missing.
We will now provide the theoretical tools for achieving this loss-evaluation based on the labels.
Even though SoftHebb is a generative model, it can be used for discrimination of the input classes C k , using Bayes’ theorem.
More formally, the proof of Theorem 2.3 involved showing that SoftHebb minimizes the KL divergence of the model q ( x ) from the data p ( x ) .
Based on this it can be shown that the algorithm also minimizes its cross-entropy H causesQ :
= H ( P ( C ) , Q ( C | x )) of the causes
Q ( C k | x ) that it infers, from the true causes of the data P ( C k ) : w SoftHebb = arg min w H causesQ .
An additional consequence is that by minimizing H causes , SoftHebb also minimizes its label-based cross-entropy H labelsQ :
= H ( P ( D d ) , Q ( D d )) between the true labels P ( D d ) and the implicitly inferred labels Q ( D d ) :
[Equation]
This is because, in Eqs.
70 and 71, the dependence of the labels on the true causes P ( D d | C k ) is ﬁxed by the data generation process.
To obtain Q ( D d | x ) and measure the cross-entropy, the causal structure P ( D d | C ) is missing, but it can be represented by a supervised classiﬁer Q 2 ( D d | Q ( C | x )) of SoftHebb’s outputs, trained using the labels D d .
Therefore, by (a) unsupervised training of
SoftHebb, then (b) training a supervised classiﬁer on top, and ﬁnally (c1) repeating the training of
SoftHebb with the same initial weights and ordering of the training inputs, while (c2) measuring the trained classiﬁer’s loss, we can observe the cross-entropy loss H labels of SoftHebb while it is being minimized, and infer that H causes is also minimized (Eq. 72).
We call this the post-hoc cross-entropy method, and we have used it in our experiments (Section 3.2 and Fig.
1 C and D) to evaluate the learning process in a theoretically sound manner.