Deep learning in stochastic stabilized supralinear networks
Anonymous Author(s)
AffiliationAddress
Abstract
There continues to be a trade-off between the biological realism and performanceof neural networks.
Contemporary deep learning techniques allow neural networksto be trained to perform challenging computations at (near) human-level, but thesenetworks typically violate key biological constraints.
More detailed models ofbiological neural networks can incorporate many of these constraints but typicallysuffer from subpar performance and trainability.
Here, we narrow this gap bydeveloping an effective method for training a canonical model of cortical neuralcircuits, the stabilized supralinear network (SSN), that in previous work had tobe constructed manually or trained with undue constraints.
SSNs are particularlychallenging to train for the same reasons that make them biologically realistic:they are characterized by strongly-connected excitatory cells and expansive firingrate non-linearities that together make them prone to dynamical instabilities unlessstabilized by appropriately tuned recurrent inhibition.
Our method avoids suchinstabilities by initializing a small network and gradually increasing network sizevia the dynamics-neutral addition of neurons during training. We first show how
SSNs can be trained to perform typical machine learning tasks by training an SSNon MNIST classification.
We then demonstrate the effectiveness of our method bytraining an SSN on the challenging task of performing amortized Markov chain
Monte Carlo-based inference under a Gaussian scale mixture generative model ofnatural image patches with a rich and diverse set of basis functions – somethingthat was not possible with previous methods.
These results open the way to trainingrealistic cortical-like neural networks on challenging tasks at scale.
Introduction
Biological neural network models of the brain have been proposed and studied long before theadvent of machine learning [1–3].
The primary goal of these models was to understand someof the fundamental network mechanisms employed by the brain but were not capable of actuallyachieving the brain’s performance on its wide range of complex computations.
More recently,significant progress has been made in the related field of artificial neural networks, especially atachieving competitive performance in a multitude of image, language and speech-related tasks.
While biologically-motivated models may occasionally find success in these settings [4, 5], recentadvances in state-of-the-art models have mainly been artificial
This has led to investigationson improving their potential biological-plausibility [9–11], usually at a cost on performance [12].
In systems neuroscience, neural networks have also been trained on laboratory tasks employed intypical experiments [13, 14].
These approaches provided important insights into the contributions ofexperimentally found macroscopic neural dynamics and representations to the successful performanceof such tasks.
For example, they described the presence of line attractors during context-dependent evidence accumulation [15], mixed task selectivity for flexible task performance [16], and gridcell responses for path integration [17] and spatial localization [18].
However, these networksdid not incorporate some of the most salient constraints on the detailed organization of corticalcircuits.
For example, they were purely feedforward [19], utilized neuronal transfer functions thatwere either outright saturating (e.g. tanh) or at best lacked the superlinear characteristics of corticalneurons (as in ReLUs, or the rectified softplus)
[19, 13, 16, 20, 15], had noiseless dynamics [14], noseparation of excitatory and inhibitory cells [19, 20, 15], and sometimes even employed artificialgating mechanisms such as LSTMs [17].
Due to these properties, these networks remained abstractedaway from biological neurons in key aspects, and hence offered limited insight into the neuron-levelmechanisms that drive their computations.
To bring the modelling of neural circuit mechanisms underlying challenging computations at (or near)single-cell resolution within reach, we develop a novel method to train an experimentally supportedneural network model, the stabilized supralinear network (SSN) [21], on challenging tasks.
Critically,the SSN satisfies many of the key constraints of cortical circuits: it has separate but recurrentexcitatory and inhibitory populations, expansive (rectified power-law) single neuron non-linearities,realistic single-neuron time constants, and no reliance on artificial gating mechanisms.
In its originalform, the SSN continues a venerable tradition of handcrafted excitatory-inhibitory (E-I) networks,whose dynamics have been extensively studied [22, 23], and that helped reveal key consequencesof the cortex operating in a balanced E-I regime [3, 24–26].
In particular, the SSN accounts for theexperimentally-observed effects of stimulus tuning, sublinear response summation and surroundsuppression of neural responses in sensory cortices [27, 28], as well as oscillations, bistable andpersistent responses [29].
Furthermore, when extended to include noise, the resultant stochastic SSN produces realistic, stimulus-modulated population-wide patterns of noise variability [30].
Thiscombination of features makes the SSN an ideally suited candidate for developing cortical-likemodels of neural computations.
However, the same features that make the SSN an attractive substrate for its biological realism, alsomake it particularly challenging to train it.
This is because, in SSNs, excitatory cells are typicallystrongly connected with one-another in order to implement non-trivial nonlinear transformations oftheir inputs, such as divisive normalization, which underlie many of the experimentally observedphenomena for which they account.
In addition, single neuron non-linearities are expansive to reflectthe experimental finding that cortical neurons almost exclusively use the convex (non-saturating) partof their firing rate non-linearities under physiological conditions, including for the strongest stimuli[31] (even if they can be trivially driven to saturation by artificial stimuli, due to the refractory periodof the mechanism generating action potentials).
Thus, stable dynamics cannot be achieved by relyingon saturating single neuron non-linearities as in some previous work [32].
These properties make SSNs particularly susceptible to dynamical instabilities resulting in run-away excitation.
This isusually prevented by appropriately tuned recurrent inhibition from inhibitory cells, requiring carefultuning of parameters [21, 29].
At the same time, neural noise can further exacerbate the instabilityproblem as even if appropriate E-I balance is achieved on average, a momentary imbalance dueto noise may still snowball into a full-fledged explosion of activities.
This makes the training ofsuch networks difficult: as the training procedure explores different parts of the parameter space,it will likely visit parameter settings which disrupt E-I balance and thus lead to instability and animmediate failure of training.
Indeed, in the few cases in which the training of SSNs was attempted,either noiseless neurons were used [33, 34], or the network was so heavily under-parameterized (15parameters for a total of 100 neurons) that it substantially limited its expressivity [35].
Here, we face this challenge by developing an approach that allows us to employ deep learningalgorithms for task optimization while still staying within an inhibition-stabilized parameter regimeof the SSN.
Additionally, the effectiveness of our method allows SSNs to be trained for general tasksbeyond specific narratives in neuroscience, which we demonstrate by training an SSN (a corticalneural circuit) for MNIST classification [36] (an artificial deep learning task) and achieving onlyslightly lower accuracy scores.
To demonstrate the superiority of our approach in a directly comparable way, we train an SSN toperform the same probabilistic inference task as an earlier attempt for training stochastic SSNs [35].
This task requires the SSN to act as the recognition model for a probabilistic generative model ofnatural image patches, the Gaussian scale mixture generative model (GSM) [37, 38], by performingamortized approximate Markov chain Monte Carlo-based inference.
That is, in response to an input(observation), the network needs to produce trajectories in the state space of its neurons (encoding latent variables of the GSM)
such that the distribution of multi-neuron response patterns sampled bythese trajectories approximately matches the joint posterior distribution of the GSM for correspondinginput.
Notably, the network needs to achieve this with a single set of parameters (prominently,synaptic weights) for a large number of inputs.
This is a challenging task because the objectiverequires the network to maintain finite levels of variability while modulating the co-variability of itsneurons in a stimulus-dependent way, which is only possible if stochasticity of neural noise is notsuppressed and recurrent connections are sufficiently strong – a combination that pushes the networktowards instability.
We show that our approach is able to successfully train SSNs that have threeorders of magnitude more parameters than what was previously possible to optimize.
Our trained SSN performs accurate inference under a GSM with a set of basis functions that is sufficiently rich forallowing high-quality reconstructions of CIFAR-10 [39] images.
This GSM represents an impossibletarget for previous, heavily constrained approaches for training SSNs.
These results validate ourapproach and open the way to training SSNs for a large variety of different tasks, and to using theresulting networks to study the circuit mechanisms of neural computations at a single cell resolution.
Methods
2.1 Stochastic supralinear stabilized network
The SSN is a canonical model of cortical circuits that is able to explain a variety of neural responsecharacteristics [21, 27] through its straightforward dynamics described by,
[Equation]
(1) where neurons are characterized by their membrane potentials u and time constants τ .
f ( h ) represents a non-linear transformation of the external input h ,
[Equation]
(2) for parameters θ h
= { θ 1 , θ 2 , θ 3 } .
W denotes the network weights, which transmit neuron firingrates r , given by a threshold expansive non-linearity,
[Equation]
(3) for γ > 1 and constant k .
Note that there is no built-in upper saturation to the model, whichpresents potential instability problems – excitations within the network can exponentially growunbounded through mutual and self excitation.
The network relies purely on inhibition to stabilize,which therefore restricts the parameter space in which the network is stable.
The network distinctlycomprises of excitatory and inhibitory neurons, which in turn imposes restrictions on the polarity ofthe elements of W .
Specifically, the columns of W must have the same sign (positive for excitatoryneurons, negative for inhibitory neurons).
Finally, η represents noise, correlated across time withtime constant τ η , as well as across neurons with covariance Σ η .
2.2 Theoretical motivation
We note that under the assumption of reasonable noise characteristics, it is always possible to add anew neuron to an existing stable SSN while preserving stability in a non-trivial manner.
We begin bydefining the newly-added neuron as a "twin" of an existing neuron. The new twin needs to have thesame incoming weights as its original counterpart.
This way, the newly-added neuron is guaranteedto be stable, since it is receiving the same input as an existing stable neuron. Secondly, all outgoingweights of both twins are halved, so that other neurons in the network now receive half outputfrom two identical neurons, effectively keeping the total output unchanged.
These other neuronsare therefore still stable as they were originally.
When implemented together, the result is a stablenetwork with an extra neuron. We refer to this process as a dynamics-neutral addition of a neuron to anetwork.
This idea forms the mathematical foundation of our proposed training method. Formally, letneuron n be the cloning candidate.
The membrane potential vector u now has an additional element,
[Equation] [Equation]
[Figure]
initialization phase unstable stable where ¬ n represents the indices of all neurons that are not n . Similarly, we split the weight matrix Winto four parts, { W ( ¬ n, ¬ n ) , W ( ¬ n,n ) , W ( n, ¬ n ) , W ( n,n ) } We update the network weights according 137 to,
[Equation]
The synaptic input into all neurons is now given by,
[Equation]
where,
[Equation]
which is the same input received before the addition of the new neuron, as claimed. Note that duringoptimization, symmetry between the twin neurons are broken due to them receiving statistically equalbut nonetheless uniquely-generated noise inputs.
The external input may also be different dependingon the training objective.
These factors wear down the absolute guarantee of stability, but it is stillreasonable to build on this intuitively high chance of stability as the non-linearity of SSN networkdynamics makes any further analytical analyses difficult.
Even with a reliable way of growing a network, we still explicitly require an existing (smaller) networkas a starting point. By "reverse-induction", we recognize that the simplest and most reliable solutionis to initialize the whole process by generating the smallest SSN possible – a network with onlyone excitatory and one inhibitory neuron. This can be achieved by brute-force or by some targetedinitialization specific to the training objective in order to get a head-start on performance optimization.
The full network can then be gradually constructed by adding neurons in a dynamics-neutral way aspreviously described. The entire training method is summarized in Fig 1.
[Figure] 2.4 Training reliability In order to quantify the reliability of our training method, we propose and calculate three differentmetrics that will help to show how our method is effective against instability problems typicallyencountered in training SSNs.
Proportion unstable in separate instances of network building.
We repeat the entire
training method, starting from re-initializing a small network to fully-building it up in a total of 100 separate instances
and counted the proportion of unstable trials.2. Proportion unstable when shuffling weights of stable networks.
In each of the 100instances, we shuffled the elements of the same type (E-E, E-I, I-E and I-I) within the weightmatrix for 100 trials and counted the number of unstable configurations. We then report themean and standard error over all instances.3.
Proportion unstable of randomly generated networks.
We randomly generate 10000initializations of the full network and counted the proportion of unstable configurations. The elements of the weight matrices are initialized such that the variance of their absolutevalues match that of our trained stable network. We then enforce polarity rules according to
whether the connecting neurons are excitatory or inhibitory.  MNIST Classification Task We seek to demonstrate how SSNs can be generalized for typical deep learning tasks by training an
SSN to perform MNIST classification [36]. The nature of its dynamics in (1) implies that neuralactivity evolves over time (at a rate determined by the time constants) in response to external inputinstead of producing an instant output.
The presence of noise further complicates the accessibilityof obtaining a straightforward readout from the network. Naturally, these issues are ever-present inexperimental neural recordings and contribute to the biological-realism of the model. In artificialneural networks, an input is processed across different layers according to their respective networkarchitectures and the result is an immediate output afterwards. As such, in order to train an SSN like
[Table]
an artificial neural network, we choose to evolve the network over time until it reaches steady-state,and pass its mean membrane potentials into a readout (softmax) layer in order to produce predictions.
An overview of the training objective is depicted in Fig 2.
We aim to train an SSN comprising of40 excitatory and 10 inhibitory neurons. Only the excitatory neurons will receive external input (i.e. MNIST images) and be decoded in the readout layer, although the entire network (including theinhibitory neurons) is evolved over time, as is commonly observed in biology. In order to transform
MNIST images into input for the SSN, we optimize an autoencoder (also comprising of 3 layers) toreduce input dimensionality from 784 to 40.
As a base comparison, we also separately train 3-layer
MLPs on both the original data and the encoded data from the autoencoder.
Note that we only use10000 images as our training data, and use the remaining 50000 images as our test data.
Training results are summarized in Table 1. We see that our dimensionality-reduction step has notsignificantly impacted the performance of the MLP, meaning that we have retained most of theinformation within the encoded images. Our trained SSN also performs at only slightly lower levels,despite having noise injected at every time step and also having to maintain dynamic stability. Inall 100 separate networks that we have built, none of the instances encountered instability problems.
Furthermore, we observe that it is virtually not possible to initialize a stable random configuration.  Amortized Inference under Generative Model
We now shift our focus to a biological narrative.
For a more direct comparison with our method,we will train an SSN to perform amortized sampling-based probabilistic inference under the GSM –something that was previously attempted with limited success [35]. An overview of the GSM andtraining objective of the SSN is summarized in Fig 3.
4.1 Gaussian scale mixture model
The GSM [37, 38] is proposed as an internal generative model of natural image patches in V1.
Animage x is generated by,
[Equation]
8) for projective fields A , latent activations y , global scalar contrast level z and pixel noise η x . Theprojective fields are 2-D Gabor filters, each of which is characterized by four parameters,
[Equation]
where x ′ and y ′ are pixel coordinates rotated by ϕ .
The parameters of A are summarized asθ A = { x 0 , y 0 , ϕ , σ } .
Note that A is a two-dimensional matrix with each row corresponding to +
GSM
[Figure]
prior
[Figure]
noise latent activations latent prior
[Figure]
[Figure]
fields) × [Equation] × input function neural network stochastic SSN network weights noise covariance the flattened pixels of single 2-D Gabor filter. An ideal observer, when presented with an image x ,computes a posterior distribution over y and z by incorporating their respective priors p ( y ) and p ( z ) ,
[Equation]
after which the contrast level z is marginalized, resulting in a posterior distribution over y ,
[Equation]
We construct our GSM by first maximizing fraction of variance explained by the projective fieldson CIFAR-10 images and subsequently optimize the remaining model parameters by maximizinglikelihood of observing the images. Additional details can be found in Appendix A. 4.2 Sampling-based probabilistic inference Given an observed image as input, the membrane potential response of the SSN, as determined by (1),is optimized to represent statistical samples for the posterior distribution over the latent variables ofthe GSM, as computed by an ideal observer using (13).
This inference method is therefore describedas "sampling-based".
We implement this by optimizing the model with respect to the mean-squarederrors between the first and second order moments of both distributions,
[Equation]
for constant coefficients λ µ and λ Σ .
In this case, both excitatory and inhibitory neurons receive anexternal input corresponding to its respective receptive fields, but similar to the previous example,only the membrane potentials of excitatory neurons are considered in the computation of the costfunction. The mathematical form of the cost is motivated by experimental practice – neural data isoften averaged over time or across trials in order to reduce noise. More recently, neural variability isproposed to also be computationally meaningful [38], and we thus seek to understand the dynamicsof how the SSN expresses its mean and variance. [Table]
4.3 Results
To provide a direct comparison, we train an SSN for the GSM constructed in [35] (denoted as SSN 1 ).
We then trained another SSN on our in-house GSM (denoted as SSN 2 ). Since target momentsare different for the two GSMs, we compute a normalized cost function for approximately faircomparison,
[Equation] In order to fully appreciate these results, we briefly describe the GSM-SSN pairs in each row of Table 2 and how they are trained.
A full comparison can be found in Appendix B.1. The first row describes the SSN from [35] optimized for the GSM from the same authors. This SSN is highly constrained; for example, the entire 100 × 100 weight matrix
is parame- terized by only 8 parameters. The same 5 rotationally-symmetric images are used for both training and testing.
Optimization is first done using Adam for a small number of iterations, followed by a biased semi-analytical method described in [40] (as well as Appendix B).
The
normalized cost of this model is set to 1 by definition
The SSN in the second row is trained by our method, with the same 5 images (as theprevious row) used as the training and test set.
The weight matrix is constrained to be block-symmetric-circulant in order to fully exploit the rotational symmetry of the images.
Each symmetric-circulant 50 × 50 block is parameterized by 26 parameters, giving a total of 104 parameters for the weight matrix.
The additional degrees of freedom has resulted in nearly an order of magnitude lower cost.
The SSN in the third row is trained on 10000 images from CIFAR-10, while the remaining 40000 images serve as the test set.
The images have no symmetry, and the 100 × 100 weight
matrix is freely represented by 10000 parameters.
Despite the increase in difficulty of the
task (matching a more complex GSM), our method was able to produce a network that
achieves a normalized cost lower than the attempt in [35].
In addition to obtaining better performance, our method is also able to consistently avoid instabilitiesduring training. Once again, in all 100 independent instances, not a single instance went unstable.
The full network
is
also unable to be randomly initialized in a stable manner, which
further motivates our approach.
Discussion
Network stability during training is a problem faced in both artificial and biological neural networks.
Typically, artificial neural networks (ANNs) can implement quick-fix solutions that will not negativelyimpact its performance, such as applying batch and layer normalizations [41–43].
Furthermore,artificial neurons may be designed to be stable, such as simply choosing a saturating activationfunction (tanh,sigmoid), or other ways of implementing bounded-input-bounded-output designs.
There also exists ANNs that are challenged by instabilities that cannot be trivially solved [44, 45],which could motivate similar network-building solutions.
In general, the concept of building networks during training has already been proposed in ANNs[46–48].
One such study combined both growing and pruning to produce efficient networks [47]. Another training approach decides on an existing neuron in the network to be split into multipleneurons [46]. Another proposed algorithm [48] involves a similar principle of adding nodes withoutimpacting existing network capabilities.
Broadly speaking, the two primary goals of these algorithmsare to reduce training load and improve performance, which are perfectly reasonable. Here, we areinstead targeting an even more basic need of "survival" by staying within a stable parameter regime,achieved using our novel method of dynamics-neutral neuron duplication.
On the same train of thought, it can be meaningful to explore the tuning of certain hyperparameters inour method, such as the number of neurons to duplicate at the same time, or the criteria in selectingwhich neuron to duplicate.
At least for the two tasks presented here, we are already able to achieve100% stability using our straightforward method, and we are therefore less motivated to explorehyperparameter settings that further optimizes for stability. On the other hand, now that stability isguaranteed, one good reason to refine the method would be to improve network performance, whichmotivates future work. It is also worthwhile to consider an orthogonal approach of adding termsto the cost function such as regularization and explore its effects on keeping network optimizationwithin a stable parameter regime.
Throughout our analysis, we have been using Adam [49] to train SSNs (additional miscellaneoustraining information can be found in Appendix C).
This is because our primary objective is to buildbiologically-realistic networks so as to study their dynamics and network characteristics after theyhave been optimized.
In other words, our objective is the end goal, and not necessarily the journey.
However, it might still be worth exploring and understanding how biologically-plausible learning canfit into our current paradigm in future investigations.
In the fast-evolving research landscape of machine learning, speed and scalability are two of themost important criteria that determines how effective an algorithm is.
In our case, building a networkup during optimization goes against these principles, because that would require computationallyexpensive function retracing operations every time the network changes in design.
We justify thiswith two reasons.
Firstly, to the best of our knowledge, there is presently no other effective way oftraining SSNs.
Secondly, the computational resources spent on optimization is still much greaterthan the resources spent on tracing, especially since there is only a finite number of tracing required(up till the network is fully built), but potentially an unlimited number of gradient computations andoptimization iterations.
On the other hand, we may also improve on our method by minimizing thenumber of retracing operations required (for example, by adding more neurons per building step).
Itwill also be meaningful to perform investigations on how to maintain the balance between networkbuilding and its associated computational costs.
Conclusion
We present an effective method of training the SSN, a canonical model of cortical circuits which ishighly prone to dynamic instabilities, without simplifying assumptions or undue constraints.
We showhow the SSN can be trained to perform the task of classifying MNIST images, thereby clearly showingthe generality of both the SSN and our method to train it.
We also achieve better performance thanprevious approaches in the challenging task of training SSNs to perform sampling-based probabilisticinference. In repeated trials of the two tasks, our method has not produced any unstable networks,thereby attesting to its robustness.
Our success opens up the possibility of training SSNs on othermeaningful cognitive tasks in hopes of unraveling biological network dynamics underlying complexcomputations performed by the brain.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] In our abstract and introduction, we claim that we
have developed a deep learning approach to train supralinear stabilized networks
This method can be found in section 4.
We also claim that we have successfully trainedthe network to perform sampling-based inference on an internal generative model ofnatural images.
These results can be found summarized in Fig 3.
Finally, we claimthat our method is an order of magnitude better than previous approaches.
This can befound in Table 2.
(b) Did you describe the limitations of your work?
[Yes] We have highlighted somepotential limitations in a dedicated section, see section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] We have
not found any potential negative impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The link to ourcode can be found in the supplementary material.
Additional information required to replicate our results are included in the supplementary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] All parameters of the model are clearly introduced early in the
text.
Values of non-optimized parameters can be found in the supplementary materials.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
N/A
(d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] Computational resources and
related information can be found in the supplementary material.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We have used
CIFAR-10 and MNIST and have cited the creators.
(b) Did you mention the license of the assets?
[Yes] We have linked the licenses in the
supplementary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] CIFAR-10 and MNIST are publicly available datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] This can be found in the supplementary.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]