Deep Learning in Stochastic Stabilized Supralinear Networks
Anonymous Author(s)
AffiliationAddress email
Abstract
There continues to be a trade-off between the biological realism and performanceof neural networks.
Contemporary deep learning techniques allow neural networksto be trained to perform challenging computations at (near) human-level, but thesenetworks typically violate key biological constraints.
More detailed models ofbiological neural networks can incorporate many of these constraints but typicallysuffer from subpar performance and trainability.
Here, we narrow this gap bydeveloping an effective method for training a canonical model of cortical neuralcircuits, the stabilized supralinear network (SSN), that in previous work had tobe constructed manually or trained with undue constraints.
SSNs are particularlychallenging to train for the same reasons that make them biologically realistic:they are characterized by strongly-connected excitatory cells and expansive firingrate non-linearities that together make them prone to dynamical instabilities unlessstabilized by appropriately tuned recurrent inhibition.
Our method avoids suchinstabilities by initializing a small network and gradually increasing network sizevia the dynamics-neutral addition of neurons during training.
We first show how
SSNs can be trained to perform typical machine learning tasks by training an SSNon MNIST classification.
We then demonstrate the effectiveness of our method bytraining an SSN on the challenging task of performing amortized Markov chain
Monte Carlo-based inference under a Gaussian scale mixture generative model ofnatural image patches with a rich and diverse set of basis functions – somethingthat was not possible with previous methods.
These results open the way to trainingrealistic cortical-like neural networks on challenging tasks at scale.
Introduction
Biological neural network models of the brain have been proposed and studied long before theadvent of machine learning [1–3].
The primary goal of these models was to understand someof the fundamental network mechanisms employed by the brain but were not capable of actuallyachieving the brain’s performance on its wide range of complex computations.
More recently,significant progress has been made in the related field of artificial neural networks, especially atachieving competitive performance in a multitude of image, language and speech-related tasks.
While biologically-motivated models may occasionally find success in these settings [4, 5], recentadvances in state-of-the-art models have mainly been artificial
This has led to investigationson improving their potential biological-plausibility [9–11], usually at a cost on performance [12].
In systems neuroscience, neural networks have also been trained on laboratory tasks employed intypical experiments [13, 14].
These approaches provided important insights into the contributions ofexperimentally found macroscopic neural dynamics and representations to the successful performanceof such tasks.
For example, they described the presence of line attractors during context-dependent evidence accumulation [15], mixed task selectivity for flexible task performance [16], and grid cellresponses for path integration [17] and spatial localization [18].
However, these networks did notincorporate some of the most salient constraints on the detailed organization of cortical circuits.
To bring the modeling of neural circuit mechanisms underlying challenging computations at (ornear) single-cell resolution within reach, we develop a novel method to train an experimentallysupported neural network model, the stabilized supralinear network (SSN) [21].
Critically, the SSNsatisfies many of the key constraints of cortical circuits: it has separate but recurrent excitatoryand inhibitory populations, expansive (rectified power-law) single neuron non-linearities, realisticsingle-neuron time constants, and no reliance on artificial gating mechanisms.
In its original form,the SSN continues a venerable tradition of handcrafted excitatory-inhibitory (E-I) networks, whosedynamics have been extensively studied [22, 23], and that helped reveal key consequences of thecortex operating in a balanced E-I regime [3, 24–26].
In particular, the SSN accounts for theexperimentally-observed effects of stimulus tuning, sublinear response summation and surroundsuppression of neural responses in sensory cortices [27, 28], as well as oscillations, bistable andpersistent responses [29].
Furthermore, when extended to include noise, the resultant stochastic SSNproduces realistic, stimulus-modulated population-wide patterns of noise variability [30].
However, the same features that make the SSN an attractive substrate for its biological realism, alsomake it particularly challenging to train it.
This is because, in SSNs, excitatory cells are typicallystrongly connected with one-another in order to implement non-trivial nonlinear transformations oftheir inputs, such as divisive normalization, which underlie many of the experimentally observedphenomena for which they account.
In addition, single neuron non-linearities are expansive to reflectthe experimental finding that cortical neurons almost exclusively use the convex (non-saturating)part of their firing rate non-linearities under physiological conditions, including for the strongeststimuli [31] (even if they can be trivially driven to saturation by artificial stimuli, due to the refractoryperiod of the mechanism generating action potentials).
These properties make SSNs particularlysusceptible to dynamical instabilities resulting in run-away excitation.
This makes the training ofsuch networks difficult: as the training procedure explores different parts of the parameter space,it will likely visit parameter settings which disrupt E-I balance and thus lead to instability and animmediate failure of training.
Indeed, in the few cases in which the training of SSNs was attempted,either noiseless neurons were used [33, 34], or the network was so heavily under-parameterized thatit substantially limited its expressivity [35].
Here, we face this challenge by developing an approachthat allows us to employ deep learning algorithms for task optimization while still staying withinan inhibition-stabilized parameter regime of the SSN.
Additionally, the effectiveness of our methodallows SSNs to be trained for general tasks beyond specific narratives in neuroscience, which wedemonstrate by training an SSN (a cortical neural circuit) for MNIST classification [36] (an artificialdeep learning task) and achieving only slightly lower accuracy scores.
To demonstrate the superiority of our approach in a directly comparable way, we train an SSN toperform the same probabilistic inference task as an earlier attempt for training stochastic SSNs [35].
This task requires the SSN to act as the recognition model for a probabilistic generative model ofnatural image patches, the Gaussian scale mixture generative model (GSM) [37, 38], by performingamortized approximate Markov chain Monte Carlo-based inference.
That is, in response to an input(observation), the network needs to produce trajectories in the state space of its neurons (encodinglatent variables of the GSM)
such that the distribution of multi-neuron response patterns sampled bythese trajectories approximately matches the joint posterior distribution of the GSM for correspondinginput.
Notably, the network needs to achieve this with a single set of parameters (prominently,synaptic weights) for a large number of inputs.
This is a challenging task because the objective requires the network to maintain finite levels of variability while modulating the co-variability of itsneurons in a stimulus-dependent way, which is only possible if stochasticity of neural noise is not suppressed and recurrent connections are sufficiently strong – a combination that pushes the network towards instability.
We show that our approach is able to successfully train SSNs that have threeorders of magnitude more parameters than what was previously possible to optimize.
Our trained
SSN performs accurate inference under a GSM with a set of basis functions that is sufficiently rich forallowing high-quality reconstructions of CIFAR-10 [39] images.
This GSM represents an impossibletarget for previous, heavily constrained approaches for training SSNs.
These results validate ourapproach and open the way to training SSNs for a large variety of different tasks, and to using the resulting networks to study the circuit mechanisms of neural computations at a single cell resolution.
Methods
2.1 Stochastic supralinear stabilized network
The SSN is a canonical model of cortical circuits that is able to explain a variety of neural responsecharacteristics [21, 27] through its straightforward dynamics described by,
[Equation]
where neurons are characterized by their membrane potentials u and time constants τ .
f ( h ) represents a non-linear transformation of the external input h ,
[Equation]
) for parameters θ h = { θ 1 , θ 2 , θ 3 } .
W denotes the network weights, which transmit neuron firingrates r , given by a threshold expansive non-linearity,
[Equation]
for γ > 1 and constant k .
Note that there is no built-in upper saturation to the model, whichpresents potential instability problems – excitations within the network can exponentially growunbounded through mutual and self excitation.
The network relies purely on inhibition to stabilize,which therefore restricts the parameter space in which the network is stable.
The network distinctlycomprises of excitatory and inhibitory neurons, which in turn imposes restrictions on the polarity ofthe elements of W .
Specifically, the columns of W must have the same sign (positive for excitatoryneurons, negative for inhibitory neurons).
Finally, η represents noise, correlated across time withtime constant τ η , as well as across neurons with covariance Σ η .
2.2 Theoretical motivation
We note that under the assumption of reasonable noise characteristics, it is always possible to add anew neuron to an existing stable SSN while preserving stability in a non-trivial manner.
We begin bydefining the newly-added neuron as a "twin" of an existing neuron.
The new twin needs to have thesame incoming weights as its original counterpart.
This way, the newly-added neuron is guaranteedto be stable, since it is receiving the same input as an existing stable neuron.
Secondly, all outgoingweights of both twins are halved, so that other neurons in the network now receive half outputfrom two identical neurons, effectively keeping the total output unchanged.
These other neuronsare therefore still stable as they were originally.
When implemented together, the result is a stablenetwork with an extra neuron.
We refer to this process as a dynamics-neutral addition of a neuron to anetwork.
This idea forms the mathematical foundation of our proposed training method.
Formally, letneuron n be the cloning candidate.
The membrane potential vector u now has an additional element,
[Equation]
where ¬ n represents the indices of all neurons that are not n .
Similarly, we split the weight matrix Winto four parts, { W ( ¬ n, ¬ n ) , W ( ¬ n,n ) , W ( n, ¬ n ) , W ( n,n ) } We update the network weights according 122 to,
[Equation]
The synaptic input into all neurons is now given by,
[Equation]
where,
[Equation]
[Figure]
which is the same input received before the addition of the new neuron, as claimed.
Note that duringoptimization, symmetry between the twin neurons are broken due to them receiving statistically equalbut nonetheless uniquely-generated noise inputs.
The external input may also be different dependingon the training objective.
These factors wear down the absolute guarantee of stability, but it is stillreasonable to build on this intuitively high chance of stability.
2.3 Training method
Even with a reliable way of growing a network, we still explicitly require an existing (smaller) networkas a starting point.
By "reverse-induction", we recognize that the simplest and most reliable solutionis to initialize the whole process by generating the smallest SSN possible – a network with onlyone excitatory and one inhibitory neuron.
This can be achieved by brute-force or by some targetedinitialization specific to the training objective in order to get a head-start on performance optimization.
The full network can then be gradually constructed by adding neurons in a dynamics-neutral way aspreviously described.
The entire training method is summarized in Fig 1.
2.4 Training reliability
In order to quantify the reliability of our training method, we propose and calculate three differentmetrics that will help to show how our method is effective against instability problems typicallyencountered in training SSNs.
Proportion unstable in separate instances of network building.
We repeat the entire
training method, starting from re-initializing a small network to fully-building it up in a total
of 100 separate instances
and counted the proportion of unstable trials
Proportion unstable when shuffling weights of stable networks.
In each of the 100instances, we shuffled the elements of the same type (E-E, E-I, I-E and I-I) within the weightmatrix for 100 trials and counted the number of unstable configurations when subjectedto the first 100 images from either MNIST or CIFAR-10.
We then report the mean andstandard error over all instances.
Proportion unstable of randomly generated networks.
We randomly generate initializations of the full network and counted the proportion of unstable configurations.
The elements of the weight matrices are initialized such that the variance of their absolutevalues match that of our trained stable network.
We then enforce polarity rules according to
whether the connecting neurons are excitatory
or
inhibitory.
[Figure]
MNIST Classification Task
We seek to demonstrate how SSNs can be generalized for typical deep learning tasks by training an
SSN to perform MNIST classification [36].
The nature of its dynamics in (1) implies that neuralactivity evolves over time (at a rate determined by the time constants) in response to external inputinstead of producing an instant output.
The presence of noise further complicates the accessibilityof obtaining a straightforward readout from the network.
Naturally, these issues are ever-present inexperimental neural recordings and contribute to the biological-realism of the model.
In artificialneural networks, an input is processed across different layers according to their respective networkarchitectures and the result is an immediate output afterwards.
As such, in order to train an SSN likean artificial neural network, we choose to evolve the network over time until it reaches steady-state,and pass its mean membrane potentials into a readout (softmax) layer in order to produce predictions.
An overview of the training objective is depicted in Fig 2.
We aim to train an SSN with 80 excitatoryand 20 inhibitory neurons which is a biologically-realistic E-I ratio.
We also train an SSN with 50excitatory and 50 inhibitory neurons so that we may compare with the performance of previousmodels [35].
Only the excitatory neurons will receive external input (i.e. MNIST images) and bedecoded in the readout layer, although the entire network (including the inhibitory neurons) is evolvedover time, as is commonly observed in biology.
In order to transform MNIST images into input forthe SSN, we optimize an autoencoder (also comprising of 3 layers) to reduce input dimensionalityfrom 784 to the number of excitatory neurons.
As a base comparison, we also separately train 3-layer
MLPs and perform logistic regression on the encoded data from the autoencoder.
Training results are summarized in Table 1.
Our trained SSNs at both E-I ratios perform at onlyslightly lower levels than their corresponding MLPs, and also better than logistic regression despitehaving noise injected at every time step and also having to maintain dynamic stability.
In all 200separate instances of networks built, none of the instances encountered instability problems.
Shufflingthe weights within their connection-blocks also resulted in mostly unstable configurations, suggestingthat the magnitude of the weights do not play as much of a role as its exact distribution.
Previous models [35] are not able to reasonably perform the task because they have a highlyconstrained parameterization designed to exploit rotational symmetries in the images which do
[Table] not exist in the MNIST dataset. However, we note that it is still performing above chance level(which is 0.1). Furthermore, we observe that it is virtually not possible to initialize a stable randomconfigurations of networks in both E-I ratios.
Amortized Inference under Generative Model
We now shift our focus to a biological narrative.
For a more direct comparison with our method,we will train an SSN to perform amortized sampling-based probabilistic inference under the GSM –something that was previously attempted with limited success [35].
An overview of the GSM andtraining objective of the SSN is summarized in Fig 3.
4.1 Gaussian scale mixture model
The GSM [37, 38] is proposed as an internal generative model of natural image patches in V1.
Animage x is generated by,
[Equation]
for projective fields A , latent activations y , global scalar contrast level z and pixel noise η x .
Theprojective fields are 2-D Gabor filters, each of which is characterized by four parameters,
[Equation]
where x ′ and y ′ are pixel coordinates rotated by ϕ .
The parameters of A are summarized asθ A
= { x 0 , y 0 , ϕ , σ } .
Note that A is a two-dimensional matrix with each row corresponding tothe flattened pixels of single 2-D Gabor filter. An ideal observer, when presented with an image x ,
+
GSM
[Figure]
prior
[Figure]
noise latent activations latent prior projective fields (receptive fields)
[Equation]
input function neural network stochastic SSN network weights noise covariance computes a posterior distribution over y and z by incorporating their respective priors p ( y ) and p ( z ) ,
[Equation]
after which the contrast level z is marginalized, resulting in a posterior distribution over y ,
[Equation]
We construct our GSM by first maximizing fraction of variance explained by the projective fieldson CIFAR-10 images and subsequently optimize the remaining model parameters by maximizinglikelihood of observing the images.
Additional details can be found in Appendix A.
4.2 Sampling-based probabilistic inference
Given an observed image as input, the membrane potential response of the SSN, as determined by (1),is optimized to represent statistical samples for the posterior distribution over the latent variables ofthe GSM, as computed by an ideal observer using (13).
This inference method is therefore describedas "sampling-based".
We implement this by optimizing the model with respect to the mean-squarederrors between the first and second order moments of both distributions, [Equation] [Equation]
[Equation]
14) for constant coefficients λ µ and λ Σ .
In this case, both excitatory and inhibitory neurons receive anexternal input corresponding to its respective receptive fields, but similar to the previous example,only the membrane potentials of excitatory neurons are considered in the computation of the costfunction.
The mathematical form of the cost is motivated by experimental practice – neural data isoften averaged over time or across trials in order to reduce noise.
More recently, neural variability isproposed to also be computationally meaningful [38], and we thus seek to understand the dynamicsof how the SSN expresses its mean and variance.
[Table]
4.3 Results
We train an SSN for the GSM constructed in [35] (denoted as SSN 1 ) and train another SSN on ourin-house GSM (denoted as SSN 2 ). We do the same for existing models in [35].
Since target momentsare different for the two GSMs, we compute a normalized cost function so that we can comparenetworks trained across different GSMs,
[Equation]
In order to fully appreciate these results, we briefly describe the GSM-SSN pairs in each row of
Table 2 and how they are trained.
A full comparison can be found in Appendix B.
The first row describes the SSN from [35] optimized for the GSM from the same authors.
This SSN is highly constrained; for example, the entire 100 × 100 weight matrix is parameterized by only 8 parameters.
The same 5 rotationally-symmetric images are used for bothtraining and testing, attributing to the high computational costs associated with the method.
Optimization is first done using Adam for a small number of iterations, followed by a biasedsemi-analytical method described in [40] (as well as Appendix B).
The normalized cost ofthis model is set to 1 by definition.2.
The SSN in the second row is trained by our method, with the same 5 images (as theprevious row) used as the training and test set.
The weight matrix is constrained to beblock-symmetric-circulant in order to fully exploit the rotational symmetry of the images.
Each symmetric-circulant 50 × 50 block is parameterized by 26 parameters, giving a totalof 104 parameters for the weight matrix.
The additional degrees of freedom has resulted innearly an order of magnitude lower cost.3.
The third row shows the SSN from [35] trained on our GSM.
Due to its highly-constrainedparameterization (as mentioned in the description of the first row) and built-in rotationalsymmetry assumptions, the model is not able to succeed in matching the moments of the GSM posterior moments when observing CIFAR-10 natural images and fails to convergeduring training.4. The SSN in the fourth row is trained on all 50000 training images from CIFAR-10.
Theimages have no symmetry, and the 100 × 100 weight matrix is freely represented by 10000parameters.
Despite the increase in difficulty of the task (matching a more complex GSM),our method was able to produce a network that achieves a normalized cost lower than theattempt in [35].
Here, our method provides two advantages. Firstly, the additional degrees of freedom gained by thenetworks from having more free parameterizations allow them to achieve a much lower cost whentrained on the same GSM (first two rows). Secondly, our stability-focused approach has successfullytrained a network to perform inference under a complex GSM of natural images, a feat which waspreviously impossible (next two rows).
Once again, in all 100 independent instances, not a singleinstance went unstable.
The full network is also unable to be randomly initialized in a stable manner,which further motivates our approach.
Discussion
Network stability during training is a problem faced in both artificial and biological neural networks.
Typically, artificial neural networks (ANNs) can implement quick-fix solutions that will not negativelyimpact its performance, such as applying batch and layer normalizations [41–43].
Furthermore,artificial neurons may be designed to be stable, such as simply choosing a saturating activationfunction (tanh,sigmoid), or other ways of implementing bounded-input-bounded-output designs.
There also exists ANNs that are challenged by instabilities that cannot be trivially solved [44, 45],which could motivate similar network-building solutions.
In general, the concept of building networks during training has already been proposed in ANNs[46–48].
One such study combined both growing and pruning to produce efficient networks [47].
Another training approach decides on an existing neuron in the network to be split into multipleneurons [46].
Another proposed algorithm [48] involves a similar principle of adding nodes withoutimpacting existing network capabilities.
Broadly speaking, the two primary goals of these algorithmsare to reduce training load and improve performance, which are perfectly reasonable.
Here, we areinstead targeting an even more basic need of "survival" by staying within a stable parameter regime,achieved using our novel method of dynamics-neutral neuron duplication.
On the same train of thought, it can be meaningful to explore the tuning of certain hyperparameters inour method, such as the number of neurons to duplicate at the same time, or the criteria in selectingwhich neuron to duplicate.
Now that stability is guaranteed, we are motivated to refine the methodby exploring other hyperparameters, which motivates future work.
It is also worthwhile to consideran orthogonal approach of adding terms to the cost function such as regularization and explore itseffects on keeping network optimization within a stable parameter regime.
Throughout our analysis, we have been using Adam [49] to train SSNs (additional miscellaneoustraining information can be found in Appendix C).
This is because our primary objective is to buildbiologically-realistic networks so as to study their dynamics and network characteristics after theyhave been optimized.
In other words, our objective is the end goal, and not necessarily the journey.
However, it might still be worth exploring and understanding how biologically-plausible learning canfit into our current paradigm in future investigations.
In the fast-evolving research landscape of machine learning, speed and scalability are two of themost important criteria that determines how effective an algorithm is.
In our case, building a networkup during optimization goes against these principles, because that would require computationallyexpensive function retracing operations every time the network changes in design.
We justify thiswith two reasons.
Firstly, to the best of our knowledge, there is presently no other effective way oftraining SSNs.
Secondly, the computational resources spent on optimization is still much greaterthan the resources spent on tracing, especially since there is only a finite number of tracing required(up till the network is fully built), but potentially an unlimited number of gradient computations andoptimization iterations.
It will also be meaningful to perform investigations on how to maintain thebalance between network building and its associated computational costs.
Conclusion
We present an effective method of training the SSN, a canonical model of cortical circuits which ishighly prone to dynamic instabilities.
We show how the SSN can be trained to perform the task ofclassifying MNIST images, thereby clearly showing the generality of both the SSN and our methodto train it.
Our success opens up the possibility of training SSNs on other meaningful cognitive tasksin hopes of unraveling biological network dynamics underlying complex computations performed bythe brain.
Checklist
For all authors...
(a)
Do
the main claims made in the abstract and introduction accurately
reflect the
paper’scontributions and scope?
[Yes] In
our abstract and introduction, we claim that wehave developed a deep learning approach to train supralinear stabilized networks.
Thismethod can be found in section 4.
We also claim that we have successfully trainedthe network to perform sampling-based inference on an internal generative model ofnatural images.
These results can be found summarized in Fig 3.
Finally, we claimthat our method is an order of magnitude better than previous approaches.
This can befound in Table 2.
(b) Did you describe the limitations of your work?
[Yes] We have highlighted some potential limitations in a dedicated section, see section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] We have
not found any potential negative impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The link to our code can be found in the supplementary material.
Additional information required to
replicate our results are included in the supplementary.
(b) Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] All parameters of the model are clearly introduced early in the
text.
Values of non-optimized parameters can be found in the supplementary materials.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
N/A
(d) Did you include the total amount of compute and the type of resources used (e g.,type of GPUs, internal cluster, or cloud provider)?
[Yes] Computational resources and
related information can be found in the supplementary material.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We have used
CIFAR-10 and MNIST
and have cited the creators.
(b) Did you mention the license of the assets?
[Yes] We have linked the licenses in the
supplementary.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] CIFAR-10 and MNIST are publicly available datasets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes] This can be found in the supplementary.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]