Emergence of Hierarchical Layers in a Single Sheet of Self-Organizing Spiking Neurons
Anonymous Author(s)
AffiliationAddress email
Abstract
Traditionally convolutional neural network architectures have been designed bystacking layers on top of each other to form deeper hierarchical networks.
The cortex in the brain however does not just stack layers as done in standard convolutionneural networks, instead different regions are organized next to each other in a largesingle sheet of neurons.
Biological neurons self organize to form topographic maps,where neurons encoding similar stimuli group together to form logical clusters.
Here we propose new self-organization principles that allow for the formation ofhierarchical cortical regions (i.e. layers) in a completely unsupervised manner without requiring any predefined architecture.
Synaptic connections are dynamicallygrown and pruned, which allows us to actively constrain the number of incomingand outgoing connections.
This way we can minimize the wiring cost by taking intoaccount both the synaptic strength and the connection length.
The proposed methoduses purely local learning rules in the form of spike-timing-dependent plasticity(STDP) with lateral excitation and inhibition.
We show experimentally that theseself-organization rules are sufficient for topographic maps and hierarchical layersto emerge.
Our proposed Self-Organizing Neural Sheet (SONS) model can thusform traditional neural network layers in a completely unsupervised manner fromjust a single large pool of unstructured spiking neurons.
Introduction
Neurons are highly organized in the brain such that neurons encoding similar features are clusteredcloser together [1, 2].
These clusters of neurons form large hierarchical modules along the cortex.
There is the formation of the major distinct cortical regions (visual cortex, auditory cortex, motorcortex etc.) and also subdivisions within each region (e.g. V1, V2, V3, etc.)
Furthermore theseregions are divided into cortical columns [3], where nearby cortical columns also encode similarfeatures.
For example neurons encoding edges of similar orientations are clustered together in thevisual cortex, forming so called topographic maps [4, 5, 6].
These topographic maps have beenextensively studied and are found everywhere along the cortex [1, 2].
Classically deep neural networks (DNNs) are constructed by stacking multiple layers on top of eachother, manually designing the neural architecture [7].
Recent techniques in neural architecture searchdo try to find more optimal architectures for a given problem [8, 9], but they still consist of modulesof layers that are combined to optimize some objective function.
This creates an inherent bias in thetype of information the network can process, and each architecture has to be optimized for the task athand.
In this paper a different approach is taken instead, we define local learning rules such that aneural architecture of multiple hierarchical regions automatically emerges.
This enables us to startwith a large pool of completely unstructured neurons that self-organize into logical modules.
An important distinction should also be made between cortical regions and ’layers’ in relation tobiological networks.
Although Convolutional Neural Networks (CNNs) [7] consist of stackingmultiple layers of neurons on top of each other, these layers are actually closer to the behaviour ofcortical regions like V1, V2 etc.
where each subsequent region increases in abstraction level.
Thereare also layers in biological neural networks (i.e. the brain), generally divided into 6 groups (Layer Ito Layer VI), but they do not serve the same function as layers in CNNs [10, 11].
Within this paperwe use the layer terminology as used in CNNs, and treat them to be equivalent to the cortical regionsin biological brains (see also figure 1).
1.1 Related work
Several spiking models exist to model biological spiking neural networks (SNNs) [12, 13, 14].
Oneof the simplest is the integrate-and-fire model (IAF) [15], which sums up the incoming input overtime and fires a spike when reaching some threshold.
More advanced approaches also exist, likethe Hodgkin-Huxley [16] and Izhikevich model [17], which can exhibit more diverse behaviourlike bursting [18] and spike frequency adaptation [19].
Although spiking neural networks are stillcomputationally more expensive relative to DNNs, they are biologically more realistic and allowfor extremely sparse processing [20].
They can also perform synaptic weight updates that takeinto account the timing of each individual spike through spike-timing-dependent plasticity (STDP)[21, 22, 23].
There has also been previous works on the formation of neural topographic maps [24, 25, 26].
Oneof the most common methods to form such maps are Self-Organizing Maps (SOMs), which arecapable of producing a low dimensional representation of the input, mapping similar concepts closertogether [27].
These are typically trained by randomly initializing a single layer artificial neuralnetwork (ANN) [28], and then matching a new input example to the unit that has the highest response(winner-take-all).
This winner-take-all approach however has the limitation of only allowing a singleunit to be active in the entire layer.
They also can only form a single layer at a time.
Combining bothself-organizing maps and spiking neural networks has also been explored before [29, 30, 31].
Thesecan form realistic topographic like maps as found in the visual cortex [4].
Hierarchical maps canalso be created by manually stacking multiple maps.
However, the main limitation is still that thelayer architecture has to be predefined, which is what is universally done in both spiking and standardneural networks.
1.2 Contributions
The main motivation behind the work in this paper is the observation that cortical regions (i.e. layers)are formed on a 2-dimensional sheet instead of through vertical stacking as done in CNNs [1].
Thismeans neurons have to wire together laterally while maintaining the same spatial mapping.
In CNNsthis is trivial as it is possible to just connect downwards to the previous layer in the same region.
However, this is no longer possible in a single sheet approach, as everything is laid out in just 2dimensions.
Achieving this type of lateral mapping offers greater flexibility to adapt to the inputdomain, as we are no longer constrained by predefined fixed layer and kernel sizes.
Features arealso encoded very differently in biological neural networks, forming topographic maps that tend toautomatically cluster neurons that encode similar concepts closer together.
This means neurons havea clear spatial importance that is mostly ignored in standard DNNs.
If we can derive self-organization principles that automatically form hierarchical layers we wouldno longer have to pre-design a neural architecture, or be constrained by fixed kernel sizes and theinherently predefined structure in convolution layers.
Our contributions in this paper can thus be summarized as follows:
• We propose a Self-Organizing Neural Sheet (SONS) model that consists of a single largepool of unstructured spiking neurons in 2D space.
This model is capable of formingcompletely unsupervised its own network topology and localized topographic maps througha simple set of self-organization principles.•
We propose a dynamic rewiring method for spiking neural networks that can grow newsynaptic connections and prune existing ones at run time.
This allows us to optimize thewiring cost of the network by actively constraining the number of incoming and outgoingconnections.
Additionally, it provides us the ability to form large sparse synaptic weightmatrices, making it possible to efficiently simulate tens of thousands of neurons and updatemillions of synapses in parallel.• We derive a stochastic Integrate and Fire (Stochastic IAF) neuron model with synaptic weightnormalization that results in a stable homeostatic neuron model capable of stochasticallyinitiating synaptic growth.•
We show that the proposed stochastic integrate and fire model together with lateral excitationand inhibition and spike-timing-dependent plasticity (STDP) allows for the formation oflogical topographic maps.
These maps cluster neurons that encode similar features closertogether.•
We show that the proposed SONS model consisting of just a single sheet of initiallyunstructured spiking neurons is capable of forming localized receptive
fields and multiplehierarchical layers in a fully unsupervised manner.
Self-Organizing Neural Sheets
In order to achieve our desired objective of having a single sheet of spiking neurons self organizeinto logical topographic maps and different hierarchical layers we derive several self-organizationprinciples.
These involve a new type of stochastic integrate and fire spiking model with synapticscaling to avoid exploding weights, lateral excitation and inhibition to form topographic maps, anddynamic rewiring rules to minimize overall wiring cost.
The following subsections will describe eachof these components in more detail, as well as the initial preprocessing steps.
2.1 Preprocessing
In order to properly take static images as input we first we have to convert them into temporal spikingpatterns (see figure 2).
This is done by computing ON and OFF cell responses through center surroundfields.
These ON and OFF cells also exist in the brain [32], and respond naturally to blobs with eithera dark or light center.
These can be computed efficiently using standard Gaussian convolutions onthe image.
We can then bin the response of these cells (normalized between 0.0 and 1.0), and fire aspike when the bin index is equal to the current time step.
For example if we have 10 bins and an
ON cell has an output response of 1 .
it will fire at t = 0 , while a response of 0 .
8 will fire at t = 2 .
This means higher intensity input will fire earlier.
After this preprocessing step we can directly passthe spike responses of the ON and OFF cells to the neural sheet.
These cells form the sensory inputneurons.
It is important to note that in contrast to traditional methods of connecting to the input in theprevious layer, these sensory neurons are directly part of the single neural sheet.
Figure static image to temporal spikes.
We first use center surround receptive fields to obtain ON/OFF cell output responses, which activate when either their center is light or dark.
We then convert the real valued output of each cell into binary spikes by binning each value.
Finally, the resulting input spikes at each time step become directly part of the entire neural sheet, and other neurons within the sheet can now connect to these sensory neurons.
2.2 Stochastic Integrate and Fire neurons
In order to compute the output of each neuron in the neural sheet, we first calculate the change inmembrane potential at each time step.
The membrane potential is the value of the neuron beforespiking.
The change in membrane potential du i is simply the sum of the incoming spikes for neuron i .
That is du
i
=
(cid:80) kj =
0 w ij x j , where w ij the synaptic weight value, and x j a binary number indicatingwhether the incoming connected neuron j spiked or not.
We then get that u t +1 =
u t + du t .
This issimilar to a standard forward pass in DNNs, except we have spikes as inputs, and we sum over time.
After obtaining the membrane potential we can apply a model responsible for generating spikes.
We require a spiking neuron model that has some inherent stochastic behaviour in order to properlygrow new synaptic connections from scratch.
To this end we derive a stochastic leaky integrate andfire model (Stochastic IAF).
The probability of this stochastic IAF firing scales proportionally to themembrane potential u .
This means the higher the summated input, the more likely it is the neuronfires.
The probability of the stochastic IAF neuron firing is as follows:
[Equation]
Where a indicates a power parameter that suppresses low membrane potentials u t further (in ourcase a = 5 ).
This reduces the overall firing probability, as u t is always between 0 and 1 due tosynaptic scaling.
b is an inherent bias, which gives a small probability of spiking to occur even whenthe membrane potential is zero (in our case b = 0 . 0001 ).
This essentially means that the higher themembrane potential is, the more likely it will fire.
This was found to work well in practice, and resultsin spiking behaviour that is dependent on the actual membrane potential u t , but also has sufficientstochastic behaviour to initiate synaptic growth.
After spiking we reset the membrane potential to− 1 . 0 , which causes a hard refractory period.
This makes it impossible for the neuron to fire againuntil it is reset back to 0 , which happens when a new image is presented.
2.3 Spike-Timing-Dependent Plasticity (STDP)
Synaptic connection weights in the SONS model are modified through the standard STDP function[21, 22, 23], which takes into account both the pre and post synaptic spike timing.
It increases thestrength of the synapse if the post-synaptic neuron fires after the pre-synaptic neuron, and decreasesit if the post-synaptic neuron fires before the post-synaptic neuron.
This means causal correlationsbetween neurons will strengthen the synaptic connection, making the neuron more likely to fire tosimilar input patterns in the future.
The update equation to the weight matrix is as follows:
[Equation]
where A and τ are parameters that determine the shape of the STDP function.
In all our experiments
A and τ were simply set to 1.0.
t is the relative timing difference between the pre and post synapticspike and dW is the resulting change in the synaptic weight value .
For updating the weights we get W t +1 =
W t + η dWd
p,q ) where η is the learning rate and d ( p, q ) is thedistance between the neurons of a given synapse.
The weight update is thus scaled by the synapticlength, meaning longer connections take more time to achieve high values, making it more likelythey get pruned.
In our experiments η was set to 0 .
2.4 Synaptic scaling
To achieve a type of homeostatic balance in the neurons, synaptic weight values are rescaled to havea fixed norm value.
This is sometimes referred to as synaptic scaling and it also occurs in biologicalneural networks [33].
This ensures that the weights do not grow indefinitely causing an explosion inthe membrane potential and uncontrolled spiking behaviour.
Rescaling is simply done by dividing allthe incoming weights of each neuron by the L2 norm after each STDP update to the weights.
This issimilar to weight normalization [34] in standard neural networks, which also helps with regularizationin those cases.
The weight are thus rescaled as follows:
[Equation]
Where || w i || 2 is the L2 norm over all incoming synaptic weights j of neuron i .
This also allows usto directly use a probabilistic spiking model, as the norm and resulting membrane potential neverexceeds 1.
Since we have a stochastic spiking model we can still spike when the membrane potentialreaches close to 1, so we do not require all inputs to be active in order to reach the threshold.
2.5 Lateral excitation and inhibition
To overcome the limitation in the classical SOM model, where only a single unit is allowed to beactive, we instead use lateral excitation and inhibition.
This allows us to learn multiple topographicmaps within the sheet, and form proper layers and localized receptive fields.
Lateral excitation andinhibition can be efficiently implemented through a difference of gaussian kernel (DoG), whichconsists of a positive Gaussian kernel with a small standard deviation subtracted by another Gaussiankernel with a larger standard deviation.
This lateral excitation and inhibition method forces close neighbours of spiked neurons to activate,causing them to spike as well.
This results in a type of spike wave around the initially activated neuron.
After the neighbours spike, they will also undergo a STDP update on their synaptic connections andthus form similar connections to the same input.
Neurons close together will therefore automaticallylearn similar features.
The larger overall lateral inhibition ensures only a localized subset of neuronsactivate, which results in a type of localized winner take all strategy.
More details of the exactequations and parameters used are given in the appendix.
Receptive fields can also form in this manner (c).
When outgoing constraints are violated, connections are pruned (c.1), and new connections can be grown from unused neurons (c.2).
Minimizing the wiring cost under constraints thus causes neurons to automatically distribute their responsibility.
Layer formation similarly can emerge (d).
When outgoing constraints are violated connections are pruned (d.1), and different connections can grow (d.2).
Since the number of allowed outgoing connections is limited new connections can only be made to neurons higher in the hierarchy.
2.6 Dynamic rewiring and synaptic competition
Our SONS model also has a dynamic rewiring component, which dynamically grows and prunessynaptic connections.
This allows us to significantly scale up the network, as we no longer needfully connected weight matrices.
Additionally, hard constraints on the number of allowed incomingand outgoing connections causes local competition between neurons, where weaker connections arepruned, while stronger ones are kept.
If we prune connections based on the strength and length of thesynapse, it automatically causes the network to indirectly minimize the overall wiring cost.
This process of minimizing the wiring cost inherently results in the emergence of a type of hierarchy,since neurons only have a limited number of allowed outgoing connections.
It also causes neuronsto automatically distribute their receptive fields, as they have to compete to connect to each inputsensory neuron (see figure 3).
The growth process could be seen as a type of Monte-Carlo sampling [35], where every m timesteps we sample new possible connections.
Each neuron that spiked grows new connections to otherneurons that also recently spiked.
This lets us grow connections to new neurons that already have ahigh potential to correlate.
We also only grow some minimum distance from the neuron and onlygrow in a single direction, this ensures we do not connect purely to local neurons that are alreadyactivated through lateral excitation or create complex feedback loops.
A higher minimum distanceessentially enables larger local topographic maps to emerge.
In our experiments we set the minimumdistance to 20, within a neural sheet of 100x200.
Each time we grow new connections, we first prune existing ones to make space (if we violate aconstraint on the number of allowed connections).
Pruning is done by sorting the incoming andoutgoing connections of each neuron i by their synaptic weight value.
We then temporarily freeze thetop-k weights with the highest magnitudes | w ij | , and remove all other connections to free up space.
In our case k was set to 64 for the incoming and outgoing connections, which allows for 8 x 8 kernelsto form.
This also makes overlapping receptive fields and local synaptic competition possible.
Forthe total capacity we allow 3 k synapses, letting us grow 2 k synapses per step.
Method
We evaluate our method with several experiments.
First we generate topographic maps by simulatinga sheet of 32x32 spiking neurons.
We compare results of our proposed Self-Organizing Neural Sheet(SONS) with standard spiking neural networks (SNNs) and spiking self-organizing maps (Spiking SOM).
As input we take 7x7 patches from the standard freely available MNIST dataset [36] andconvert them to temporal spike patterns as was described in figure 2.
Each time a new image ispresented all membrane potentials are reset back to zero to ensure a clean slate.
Patches are presentedfor 20 time steps, and in total the sheet is simulated for 200k timesteps until convergence (applyingthe self-organization rules at each step as described).
Although the dataset is relatively simple, itcan still serve as a good baseline for the evaluation of unsupervised spike-based methods.
We thenevaluate our method on the full images.
In this case we use a large sheet of 100x200 spiking neurons(20k neurons).
Images are presented for 60 time steps, and the entire sheet is simulated for 3600ktime steps until convergence.
Existing methods fail in this case as they require fully connected weightmatrices.
However, we do perform ablation studies in order to study the effect of each component.
The implementation was mainly done in Pytorch [37], together with a custom cuda-accelerated datastructure for efficiently growing and pruning connections.
All experiments were run on a single Titan
V and 12-core i7 CPU.
Please see the appendix for full implementation details.
Evaluation metric To evaluate the quality of the topographic maps and resulting synaptic connections we can calculate both the synaptic strengths of the top-k strongest connections and the lengthof each of those connections.
Ideally the strongest connections go up in strength while reducing inlength.
High strong local connectivity indicates that nearby neurons are highly correlated, which isthe desired property in good topographic maps.
This allows us to derive the following desired lossmetric which also indicates the overall wiring cost of the network. [Equation] [Equation]
[Equation]
Where n is the number of neurons, and k the number of synapses per neuron.
p and q are pointsindicating the xy positions of each neuron.
So w ( p i , q j ) is the synaptic weight between neuron i andj as learned through STDP given in eq.
and normalized through eq. 3, and d ( p i , q i ) is simply theeuclidean distance between neuron i and j .
Results
To keep comparisons fair, we use the same underlying implementation and experimental setup for allour experiments.
The main difference of SONS to SNNs is that SONS have lateral excitation andinhibition together with dynamic growth and pruning of synaptic connections.
SOMs do have lateralexcitation and inhibition but no dynamic connections, they can thus form topographic maps, but areunable to form receptive fields or hierarchical layers.
SNNs and SOMs also require fully connectedweight matrices, and thus scale poorly to a large number of neurons.
topographic maps The resulting learned topographic maps can be found in Figure4.
Note that the sensory neurons do not have weights or connections and therefor do not encode anyfeatures and are not visualized.
It can be seen that SNNs learn proper edge filters as expected, but donot self-organize into logical clusters.
SOMs and SONS do form more organized representations,where similar edges cluster closer together.
Training progress can also be found in figure 5 and thefinal performance is given in Table 1.
This shows that our proposed SONs also further minimizewiring cost as opposed to SOMs, meaning features are better organized such that it minimizes theoverall wiring length toward the input.
It’s also important to note that although our model and SOMsobtain similar results, SOMs do not scale to larger sheet sizes, and only work within this single-layerexperiment (in contrast to SONS).
, which have no connectivity.
Compared to standard SNNs, neurons in our SONS model can be seen to form more logical clusters, clustering learned edges of similar orientations closer together.
Spiking SOMs perform similarly in this setting, but they only work on single layers, are not scalable to larger sheets, and they do not minimize the overall wiring cost.
[Figure]
[Table]
Emergence of receptive fields and layers Figure 6 illustrates the full image experiment usingour SONS model.
The total simulated sheet consists of 20k neurons organized into a 100x200 grid.
Receptive fields and layers can be seen to emerge within the single sheet of spiking neurons.
Thenetwork is initially completely random and unstructured, but it can be seen that the spiking neuronsproperly self-organize to be responsible for different parts of the sheet after training.
Neurons in thesheet properly connect locally to nearby neurons, minimizing the overall wiring length.
Neurons growlocalized receptive fields and properly maintain the spatial mapping across the sheet.
Topographicmaps of edge filters also emerge.
These are smaller localized edge filters that logically cluster togetherby their orientations.
They also connect to the same part of the visual field.
A visualization of the spiking behaviour within this sheet can be found in figure 7.
Spikes can beseen to propagate from the input across multiple logical hierarchical ’layers’.
These layers emergedcompletely independently and were not predefined.
By calculating the average spike response to eachdigit we can also observe higher order neurons that properly cluster similar digits together, whichmeans topographic maps also form at higher abstraction levels.
The connectivity of a single neuron is also highlighted (resulting receptive fields), which shows it formed localized receptive fields that maintain a proper spatial mapping to the input.
The black area at the top indicates the input sensory neurons, which get overwritten with the actual spiking pattern of the presented stimuli image (it has no incoming synapses).
Initially the sheet is completely randomly connected with no meaningful receptive fields or edge filters (a).
Through our derived self-organization rules the sheet converges to an organized connectivity pattern, where synapses from each neuron connect to the same region and form meaningful edge-like feature detectors (b
Ablation studies Experiments were also performed to evaluate the importance of each of the components of the self organization rules.
Removing the STDP update naturally results in a static networkwhere no learning occurs and all weights stay the same.
Removing synaptic weight normalizationdestroys the homeostasis of the spiking neurons.
Without some regularization or balance on thesynaptic weights the weights just grow indefinitely, causing all neurons to always spike.
Removingthe dynamic growth and pruning of synaptic connections results in a statically wired network.
It isstill possible to learn very sparse individual features, however no receptive fields or meaningful layerscan form and the network is just randomly wired.
Also, without lateral excitation and inhibition notopographic maps form, similar to the case of using standard SNNs in figure 4.
Additional detailsrelated to these experiments can be found in the appendix.
Discussion
We showed that our derived self-organization rules are capable of producing proper hierarchicallayers and topographic maps.
The biggest limiting factor of the results obtained is that our modelrequires extreme scalability.
We were able to simulate 20 thousand neurons, but the visual cortexhas over 100 million neurons, and the entire cortex has more than 20 billion [38].
This means thesize of the topographic maps we obtain is fairly limited.
However, this type of single sheet modelis inherently extremely scalable, as we can simply increase the entire neural sheet size.
Furtherexperiments would have to be done to investigate the extend to which we can scale, and to confirmwhat type of self-organized behaviour we might obtain in extremely large networks.
There are also several possible future research directions to further improve the proposed selforganizing neural sheets.
Since the network consists purely of simple spiking neurons, it would beinteresting to apply it to neuromorphic chips or more specialized hardware [39], which could allowfor much greater scalability and reduce the overall environmental impact of training extremely largemodels.
Another direction is to allow for spatio-temporal maps to form that can learn topographicmaps of temporal sequences.
It would then be possible to apply it to different input modalities (e.gaudio waves).
Better visualization methods would also be desirable, to gain a better understanding ofthe behaviour of each neuron in the sheet and overall neural spiking dynamics [40].
This paper presents a first step towards a different type of neural network that consists just of a largesingle sheet of self-organizing spiking neurons.
This neuronal sheet does not require any manuallypredefined neural architecture of layers and connections, but it can instead dynamically rewire andform new connections and hierarchical layers as needed. Checklist For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] See contributions subsection
(b) Did you describe the limitations of your work?
[Yes] See Discussion section
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] We donot believe the paper has any direct negative societal impact(d)
Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
The work isnot theoretical
(b) Did you include complete proofs of all theoretical results?
[N/A]
The work is not
theoretical
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] Given insupplemental materials
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] Given in supplemental materials
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Given in tables and supplemental materials
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Implementation paragraph in
Method section
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
No existing assetswere used
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material
or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’reusing/curating?
[N/A](e)
Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
No human subjects were involved(b)
Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A](c)
Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]