[Re] Temporal Spike Sequence Learning via Backpropagation for Deep Spiking Neural Networks
Anonymous Author(s)
AfﬁliationAddress email
Reproducibility Summary
Scope of Reproducibility
In this report, we reproduce the results of a novel learning method for Spiking Neural Networks (SNN) proposed by
Zhang and Li (2020)
The proposed learning method utilises biologically more plausible neuron interactions than existing SNN algorithms.
The original paper claims that the method can produce higher performance than other state-of-the-art SNN learning algorithms on ML benchmarking datasets whilst utilising many fewer timesteps. In order to test this claim, we reproduced the results of two datasets; MNIST and CIFAR-10.
Methodology
For the reproduction of the experiments in the paper [23], we used the author’s original source code with minimaladditions; logging facilities and plotting functionality.
We also performed an additional hyperparameter searchexperiment for the MNIST dataset.
The experiments were run on two different GPUs, an NVIDIA Tesla V100-PCIE32GB GPU and an NVIDIA Titan RTX.
The total GPU runtimes were 150h 26m and 56h 4m, respectively.
Additionalto the experiments performed, we inspected the theoretical equations in the original paper.
We then scrutinised thesource code and the speciﬁc implementations of the mathematical equations.
Results
Due to high computational requirements, we reproduced two out of four experiments from the original paper.
Overall,the results match within a reasonable margin as reported in the paper.
A Bayesian hyperparameter search, throughdifferent combinations of parameters, revealed some insights about the stability and the speed of the training process.
What Was Easy
The original paper is well-written, with clear explanations of the models and the learning algorithm.
We also thank theauthors for publishing their source code online.
This made the reproduction study easier and more fruitful.
The sourcecode was written in an understandable way and the authors provided clear general instructions to rerun the networks.
What Was Difﬁcult
The computationally demanding nature of the networks yields it challenging for us to reproduce all the experiments inthe paper.
Particularly, SNNs require multiple timesteps that leads proportionally longer runtime.
Additionally, wediscovered some parameters that were hardcoded and undocumented in the source code without explanations.
We couldnot ﬁnd their particular contexts in the original paper.
Communication with Original Authors
We contacted the authors on several occasions to ask questions about the paper and the undocumented parameters in thesource code. They kindly clariﬁed all our questions. We also provided some feedback regarding theoretical equationsand code implementation. Submitted to ML Reproducibility Challenge 2020.
Do not distribute.
Introduction
Spiking Neural Network (SNN) models are based on biological networks which utilise spikes as a method of informationtransmission.
Spiking is highly energy efﬁcient, meaning these networks provide attractive computational solutions [5].
Several neuromorphic chip hardware have been developed to run spiking networks [1, 3], however, neural networkalgorithms that fully utilise their capabilities have yet to be realised.
Deep neural networks have received increasinginterest over recent years, inspiring the development of efﬁcient deep spiking algorithms (Deep-SNNs) that can berun over multiple layers [17].
Deep-SNNs vary in both spatial and temporal aspects, leading to complicated networkdynamics.
The utilisation of spiking codes for machine learning tasks is therefore nontrivial.
This is speciﬁcallydue to the challenges of employing backpropagation—the typical basis for calculating weight updates.
Previousspiking network algorithms have solved the problem of non-differential discrete spike events via surrogate gradients orapproximation using continuous activation functions.
However, these techniques destroy crucial temporal aspects of
SNNs—previous spikes of a neuron affect future spikes.
In addition, the use of spikes in previous SNN models hasrequired large temporal latency—a greater number of timesteps, in order to achieve more accurate performance ontasks.
This makes scaling to deep network architectures with many layers computationally expensive.
In this report, wereview the proposed method of “temporal spike sequence learning via backpropagation” (known as TSSL-BP) whichclaims to deal with both of these problems by considering inter and intra-neuron spiking dependencies, and reducingthe required number of timesteps.
Previous studies [19, 6, 15, 10, 7, 18, 16, 22] have demonstrated increasing accuracy of spike-based network algorithmson image classiﬁcation datasets such as MNIST [9], Neuromorphic MNIST (N-MNIST)
[12], Fashion-MNIST [20],and CIFAR-10 [8].
TSSL-BP demonstrates increased accuracy on all these datasets, including a 3.98% increase for themore challenging CIFAR-10.
Critically, the network can not only perform at higher accuracy, but requires much shortertime-window.
1.1 TSSL-BP Overview
The aim of the TSSL-BP algorithm is to learn a desired ﬁring sequence, that can be set arbitrarily.
The error function forthe network to be minimised is the distance between the produced spiking pattern and the desired sequence (target).
In
TSSL-BP, the loss is deﬁned as the sum of the squared error over all neurons for each timestep.
The distance betweenthe actual and desired spiking times is
[Equation]
1) where E [ t k ] is error at discrete timestep t k , d
and s are the desired and actual spike trains, and (cid:15) is a kernel functionmeasuring the Van Rossum distance between them [13].
The spike trains are binary sequences within a certaintime-window.
1.1.1 Leaky Integrate-and-Fire Model
Spikes in the neural network model are generated using the standard Leaky Integrate-and-Fire (LIF) model [4].
Thismodel describes a neuron’s membrane potential over time and generates a spike if the potential value is a higher than aspeciﬁed threshold.
Incoming spikes are converted into a postsynaptic current (PSC) a j ( t ) .
The neuronal membranepotential u i ( t ) for neuron i is
[Equation]
where R and τ m are leaky resistance and time constant of the membrane, w ij is the synaptic weight between neurons iand j , and η i is the reset function.
The PSC and the reset function are deﬁned as,
[Equation]
where s j is the spike times of neuron j , ν is reset kernel and (cid:15) is spike response.
The spike response kernel is
[Equation]
where τ s is synaptic time constant.
Finally, the ﬁring output is determined by the Heaviside step function, H ( . ) , producing all-or-none spiking dependingon whether the membrane potential is over a speciﬁed threshold V th :
[Equation] [Equation]
Intra -neuron Dependencies
Spiking outputs are binary; all-or-none discrete events, meaning the activation function is not differentiable.
Some
SNN implementations [18, 16, 21] have circumvented this issue via use of surrogate gradient methods [11].
However,these methods degrade training performance due to discrepancies between the target loss and gradient.
More recent
SNN methods have reduced the number of timesteps required, however, still utilise continuous activation functions asapproximations of the spiking neurons [19].
The approach of TSSL-BP considers the precise temporal dependencies ofboth spiking between neurons (inter) and within neurons (intra).
Inter-neuron dependencies arise as the spikes from thepresynaptic neurons cause changes in the postsynaptic neurons’ current.
Furthermore, due to the membrane potentialreset kernel in the LIF model, the timing of spikes within a neuron can be dependent on previous spikes within a certaintime-window.
This affects the precise timing of spikes and the subsequent postsynaptic current.
We refer the reader tosee the original paper for the derivation.
Scope of reproducibility
In this reproduction, we test the following claims of the paper:
1. TSSL-BP performs at a higher accuracy than previous spiking neural nets on supervised learning problems using the MNIST and CIFAR-10 datasets.
A large increase of 3.98% on CIFAR-10 can be achieved, which is
a challenging dataset for SNNs.
Compared to previous SNNs, TSSL-BP utilizes a lower latency of spikes, whilst still maintaining high accuracy
on the datasets.
To test these claims, we ﬁrst checked the mathematical derivations provided in the paper, supplementary material andthe source code implementation.
Second, we tested the claims about performance accuracy by reproducing the resultsfor the TSSL-BP method on the MNIST and CIFAR-10 datasets.
An additional hyperparameter search was conductedfor the MNIST dataset to assess whether the parameters selected in the original paper were justiﬁable.
Methodology3.1 Model Descriptions
[Figure] The neural network architecture and the learning algorithm were implemented in PyTorch ML framework.
The networkarchitecture is an adapted ConvNet, modiﬁed to allow spike generations.
The network architecture used for MNISTdataset was 15C5-P2-40C5-P2-300 , it has 210,375 trainable parameters in total.
The CIFAR-10 dataset was tested ontwo different network architectures.
The ﬁrst one ( CNN1 ) was 96C3-256C3-P2-384C3-P2-384C3-256C3-1024-1024and the second ( CNN2 ), larger network 128C3-256C3-P2-512C3-P2-1024C3-512C3-1024-512 .
For both, a ﬁxedprobability of 0.2 dropout was applied after each layer.
The networks have 21,156,384 and 44,999,040 trainableparameters, respectively.
Outputs of the model are converted to spike rates and tested against desired spike rates foreach class.
The target class is given a desired spike count, and other classes are set to target zero spikes.
At the beginning of network training, low initial weight values can lead to an absence of ﬁring activity, meaningbackpropagtion with TSSL-BP is not possible.
To solve this, the authors suggests to apply a warm-up mechanism,where an average ﬁring rate threshold is set for each layer.
When an activity is above this threshold, TSSL-BP is applied.
If the ﬁring activity is very low the warm-up is applied.
In this case, the activation function is approximated using acontinuous sigmoid function of membrane potential, allowing backpropagation without spiking.
3.2 Datasets and Hyperparameters
MNIST is a prominent benchmarking dataset of handwritten digits for image recognition tasks.
It includes 70,000gray-scale input images of size of 28 × 28 for 10 classes.
Amongst these, 60,000 images are used for training and 10,000for testing.
CIFAR-10 is another well-known, image recognition benchmarking dataset.
It includes 60,000 colour inputimages of size 32 × 32 for 10 classes.
Amongst these, 50,000 images are used for training and 10,000 for testing.
For both the MNIST and CIFAR-10 datasets, preprocessing was performed as in the original paper.
For each image, short time-windows of real-valued spike currents are generated from pixel intensities.
For the reproduction of the experiments, we used the same parameters as in the original paper and source code providedby authors.
For both MNIST and CIFAR-10 datasets, batch size was 50, τ s (synaptic time constant) was 3, τ m(membrane time constant) was 5, time-window was 5 (with desired count of 4 and undesired 1).
For MNIST, thenetworks were trained for 100 epochs at a learning-rate of 0.0005, and for CIFAR-10, 150 epochs at a learning-rate of0.0002.
An additional Bayesian hyperparameter search (12 runs) was performed for the MNIST dataset.
The details ofthis experiment can be found in Section 4.3.
3.3 Experimental Setup and Computational Requirements
As in the original paper, we ran ﬁve trials each of the MNIST network and CNN1 network (CIFAR-10 dataset).
Due toruntime limitations, CNN2 network was trained twice and the best performing is reported here.
All performance datawere measured by accuracy.
Our additional code is available online.
3 Weights and Biases API was implemented on the model code to track modellearning and assist with analysis [2].
Reproduction of the results for MNIST and CIFAR-10 datasets were run on a
NVIDIA Tesla V100-PCIE-32GB GPU, and the hyperparameter search for MNIST dataset was run on a NVIDIA Titan
RTX.
The detailed GPU runtimes are:
• For the MNIST dataset, the mean run time was 1h 21m and the total GPU time for 5 runs was 6h 47m.•
For the CIFAR-10 dataset, the mean run time for CNN1 network was 18h 33m hours and the total GPU time for5 runs was 92h 43m.•
For the CIFAR-10 dataset, the mean run time for CNN2 network was 25h 28m and the total GPU time for 2runs was 50h 55m.
• For the MNIST dataset hyperparameter search, the total runtime of 12 runs was 56h 4m.
Results
This section reports reproduction of two experiments from the paper; MNIST and CIFAR-10, and an additionalhyperparameter search for MNIST.
Overall, our results supports the claims in the original paper.
Accuracy scores foreach network reproduction were within reasonable margin of the original paper.
4.1 Result 1: MNIST
We conducted ﬁve runs for MNIST dataset, with the same hyperparameters as the original paper.
Our reproductionproduced a mean accuracy of 99.40% (see Table 1).
Compared to the paper, this was within reasonable margin, original99.50% with ours 0.1% lower.
Between the best performance of each, the difference was only 0.06%.
[Table]
For the MNIST dataset, we conﬁrm that TSSL-BP outperforms most other SNNs [7, 10, 16, 18].
As in the original paper,it performs marginally below ST-RSBP network [22].
ST-RSBP achieves 99.57%, versus 99.40% for our replication and99.50% reported in the original paper.
However, ST-RSBP with the same network architecture requires 400 timesteps,versus only 5 for TSSL-BP.
Given this, the performance for TSSL-BP is highly comparable to other SNNs.
This resultalso provides support for the claim that TSSL-BP can perform well even with few time steps.
4.2 Result 2: CIFAR-
For CIFAR-10, the results of our replication can be found in Table 2.
For the smaller CNN1 network, the original paperdemonstrates best accuracy increases of 3.98% over STBP algorithm [19].
For our reproduction, we achieved meanaccuracy of 88.96% versus 88.98% in the original paper.
Best performance for the reproduction was slightly lower,89.07% versus 89.22%.
Compared to STBP, the best performance increase is 3.98% original and 3.83% reproduction.
[Table]
In the original paper, it is unclear how many times the CNN2 architecture was run.
We contacted the authors in thatregard and they clariﬁed that the network was run only once.
The accuracy reported in the original work is 91.41%, a0.88% increase on STBP (with NeuNorm) [19].
In this reproduction, we ran the network twice and selected the highestaccuracy of the two, 89.61% versus 89.53%.
This is a lower performance than STBP, with a reduction of 0.92%.
It isunclear whether TSSL-BP would consistently score lower over more trials, or whether the lower score obtained herewas due to network stochasticity.
Nevertheless, TSSL-BP utilises marginally fewer timesteps, with a reduction fromeight for STBP (with NeuNorm) to ﬁve.
The authors also report that there are no additional optimisations on TSSL-BPthat are used in the comparable SNNs [19], such as neuron normalisation and population decoding.
4.3 Results Beyond Original Paper
We investigated the hyperparameters used in the MNIST network, the original parameters can be found in Table 3.
As itis costly to run the network and impractical to search for large numbers of parameters, we performed a brief Bayesianhyperparameter search for learning-rate, number of epochs and time-window.
We utilised the sweep functionality from
Weight & Biases ML developer tools.
For this search, we used Bayesian optimisation with the objective functionimproving the test-accuracy.
We ran this optimisation 12 times, with parameters to be selected from:
• epochs ∈ { 50 , 100 , 150 , 200 }
• learning-rate ∈ { 0 . 0001 , 0 . 0005 , 0 . 001 , 0 . 005 , 0 . 01 }
• time-window ∈ { 5 , 10 , 20 }
[Table]
Figure 1 shows an overview of the different parameters used and the corresponding test-accuracy after completingtraining.
Within the limited number of runs, one can see that a longer time-window leads to higher accuracy.
Although,there is a trade-off—the longer time-window increases runtime, due to longer input processing.
On the other hand, the
Bayesian optimisation method was favouring longer time-windows.
Figure 2 shows training accuracy obtained at each epoch for 12 runs.
We observed sudden changes in three of the runsduring training.
This may be due to an instability of the learning algorithm—this could be explored in future work.
Surprisingly, on the other hand, the test performance did not change.
Amongst the unexpected runs, one had a peculiarlearning process (see purple curve).
The training performance accuracy periodically changed.
Another observationis that some of the runs (e.g. top orange curve) increased to their maximum performance quite early on (less than 25epochs) and the performance did not improve thereafter, suggesting that the number of epochs may have been set toohigh.
Figure 3 demonstrates the distribution of learning-rates against number of epochs, and the colours represent testaccuracy.
It is clear that the effect of epoch is not substantial, though, learning-rate impacts greatly; lower learning-rateleads to better performance.
Results suggest that approximately the same performance could be achieved with half the amount of epochs, if the learning-rate is chosen appropriately, e.g. 0.001.
For this learning-rate setting, three runswere performed.
The test-accuracy results were 99.3%, 99.43% and 99.37% for number of epochs 50, 150 and 200,respectively.
On the other hand, the time it takes to run each of them was 1h 50m, 5h 32m and 7h 23m. From this, thereis no signiﬁcant beneﬁt of running the MNIST experiment for a large number of epochs.
Overall, the highest performance was 99.46%, using time-window of 20 for 200 epochs at learning-rate of 0.0005,however, this took 7h 22m. The lowest computational cost was 1h when using time-window of 5 for 100 epochsat learning-rate of 0.005 with a performance of 98.85%.
Comparing this experiment with the paper, the originalhyperparameters selected in the paper were well-optimised.
Discussion
Overall, the reproduction study was fairly straightforward.
The authors were helpful and provided clear explanations.
Due to time constraints and limited resources available, we could only reproduce two out of the four experiments fromthe paper.
However, our choice of the reproduced experiments was deliberate; we chose a well-known and relativelysimpler dataset (MNIST) and another well-known but more complicated dataset (CIFAR-10).
The reproduced accuracyresults were within reasonable ranges to the authors’ original paper results.
For both MNIST and CIFAR-10 (using a smaller network structure) TSSL-BP outperforms most other SNN algorithms.
Furthermore, we found that the reducednumber of timesteps is sufﬁcient to reproduce these accuracies, supporting claim two.
We performed an additional hyperparameter search to investigate the proposed learning algorithm’s abilities outsideof the selected parameter domain.
This investigation revealed some interesting properties; such as that comparableperformance can be achieved in a shorter runtime, and that the learning algorithm may have some instabilities.
Theseinsights can lead to further experimentation and perhaps further novel contributions in the future.
What was easy Authors provided the necessary source code for the learning algorithm and most network setups.
This made the reproducibility study more fruitful.
The only missing network setup was for CNN2 , however, this caneasily be reproduced by amending the layer sizes provided in the CNN1 ﬁle.
We used W&B API for logging theresults and plotting facilities, this made the collaboration experience easier and allowed us to monitor the network runsasynchronously.
What was difﬁcult Some parameters included in the original source code were undocumented.
These are: a = 0.2(line 89 in functions/tsslbp.py) , th = 1/(4 * tau_s) (line 56 in functions/tsslbp.py) , andtheta = 1.1 (line 31-32 in layers/pooling.py) .
It was also unclear how the network weight clipping was determined (-8 and 8 for line 90 at tsslbp.py and -4 and 4 for weight_clipper function inlayers/linear.py and layers/conv.py ). We contacted the authors to clarify these; a and th are used as rescalingfactors and theta was not used in the code (i.e. redundant). Moreover, the authors conﬁrmed that the particular valuesare empirically found and manually tuned.
Finally, the network warm-up mechanism could be cumbersome for morecomplicated datasets.
Communication with original authors We communicated with the author at the NeurIPS poster session, where ourinitial questions were answered surrounding the method, implementation and goals of the network.
Following thiswe provided the authors with feedback on the details of some of the equations via email.
We also enquired about theundocumented parameters given in the source code and the number of runs performed for the CIFAR10 dataset using
CNN-2 network.
We thank the authors for their engagement with this process.
Future Work We attempted to test the algorithm on another neuromorphic dataset used to benchmark SNN—a dynamicvision sensor (DVS) version of the CIFAR-10.
During initial testing we found that large computational resources arerequired, and therefore did not proceed.
We have provided the code for preprocessing of the dataset (based on [14]) inour code repository for future works to utilise.
The CNN2 network reproduction demonstrated lower accuracy than the STBP algorithm [18].
It is unclear whetherthis is due to variability of running a low number of trials or a more general trend.
As fewer timesteps were used for the TSSL-BP implementation when comparing, it is possible TSSL-BP would achieve higher accuracy with the samenumber of timesteps.
Future work could investigate whether comparing in this case yields higher accuracy as claimedby the authors.