A High Performance and Low Latency Deep Spiking Neural Networks Conversion Framework
Anonymous Author(s)
AffiliationAddress email
Abstract
Spiking Neural Networks (SNN) are promised to be energy-efficient and achieve
Artificial Neural Networks (ANN) comparable performance through conversionprocesses.
However, a converted SNN relies on large timesteps to compensatefor conversion errors, which as a result compromises its efficiency in practice.
In this paper, we propose a novel framework to convert an ANN to its SNNcounterpart losslessly with minimal timesteps.
By studying the errors introducedby the whole conversion process, an overlooked inference error is reveald besidesthe coding error occured during converting.
Inspired by the quantization awaretraning, a QReLU activation is introduced during training to eliminate the codingerror theoretically.
Furthermore, a buffered non-leaky-integrate-and-fire neuronthat utilizes the same basic operations as in conventional neurons is designed toreduce the inference error.
Experiments on classification and detection tasks showthat our proposed method attains ANNs level performance using only 16 timesteps.
To the best of our knowledge, it is the first time converted SNNs with low latencydemonstrate their capability to achieve high performance on nontrivial vision tasks.
Source code will be released later.
Introduction
Recently, Spiking Neural Networks (SNN) have attracted a great deal of researchers’ attentiondue to their essential advantages, such as spatio-temporal information processing capability andenergy-efficient with low power consumption.
Unlike conventional Artificial Neural Networks (ANN)communicating with every neuron between layers during every inference step, SNN only passessparse events when particular neurons are fired.
This compute-on-demand property of SNN isvery friendly to low-power hardware, and suitable for neuromorphic platforms, such as event-basedcamera [11] and brain-inspired chips [32].
Such dedicated bionic systems can significantly reducememory usage and energy consumption, which is very appealing to a wide range of applications likeautonomous mobile robots.
Despite the promising characteristics, it is still a challenging problem to train high-performance deep
SNNs for practical tasks.
Directly applying the backpropagation approach is infeasible because of thenon-differentiable nature of the spike activity.
Taking advantage of the ANN’s success, conversionmethods have been proven to be a very promising direction in achieving high performance SNN.
Caoet al.
[4] first convert ANNs to SNNs by demonstrating the relationship between spiking neuron and
ReLU function for rate-based methods.
Diehl et al.
[9] propose a weight normalization approachto mapping weights from ANN to SNN.
Later, Rueckauer et al.
[39] give a theoretical analysis on
ANN-SNN conversion.
A reset-by-subtraction IF neuron is proposed to better handle the accuracydegradation during the conversion.
They also propose the weight normalization algorithm that usespercentile function instead of max value to achieve better performance.
Kim et al.
[23] propose achannel-wise normalization to further reduce the performance gap between ANNs and SNNs .
Although great progress has been made, those converted SNNs suffer from accuracy-latency tradeoff [9, 31].
As a result, the converted SNNs need longer time to achieve comparable precisionwith their counterpart ANN during inference and compromises the efficiency in practice.
To dealwith this problem, hybrid methods [37, 36] are proposed to further finetune a converted SNN withapproximated gradients to decrease the inference time.
On the other hand, for the theoretical traininggaps between ANNs and SNNs, Ding et al.
[10] propose Rate Norm to better choose the scale factorand introduce extra loss to decrease the latency.
Deng and Gu [8] utilize training samples to transfersthe weights to the target SNN by combining threshold balance and soft-reset mechanisms.
However,these works only focus on reducing the errors introduced by mapping feature values to spike trains.
With an in-depth analysis of the entire coversion process, we recognize two types of conversionerrors, namely coding error and inference error, that degrade the performance of SNN.
As otherconversion approaches mainly focus on the coding errors, our motivation is to reduce both the codingand inference errors.
Inspired by the analysis in Rueckauer et al.
[39], we notice that the SNNwith step function resembles quantized ANNs.
By establishing a strict equivalency between spikingneuron and quantized activation function, the coding error can be eliminated completely under limitedtimesteps.
Further, we observe that the assumption in rate-based conversioin is inconsistent withthe actual neuron behavior and the SNN’s dynamics during the inference.
Most of the previousconversion-related literature overlook this problem, which limits the performance of the algorithmsand leads to longer simulation time to achieve comparable results of the original ANN.
In this paper, we propose a novel High Performance Conversion (HPC) method to drastically closethe gap between ANNs and SNNs.
Specifically, a QReLU activation is proposed to eliminate thecoding error.
Since QReLU is applied to the ANN training phase, no extra effort is required duringthe conversion.
Besides, a noval buffered non-leaky IF neuron is designed to compensate for theinference error and retain SNN’s efficiency during inference.
Utilizing a new form of activationfunction and novel efficient spiking neuron, SNN obtained by our proposed approach achieves highperformance and low latency simultaneously.
With only 16 timesteps, HPC achieves state-of-the-artperformance across classification and detection tasks.
Our major contributions are summarized as:
• An overlooked inference error is discovered.
The conversion errors including both coding
errors and inference errors are theorically analized to clearify the reason of performance
degration during conversion.
• A novel high performance conversion method is proposed to deal with the two types ofconversion errors.
The coding errors are eliminated completely in theory and the inferenceerror problem is relieved to a great extent.
• Comprehensive experiments on different tasks demonstrate the efficacy of the proposedmethod.
To the best of our knowledge, it is the first time converted SNN achieves ANN
level performance while maintaining high efficiency and low latency across various tasks.
Related Work
Up to now, there exist two main categories of supervised training strategies for SNN: approximatederror backpropagation methods, and conversion methods.
Our proposed work is mainly related to ANN-SNN conversion methods with rate-based coding [9,35, 4, 39].
The goal of conversion approaches is to remain the high performance of ANN in SNNframework.
Recently, Xing et al.
[44] extend the territory of conversions by adding supports tomore structures such as softmax activation and residual block.
Instead of using longer timestepsto aproximate ANN accurately, conversion errors are analized and compensated by adjusting theparameters of converted SNN [27].
Rate Norm is proposed [10] to better choose the scale factor.
Deng and Gu [8] utilize training samples to transfers the weights to the target SNN by combiningthreshold balance and soft-reset mechanisms.
As a general approach, Kim et al.
[23] show thatconversion can be applied to detection tasks.
However, only coding error are considerd in thosework.
We notice that the inference error or “unevenness error” is described in the parallel work [3].
However, the “unevenness error” is ignored during the analysis of conversion error in their work.
Other researchers [2, 29, 21] try to approximate the gradient to train the SNN by replacing the threshold function with other functions.
A comprehensive summary of surrogated gradient backpropagationmethods can be found in [30].
Huh [20] introduce differentiable synapse and neuron models to SNNs.
Since spike trains are normaly sparse, gradients in SNNs are naturelly sparse comparedto traditional ANNs.
An sparse spiking gradient descent approach is introduced [34] to speed upthe backpropgation.
To enable very deep spiking neural network, modified residual module withelement-wise functions [14] is proposed to mimic directly gradient path of ResNet [17].
Ratherthan compute the approximated gradient, a neuron with extra inner state is introduced in [42] toestimate the gradient.
Different from [42], the buffered neruon proposed in this literature utlizes extrainner state to compensate for inference error.
Backpropagation trained SNNs show the potential ofachieving ANN level performance.
However, extra efforts are required to train the SNN properly,making it hard to train and limit its application on real world tasks.
Recently, combined methods [37, 36] emerges to alleviate the drawbacks that conversion methodsrequire larger timesteps to achieve competative performance.
These methods often contain two steps:conversion and fine-tuning.
The SNN converted in the first step is served as a weight initialization fora further fine-tuning procedure using backpropagation.
With the help of roughly mapped weights,several epochs of fine-tuning yield nearly loss-less performance with fewer timesteps [37].
However, compared to conversion methods, extra efforts are required to fine-tune the SNN.
Also, thebackpropagation imposes additional limitations to the hybrid approach, which further restrains itsapplication.
Methods
In this section, we first analyse the theoretical equivalency and gaps between a target SNN and anoriginal ANN.
Then, a quantized activation is proposed to minimize the coding error during theconversion.
A buffered non-leaky IF neuron is presented to reduce the inference error.
Finally, weintegrate the proposed conversion framework to further decrease the difference between SNN and
ANN.
3.1 Theoretical Errors and Analysis
Following Rueckauer et al.
[39], we here illustrate the equivalency between SNN and ANN andintroduce two types of errors when using the conversion methods with rate-based coding.
The target
SNN shares a similar overall architecture with the original ANN except two significant discrepancies.
First, neurons in the SNN communicate with each other by spike trains S ( t ) where S ( t ) ∈ { 0 , 1 } foreach timestep t ∈ { 1 , ..., T } , while the activation functions in ANN utilize real-value.
T is the totaltimesteps.
Second, a non-leaky integrate-and-fire (IF) neuron is used instead of the ReLU activationin conventional ANN.
The membrane function of the non-leaky IF neuron j can be described as:
[Equation]
where V j ( t ) is the membrane potential of the j th neuron and t is the timestep.
N j is the set of allinput neurons in the last layer connecting to j th neuron.
w sij is the weight and connection strengthbetween the input i th neuron and current j th neuron.
V th is the threshold for the neuron to fire asignal.
S i ( t ) and S j ( t ) are input and the resulting output spike trains of current neuron, respectively.
The firing signals are generated as follow:
[Equation]
where V ′ j ( t ) =
V j ( t − 1) + V th (cid:80) N j w sij S i ( t ) denotes the membrane potential before signal firingand potential reset operation.
Considering ANN-SNN conversion and rate-based coding, we can compute spike rate as r = 129 (cid:80)
Tt =1 S ( t ) /T for a total timestep T .
The goal of a conversion process is to derive a function transfersthe parameters of an ANN to the SNN as w s = f
( w a ) so that each neuron in the SNN produces aspike rate r ∝ a after T timesteps, where a is the corresponding activation value of original networks.
To this end, we can derive the spike count of the spike by summing Eqn.
1 over totaltimesteps T :
[Equation]
[Figure]
where V j (1) is the initial membrane potential.
For convenience, we set V j (1) =
in this workwithout loss of generality.
Eqn.
3 gives the relation between the total spike count of current neuronand its input neurons’ spike count besides the internal membrane potential.
For an ideal conversion, the spiking neuron intergrates membrane potential uniformly accross timeand introduces no error during inference.
Under this ideal condition, the output spike count shouldfullfill the following equation,
[Equation]
where V a = V th (cid:80) i ∈N j w sij (cid:80)
Tt =1 S
i ( t ) denotes the accumulated membrane potential.
Please referto the appedix for detail proof.
Assuming Eqn.
4 hold, we can derive the ideal output spike rate byaveraging Eqn.
4 over timesteps T :
[Equation]
where R ∗ is defined as follow.
[Equation]
For the ideal ANN-SNN conversion situation, when R ∗ is approaching zero, the formal expression ofa simple IF neuron becomes similar to the ReLU function in processing information.
Thus, the weightof a SNN can be transformed from its counterpart ANN directly.
Since spike rate r ∗ is bounded, alinear mapping function is utilized to convert ANN weights W s =
f
( W a )
=
λ i λ j W a , in which λ i isthe scale factor for layer i .
3.1.1 Coding Error
As shown in Eqn. 5, R ∗ can be viewed as a coding error term for the ideal conversion processduring the ANN-SNN weight mapping.
As V a V th − (cid:106) V a V th (cid:107) <
1 for V a > 0 , the upper bound of theconversion error R ∗ < 1 /T continuously decreases when the timesteps is set increasingly.
Due toerrors accumulated between layers, the converted SNNs with deep layers require large timesteps T tomaintain high fidelity and performance [39, 23].
On the other hand, the large timesteps impair theefficiency of the SNN.
This phenomenon is also known as accuracy-latency trade-off [9, 31].
3.1.2 Inference Error
Besides the conversion error, the assumption in Eqn.
4 can be contravened easily due to essentialnature of SNN neurons during the inference.
Intuitively, the inference process of SNN is quitedynamic and spiking neurons are unable to accumulate membraine potential uniformly accrosstimesteps as Eqn. 4, which leads to the divergence of overall output spike counts.
We refer to thistype of error as inference error .
We conclude two situations where the Eqn. 4 is violated during inference: over-fire and under-firephenomenons.
Over-fire indicates the situation where number of total generated spikes is greaterthan number of expected spikes (cid:80)
Tt =1 S j ( t ) > (cid:80) Tt =1 S ∗ j ( t ) .
Volatile membrane potential maycross the threshold V ′ j ( t ) ≥
V th and cause spike S j ( t )
= 1 that makes the number of generatingspikes exceeding the number of expected spikes.
Conversely, under-fire refers to the situation whereinsufficient spikes have been generated (cid:80) Tt
=1 S j ( t ) <
(cid:80)
Tt =1 S ∗ j ( t ) .
With inadequate spikes, underfire neurons will have membrane potential large than V th at the end of inference.
Fig.
1a shows anexample of these two situations.
3.2 Quantized ReLU Activation
To deal with the coding error, we propose a Quantized ReLU activation function in ANN trainingstage to accurately transform the weights to SNN with limited timesteps.
According to Eqn.4, we canreformulate Eqn. 5 in a different way :
[Equation]
Let x s = (cid:80) N j w s r
i be the weighted sum of input spike rates in SNNs.
Taking maximal spike rateinto consideration, IF neuron(Fig.
1b) can be described with step function as follow,
[Equation]
The r max is the maximum spike rate of a spike train, which is 1 for conventional spike trains.
Theproposed function is strictly equivalent to the IF neuron instead of increasing total timesteps T toapproximate the ReLU function with a fine-grain step function.
By mapping spike rate and activationvalue a = λr , the proposed activation function is derived through Eqn. 8 :
[Equation]
We name the proposed activation function as QReLU, since it can be seen as a quantized version ofthe ReLU function.
The approximate gradients are given by the PACT algorithm [6] originated from
STE [1]:
[Equation]
The scale factor λ here is also referred to as quantization boundary and is learned during the ANNtraining process.
Guaranteed by the success of quantization techniques [1, 6], a neural networkconsisting of QReLU is capable of achieving similar performance compared to the original ANN.
By mapping weights from an ANN equiped with QReLU activation, the coding error R ∗ is eliminated intheory.
3.3 Buffered Non-leaky IF Neuron
For the inference error, we observe that reduing over-fire problem alone can boost performancesiginificantly based on our experiments.
To further improve our conversion framework, we propose anovel buffered non-leaky IF neuron to relieve the over-fire situation to compensate for the inferenceerrors.
The proposed buffered neuron utilizes a recurrent mechanism to regulate the spikes.
In the proposedneuron, an additional buffered potential V b is introduced as follows:
[Equation]
Similar to the membrane potential, the initial buffer potential V bj (1) is set to zero for convenience.
The spike generating process is described as
[Equation]
Intuitively, the proposed Buffered Non-leaky IF Neuron generates negative spikes to regulate the totalspikes count for the inference errors.
Thus, the over-fire problem can be alleviated.
Please refer toappendix for detail proof.
As same as ordinary IF neurons, only addition operation and thresholdfunctions are involved in the proposed neuron.
The inference efficiency is preserved since the extrapower required by transmitting sign bit between neurons is negligible [45].
3.4 Integration with Multiple Bits Spike Train
To further improve the performance of the proposed conversion framework, we integrate shiftbased multi-bits spike train into the proposed framework.
Unlike the previous work [45] whichcompressed an existing SNN to speed up the inference, we extend the definition of the spike as
S ( t ) ∈ { 2 n | n ∈ N , 2 n ≤ S max } to facilitate the shift operation directly during the conversion.
Themulti-bits spike train version of Eqn.13 is as follow:
[Equation]
A (
[Equation]
th (cid:107)
[Equation]
, if V ′ j ( t ) >
V th
A (min (cid:18) (cid:106)
[Equation]
j ( t )
[Equation]
th (cid:107) , (cid:22) V bj ( t − 1) V th (cid:23)(cid:19) ) , if V ′ j ( t ) <and V bj ( t ) ≥ V th 0 , otherwise, (14) where the adjust function A ( · ) is defined as
[Equation]
Multiple bits spike train carries more information than ubiquitous on-off spikes [5, 22, 33, 47].
However, expensive multiplications are required during the integration with non-uniform spikes.
Toavoid multiplication, we employ the log 2 adjust function and shift operation [45] as an alternative andachieve high energy efficiency.
The adjust function can be implemented with a conditional function.
For QReLU, as the maximal ratio coding is equal to maximal spike counts, r max = (cid:80)
Tt =1 S max /T =
S max , the multi-bits version of Eqn.
9 can be described as follow:
[Table]
[Table]
[Table]
[Equation]
Here the λ b = λS max is the quantization boundary and P = TS max is the representation power .
As implied by the definition of the representation power, even the minimal shift extension with
S max
= 2 can maintain the same representation power with half timesteps.
This phenomenon isnamed strength-latency trade-off .
Experiments
In this section, we conduct experiments on both classification and detection tasks to demonstrate theefficiency and effectiveness of the proposed method.
Since the equivalency of features and spike ratesare universal throughout the entire network, real-valued input is directly utilized in the converted
SNN.
We select a unity threshold V th = 1 for all our experiments.
Max spike value S max = 2 isused for all experiments if not stated otherwise.
Other implementation details can be found in theappendix.
4.1 Visual Object Recognition and Detection Performance
We test our proposed High Performance Conversion framework (HPC) on visual object recognitionand detection tasks.
A ResNet18 [17] like architecture is utilized as backbone networks to conduct experiments on the CIFAR10 [25] dataset and a VGG16 [41] like network is trained on the
ImageNet [7] dataset.
Table 1 reports the classification accuracy of the proposed HPC compared [Figure] with other methods.
Top-1 accuracy is used to evaluate the performance.
Since the performance ofconverted SNN is closely related to the ANN baseline, a performance drop ∆ is also reported.
Asillustrated in Table 1, the proposed HPC can perform nearly lossless conversion while maintaininglow inference latency.
As a general method, we also test our HPC with tiny-yolo [38] structure on both PASCAL VOC [13]and MS COCO [28] datasets.
Table 2 summarizes the AP50 performance of the proposed method.
We can see that our proposed HPC can achieve state-of-the-art conversion performance within 32timesteps, which is over 100 times faster than the previous SNN detection work [23].
4.2 Energy Efficiency
To evaluate the energy efficiency of the proposed method, operation ratio is adopted to measure theoverall efficiency as described in Eqn. 17.
In contrast to actual energy consumption which may varybetween hardware devices [32], counting-based metric [36] is straightforward and representativefor different methods.
For a fair comparison, integration operations triggered by spikes for bothmembrane potential and buffer potential are counted in our experiments.
[Equation]
Operation ratios, as well as estimated energy consumption ratio, are summarized in Table 3.
Herewe roughly estimate that multiplication consumes 4 times more energy than addition accordingto Horowitz [18].
The results on the CIFAR10 dataset show a strong correlation between timestepsand energy efficiency.
To further study energy efficiency, we test the relationship between operationratio and networks performance.
As depicted in Fig.
2a, under the same timesteps setting, thespike ratio continuously decreases with the overall performance.
Our experiments confirm a generalrelationship between inference energy and performance, which holds even under constant latency.
Fig.
2b depicts the distributions of spike values in classification experiments.
Without loss of generality, we conclude that only 20% of the spikes require shift operation.
As measured by Gudovskiyand Rigazio [15], the shift operation only cost 10% of energy consumed by addition operation.
Thus,multi-bits spike only use 2% extra energy, which is negligible.
4.3 Ablation Study
To reveal the effectiveness of the proposed framework, we conduct ablation studies on both classification and detection tasks.
Specifically, we remove the quantized training technique and thebuffered IF neuron sequentially and compare the results.
The baseline ANN is trained and fixed forexperiments without QReLU component, meanwhile quantized ANN networks are trained separatelyfor different timestep configurations.
Without the QReLU, the scale factor is calculated using thepercentile function with the parameter empirically set to 99 . 0 .
Channel-wise normalization is utilizedto ensure competitive performances.
Since conventional converted SNN requires large timesteps to [Figure] achieve plausible performance, we only report those performances with 32 or more timesteps for the
CIFAR10 dataset and 64 for others.
Fig.
3 validates the effectiveness of the proposed techniques.
Compared to conventional SNNs,the SNN integrates with buffered neurons only requires a quarter of timesteps to reach the sameperformance.
As the buffered neuron mitigates only the over-fire problem, we conclude that theover-fire situation is the primary cause of performance degradation during inference.
The introductionof quantized training further enhances the performance of converted SNN under limited timesteps.
Similar trends can be observed across experiments.
Conclusion and Limitation
In this paper, we proposed a novel High Performance Conversion (HPC) method to simultaneouslyachieve high-performance SNN inference with low latency.
Efforts had been made to reduce bothcoding errors and often overlooked inference errors.
A novel QReLU activation was proposed toeliminates the conversion error.
Meanwhile, a buffered non-leaky IF neuron was designed to mitigatethe over-fire problem and boost the inference performance while maintaining its simplicity andefficiency.
For the first time, efficient SNN converted without extra fine-tuning revealed its capabilityto achieve state-of-the-art performances in nontrivial vision tasks.
However, there are limitations.
Since no actual hardware is involved, the efficiency is estimated usingenergy consumed by simple operations.
As computing platform evolves rapidly, it is beyond ourreach to accomplish real neuromorphic hardware which can be researched in further work.
For furture work, we believe further study on novel structures such as attention mechanisms SE [19]and transformer [12] can extend the current framework and may lead to better understanding of theunderlying perception mechanism.
Moreover, applications of SNN on video-like input that achieves
ANN level performance with great efficiency are another promising direction.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A] Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No](d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results.
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] Source code
will be released later.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Training details can be found in the appendix.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Performance stability is reported in the appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Training details can be found inthe appendix.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[No] Popular datasets are used.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]