O PTIMISING E VENT -D RIVEN S PIKING N EURAL N ETWORK WITH R EGULARISATION AND C UTOFF
Anonymous authors
Paper under double-blind review
A BSTRACT
Spiking neural networks (SNNs), next generation of artificial neural networks (ANNs) with the benefit of energy efficiency, have achieved the accuracy close to its ANN counterparts, on benchmark datasets such as CIFAR10/100 and ImageNet.
However, comparing with frame-based input (e.g., images), eventbased inputs from e.g., Dynamic Vision Sensor (DVS) can make a better use of
SNNs thanks to the SNNs’ asynchronous working mechanism.
In this paper, we strengthen the marriage between SNNs and event-based inputs with a proposal to consider anytime optimal inference SNNs, or AOI-SNNs, which can terminate anytime during the inference to achieve optimal inference result.
Two novel optimisation techniques are presented to achieve AOI-SNNs: a regularisation and a cutoff.
The regularisation enables the training and construction of SNNs with optimised performance, and the cutoff technique optimises the inference of SNNs on event-driven inputs.
We conduct an extensive set of experiments on multiple benchmark event-based datasets, including CIFAR10-DVS, N-Caltech101 and DVS128 Gesture.
The experimental results demonstrate that our techniques are superior to the state-of-the-art with respect to the accuracy and latency.
I NTRODUCTION
SNNs have recently attracted significant research and industrial interests thanks to its energy efficiency and low latency Pfeiffer & Pfeil (2018), and there are neuromorphic chips such as Loihi
Davies et al.
(2018) and TrueNorth Akopyan et al.
(2015) on which SNNs can be deployed.
Mechanistically, SNNs mimic biological neurons, and the neurons process and forward spikes independently.
With such an asynchronous working mechanism, only a (small) subset of neurons will be activated during inference.
That is, energy efficiency is inherent to SNNs.
The asynchronous mechanism also suggests that event-based input may make a better use of SNNs.
Actually, neuromorphic sensors such as Dynamic Vision Sensor Lichtsteiner et al.
; Delbr ¨ uck et al. (2010); Gallego et al.
and
Dynamic Audio Sensor (DAS) Anumula et al.
(2018) have been developed to generate binary “events”, which are ideal inputs to SNN.
For example, unlike conventional frame-based cameras which measure the “absolute” brightness at a constant rate, DVS cameras are bio-inspired sensors that asynchronously measure per-pixel brightness changes (called “events”), and output a stream of events that encode the time, location and sign of the brightness changes Gallego et al. DVS reveals the sparsity and asychronicity in recognition systems for computational efficiency Amir et al. ; Messikommer et al.
Kim et al. To deal with event-based input, we propose to consider anytime optimal inference SNNs, or AOI-SNNs, which allow the termination at any time during the inference on a spike train (i.e., an input) and return the best possible inference result.
Such SNNs enable the cutoff during the inference without (significantly) compromising the performance, and thus can achieve the best in terms of accuracy and latency.
Regarding the training of SNNs, a mainstream approach is through ANN-to-SNN conversion, which adopts the mature training regime of ANNs to first train a high-accuracy ANN, and then convert it into SNN.
Such conversions via ANNs have resulted in research to focus on achieving the nearzero conversion loss.
However, existing conversion methods Deng & Gu (2021); Bu et al. (2022);
Han et al.
(2020) mostly conduct empirical experiments on frame-based benchmark datasets such as ImageNet Deng et al.
(2009) and CIFAR10/100 Krizhevsky & Hinton (2009).
In this paper, we will focus on event-based input, and therefore the AOI-SNNs, and explore effective training and inference methods to improve accuracy and latency together.
When considering ANN-to-SNN conversions to deal with DVS inputs, there are two possible ways.
The first one aggregates the sparse events in the DVS stream into a frame-based input, on which the
SNN processes as a whole.
This resembles the ANN processing a static input (such as an image).
As explained in Section 3.1, the frame-based input will base on the average spike rate, neglecting the spike timing information.
The second is to directly work with the event-based input, by considering e.g., AOI-SNNs.
An obvious benefit is that SNNs can exploit sparse events in the DVS input, enabling energy-efficient operation and reduced latency.
In addition, unlike frame-based input, the event-based input does not need an encoder at or before the first layer, which allows SNNs to operate asynchronously and achieve extra low-latency (further explained in Section 3.1).
This paper makes two key technical contributions.
Firstly, we propose a regularisation technique to influence the activation distribution during ANN training, which results in an SNN that can classify with less input information.
As will be discussed in Related Work (Section 2), with our proposed regulariser, we can train an ANN without clipping and do not need to apply any quantisation-aware technique.
Experiments in Section 5.1 show that we can achieve better accuracy than the state-ofthe-art methods on both direct training and ANN-to-SNN conversion.
Clipping (and quantisationaware) techniques have been the status quo in this area due to the recent progress Li et al.
Bu et al. (2022); Deng & Gu (2021); Wu et al.
and our result suggests that there is an alternative, and probably better, way to get an improved SNN.
Instead of simulating non-differentiable
SNN activations during ANN training, our regulariser enables the attainment of a better distribution of SNN current by actively regularising the activations of the possible misclassifications.
The regulariser is based on a new theoretical result (Section 4.1) that a smaller ratio of threshold voltages to average accumulated current can result in an SNN that can achieve optimised performance at any time during the inference.
The second contribution is that, instead of setting the inference length to always be T total , we can explore an early cutoff mechanism that enables the SNN model to automatically achieve optimal latency and energy efficiency.
As shown in Figure 1, the SNN model will run a monitoring mechanism to determine when it is sufficiently confident to make a decision.
Once such a decision is made at time t < T total , a cutoff action is triggered so that the SNN will not take future inputs until the time T total .
Therefore, not only will this lead to lower latency (because decision is made at time t rather than T total ), but it will be also more energy-efficient (because no spike will be generated after time t ).
R ELATED WORK
[Table]
et
[Table]
[Table]
Wu et al
[Table]
Li et
[Table]
The application of SNNs to a data source can be separated into two phases: training and inference.
Broadly speaking, the training algorithms for SNNs can be categorised into direct training (DT) and
ANN-to-SNN conversion.
Recently, Spike-based Error Backpropagation Wu et al.
Fang et al.
(2021b)
; Deng et al.
; Yao et al.
(2021) direct train a neural network to process the temporal information of input spikes.
However, either direct training or conversion algorithm Kugele et al.
; Wu et al.
(2022) needs to collapse the input spikes into frames for the training.
More specifically, the first layer in former SNN needs to wait for the full spike train within one frame to generate one spike, while the latter can respond very fast as long as the SNN receives spikes.
Normally, the number of frames in direct training is kept small to reduce training complexity and determines the latency of SNN in inference.
In contrast, ANN-to-SNN conversion can incorporate the maximum number of spikes during training to consider the SNN with optimal latency.
For the ANN-to-SNN conversion, early studies Rueckauer et al.
; Diehl et al.
(2015) use the maximum value of activation to normalise the weights from ANN, and Sengupta et al.
(2019) proves that the normalisation can also be achieved by greedily searching for the optimal threshold using the input spike train.
A unified conversion framework is studied in Wu et al.
Besides, there are hybrid methods Rathi et al.
Rathi & Roy (2021) that combine conversion and direct training.
Tandem Learning Wu et al.
(2021) leverages the gradient from ANN to update SNN during training.
The first two columns of Table 1 present the technical ingredients of different conversion methods for the training phase.
Recent work Deng & Gu (2021); Wu et al.
(2022) shows that, outlier elimination (OE) in ANN activations can be implemented by applying clipping operation after the
Rectified Linear Unit (ReLU).
Based on this, Li et al.
(2021); Bu et al.
(2022) further minimise the quantisation error by Quantisation-aware (QA) training.
Different from the above methods, we develop a new regulariser to achieve the better performance without clipping, and moreover, noticeably, we are free from applying QA training.
For the inference phase, as indicated in the last two columns of Table 1, the soft-reset mechanism
Rueckauer et al.
(2017) and the additive white noise to membrane potential Deng & Gu (2021); Wu et al.
(2022); Bu et al.
(2022) can significantly increase the conversion efficiency.
To the best of our knowledge, there is no existing work on cutoff in the inference phase, and our confidence-based method is the first of its kind.
P RELIMINARY
In this section, we discuss the event-based input in spiking neuron and introduce the ANN-to-SNN conversion.
To facilitate the analysis, we use bold symbol to represent vector, l to denote the layer index, and i to denote the index of elements.
For example, a l is a vector and a l i is the i -th element in a l .
Inference time t represents the time length of input.
T total denotes the maximum time length of input and it can be various depending on dataset.
W l is weight matrix at the l -th layer.
3. I NTEGRATED - AND - FIRE MODEL
Conversion-based SNN uses integrate-and-fire (IF) neuron as the basic computing unit to approximate ReLU in ANN Wu et al.
Figure 2 illustrates the inference process in IF neurons.
The input spike train X i ( t ) charges the membrane potential V i ( t ) with weighted current.
The weighted current and bias current are translated from the weight W l and bias b l in ANN.
When V i ( t ) reaches the threshold V thr , the neuron will generate a spike and then reset the V i ( t ) by subtracting V thr .
The reset by subtraction mechanism was firstly suggested in Rueckauer et al.
(2017) to reduce information loss during inference.
The dynamics of IF neuron can be described as
[Equation]
where θ l ( t ) is a step function i.e., θ li ( t )
= 1 if V li ( t − 1) + Z li ( t ) ≥ V lthr and θ li ( t )
=
otherwise.
Z l ( t ) is the input current such that
[Equation]
when l > 1 .
[Equation] For the event-based inputs (e.g., from a DVS sensor) , Z l ( t ) at the first layer, i.e., Z 1 ( t ) , can be initialised as
[Equation]
where X ( t ) is the time-dependent spike train, i.e., the input may change the charging current with time during the inference.
To consider the temporal information, we split the spike train into F frames and duration of each frame is equal to T
= T total /F , where F ∈ Z
+ .
We write ¯ X f to represent the average spiking rate of f -th frame, i.e., ¯ X f = 1 /N max (cid:80) T · f t = T · ( f − 1) X ( t ) such that
N max is the maximum spikes of all training frames in dataset D .
The spiking resolution of X ( t ) can be roughly computed as S r = N max /T .
For event-based input, SNN can manifest faster inference due to immediate response after receiving the first spike, and it completes the inference whenever the spike train ends, i.e., at T total .
The event-based benchmarks are further introduced in AppendixA.4.
This characteristic makes it possible that the inference time is dynamic for different inputs.
In this paper, with the cutoff technique as in Section 4.3, we will show that the average latency of the inference in SNN can be further reduced (to some t ≤ T total ).
3. T EMPORAL TRAINING
Regarding the direct training of SNN in Fang et al.
(2021b)
; Deng et al.
(2022), the resulted SNN can make decision by averaging output spikes of consecutive frames.
Assuming that the inference of each frames is independent, such process can be approximated in ANN training by letting the loss function be
[Equation]
where Y f is output of ¯ X f after softmax, ˆ Y is the ground truth and L CE is cross-entropy loss.
Temporal training loss ( L TT ) was suggested in Deng et al.
(2022) that achieves better generalisation.
To simplify the theoretical analysis, we let F = 1 in section 3.3 & 4.
The further explanation of temporal training is given in appendix A.5, including the impact of F on ANN training and extension of theories to F > 1 .
To ensure the independence between frames, the membrane potential except output layer needs reset after each frame, which is practical in hardware implementation Frenkel &
Indiveri (2022); Khodamoradi et al. (2021).
3. ANN- TO -SNN CONVERSION
The conversion method is mainly based on integrated-and-fire (IF) neuron, which generates spikes depending on positive accumulated current, corresponding to ReLU activation in ANN.
An existing conversion method Wu et al.
(2022) uses current normalisation methods by letting
[Equation]
where a 1 is the output of ReLU activation at the first layer of ANN.
The spiking rate of each SNN neuron at layer l is defined as r l ( t ) =
N l ( t ) /t , where N l ( t ) is the number of spikes received up to time t by neuron at layer l .
The relationship between spiking rate in SNN and activation in ANN have been theoretically proved in Wu et al.
(2022), which gives
[Equation]
where ∆ l ( t ) ≜ V l ( t ) /
( tS r V lthr ) represents the residual spiking rate.
The spiking rate at the first layer can be initialised as r 1 ( t ) =
a 1 /V 1 thr − ∆ 1 ( t ) .
Note that, we use tS r to represent the timestep in Wu et al.
Then, the current normalisation can be achieved by
[Equation]
where λ l be the maximum value of the activation at layer l .
For temporal training, the temporal input frames share the same λ l .
M ETHODS
We introduce two novel techniques: one is for the training and the other for the inference.
Section 4.
presents the theoretical underpinning of the regulariser, which in turn is introduced in Section 4.2.
This is followed by the introduction of cutoff mechanism in Section 4.3 for the inference.
4. A NYTIME OPTIMAL INFERENCE SNN S
The regulariser is based on an investigation into the design of AOI-SNNs.
An AOI-SNN is able to perform optimally under different settings on the inference time for processing an input.
When t is large enough to make ∆ l ( t ) negligible, we define the desired spiking rate as follows:
[Equation]
We start from establishing a theoretical underpinning between spiking rate r l ( t ) and its desired value r ld .
Let ϕ l denote the angle between r l d and r l ( t ) .
Then, we follow Banner et al.
(2018) to use cosine similarity between r ld and r l ( t ) , i.e., cos( ϕ l ) , for the measurement of the performance of SNN by t .
Actually, Banner et al.
(2018) shows that the cosine similarity between full precision and quantised neural network has a high correlation with the final accuracy of the quantised neural network.
Similarly, we expect that higher cosine similarity between r l ( t ) and r ld can result in less accuracy drop by t .
The following theorem states that, for any t , the performance of the SNN is of negative correlation with threshold V lthr , and positive correlation with L 2 norm over a l , as stated in the below theorem.
Theorem 4.1 For any inference time, assuming that the residual spiking rate ∆ l ( t ) is independent from r ld , the cosine similarity between r ld and r l ( t ) is inversely proportional to the ratio of threshold to average accumulated current,
[Equation]
where n l is the dimension of a l and ∥ a l ∥ 2 / √ n l denotes the average accumulated current.
We give a proof sketch of the theorem.
Because ∆ l ( t ) is independent from r ld , the angle between these two vectors tends to be π/ 2 at high dimension.
Then, by r l ( t ) =
r ld − ∆ l ( t ) , we get a right angle triangle with r ld and ∆ l ( t ) as the legs, and r l ( t ) as the hypotenuse, as illustrated in Figure 3.
Moreover, we have
[Equation]
We are interested in increasing the lower bound of Equation 9, so that we have greater cos( ϕ l ) for different t .
Combining with Equations (6) and (8), we have
[Equation]
Assuming that elements in V l ( t ) satisfy uniform distribution over the time t and they are in [0 , V thr ] , we can derive E ( ∥ V ( t ) / ( tS r ) ∥ 2 ) ≤ √ n l V thr / ( √ 3 tS r ) (proof in Appendix A.2).
Moreover, at high dimensions, the relative error made as considering E ( ∥ V ( t ) / ( tS r ) ∥ 2 ) instead of the random variable ∥ V ( t ) / ( tS r ) ∥ 2 becomes asymptotically negligible Biau & Mason (2015); Banner et al.
Therefore, Equation 10 can be computed with the following lower bound
[Equation]
which explicitly explains that (1) the increase of t to t ≫ √ n l V lthr / ∥ a l ∥ 2 can increase the lower bound and (2) it is possible to minimise term √ n l V lthr / ∥ a l ∥ 2 for developing an SNN with optimised performance at any time during the inference.
In other words, an AOI-SNN expects a good (small) ratio of threshold voltage V lthr to average accumulated current, i.e., ∥ a l ∥ 2 / √ n l , while not degrading
SNN classification performance.
The point (2) corresponds with the theorem.
4. R EGULARISER FOR OUTLIER ELIMINATION (ROE)
This section shows how to design a regulariser based on Theorem 4.1.
Recall from Equation (7) that V lthr is determined by λ l and λ l − 1 , where λ l is the maximum value of activation in the l -th layer.
To simplify the complexity of optimisation, the impact of 1 /λ l − 1 is omitted and the ratio of threshold to expected current approximately becomes proportional to λ l / ∥ a l ∥ 2 .
Therefore, we design a regulariser to minimise term λ l / ∥ a l ∥ 2 to develop an AOI-SNN.
Firstly, we use matrix
A l to represent a batch of a l during training.
Secondly, we simply use maximum value in A l to approximate λ l , i.e., λ l ≈ ∥
A l ∥ max .
Then, we write ∥
A l ∥ 2 ,q = ( (cid:80) j
( (cid:80) i
A 2 ij )
q/ 2 ) 1 /q to denotes the L 2 ,p over A l , where A lij presents j -th a l i in the batch and q ∈ Z .
Finally, we can let the penalty term be the ratio between ∥ A l ∥ max and ∥
A l ∥ 2 ,q with scale constant √ n l , i.e.,
[Equation]
We let q be −∞ so that the penalty term can focus on the inputs with relatively small accumulated current in the batch.
The final training objective is
[Equation]
[Figure]
[Equation]
(13) where α is a hyper-parameter to balance two loss terms.
Logarithm is applied to reduce the impact from extremely large value.
The regularisation-based training is to train an ANN based on ¯ X f resulting in an SNN [Figure] then SNN operates with the event-based input (Equation 3).
A small R ( A ) l implies that it is less possible for λ l to be an outlier and ∥ a l ∥ 2 is generally large [Figure]
4. C UTOFF MECHANISM TO REDUCE INFERENCE TIME
[Figure]
Thanks to the asynchronous working mechanism [Figure] event-driven SNNs can predict when only part of the spike train is processed.
Nevertheless, a naive cutoff on the length of spike train (or the sampling time of event sensor) can easily result in accuracy loss.
In this section, we suggest a principled method to determine the inference time.
Technically, a new metric, called confidence rate and denoted as C (ˆ t, D { S gap > β } ) , is defined based on the statistical characteristics of processing a set D of inputs with respect to the discrete inference time ˆ t and S gap .
S gap > β operates as a condition to identify the samples in D that are suitable for cutoff.
Actually, we are able to plot a curve of confidence rate C (ˆ t, D { S gap > β } ) with respect to the time ˆ t and β , respectively.
During the processing of an individual input X , we will monitor another variable S gap , and once it is able to ensure the confidence rate can reach certain degree, an early cutoff signal can be sent (see Figure 1).
The following provides the details.
We write X [ˆ t ] = (cid:80) ˆ tt =
X ( t ) to denote the accumulation of X ( t ) from 0 up to ˆ t .
Then, we let f ( X [ˆ t ]) return the prediction of f based on the partial input X
[ˆ t ] .
Based on this, we define a function
[Equation]
to express the earliest time from which the model f is able to confidently and correctly classify according to the partial input.
1 ( · ) is the indicator function,
i.e., 1 ( x 1 = x 2 ) = 1 and 1 ( x 1 ̸
= x 2 )
= 0 .
1 ( f ( X [ˆ t 1 ]) =
y ) suggests that f ( X [ˆ t ]) is the same as the ground truth y .
Then, recall that
N L ( t ) is the number of spikes received by t by the output layer L .
We write Top k ( N L ( t ))
as the top k spikes that occur in some neuron of layer L .
Then, we let
[Equation]
(15) be a variable denoting the gap of top-1 and top-2 number of spikes.
A large S gap implies little possibility of switching the prediction results during inference.
Then, we let D {·} denote the inputs in subset of D that satisfy a certain condition.
Now, we can define the confidence rate as follows:
[Equation]
which intuitively computes the percentage of inputs in D that can achieve the prediction success on or before a prespecified time ˆ t , i.e., g ( X ) ≤ ˆ t .
| D { S gap > β }| denotes the number of samples in
D satisfying the condition.
It is not hard to see that, when ˆ t = 0 , C (ˆ t, D { S gap > β ) is also 0 , and with the increase of time ˆ t , C
(ˆ t, D { S gap > β ) will also increase until reaching 1 .
Our algorithm searches for a minimum β ∈ Z + at a specific ˆ t , as expressed in the following optimisation objective:
[Equation]
(17) where ϵ is a pre-specified constant such that 1 − ϵ represents an acceptable level of confidence for activating cut-off, and a set of β is extracted under different ˆ t using training samples.
Equation 16 is visualised in Figure 4 that shows the impact of inference time and β on confidence.
Time ratio denotes the normalised inference time.
We characterise the confidence metric with training samples and eventually use testing samples for evaluation.
Note that, on Cifar10-DVS, all model achieves 100% for training accuracy, however, they perform differently on confidence.
With regularisation, SNN-ROE can further improve the confidence than SNN-QA, e.g., it is 0.01 higher at 0 .
125 T total and 0.07 higher at 0 .
25 T total .
Therefore, SNN-ROE can have a better performance at any time during the inference, as there are more inputs join the early cutoff.
Figure 4b presents that the input with large S gap has more consistent prediction over time, which supports the use of
S gap > β as the cutoff condition.
E XPERIMENT
We implement the ROE and conduct an extensive set of experiments to validate it.
We consider its comparison with the state-of-the-art CNN-to-SNN conversion methods.
In this section, ‘SNNQA’ denotes the method in Bu et al.
(2022), which includes both COE and QA during training, and outperforms the other methods on image input.
In contrast, ‘SNN-COE’ denotes the SNN with only COE.
Our proposed method is denoted by ‘SNN-ROE’.
To reduce the accuracy loss during inference, we followed Wu et al.
(2022); Bu et al.
(2022) to add extra current V lthr / 2 to each neuron.
Our method is validated against three event-based datasets, e.g., Cifar10-DVS Li et al.
(2017), NCaltech101 Orchard et al.
(2015) and
DVS128 Gesture Amir et al.
We train the neural network using Tensorflow with Keras API and convert it into SNN by SpKeras Wu et al.
As Bu et al.
(2022) did not cover the event-based input, we replicate their work as our baseline for comparison and set the quantisation length of SNN-QA to 16 for all datasets, which yields optimal performance.
Note that, we use original input from DVS camera without any pre-processing for the inference so that SNN can remain asynchronous to the input events.
The details of training setting are described in Appendix A.3.
5. E XPERIMENTAL RESULTS
This section presents a comparison between SNN-ROE, SNN-QA and SNN-COE on accuracy w.r.t time ratio and performance improvement after cutoff.
The inference time t =
Time Ratio × T total , where T total is equal to 1.3 s , 0.3 s and 1.2 s for Cifar10-DVS, N-Caltech101 and DVS128 Gesture respectively.
It is not hard to see, with cutoff, the performance of all models is improved in Figure 5(a, b, c).
For example, the accuracy curve moves above its original curve, which means that same accuracy can have less inference time for same model.
Thanks to the increase of confidence (recall the results in
Figure 4a)
, SNN-ROE can have general higher accuracy before the time point (red dash line) and it shows consistent results in different datasets.
The confidence evaluation for N-Caltech101 and
DVS128 Gesture is provided in Figure A-3.
It has been argued in Yao et al.
(2021) that the temporal information in Cifar10-DVS is not the dominant information, which is similar in N-Caltech101.
Unlike N-Caltech101 and Cifar10-DVS, the correlation between the temporal events in DVS
Gesture is high.
This phenomenon can also be observed from the Figure A-1.
Therefore, we set
F = 1 on N-Caltech101 and Cifar10-DVS for efficient training and F = 4 on DVS128 Gesture to incorporate temporal information.
Moreover, the result in Figure 5c shows that cutoff also can improve SNN trained with F > 1 .
Figure 5d presents that the time ratio becomes adaptive after applying cutoff and regulariser can generally increase the cutoff performance with more accurate predictions at early inference time.
We collect the results around the time point in Table 2, which has similar inference time, to show the performance of each model on cutoff.
SNN-ROE achieves the superior performance on accuracy and latency.
Spiking resolution S r is calculated to estimate maximum average spikes per second.
[Figure]
[Figure] [Figure]
The results around red dash line are summarised in Table
[Table]
[Table]
-Caltech ( S r =5230,
F =1)
[Table]
DVS128Gesture ( S r =506.67,
F =4)
[Table]
C ONCLUSIONS
This paper promotes anytime optimal inference SNNs (AOI-SNNs), which maintain the optimal performance throughout the inference stage, and therefore are suitable for event-driven inputs such as those from dynamic vision sensor or dynamic audio sensor.
Two technical novelties are proposed to optimise the attainment of AOI-SNNs, one for the training stage and the other for the inference stage.
Our experiments demonstrate the superior performance with respect to the accuracy and latency, comparing to the state-of-the-art. A A PPENDIX A.
[Table]
OTATION TABLE
Symbol
Definition
Symbol
Definition i
Element index l
Layer index
T total
Total inference time
F
Number of input frame
T
Duration of one frame t
Inference time
N max
Maximum spikes in training dataset
S r
Spiking resolution
V l ( t )
Membrane potential
Z l ( t )
Weighted current θ l ( t )
Step function
V thr
Threshold voltage
W l
Weight b l
Bias
X ( t )
Input spike train ˆ X f
Spiking rate of f-th frame
Y f
Output of ˆ X f
Y
Ground truth
N l ( t )
Number of spikes received r l ( t )
Spiking rate ∆ l ( t )
Residual spiking rate λ ld
Maximum value of activation r l d
Desired spiking rate ϕ l
Angular between r l d and r l ( t ) a l
Activation n
Dimension of a l
A l
A batch of activation
L TT
Temporal training loss ˆ t
Discrete inference time
C (ˆ t, · )
Confidence rate g ( X )
Gap of top-1 and top-2 number of spikes β
Constant value for cutoff
D
Training dataset
R ( · )
Regulariser term A. I NEQUATION PROOF
We follow Banner et al.
(2018) to derive the bound of expected norm of a random variable vector.
By Jensen’s inequality, it gives
[Equation]
As the V i ( t ) is a uniform random variable in range [0 , V thr ] , the expected value of V 2 i ( t ) can be computed as follows
[Equation]
which yields
[Equation]
Since t and S r are constant values, the following inequality holds
[Equation]
A. E XPERIMENT SETUP
The network architectures for difference datasets are given in Table A-1, which are modified from
VGG-11 Simonyan & Zisserman (2014) for Cifar10-DVS & N-Caltech101 and VGG-like structure
Fang et al.
(2021b) for DVS128 Gesture.
[Table]
Batch Normalisation Ioffe & Szegedy (2015) is applied after each convolutional and fully-connected layer to accelerate the convergence of ANN training.
For all experiment, the learning rate is set to 0.
and decays to zero after 300 epochs based on cosine decay schedule Loshchilov & Hutter (2016).
Weight decay is set to 0.0005.
We set α to 0.003 for the regulariser proposed in Section 4.2 and use pixel shifting as the data augmentation for all models, i.e., both width and height are randomly shifted by the range
Dropout is applied after fully-connected layer for DVS128 Gesture to improve the training and the dropout rate is 0.2.
We set the batch size to 128 for F = 1 andfor F > 1 to reduce memory consumption.
For SNN training, we inherit the conversion methods and most of the notation from Wu et al.
Bu et al. (2022).
Particularly, the relationship between ReLU and IF is from Wu et al.
and the relationship between quantised ReLU and IF is from Bu et al. Moreover, ROE minimisation is based on Wu et al. and operates as a penalty term during ANN training. A. E VENT - BASED DATASETS
The samples in the event-based datasets record the event addresses with on/off events over a period of time.
For Cifar10-DVS, it consists of 10,000 samples extracted from Cifar10Krizhevsky & Hinton (2009).
Each sample has 128 × 128 spatial resolution.
The length of each spike train is less equal to 1 .
3 s .
For N-Caltech101, it has 8709 samples categorised into 101 classes.
The number of samples in each class ranges from 31 to 800.
The length of each spike train is about 0 .
3 s .
The width in x-direction does not exceed 240 pixels and in y-direction does not exceed 180 pixels.
For this two datasets, we use 90% samples in each class for training and 10% for testing.
DVS128 Gesture consists of 1341 samples with 11 categories.
Each sample is repetitive over 6 . 0 s .
A. APPLYING TEMPORAL TRAINING IN ANN
[Figure]
[Figure]
Y
Output
Y f …
L TT
Similar to the direct training in Fang et al.
(2021b), the temporal training in ANN, shown in Figure
A-1, reshapes the temporal frames before forwarding them into the neural network and computes average loss after multiple outputs for optimisation.
However, temporal training uses ReLU as the activation function and has no iterative operation during forward propagation.
Although it ignores the correlation between the neighbouring frames in hidden layers, our experiment shows that SNN still can achieve good performance.
Normally, iterative operation can be expensive when the number of iteration is large, i [Figure] g., large memory required Fang et al.
(2021b). Figure A-2 presents that the increase of F can improve the accuracy on DVS128 Gesture, while it has little effect on Cifar10DVS.
We did not examine F in N-Caltch101 due to its large size.
[Figure]
Moreover [Figure] since temporal training treats consecutive frames as individual frames and generates most spike for the prediction, regulariser and cutoff can be directly applied when F > 1 ,
A.
A DDITIONAL EXPERIMENTS
Figure A-3 shows the confidence comparison on N-Caltech101 and DVS128 Gesture.
The results are consistent, SNN-ROE can have better confidence at early inference time.
[Figure]
Figure A-4 shows that the proposed regulariser improves cos ( ϕ ) ,
i
[Figure]
e., the cosine similarity between spiking rate r ( t ) and desired spiking rate r d , over different layer and reduces the conversion error at early inference time.
After increasing α above 0.003, the improvement in cos ( ϕ ) becomes limited [Figure]
Therefore, we remain α = 0 .
003 in other experiments.
Figure A-5 shows the difference of cos ( ϕ ) on SNN using ROE, QA and COE.
SNN-ROE has general higher cos ( ϕ ) than the other methods.
Meanwhile, Figure A-4 and A-5 also show that the increase of inference time benefits cos ( ϕ ) . [Figure]
[Figure]
Figure A-6 indicates that the change of α and batch size can influence the performance of resulted
SNN.
It can be easily found that ROE-SNN can achieve better accuracy at 0 .
125 T total with larger batch size.
However, too small or large batch size can significantly degrade the normal ANN training.
Thus, we set the batch size wisely that can result in an optimal SNN.
Figure A-7 presents the activation distributions for ROE, COE and QA over different layers.
Comparing with QA, ROE helps ANN retain a Gaussian-like distribution.
On the other hand, ROE brings the 99 .
99 th percentile value closer to the mean of the activation than COE.
[Figure]
We use method @ batch size to present training method with the setting of batch size. ’ ROE’ represents α = 0 . 003 and ’Normal’ P r o b a b ili t y ( y - a x
i s )
R O E
Activation (x-axis) 99.99%
Layer
Layer
Layer
Layer
C O E
Q A