O PTIMAL C ONVERSION OF C ONVENTIONAL A RTIFICIAL N EURAL N ETWORKS TO S PIKING N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Spiking neural networks (SNNs) are biology-inspired artiﬁcial neural networks (ANNs) that comprise of spiking neurons to process asynchronous discrete signals.
While more efﬁcient in power consumption and inference speed on the neuromorphic hardware, SNNs are usually difﬁcult to train directly from scratch with spikes due to the discreteness.
As an alternative, many efforts have been devoted to converting conventional ANNs into SNNs by copying the weights from ANNs and adjusting the spiking threshold potential of neurons in SNNs.
Researchers have designed new SNN architectures and conversion algorithms to diminish the conversion error.
However, an effective conversion should address the difference between the SNN and ANN architectures with an adaptive approximation on the loss function, which is missing in the ﬁled.
In this work, we analyze the conversion error by recursive reduction to layer-wise summation and propose a novel strategic pipeline that transfers the weights to the target SNN by combining threshold balance and soft-reset mechanisms.
This pipeline enables almost no accuracy loss between the converted SNNs and conventional ANNs with only ∼ 1 / 10 of the typical SNN simulation time.
Our method is promising to get implanted onto embedded platforms with better support of SNNs with limited energy and memory.
I NTRODUCTION
Spiking neural networks (SNNs) are proposed to imitate the biological neural networks(Hodgkin & Huxley, 1952a; McCulloch & Pitts, 1943) with artiﬁcial neural models that simulate biological neuron activity, such as Hodgkin-Huxley(Hodgkin & Huxley, 1952b), Izhikevich(Izhikevich, 2003) and Resonate-and-Fire(Izhikevich, 2001) models.
The most widely used neuron model for SNN is the Integrate-and-Fire (IF) model
(Barbi et al., 2003; Liu & Wang, 2001) where a neuron in the network emits a spike only when the accumulated input exceeds the threshold voltage.
This setting makes SNNs more similar to biological neural networks.
The past two decades have witnessed the success of conventional artiﬁcial neural networks (named as ANNs for the ease of comparison with SNNs), especially with the development of convolutional neural networks including AlexNet (Krizhevsky et al., 2012), VGG (Simonyan & Zisserman, 2014) and ResNet (He et al., 2016).
However, this success highly depends on the digital transmission of information in high precision and requires a large amount of energy and memory.
So the traditional
ANNs are infeasible to deploy onto embedded platforms with limited energy and memory.
Distinct from conventional ANNs, SNNs are event-driven with spiking signals thus more efﬁcient in the energy and memory consumption on embedded platforms (Roy et al., 2019).
By far, SNNs have been implemented for image (Acciarito et al., 2017; Diehl & Cook, 2014; Yousefzadeh et al., 2017) and voice (Pei et al., 2019) recognition.
Although potentially more efﬁcient, current SNNs have their own intrinsic disadvantages in training due to the discontinuity of spikes.
Two promising methods of supervised learning are backpropagation with surrogate gradient and weight conversion from ANNs.
The ﬁrst routine implants ANNs onto SNN platforms through realizing the surrogate gradient with the customized activation function (Wu et al., 2018).
This method can train SNNs with close or even better performance than the conventional ANNs on some small and moderate dataset (Shrestha & Orchard, 2018; Wu et al., 2019;
Zhang & Li, 2020; Thiele et al., 2019).
However, the training procedure requires a lot of time and memory and suffers from the difﬁculty in convergence for large networks such as VGG and ResNet.
The second routine is to convert ANNs to SNNs by co-training a source ANN for the target SNN that adopts IF model and soft-reset mechanism (Rueckauer et al., 2016; Han et al., 2020).
When a neuron spikes, its membrane potential will decrease by the amount of threshold voltage instead of turning into the resting potential of a ﬁxed value.
A limitation of this mechanism is that it discretizes the numerical input information equally for the neurons on the same layer ignorant of the variation in activation frequencies for different neurons.
As a consequence, some neurons are difﬁcult to transmit information with short simulation sequences.
Thus the converted SNNs usually require huge simulation length to achieve high accuracy (Deng et al., 2020) due to the trade-off between simulation length and accuracy (Rueckauer et al., 2017).
This dilemma can be partially relieved by applying the threshold balance on the channel level Kim et al.
and adjusting threshold values according to the input and output frequencies (Han et al., 2020; Han & Roy, 2020).
However, as far as we know, it remains unclear how the gap between ANN and SNN formulates and how the simulation length and voltage threshold affect the conversion loss from layers to the whole network.
In addition, the accuracy of converted SNN is not satisfactory when the simulation length is short as tens.
Compared to the previous methods that focus on either optimizing the conversion process or modifying the SNN structures, we theoretically analyze the conversion error from the perspective of activation values and propose a conversion strategy that directly modiﬁes the ReLU activation function in the source ANN to approximate the spiking frequency in the target SNN based on the constructed error form.
Our main contributions are summarized as follows:
• We theoretically analyze the conversion procedure and derive the conversion loss that can be optimized layer-wisely.
• We propose a conversion algorithm that effectively controls the difference of activation values between the source ANN and the target SNN with much shorter simulation length than existing works.
• We both theoretically and experimentally demonstrate the effectiveness of the proposed algorithm and discuss its potential extension to other problems.
P RELIMINARIES
Our conversion pipeline exploits the threshold balancing mechanism between ANN and SNN with modiﬁed ReLU function on the source ANN to reduce the consequential conversion error (Fig.1 A).
The modiﬁcation on regular ReLU function consists of thresholding the maximum activation and shifting the turning point.
The thresholding operation suppresses the excessive activation values so that all neurons could be activated within a shorter simulation time.
The shift operation compensates the deﬁcit of the output frequency caused by the ﬂoor rounding when converting activation values to output frequencies.
We here introduce common notations used in the current paper.
Since the infrastructures of the source ANN and target SNN are the same, we use the same notation when it is unambiguous.
For the l -th layer, we denote W l as the weight matrix, and z l , z (cid:48) l as layer- l ’s pre-activation output corresponding to ANN and SNN.
The threshold on activation value added to the ReLU function in the source ANN is y th and the threshold voltage of the spiking function in the target SNN is V th .
The SNN is simulated in T time points, where v l ( t ) = { v li ( t ) } is the vector collecting membrane potentials of neurons at time t , and θ l = { θ li ( t ) } records the output to the next layer, i.e the post synaptic potential (PSP) released by l -th layer to the ( l + 1) -th layer.
Suppose the l -th layer receives the t -th input x l − 1 ( t ) and x (cid:48)
l − 1 ( t ) from the ( l − 1) -th layer at time point t corresponding to ANN and SNN, the forward process of the source ANN and target SNN can be described by
[Equation]
where h l ( · ) and h (cid:48) l ( · ) denote the activation function for the ANN and SNN respectively.
For the source ANN, the activation function is modiﬁed as the threshold ReLU function, i.e.
[Equation]
where y th is the threshold value added to the regular ReLU function.
In the following sections, we ﬁrst derive the equations for weight transform from the source ANN to target SNN.
Next, we prove that the overall conversion error can be decomposed to the summation of error between source activation value and target output frequency on each layer.
Finally, we estimate an optimal shift value for each layer based on the threshold balancing mechanism and formulate the overall conversion error.
Putting these three parts together, we demonstrate both that our approach almost achieve the optimal solution when converting ReLU-based ANNs to spike-based SNNs.
C ONVERSION E QUATION FROM ANN TO SNN
We ﬁrst derive the formula of converting the source ANN to target SNN with the threshold balancing mechanism.
For the l -th layer in the SNN, suppose it receives its ( t + 1) -th input x l ( t + 1) at time point t +1 .
Then there will be an additional membrane potential W l · x (cid:48) l ( t )+ b (cid:48)
l added to its membrane potential v l ( t ) at time point t , resulting in a temporal potential
[Equation]
If any element in v ltemp ( t + 1) exceeds V th , it would release a spike with potential V th , decrease its membrane potential by V th and update the membrane potential to v l ( t +1) at time point t +1 .
Thus we can write the membrane potential update rule as:
[Equation]
where θ l ( t + 1) equals V th to release PSP to the next layer if v ltemp ( t + 1) ≥ V th and remain as 0 if v ltemp ( t + 1)
<
V th .
We accumulate the Eqn.4 from time 1 to T , divide both sides of the equation by T , and have
[Equation]
We use a (cid:48) l
= (cid:80) Tt =1 x (cid:48)
l ( t ) /T
to denote the averaged input to the l -th layer.
Then Eqn5 gives that the input to the ( l + 1) -th layer equals the expected PSP of the l -th layer, i.e. a (cid:48)
l +1 =
(cid:80) Tt
=1 θ l ( t ) /T .
If we set the initial membrane potential v l (0) as zero, we can reformulate the Eqn.5 as:
[Equation]
When the threshold potential V th is set greater than the maximum of the source ANN’s activation values, the remained potential v l ( T ) would be less than V th thus would not be output ﬁnally.
With the clip-operation, the output then can be expressed as
[Equation]
where clip( x, 0 , T ) = 0 when x ≤ 0 ; clip( x, 0 , T ) =
x when 0 < x < T ; and clip( x, 0 , T )
=
T when x ≥ T .
So the output function of SNN is actually a step function (see the blue curve in Fig.
B).
The clipping by Eqn.7 is accurate in most cases but can be slightly different from Eqn.6 when the summation of membrane potential cancel between the positive and negative parts while the positive parts get spiked along the sequence.
The forward equation for ANN is:
[Equation]
which is shown as the green line in Fig.1 B. Since the SNN output is discrete and in the form of ﬂoor rounding while the ANN output is continuous, there actually would be an intrinsic difference in a (cid:48)
l +1 and a l +1 as shown in Fig.1 B,C
even if we equalize a (cid:48) l and a l .
We will analyze how to minimize this difference later.
D ECOMPOSITION OF C ONVERSION E RROR
The performance of the converted SNN is determined by the source ANN performance and the conversion error.
While the former one is isolated from the conversion, we discuss how to optimize the latter one here.
The loss function L can also be viewed as function of the last layer output thus the conversion error can be formulated as
[Equation]
where the expectation is taken over the sample space.
For the l -th layer in the SNN, we can reversely approximate its output with the source ANN’s activation function h ( · ) and get
[Equation]
where ∆ a l (cid:48) is the error caused by the difference between the activation functions of the ANN and
SNN.
The total output error ∆ a l between the source ANN and target SNN on the l -th layer, i.e. the difference between the activation a l and a (cid:48) l can be approximated as
[Equation]
where the last approximation is given by the ﬁrst order Taylor’s expansion, and B l is the matrix with the ﬁrst derivatives of h l on the diagonal.
Expand the loss function in Eqn.9 around a L , we get
[Equation]
(12) where the last approximation is given by the 2-order Taylor’s expansion, and H a L is the Hessian of
L w.r.t a L .
As L is optimized on the source ANN, we can ignore the ﬁrst term here.
Thus it sufﬁces to ﬁnd ∆ a L to minimize the second term.
By substituting Eqn. 11 into Eqn12, we further have
[Equation]
where the interaction term can either be ignored by decoupling assumption as in (Nagel et al., 2020) or dominated by the sum of the other two terms by Cauchy’s inequality.
Applying similar derivation as in (Botev et al., 2017), we have H a l − 1 =
B l W Tl H a l W l B l .
Then Eqn.13 can reduce to
[Equation]
Here the term H a l can either be approximated with the Fisher Information Matrix (Liang et al., 2019) or similarly assumed as a constant as in (Nagel et al., 2020).
For the simplicity, we take it as a constant and problem of minimizing the conversion error then reduce to the problem of minimizing the difference of activation values for each layer.
L AYER - WISE AND T OTAL C ONVERSION E RROR
In the previous section, we analyze that minimizing the conversion error is equivalent to minimizing output errors caused by different activation function between ANN and SNN on each layer.
Here we further analyze how to modify the activation function so that the layer-wise error can be minimized.
First, we consider two extreme cases where the threshold V th is so large that the simulation time
T is not long enough for the neurons to ﬁre a spike or so small that the neuron spikes every time and accumulates very large membrane potential after simulation (upper bound of Eqn.7).
For these two cases, the remaining potential contains most of the information from ANN and it is almost impossible to convert from the source ANN to the target SNN.
To eliminate these two cases, we apply the threshold ReLU instead of the regular ReLU and set the threshold voltage V th in the SNN as the threshold y th for ReLU in the ANN.
Next, we further consider how to minimize the layer-wise squared difference E || ∆ a (cid:48) l || 2 =
E || h (cid:48)
l ( W a (cid:48)
l − 1 ) − h l ( W a (cid:48)
l − 1 )
|| 2 .
For the case of using threshold ReLU for h l , the curve of h l and h (cid:48) l are shown in Fig.1 B with their difference in Fig.1 C. We can see that when the input is equal, h l and h (cid:48) l actually have a systematic bias that can be further optimized by either shifting h l or h (cid:48) l .
Suppose that we ﬁx h (cid:48) l and shift h l by δ , the expected squared difference would be
[Equation]
If we assume that z = W a (cid:48)
l − 1 is uniformly distributed within intervals [( t − 1)
V th /T, tV th /T ] for t = 1 , ..., T , optimization of the loss in Eqn.15 can then approximately be reduced to
[Equation]
Thus the total conversion error can be approximately estimated as
[Equation]
which explicitly indicates how the low threshold value and long simulation time decrease the conversion error.
In practice, the optimal shift may be different from V th / 2 T .
As we illustrate above, the effect of shift is to minimize the difference between the output of the source ANN and the target
SNN rather than optimizing the accuracy of SNN directly.
Thus the optimal shift are affected by both the distribution of activation values and the level of overﬁt in the source ANN and target SNN.
The conversion pipeline is summarized in Algorithm 1 where the source ANN is pre-trained with threshold ReLU.
The threshold voltage V th can also be calculated along the training procedure.
Require: Pre-trained source ANN, training set, target SNN’s simulation length T .
Ensure: The converted SNN approximates the performance of source ANN with an ignorable error.
R ELATED W ORK
Cao et al.
ﬁrst propose to convert ANNs with ReLU activation function to SNNs.
This work achieves good results on
Following Cao et al.
(2015), the weight-normalization method is proposed to convert a three-layer
CNN structure without bias to SNN (Diehl et al., 2015; 2016).
The spike subtraction mechanism (Rueckauer et al., 2016), also called soft-reset (Han et al., 2020), is proposed to eliminate the information loss caused by potential resetting.
More recently, Sengupta et al.
(2018) use SPiKE-NORM to obtain deep SNNs like VGG-16 and ResNet-20 and Kim et al.
(2019) utilize SNN for object recognition.
The most commonly used conversion method is the threshold balancing (Sengupta et al., 2018), which is equivalent to the weight normalization (Diehl et al., 2015; Rueckauer et al., 2016).
The neurons in the source ANN are directly converted into those in the target SNN without changing their weights and biases.
The threshold voltage of each layer in the target SNN is the maximum output of the corresponding layers in the source ANN.
The soft reset mechanism (Rueckauer et al., 2017; Han et al., 2020) is effective in reducing the information loss by replacing the reset potential
V rest in the hard reset mechanism with the difference between membrane potential V and threshold voltage V th so that the remaining potential is still informative of the activation values.
Another big issue for the direct conversion is the varied range of activation for different neurons, which results in a very long simulation time to activate those neurons with high activation threshold values.
Rueckauer et al.
(2017) suggests the use of the p -th largest outputs instead of the largest one as the weights normalization scale to shrink the ranges.
Kim et al.
(2019) applies threshold-balance on the channel level rather the layer level for better adaption.
Rathi et al.
(2019) initializes the converted
SNNs with roughly trained ANNs to shorten simulation length by further ﬁne-training the SNN with
STDP and backpropagation.
Han et al.
(2020), Han & Roy (2020) adjust the threshold and weight scaling factor adapted to the input and output spike frequencies.
E XPERIMENTS
In this section, we validate our derivation above and compare our methods with existing approaches via converting CIFAR-Net, VGG-16, and ResNet-20 on CIFAR-10, CIFAR-100, and ImageNet.
See
Appendix A.1 for the details of the network infrastructure and training parameters.
7. ANN PERFORMANCE WITH THRESHOLD R E LU
We ﬁrst evaluate the impact of modifying ReLU on the source ANN from the perspective of maximum activation value distribution and classiﬁcation accuracy.
Smaller range of activation values will beneﬁt the conversion and but may potentially result in lower representativity for the ANN.
We set the threshold value y th = 1 on CIFAR-10, y th = 2 on CIFAR-100 and examine the impact of threshold on the distribution of activation values.
From Fig.2A we can see that the threshold operation signiﬁcantly reduces the variation of maximum activation values across layers and the shift-operation won’t cause any additional impact signiﬁcantly.
This suggests that with the modiﬁed
ReLU, the threshold voltage V th of spiking could be more effective to activate different layers.
We next look into the classiﬁcation accuracy when modifying the ReLU with threshold and shift.
Comparisons on CIFAR-10 and CIFAR-100 are shown in Fig.2B.
On the CIFAR-10, the performance of networks with modiﬁed ReLU actually get enhanced for all three infrastructures ( +0 . 18% on CIFARNet, +0 .
26% on VGG-16 and +1 .
28% on ResNet-20).
On the CIFAR-100 dataset, the method of adding threshold result in a slight decrease in accuracy on VGG-16 ( − 0 .
16% ), but a huge increase on ResNet-20 ( +2 .
These results support that the threshold ReLU can serve as a reasonable source for the ANN to SNN conversion.
7. I MPACT OF SIMULATION LENGTH AND SHIFT ON CLASSIFICATION ACCURACY
In this part, we further explore whether adding threshold and shift to the activation function will affect the simulation length ( T ) required for the converted SNN to match the source ANN in classiﬁcation accuracy and how shift operation affects differently in different period of simulation.
From Fig.3A, we can see that no matter applied separately or together, the threshold and shift operations always signiﬁcantly shorten the simulation length to less than 100 when the accuracy loss is much smaller than using regular ReLU with T ≈ 200 .
When T < 50 , the combination of threshold and shift achieves the fastest convergence and the adoption of threshold is more efﬁcient than shift.
However, the difference is minor when simulating with T > 100 .
This inspires us to further explore the impact of shift at different simulation length.
From the green and blue lines in Fig.2A and accuracy changes w.r.t the variation of shift scales in Fig.2B, we can conclude that the shift by V th / 2 T is almost always optimal even when T = 200 for the source ANN with regular ReLU.
However, the shift works different when the ReLU is with threshold.
When the simulation length is very short like
T = 16 , the conversion can beneﬁt hugely ( > 0 .
30 accuracy improvement) from the derived optimal shift (Fig.2C).
When the simulation length is long enough to almost eliminate the conversion error, the shift mechanism no longer works and may slightly damage the accuracy (Fig.2C).
7. C OMPARISON WITH RELATED WORK
In order to validate the effectiveness of the whole proposed pipeline, here we compare our method with SPIKE-NORM (Sengupta et al., 2018),
Hybrid Training (Rathi et al., 2019), RMP (Han et al., 2020), TSC(Han & Roy, 2020) and direct training method including STBP (Wu et al., 2019) and
TSSL-BP (Zhang & Li, 2020) through classiﬁcation tasks on on CIFAR-10, CIFAR-100 and ImageNet (Table.1).
For the network with relatively light infrastructure like CIFAR-Net, our model can converge to > 90% accuracy within 16 simulation steps.
Direct training methods like STBP and
TSSL-BP are feasible to achieve a high accuracy with short simulation length.
However, their training cost is high due to RNN-like manner in the training procedure and cannot extend to complex network infrastructures like VGG-16 and ResNet-20.
RMP and TSC achieve the highest accuracy for
SNN with VGG-16, probably a result from more accurate pre-trained source ANNs.
For ResNet-20, our model outperforms the compared models not only in accuracy but also in the simulation length.
We next pay attention to the accuracy loss during the conversion.
For VGG-16, our proposed model averaged on length 400-600 achieves a similar loss with RMP on CIFAR-10 ( ≈ 0 .
On CIFAR100 and ImageNet, our model maintains the lowest loss compared to both the ANN trained with regular ReLU and threshold ReLU.
For ResNet-20, there appears an interesting phenomenon that the conversion to SNN actually improves rather than damage the accuracy.
Compared with the source
ANN trained with regular ReLU, the accuracy of our converted SNN has a dramatical increase with 1 . 26% on CIFAR-10 and 2 .
74% on CIFAR-100.
This may be a result from the fact that the source
[Table] ANN for conversion is not compatible with the batch normalization and max pooling thus bears a deﬁcit by overﬁtting that can be reduced by the threshold and discretization of converting to SNN.
In addition to the excellence in accuracy, our model is even more advantageous in the simulation length.
When we combine the threshold ReLU with shift operation through the layer-wise approximation, our model can achieve a comparable performance with RMP for VGG-16 with 16 simulation steps on CIFAR-10 and 32 simulation steps on CIFAR-100, much shorter than that of RMP ( on CIFAR-10, 2048 on CIFAR-100).
Similar comparisons are observed on ResNet-20 with better performance than RMP, TSC, SPIKE-NORM and Hybrid Training.
On ImageNet, although we do not achieve the highest accuracy due to the constraint of accurate source ANN and short simulation length, we can still observe that the 400-600 simulation steps can return a conversion error smaller than that of RMP, TSC and SPIKE-NORM with 4096 and 2500 steps correspondingly.
When the simulation length is 512, the conversion with only shift operation works accurately enough as well.
See Appendix A.2 for the repeated tests averaged with simulation length 400 to 600 and Appendix A.3 for the comparison with RMP and TSC when the simulation length is aligned on a short level.
The proposed algorithm can potentially work for the conversion of recurrent neural network (RNN) as well.
See Appendix A.4 for an illustrative example that demonstrates the superiority of the current work over directly-training approach when the source architecture is an RNN.
D ISCUSSION
Since Cao et al.
ﬁrst proposed conversion from ANN with ReLU to SNN, there have been many follow-up works (Diehl et al., 2016;
Rueckauer et al., 2016; Sengupta et al., 2018; Han et al., 2020) to optimize the conversion procedure.
But they fail to theoretically analyze the conversion error for the whole network and neglect to ask what types of ANNs are easier to transform.
We ﬁll this gap and performed a series of experiments to prove their effectiveness.
Our theoretical decomposition of conversion error actually allows a new perspective of viewing the conversion problem from ANNs to SNNs.
Indeed, the way we decompose the error on activation values works for the problem of network quantization (Nagel et al., 2020) as well.
This is because the perturbation in edge weight, activation function and values would all affect the following layers through the output values of the current layer.
However, one limitation to point out here is that the interaction term in Eqn.13 may not be small enough to ignore when different layers are strongly coupled in the infrastructure.
In this case, the layer-wise recursion is suboptimal and blockwise approaches may help for further optimization.
The trade-off between representativity and generalizability applies for the adoption of threshold ReLU in the training of ANN and provides an intuitive interpretation on why our converted SNN outperforms the source ResNet-20 in the conversion.
We also show that shift operation is necessary for the conversion considering the difference between
ReLU and step function.
This difference has also been noticed in direct training approach as well.
Wu et al.
(2018; 2019) use modiﬁed sign function to model the surrogate gradient.
This choice makes the surrogate curve coincide with the red line in Fig.2B.
Yet we need to point out that the shift operation cannot ensure the improvement when the simulation length is long enough.
We can understand the shift operation as an approach to pull the target SNN to the source ANN.
At the same time, pulling the SNN to ANN also causes it to reduce the generalizability out of discretization.
As we describe above, the target SNN is possible to outperform the source ANN by beneﬁting more generalizability than losing the representativity.
In this case, it makes no sense to pull the target
SNN model back to the source ANN anymore.
This is potentially the reason why the optimal shift is no longer V th / 2 T when the simulation length is long enough to make the target SNN hardly gain more representativity than lose generalizability by getting closer to the source ANN.
Compared to the direct training approach of SNNs, one big limitation of converting SNNs from
ANNs is the simulation length.
Although SNNs can reduce the energy cost in forward process (Deng et al., 2020), the huge simulation length will limit this advantage.
Our method greatly reduces the simulation length required by the converted SNNs especially for large networks and can be combined with other optimization methods in Spiking-YOLO, SPIKE-NORM and RMP.
Further, as we point above about the duality between input x and edge weight w , future optimization could potentially reduce the simulation length to about 2 4 = 16 that is equivalent to 4 bits in network quantization with very tiny accuracy loss.
In future, it makes the framework more practical if we can extend it to converting source ANN trained with batch normalization [Table]
A A PPENDIX A. N ETWORK STRUCTURE AND O PTIMIZATION S ETUP
We adopt three network structures, CIFARNet (Wu et al., 2019), VGG-16 and ResNet-20 (Sengupta et al., 2018).
All the pooling layers use average pooling to adapt to SNN.
CIFARNet is a 8 layer structure like: 128C3 (Encoding)-256C3-AP2-512C3-AP2-1024C3-512C3-1024FC-512FC-Voting, where C means convolutional layer, AP means average pooling and FC means fully connected layer.
ResNet-20 is ResNet-18 adding two convolutional layers for pre-processing in front, and an activation function is added after the basic blocks (Sengupta et al., 2018).
Following (Sengupta et al., 2018), before training, we initialize convolution layers (kernel size k, and n output channels) weight a normal distribution and standard deviation (cid:112) 2 / ( k 2 n ) for non-residual convolutional layer, √ 2 / ( k 2 n ) for residual convolutional layer.
And then we add dropout layer that p = 0 .
2 between convolutional layers and p = 0 .
5 between fully connected layers.
Our work is based on Pytorch platform.
For CIFAR-10 and CIFAR-100, the initial learning rate of each network is 0.01, batch size is 128, total epochs is 300.
We use cross entropy loss function and
SGD optimizer with momentum 0.9 and weight decay 5 e − 4 .
And learning rate decay by a factor of 0.1 at epochs of 180, 240 and 270.
For the sake of better conversion, in the ﬁrst 20 epochs, we use regular ReLU for training, and then add the threshold to ReLU.
On CIFAR-100, our network is initialized with the network parameters on CIFAR-10 except the output layer (Pei et al., 2019).
The training on CIFAR-100 adopts the pre-trained weight without threshold for initialization in the ﬁrst few epochs as a compensation of the lack of batch-normalization (Han et al., 2020;
Rathi et al., 2019) to enable the convergence in early steps.
A. SNN PERFORMANCE WITH SIMULATION LENGTH 400-
We provide the average accuracy and variance of the converted SNN over the simulation length of 400-600 on CIFAR-10 (Table2) and CIFAR-100 datasets (Table3), the shift operation is shifting constant V th / 1600 , except for the output layer.
The results include (1) regular ReLU without our method, (2) threshold ReLU and (3) both threshold and shift operation.
Since simulation length of 400-600 is long enough for conversion from the source ANN with threshold ReLU to SNN, shift operation may reduce the converted SNN accuracy.
[Table] [Table]
A. SNN PERFORMANCE WITH SHORT SIMULATION LENGTH
Here we compare the performance of our method versus RMP and TSC when the simulation length is short.
The conversion loss is reported as the accuracy difference ( acc ANN − acc SNN ) between the converted SNN and the source ANN with regular ReLU and threshold ReLU (in brackets).
Our work-T/S denotes the SNN converted from the source ANN with both threshold (T) ReLU and shift (S).
The performance of using shift operation and regular ReLU is similar to TSC on CIFAR10 (Table4), but is better than TSC and RMP on CIFAR-100 dataset (Table5).
For VGG-16 on
ImageNet, our model can achieve nearly zero conversion loss with simulation length 256 (Table 6).
Using both threshold and shift at the same time reduces the conversion error the fastest when simulation length is short.
And only using threshold achieves the minimum conversion error on a relatively long simulation time. [Table] [Table]
[Table]
A. P ERFORMANCE OF SNN CONVERTED FROM RNN
Here we provide an illustrative example of how the proposed conversion pipeline can be extended to the case of converting RNN on the dataset for Sentiment Analysis on Movie Reviews (Socher et al., 2013).
The rules are slightly different from conversion in the main text considering the implicit deﬁnition of RNN’s hidden states. For the fairness of comparison, we set the same input and simulation length for the RNN and SNN and adjust the structure as follows. The source RNN adopts the threshold-ReLU as its activation function with the remaining value as the hidden state value.
The hidden value will be added to the pre-activation value at the next time point.
We add a non-negative attenuation τ to the hidden state values in order to enhance the nonlinearity.
(3) The converted SNN keeps the same infrastructure, weight parameters, attenuation value τ as the source RNN.
On the output layer, we use the threshold balancing method and loop its input multiple times to ﬁre enough spikes to obtain a good approximation to the fully-connected layer for the ﬁnal prediction.
We compare the performance of the source RNN, converted SNN and directly-trained SNN.
On the validation set, the converted SNN achieves an accuracy of 0.5430 that is close to the source RNN (acc = 0.5428), while the directly-trained SNN with surrogate gradient only gets an 0.5106 accuracy.
We also ﬁnd that using the regular ReLU instead of threshold-ReLU on the source RNN produces a big accuracy drop for the converted SNN (acc = 0.5100). Since the surrogate gradient error of the directly-trained method will accumulate over the whole simulation process while complex infrastructures and tasks usually require a long simulation to achieve an effective spiking frequency distribution, the typical directly-training approaches for SNNs are often not optimal in complex network structures and tasks. Our results illustrate the potential efﬁciency of converting SNN from RNN.
In future works, it would be promising to investigate how to design a better conversion strategy that can simultaneously combine the strength of RNN and SNN.