GLIF: A Uniﬁed Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks
Abstract
Spiking Neural Networks (SNNs) have been studied over decades to incorporate their biological plausibility and leverage their promising energy efﬁciency.
Throughout existing SNNs, the leaky integrate-and-ﬁre (LIF) model is commonly adopted to formulate the spiking neuron and evolves into numerous variants with different biological features.
However, most LIF-based neurons support only single biological feature in different neuronal behaviors, limiting their expressiveness and neuronal dynamic diversity.
In this paper, we propose GLIF, a uniﬁed spiking neuron, to fuse different bio-features in different neuronal behaviors, enlarging the representation space of spiking neurons.
In GLIF, gating factors, which are exploited to determine the proportion of the fused bio-features, are learnable during training.
Combining all learnable membrane-related parameters, our method can make spiking neurons different and constantly changing, thus increasing the heterogeneity and adaptivity of spiking neurons.
Extensive experiments on a variety of datasets demonstrate that our method obtains superior performance compared with other SNNs by simply changing their neuronal formulations to GLIF.
In particular, we train a spiking ResNet-19 with GLIF and achieve 77 .
35% top-1 accuracy with six time steps on CIFAR-100, which has advanced the state-of-the-art.
Codes are available at https://github.com/Ikarosy/Gated-LIF .
Introduction
Spiking neural network (SNN) is considered the third generation of the neural network, and the bionic modeling of its activation unit is called the spiking neuron, which brings about the spike-based communication between layers[1, 2].
Due to the high sparsity and multiplication-free operation in information processing, such spike-based communication can improve energy efﬁciency[3, 4].
Moreover, spiking neuron can effectively capture temporal information and has good performance in processing neuromorphic data[5] and anti-noise[6].
Together with artiﬁcial neural network (ANN) colleagues, SNN further shows great potential for general intelligence[7].
Thus, SNN has aroused heated research interests recently.
The commonly used spiking neuron in different SNNs[6, 8, 9, 10, 11, 12] is the leaky integrate-andﬁre (LIF) model, named vanilla LIF.
As the name suggests, it can implement three different neuronal behaviors: membrane potential leakage, integration accumulation, and spike initiation.
To better model neurons, prior works [13, 14, 15, 16, 17] have improved the vanilla LIF from the above three τ exp
S t −
U t − τ lin
V re + ×
S t
U t g t
C t +
U t
S t (a) (b) (c) σ ( x )  × ×−/
𝔾 γ
𝔾 α 𝔾 β
L t
F t
I t aspects, and proposed many LIF variants with different biological features.
For all these vanilla and variant LIF models (referred to as simplex LIF), any of the three neuronal behaviors supports only single biological feature.
For instance, the vanilla LIF model only utilizes exponential decay in membrane potential leakage, while [15] only supports linear decay.
However, different neurons in the cerebral cortex have different response characteristics[18, 19].This naturally raises an issue: could the higher neuronal dynamic diversity of spiking neurons help SNNs do better?
In this paper, we investigate the hybrid bio-features in the LIF model to answer this question.
We propose the gated LIF model (GLIF) that fuses different bio-features in the aforementioned three neuronal behaviors to possess more response characteristics.
As illustrated in Fig.
1, GLIF controls the fusion of different bio-features through gating units G α , G β , and G γ for those three neuronal behaviors that are membrane potential leakage, integration accumulation, and spike initiation, respectively.
In each gating unit, a gating factor is computed from a Sigmoid function σ ( x ) over a learnable gating parameter x to determine the proportion of each bio-feature, thus guiding the fusion of different bio-features.
Therefore, GLIF can simultaneously contain different bio-features, possessing more response characteristics.
In addition, when the gating factor is 0 or 1 , GLIF can also support single bio-feature.
As a result, GLIF can cover other different LIF models and be viewed as a super spiking neuron, greatly enlarging the representation space of spiking neurons.
Furthermore, we introduce the channel-wise parametric method to GLIF.
This method makes all membrane-related parameters in GLIF learnable and shares the same GLIF parameters channelwisely in SNNs.
Combining with learnable gating factors in GLIF, on the one hand, this method makes different channels in SNNs have completely different spiking neurons, leveraging the larger representation space of GLIF neurons to increase the neuronal dynamic diversity of spiking neurons.
Meanwhile, the heterogeneity of spiking neurons and the expressive ability of SNNs are also increased.
On the other hand, the spike neurons in SNNs are constantly changing during training, which is similar to the neuronal maturation during development[20, 21, 22], thus the adaptivity of spiking neurons being enhanced.
Our contributions are as follows:
• We propose the GLIF model, a uniﬁed formulation of the spiking neuron that fuses multiple biological features through gating units, enlarging the representation space of spiking neurons.
• We exploit the channel-wise parametric method to leverage the larger representation space of GLIF, thus increasing the neuronal dynamic diversity in SNNs.
• Our experiments demonstrate the effectiveness of the GLIF model on both static datasets and the neuromorphic dataset.
For example, our model can achieve the state-of-the-art 75 .
48% top-1 accuracy on CIFAR-10 with only 2 time steps.
Related Work
Variant LIF.
Variant LIF models with different biological features are proposed in previous arts, and those models differ in one or more of the three different neuronal behaviors: membrane potential leakage, integration accumulation, and spike initiation.
Firstly, membrane potential leakage represents the potential decay mechanism.
Exponential decay is ﬁrst exploited in the vanilla LIF model[23], and it brings neuronal dynamics with stable convergence, shrinking the potential to the resting potential steadily.
But, exponential decay is not efﬁcient enough in massive neural simulations due to its multiplication operation.
Therefore, linear decay replaces exponential decay in [13, 24, 25, 26, 27], and brings neuronal dynamics with the ﬂexibility, making it possible for conﬁgurable spiking rates [15].
Although the two types of decay behave differently in neuronal dynamics, both of them are proven effective in SNNs[6, 28].
Secondly, integration accumulation represents the potential accumulation induced by presynaptic spikes.
Commonly, spike intensities of different time steps are considered equally high.
Consequently, the discrepancies among the spikes at different time steps are ignored.
To address this issue, many prior works propose to assign different weights to the input spikes at different time steps.
These methods are implemented by manual settings[29, 30, 31] or adaptive settings[32, 33, 34] to improve biological plausibility, SNN performance, or inference efﬁciency.
Lastly, spike initiation functions as the resetting mechanism triggered by spike ﬁring.
There are two common resetting strategies: hard reset[35] and soft reset[15, 36].
When a spike is triggered, the hard reset directly sets the potential to the resting potential, while the soft reset subtracts the potential by a conﬁgurable value.
Therefore, hard reset is more stable as a strict reset mechanism, while soft reset is more ﬂexible, making conﬁgurable spiking rates feasible[15].
Compared with the LIF models above, our GLIF incorporates different neuronal dynamic features to increase the representation space of LIF models and improve the performance of SNNs.
Parametric spiking neurons.
Early existing spiking neurons often require manual tuning of the membrane-related parameters to set their neuronal dynamics, and these parameters are often shared by all neurons, which limits the diversity of spiking neurons and the expressive ability of SNNs.
In recent years, parametric spiking neurons[37, 38, 10, 28] have been proposed, which set one or more membrane-related parameters to be learnable, and update these parameters during the training of
SNNs.
[37] and [38] set the membrane leak and potential threshold learnable, while [10] and [28] choose the membrane leak only.
FS-neuron[39] is more aggressive, setting all membrane-related parameters in the spiking neuron learnable to strictly ﬁt the neuronal dynamics to the target activation function.
Compared with the above methods, GLIF adopts the channel-wise parametric method to fully parameterize the spiking neuron, including learnable exponential decay, linear decay, potential threshold, resetting voltage, input conductance, and gating factors.
Supervised direct learning of SNNs.
Supervised direct learning of SNNs follows the idea of backpropagation (BP) in ANN[40].
Bohte et al.
[41] ﬁrst adopt the surrogate gradient (SG) to empirically solve the non-differentiable term of spiking neurons by approximation with the local derivative of potential.
Based on the observation that the proper steepness of the SG curve matters more than the shape in SNN learning[42], Li et al.
[11] conduct a pre-estimation before each epoch to ﬁnd the SG curve with the optimal smoothness.
Besides, many techniques are developed to make deep SNNs converge faster, such as NeuNorm[8], tdBN[9], SEW block[43], TET[12], MS residual block[44], etc.
In this way, the problem of end-to-end training of deep SNNs can be solved.
In this paper, we choose supervised direct learning of SNNs to estimate the effectiveness of our GLIF.
Methodology 3.1 Revisiting LIF models
Vanilla LIF.
The Vanilla LIF model is commonly adopted as a spiking neuron model in SNNs, which can be discretely formulated as follows:
[Equation]
t
[Equation] [Equation]
[Equation] [Equation]
Here, (cid:12) denotes the element-wise multiplication.
U ( t,l ) is the membrane potential vector of the layer l at time-step t and can be updated in a discrete way through Eq.(1), where C ( t,l ) denotes the input vector and can be obtained by the dot-product between the synaptic weight matrix W and the output spike vector of the previous layer S ( t,l − 1) , as described in Eq.(2).
The output spike vector
S ( t,l ) is given by the Heaviside step function H ( · ) in Eq.(3), indicating that a spike is ﬁred when the membrane potential exceeds the potential threshold V
th .
When updating U ( t,l ) in Eq.(1), the three neuronal behaviors are considered.
Speciﬁcally, the time constant τ acts as the exponential decay coefﬁcient on the term τ U ( t − 1 ,l ) to implement exponential decay, C ( t,l ) is integrated over time to
U ( t,l ) , and the term (1 − S ( t − 1 ,l ) ) hard resets the membrane potential to 0 if a spike is ﬁred at the previous time step.
Bio-features & Primitives.
Bio-features represent the characteristics of the three critical neuronal behavior.
For the instance of vanilla LIF, its bio-features can be described as exponential decay, uniform-coding scheme, and hard reset.
And, the three bio-features qualitatively describe the neuronal behavior of membrane potential leakage, integration accumulation, and spike initiation, respectively.
Since each behavior is actually implemented by the corresponding term in formulas, for explicitness, we refer to those neuronal behavior-related parameters as primitives in this paper.
In this way, biofeatures can be quantitatively described by primitives.
For example, exponential decay is realized by the term τ U ( t − 1 ,l ) in Eq.(1), so τ is a primitive and determines the amplitude of exponential decay.
Thus, by choosing different primitives for each behavior, we can formulate different LIF variants.
For instance, with the linear decay primitive τ lin and the soft reset primitive τ re , we have:
U (
[Equation] [Equation]
) =
[Equation]
re
[Equation]
+ C
[Equation]
t
[Equation]
l ) , which formulates the spiking neuron in [15].
3.2 Gated LIF Model
Although variant LIF offers a different choice in modeling the spiking neuron, it still remains simplex, i.e., the bio-features of the neuronal dynamic are unitary.
Once formulated, a hard-reset LIF neuron cannot perform the soft-reset operation.
As a result, these simplex LIF models make the spiking neurons in SNNs lack dynamic diversity.
Thus, we propose the gated LIF (GLIF), a uniﬁed spiking neuron, to fuse different bio-features in the three neuronal behaviors.
GLIF is illustrated in Fig.
1 and further described as follows:
[Equation]
In Eq.(4), L ( t,l ) denotes the membrane potential vector after membrane potential leakage, I ( t,l ) denotes the vector of the incremental potential caused by integration accumulation, and F ( t,l ) denotes the vector of the potential reduction caused by spike initiation.
Obviously, the above three vectors reﬂect the consequences of the three neuronal behaviors.
In Eq.(7)(8)(9), the gating units G α ( · ) , G β ( · ) , and G γ ( · ) are proposed to formulate the three neuronal behaviors, thus computing the consequent vectors.
Computing L ( t,l ) .
In computing L ( t,l ) , the two primitives τ lin and τ exp are considered.
τ lin is the linear decay primitive, and τ exp is the exponential decay primitive.
As their names indicate, τ lin and τ exp play the roles of the linear decay and exponential decay on U ( t − 1 ,l ) .
Through the gating unit G α ( · ) , the two primitives are fused together to perform membrane potential leakage.
Following is the formulation of G α ( · ) :
[Equation]
where α is the gating factor of G α ( · ) , whose value is bounded within (0 , 1) .
When α approaches 0 , [1 − α (1 − τ exp )] and (1 − α ) tend to 1 , which means GLIF performs linear decay.
As α increases from 0 to 1 , (1 − α )
τ lin decreases from τ lin to 0 , while [1 − α (1 − τ exp )] decreases from 1 to τ exp , which means GLIF is shifting from linear decay to exponential decay.
Therefore, in Eq.(10), the two primitives τ lin and τ exp are dually fused with the help of the gating factor α , which can also balance the amplitudes of linear decay and exponential decay.
Note that the primitives in the same gating unit are referred to as the dual primitives in the following content, and correspondingly, the bio-features the dual primitives describe are referred to as the dual bio-features .
In the extreme case of α =or 1 , G α preserves only single primitive, which means GLIF covers the simplex formulations of membrane potential leakage.
Computing I ( t,l ) .
In computing I ( t,l ) , the dual primitives g t and 1 are considered to perform integration accumulation.
g t denotes the conductance primitive, namely, the time-dependent synaptic weight, which can be coded with different patterns, such as rank-order based patterns[29] and phasecoding based patterns[31].
So, we can view it as the ﬂexible-coding primitive.
While, the constant 1 means g t ≡ 1 , representing the uniform-coding primitive, which is the dominant paradigm and proven doing well in the direct learning of SNNs[45, 12, 11].
Following is the formulation of G β ( · ) :
[Equation]
(11) where β , bounded within (0 , 1) , is the gating factor of G β ( · ) .
Similarly, as β → 0 , the uniform-coding pattern is preferred rather than the ﬂexible-coding, while β → 1 is the opposite case.
When β = 0 or β = 1 , GLIF supports single spike coding pattern of integration accumulation.
Computing F ( t,l ) .
In computing F ( t,l ) , the soft-reset primitive V re and the hard-reset primitiveare coupled as the dual primitives to perform spike initiation.
Following is the formulation of G γ
[Equation]
[Equation]
where γ bounded within (0 , 1) is the gating factor of G γ ( · ) .
The term (1 − γ ) V re is the actual potential decrease caused by soft reset, while the term ( 0 + γ ) L ( t,l ) exp is the actual potential decrease caused by hard reset.
Note that L ( t,l ) exp is given by [1 − α (1 − τ exp )]
U ( t − 1 ,l ) in Eq.(10), which means
GLIF only performs hard reset on the exponential-decay-related part of the membrane potential.
In a similar way, as γ → 0 , soft reset is incrementally biased, while γ → 1 , hard reset is incrementally biased.
When γ = 0 or 1 , GLIF supports the simplex reset formulation of spike initiation.
Therefore, with gating units, the GLIF-based spiking neurons can fuse the dual bio-features in each one of the three neuronal behaviors.
When gating factors are binary, GLIF transforms into the corresponding simplex LIFs.
For example, in the case of α, β, γ = 1 , 0 , 1 , GLIF is equivalent to the vanilla LIF.
In a nutshell, GLIF uniﬁes different simplex LIFs by incorporating dual bio-features and exploits gating factors to balance the dual bio-features.
Channel-wise parametric method.
The parametric method is applied to all the membrane-related parameters in our GLIF, including primitives ( τ lin , τ exp , V re , g t , and V th ) and gating factors ( α , β , and γ ).
Given x is one of these membrane-related parameters, we can obtain x by the Sigmoid function: x = 1 1+exp( − x p ) where x p ∈ ( −∞ , + ∞ ) is a learnable parameter that changes at each training iteration.
With all these membrane-related parameters trainable, the adaptivity of the spiking neurons is increased.
Such that, GLIF neurons can adjust their neuronal response characteristics in a wide range during training.
Furthermore, prior SNNs commonly adopt the same membrane-related parameters for all spiking neurons throughout the network, totally neglecting the heterogeneity.
While in GLIF, the channelwise sharing scheme, where spiking neurons in the same channel share a unique setting of the membrane-related parameters but inter-channel spiking neurons don’t, is employed.
With the channel-wise sharing scheme and the parametric method combined, we can greatly increase the heterogeneity of spiking neurons after training.
The heterogeneity over channels and layers can be observed as we visualize the membrane-related parameters in Appendix A.1.
Training framework.
In terms of the training framework, we utilize backpropagation through time (BPTT) as the training algorithm for our GLIF SNNs.
As for the non-differentiable Heaviside step function in Eq.(6), we solve it by the surrogate gradient deﬁned as:
[Equation] [Equation]
Leak
V th V th
V th
V re V re
V re γ γ γ
Leak
V th V th
V th
V re V re
V re γ γ γ
Leak
α τ lin τ exp
Soft & hard reset
V re (c) (d) (b) (a) α β
Leak
V th V th
V th
V re V re
V re γ γ γ
Leak
V th V th
V th
V re V re
V re γ γ γ
Thus, GLIF SNNs can be trained end-to-end.
3.3 Dynamic Analysis
GLIF contains dual bio-features, which brings great diversity to the neuronal dynamics.
Here we use a simple case to analyze the dynamic properties of GLIF.
Fig.
(2) illustrates the response of the spiking neurons to the constant input, which spikes each time step until the membrane potential reaches the threshold.
As shown in the ﬁgure, GLIF neurons with different parameter conﬁgurations have completely different dynamic responses to the same input.
Bridging the gap between exponential decay and linear decay.
As shown in Fig. 2(a), compared with non-decay, on the one hand, exponential decay brings the neuronal dynamic with stability and astringency, making it increasingly difﬁcult for the membrane potential to climb over the threshold.
However, the membrane potential falls into the saturation zone and fails to reach the potential threshold, which indicates exponential decay may impair the spiking responsiveness.
On the other hand, linear decay operates equally at each time step to avoid saturation, thus maintaining the high spiking responsiveness.
However, if there are no inputs, linear decay leads to the potential decreasing in an unlimited way, which impairs the stability.
While in GLIF, the above two types of decay are the extreme cases, i.e., GLIF only acts exponential decay if α = 1 and only serves linear decay if α = 0 .
In most cases where α ∈ (0 , 1) , GLIF is eligible for exponential decay’s astringency and beneﬁts from the high spiking responsiveness linear decay brings about.
To some extent, the gating factor α makes the two types of decay complementary and determines the proportion of each of the two decay, bridging the gap between the neuronal dynamics of exponential decay and linear decay and enlarging the representation space of membrane potential leakage.
Furthermore, since τ exp and τ lin are learnable in GLIF, the gap between exponential and linear decay becomes more ﬂexible.
Therefore, the actual representation space given by τ exp , τ lin , and α could be even larger.
Incorporating time-dependent synaptic weights.
In GLIF, we assign the time-dependent synaptic weights g t to the spike train to differentiate spike stimulation levels at different time steps.
Since g t is learnable, the ﬂexible-coding patterns of different GLIF neurons may differ signiﬁcantly.
For the convenience of analysis, we add constraints to g t , assuming that g t is in the form of cosine oscillation, where β controls the amplitude.
As shown in Fig.
2(b), when β = 0 , which means the uniform-coding pattern, each time the spiking neuron receives an input spike, the membrane potential increases by the same amount.
Our GLIF is different.
When β ∈ (0 , 1) , each time the GLIF receives an input spike, the membrane potential responds differently depending on the particular stimulation level of the current input spike.
On the one hand, considering that there is no constraint on g t in practice, GLIF is more ﬂexible, and the representation space is further enlarged.
On the other hand, because the stimulus of input spikes at different times are considered differently, incorporating time-dependent synaptic weights may also help to better process temporal features.
Bridging the gap between hard reset and soft reset.
Fig. 2(c) compares the neuronal dynamics between soft reset and hard reset.
As we can see, soft reset reduces the current membrane potential by
V re .
Since V re is not related to the current membrane potential, soft reset may lead to over-resetting or under-resetting of membrane potential, which is ﬂexible for the spiking neurons.
Hard reset reduces the current membrane potential to an empirical value, e.g., 0 , restarting the potential trace all over, providing the neuronal dynamics with stability.
While in GLIF, we consider the potential reduction caused by spike initiation to be both related to V re and the current membrane potential.
Because γ ∈ (0 , 1) balances the amplitudes of the two resets, GLIF can reset the membrane potential to any possible levels between the consequent potentials of hard reset and soft reset as shown in Fig. 2(d), enlarging the representation space of spike initiation.
Experiments
In this section, we demonstrate the effectiveness of our proposed GLIF model.
We ﬁrst compare our results with other existing state-of-the-arts, then carry out ablation studies to evaluate different aspects of our proposed method.
4.1 Experimental Settings
We perform experiments on two kinds of datasets.
The ﬁrst is image classiﬁcation datasets, including
CIFAR[46] and ImageNet[47].
The other one is the event dataset, i.e., CIFAR10-DVS [48], which is converted by a popular image recognition dataset CIFAR-10 using a DVS camera.
For a fair comparison with other methods, we choose the commonly used network architectures in existing works for experiments, including ResNet-family[49, 9, 44], 7B-wideNet[43], and CIFARNet[42, 8, 50], and apply tdBN[9].
More implementation details of training those SNNs are revealed in
Appendix A.2.
4.2 Comparison with Existing Works
CIFAR.
We test our GLIF on CIFAR-10 and CIFAR-100 datasets and run three times for each experiment to report the “mean ± std”.
As shown in Table 1, our GLIF SNNs can outperform existing works.
Speciﬁcally, on CIFAR-10, we ﬁrst test our GLIF on a smaller network CIFARNet and achieve the superior accuracy over other methods [42, 8, 50], even with fewer time steps.
Since tdBN is employed in our GLIF SNNs, we then compare GLIF with STBP-tdBN[51] on ResNet-19.
By simply changing the neurons in STBP-tdBN to GLIF, our SNNs can achieve signiﬁcant improvement across all the tested time-steps.
Compared with the state-of-the-art TET[12], our method can still obtain better accuracy, and our ResNet-19 with time-step 4 even outperforms theirs with time-step 6 by 0 .
On CIFAR-100, our method can also achieve much better performance than STBP-tdBN and TET.
We also compare GLIF ResNet-18 with the highly handcrafted ResNet-18 of Dspike[11].
As listed in Table 1, our GLIF ResNet-18 can outperform Dspike ResNet-18 by at least 0 .
ImageNet.
We apply GLIF to ResNet-34 and MS-ResNet-18[44].
Table 2 shows the results on the validation set.
We ﬁrst compare our method with STBP-tdBN, which can be viewed as our baseline, on ResNet-34, and we can observe that our GLIF SNNs achieve better performance ( 69 . 09% v.s. 63 .
Since parametric technology is employed in our GLIF, we also compare GLIF with the parametric spiking neuron PLIF[43].
Without changing the model structure, our method can
[Table]
[Table] [Table]
still achieve higher accuracy than PLIF on ResNet-34 ( 67 . 52% v
[Table]
s. 67 .
Moreover, compared with MS-ResNet[44]
[Table]
whose handcrafted residual block achieves the state-of-the-art performance, we apply our GLIF to MS-ResNet-18 and can also signiﬁcantly increase the accuracy ( 68 .
11% v
[Table]
s.
These results demonstrate the effectiveness of our proposed GLIF model.
In addition, compared with other SNNs in TET[12] and Dspike[11], GLIF SNNs can also obtain higher accuracy.
CIFAR10-DVS.
To demonstrate the effectiveness of GLIF in processing spatio-temporal features, we conduct experiments on the event-based dataset CIFAR10-DVS.
Since SEW+PLIF[43] is the most close to our work for it also exploits parametric methods and tdBN, we compare with existing methods PLIF[10], SEW-PLIF[43], STBP-tdBN[9], and LIAF[28] on 7B-wideNet.
As shown in
Table 3, our GLIF SNNs can outperform all of those methods.
Speciﬁcally, GLIF outperforms
SEW-PLIF on the same network architecture 7B-wideNet, which demonstrates the effectiveness of the proposed GLIF on event-based dataset.
[Table]
[Table] [Table]
4.3 Ablation Study
Effectiveness of different bio-features.
To demonstrate the effectiveness of different bio-features that are incorporated in our GLIF, we evaluate the performance of GLIF SNNs in extreme cases where GLIF only acts single bio-feature in each neuronal behavior.
Since the gating factors control bio-features, we can obtain eight simplex LIFs by combining different gated biometrics, all of which have static binary gating factors.
We name those simplex LIFs with the values of their gating factors, e.g., 101 is the GLIF with α, β, γ = 1 , 0 , 1 that only acts exponential decay, the uniform-coding pattern, and hard reset.
On CIFAR-100, we use ResNet-18 and ResNet-19 with time-step 4 for experiments.
As shown in Table 4, our proposed GLIF outperforms all of those simplex LIFs, which demonstrates the effectiveness of fusing different bio-features.
Importance of learnable gating factors To demonstrate the effectiveness of our proposed learnable gating factors in fusing bio-features, we compare GLIF with other two GLIF variants: GLIF_s and
GLIF_f, where GLIF_s keeps gating factors static after initialization, GLIF_f removes gating-factors and simply fuses two different bio-features by addition.
On CIFAR-100, we also use ResNetand ResNet-
with time-step 4.
Table 5 lists the experimental results.
On the one hand, compared with GLIF_s, our method can get higher accuracy, which demonstrates that learnable gating factors can make GLIF better.
On the other hand, compared with GLIF_f, GLIF can also achieve better performance, this is because learnable gating factors can balance dual bio-features in each neuronal behavior.
As a result, learnable gating factors are important in GLIF.
Channel-wise v.s. Layer-wise.
Besides the investigations above, we conduct an ablation study on GLIF with different parameter-sharing schemes, which are layer-wise sharing and channel-wise sharing.
We use ResNet-19 and ResNet-18 with time-step 2 and 4 on CIFAR-100.
As listed in Table 6, GLIF, which leverages channel-wise sharing scheme, consistently improves the performance over the layer-wise sharing scheme.
Conclusion
In this paper, we propose GLIF, a uniﬁed spiking neuron, that fuses the bio-features in each neuronal behavior.
In GLIF, our method exploits the gating factor to bridge the gap between the dual biofeatures, enlarging the representation space and enhancing the neuronal dynamic diversity of spiking neurons.
Furthermore, we adopt the channel-wise parametric method to make all membrane-related parameters learnable, as well as the gating factors, and train those parameters through BPTT, increasing the neuronal adaptivity of spiking neurons.
We validate our GLIF SNNs on various benchmark datasets, showing the state-of-the-art performance compared with other methods.
Therefore, we think
GLIF could be a positive answer to the issue: could the higher neuronal dynamic diversity of spiking neurons help SNNs do better?
Additionally, the distributions of learned parameters is interesting as we visualize them in Appendix
A.1.
The initially identical parameters learn into different bell-shaped distributions layer-wisely.
We think this could be further studied to mine the biological plausibility of SNNs.
How to apply GLIF in other training frameworks, e g., ANN2SNN conversion, also remains a worth-studying problem
A Appendix A.1 Visualization of Learned Parameters
This section visualizes the distributions of the learned parameters from ResNet-18 on CIFAR-100.
[Figure] [Figure]
Fig.
3 illustrates the distributions of learned gating factors α , β
[Figure]
and γ .
Initially, they are all uniformly sampled from [0.4502, 0.5498).
After training, the property of uniform distributions remains
[Figure]
but the ranges are all narrowed and the displacement of each gating factor is rather small.
This indicates dual bio-features of each behavior tend to be considered equally effective for the whole SNN.
Such that, compared to the single bio-feature, the hybrid bio-feature are inclined to by the SNN.
Fig.
4 illustrates the distributions of learned primitives V th , reV th , τ exp and τ lin .
Two interesting phenomena can be observed.
For one thing, neurons in the same layer are inclined to the same displacement direction and the distribution of their primitives naturally form into the phenotypes of bell-shaped distributions.
This is quite similar to the hierarchical structures of brains.
For another, as a whole, V th tends to be smaller which indicates the potential required for a spike is smaller.
reV th tends to be smaller which indicates the potential loss for a soft-reset is smaller.
τ exp tends to be bigger which indicates the potential loss for decay is smaller.
τ lin overall tends to remain still.
Overall, the potential loss becomes smaller and the potential required for a spike becomes less.
Therefore, the whole SNN learns to be more active or excitatory, which is similar to the on-task state of brains.
Above all, the heterogeneity of the GLIF-based ResNet-18 is proven greatly increased through visualizing the distributions of the learned parameters.
Plus, the learned parameters somehow embodies some biological properties of brains.
Such observations could be further studied to mine the biological plausibility of SNNs.
A.2 Implementation Details
Training details.
For all of our experiments, we use the stochastic gradient descent (SGD) optimizer with a momentum of 0.9 and weight decay of 5 e − 5 to optimize the model.
No weight decay is applied on the parameters in GLIFs.
The learning rates of the GLIF parameters and the model parameters are both reduced by the cosine-annealing scheduler. Specially for CIFAR, gating factors’ learning rate is set to 1 / 10 of other parameters’.
Specially for CIFAR10-DVS, no weight decay is applied to the whole SNN and the annealing period ( T max ) of the cosine-annealing scheduler is 64.
As for the initialization of GLIF parameters, gating factors are initialized with the random values sampled from [0.4502, 0.5498), while the primitives are initialized as listed in Table 7.We train all of the models on A100 GPU, and other details for training different models are listed in Table 8.
[Table]
Architecture
[Table]
Architecture.
ResNet-18 is originally designed for ImageNet task in [49].
To process CIFAR tasks, we remove the max-pooling layer, replace the ﬁrst 7 × 7 convolution layer with a 3 × 3 convolution layer
[Table]
and replace the ﬁrst and the second 2-stride convolution operations as 1-stride, following the modiﬁcation philosophy from STBP-tdBN[9].
7B-wideNet[43] is originally designed to receive frames with 128 × 128 pixels in each channel (two in total).
Whereas, we resize each frame into 48 × 48 pixels in our DVS data pre-processing.
Therefore
[Table]
we reschedule the down-sampling strategy so that 7B-wideNet is constructed as c64k3s1BN-GLIF-{SEW Block (c64)}*3-APk2s2-c128k3s1-BN-GLIF-{SEW Block (c128)-APk2s2}*3-FC11 , where c32k3s1 represents the convolutional layer with kernel size 3 stride 1, APk2s2 is the average pooling with kernel size 2 and stride 2, SEW Block is proposed and illustrated in [43], and the symbol {}* n denotes the structure {} is repeated n times.
Notably, our implemented 7B-wideNet is the same size of [43], but has fewer operations due to the smaller input size.
Except for the above networks, we use the same architectures as the prior works.
Data Pre-processing for CIFAR10-DVS.
We ﬁrst separate the whole dataset into 9000 training samples and 1000 test samples.
In the data pre-processing of single sample of CIFAR10-DVS, we split the event stream of the sample into 16 slices and integrate the events in each slice into one frame, and resize each frame into 48 × 48 pixels.
A.3 Coarsely Fused LIF
In this section, we formulate the GLIF_f, which is a coarsely fused LIF without gating factors and utilized as a comparison in the ablation study.
In the GLIF_f, primitives are directly stacked together as follows:
[Equation]
Although, the representation space of GLIF_f is larger than GLIF due to the removal of the gating mechanism.
As listed in Table 5, GLIF_f performs even worse than GLIF_s.
This indicates the balance between dual primitives is helpful in regulating the representation space within a valid range.