GLIF: A Uniﬁed Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks
Abstract
Spiking Neural Networks (SNNs) have been studied over decades to incorporate their biological plausibility and leverage their promising energy efﬁciency.
Throughout existing SNNs, the leaky integrate-and-ﬁre (LIF) model is commonlyadopted to formulate the spiking neuron and evolves into numerous variants withdifferent biological features.
However, most LIF-based neurons support only singlebiological feature in different neuronal behaviors, limiting their expressivenessand neuronal dynamic diversity.
In this paper, we propose GLIF, a uniﬁed spikingneuron, to fuse different bio-features in different neuronal behaviors, enlargingthe representation space of spiking neurons.
In GLIF, gating factors, which areexploited to determine the proportion of the fused bio-features, are learnable duringtraining.
Combining all learnable membrane-related parameters, our method canmake spiking neurons different and constantly changing, thus increasing the heterogeneity and adaptivity of spiking neurons.
Extensive experiments on a variety ofdatasets demonstrate that our method obtains superior performance compared withother SNNs by simply changing their neuronal formulations to GLIF.
In particular,we train a spiking ResNet-19 with GLIF and achieve 77 .
35% top-1 accuracy withsix time steps on CIFAR-100, which has advanced the state-of-the-art.
Introduction
Spiking neural network (SNN) is considered the third generation of the neural network and thebionic modeling of its activation unit is called the spiking neuron, which brings about the spike-basedcommunication between layers[1, 2].
Due to the high sparsity and multiplication-free operationin information processing, such spike-based communication can improve energy efﬁciency[3, 4].
Moreover, spiking neuron can effectively capture temporal information and has good performance inprocessing neuromorphic data[5] and anti-noise[6].
Together with artiﬁcial neural network (ANN)colleagues, SNN further shows great potential for general intelligence[7].
Thus, SNN has arousedheated research interests recently.
The commonly used spiking neuron in different SNNs[6, 8, 9, 10, 11, 12] is the leaky integrateand-ﬁre (LIF) model, named vanilla LIF.
As the name suggests, it can implement three differentneuronal behaviors: membrane potential leakage, integration accumulation, and spike initiation.
Tobetter model neurons, prior works [13, 14, 15, 16, 17] have improved the LIF model from the abovethree aspects, and proposed many LIF variants with different biological features.
For all the above
LIF models (referred to as simplex LIF), any of the three neuronal behaviors supports only singlebiological feature.
For instance, the vanilla LIF model only exploits exponential decay in membranepotential leakage, while [15] only supports linear decay.
However, different neurons in the cerebralcortex have different response characteristics[18, 19].This naturally raises an issue: could higher τ exp
S t −
U t − τ lin
V re + ×
S t
U t g t
C t +
U t
S t (a) (b) (c) σ ( x )  × ×−/
𝔾 γ
𝔾 α 𝔾 β
L t
F t
I t neuronal dynamic diversity of spiking neurons help SNNs do better?
In this paper, we investigate thehybrid bio-features in the LIF model to answer this question.
We propose the gated LIF model (GLIF) that fuses different bio-features in the aforementionedthree neuronal behaviors to possess more response characteristics.
As illustrated in Fig.
1, GLIFcontrols the fusion of different bio-features through gating units G α , G β , and G γ for those threeneuronal behaviors that are membrane potential leakage, integration accumulation, and spike initiation,respectively.
In each gating unit, a gating factor is computed from a Sigmoid function σ ( x ) overa learnable gating parameter x to determine the proportion of each bio-feature, thus guiding thefusion of different bio-features.
Therefore, GLIF can simultaneously contain different bio-features,possessing more response characteristics.
In addition, when the gating factor is 0 or 1 , GLIF can alsosupport single bio-feature.
As a result, GLIF can cover other different LIF models and can be viewedas a super spiking neuron, greatly enlarging the representation space of spiking neurons.
Furthermore, we introduce the channel-wise parametric method to GLIF.
This method makes allmembrane-related parameters in GLIF learnable and shares the same GLIF parameters channelwisely in SNNs.
Combining with learnable gating factors in GLIF, on the one hand, this methodmakes different channels in SNNs have completely different spiking neurons, leveraging the largerrepresentation space of GLIF neurons to increase the neuronal dynamic diversity of spiking neurons.
Meanwhile, the heterogeneity of spiking neurons and the expressive ability of SNNs are also increased.
On the other hand, the spike neurons in SNNs are constantly changing during training, which issimilar to the neuronal maturation during development[20, 21, 22], thus the adaptivity of spikingneurons being enhanced.
Our contributions are as follows:
• We propose the GLIF model, a uniﬁed formulation of the spiking neuron that fuses multiplebiological features through gating units, enlarging the representation space of spikingneurons.•
We exploit the channel-wise parametric method to leverage the larger representation spaceof GLIF, thus increasing the neuronal dynamic diversity in SNNs.• Our experiments demonstrate the effectiveness of the GLIF model on both static datasets andthe neuromorphic dataset.
For example, our model can achieve the state-of-the-art 75 .
48%top-1 accuracy on CIFAR-10 with only 2 time steps.
Related Work
Variant LIF .
Variant LIF models with different biological features are proposed in previous arts,and those models differ in one or more of the three different neuronal behaviors: membrane potentialleakage, integration accumulation, and spike initiation.
Firstly, membrane potential leakage representsthe potential decay mechanism.
Exponential decay is ﬁrst exploited in the vanilla LIF model[23], andit brings neuronal dynamics with stable convergence, shrinking the potential to the resting potentialsteadily.
But, exponential decay is not efﬁcient enough in massive neural simulations due to itsmultiplication operation.
Therefore, linear decay replaces exponential decay in [13, 24, 25, 26, 27],and brings neuronal dynamics with the ﬂexibility, making it possible for conﬁgurable spiking rates[15].
Although the two decays behave differently in neuronal dynamics, both of them are proveneffective in SNNs[6, 28].
Secondly, integration accumulation represents the potential accumulationinduced by presynaptic spikes.
Commonly, spike intensities of different time steps are consideredequally high.
Consequently, the discrepancies among the spikes at different time steps are ignored.
To address this issue, many prior works propose to assign different weights to the input spikes atdifferent time steps.
These methods are implemented by manual settings[29, 30, 31] or adaptivesettings[32, 33, 34] to improve biological plausibility, SNN performance, or inference efﬁciency.
Lastly, spike initiation functions as the resetting mechanism triggered by spike ﬁring.
There are twocommon resetting strategies: hard reset[35] and soft reset[15, 36].
When a spike is triggered, the hardreset directly sets the potential to the resting potential, while the soft reset subtracts the potential by aconﬁgurable value.
Therefore, hard reset is more stable as a strict reset mechanism, while soft reset ismore ﬂexible making conﬁgurable spiking rates feasible[15].
Compared with the LIF models above,our GLIF incorporates different neuronal dynamic features to increase the representation space of
LIF models and improve performance of SNNs.
Parametric spiking neurons .
Early existing spiking neurons often require manual tuning of themembrane-related parameters to set their neuronal dynamics, and these parameters are often sharedby all neurons, which limits the diversity of spiking neurons and the expressive ability of SNNs.
Inrecent years, Parametric spiking neurons[37, 38, 10, 28] have been proposed, which set one or moremembrane-related parameters to be learnable, and update these parameters during the training of
SNNs.
[37] and [38] set the membrane leak and potential threshold learnable, while [10] and [28]choose the membrane leak only.
FS-neuron[39] is more aggressive, setting all membrane-relatedparameters in the spiking neuron learnable to strictly ﬁt the neuronal dynamics to the target activationfunction.
Compared with the above methods, GLIF adopts the channel-wise parametric method tofully parameterize the spiking neuron, including learnable exponential decay, linear decay, potentialthreshold, resetting voltage, input conductance, and gating factors.
Supervised direct learning of SNNs.
Supervised direct learning of SNNs follows the idea ofbackpropagation (BP) in ANN[40].
Bohte et al.
[41] ﬁrst adopt the surrogate gradient (SG) toempirically solve the non-differentiable term of spiking neurons by approximation with the localderivative of potential.
Based on the observation that the proper steepness of the SG curve mattersmore than the shape in SNN learning[42], Li et al.
[11] conduct a pre-estimation before each epoch toﬁnd the SG curve with the optimal smoothness.
Besides, many techniques are developed to makedeep SNNs converge faster, such as NeuNorm[8], tdBN[9], SEW block[43], TET[12], MS residualblock[44], etc.
In this way, the problem of end-to-end training of deep SNNs can be solved.
In thispaper, we choose supervised direct learning of SNNs to estimate the effectiveness of our GLIF.
Methodology
3.1 Revisiting LIF models
Vanilla LIF.
The Vanilla LIF model is commonly adopted as a spiking neuron model in SNNs,which can be discretely formulated as follows:
[Equation]
Here, (cid:12) denotes the element-wise multiplication.
U ( t,l ) is the membrane potential vector of thelayer l at time-step t and can be updated in a discrete way through Eq.(1), where C ( t,l ) denotes theinput vector and can be obtained by the dot-product between the synaptic weight matrix W and theoutput spike vector of the previous layer S ( t,l − 1) , as described in Eq.(2).
The output spike vector
S ( t,l ) is given by the Heaviside step function H ( · ) in Eq.(3), indicating that a spike is ﬁred when themembrane potential exceeds the potential threshold V
th .
When updating U ( t,l ) in Eq.(1), the threeneuronal behaviors are considered.
Speciﬁcally, the time constant τ acts as the exponential decaycoefﬁcient on the term τ U ( t − 1 ,l ) to implement exponential decay, C ( t,l ) is integrated over time to
U ( t,l ) , and the term (1 − S ( t − 1 ,l ) ) hard resets the membrane potential to 0 if a spike is ﬁred at theprevious time step.
Bio-features & Primitives.
Bio-features represent the characteristics of the three critical neuronalbehavior.
For the instance of vanilla LIF, its bio-features can described as exponential decay, uniformcoding scheme, and hard reset.
And, the three bio-features qualitatively describe the neuronalbehavior of membrane potential leakage, integration accumulation, and spike initiation, respectively.
Since each behavior is actually implemented by the corresponding term in formulas, for explicitness,we refer to those neural behavior related parameters as primitives in this paper.
In this way, biofeatures can be quantitatively described by primitives.
For example, exponential decay is realizedby the term τ U ( t − 1 ,l ) in Eq.(1), so τ is a primitive and determines the amplitude of exponentialdecay.
Thus, by choosing different primitives for each behavior, we can formulate different LIFvariants.
For instance, with the linear decay primitive τ lin and the soft reset primitive τ re , we have:
[Equation] [Equation]
C
[Equation]
t,l ) , which formulates the spiking neuron in [15].
3.2 Gated LIF Model
Although variant LIF offers a different choice in modeling the spiking neuron, it still remains simplex,i.e., the bio-features of the neuronal dynamic are unitary.
Once formulated, a hard-reset LIF neuroncannot perform the soft-reset operation.
As a result, these simplex LIF models make the spikingneurons in SNNs lack dynamic diversity.
Thus, we propose the gated LIF (GLIF), a uniﬁed spikingneuron, to fuse different bio-features in the three neuronal behaviors.
GLIF is illustrated in Fig.
1 andfurther described as follows:
[Equation]
[Equation]
In Eq.(4), L ( t,l ) denotes the membrane potential vector after membrane potential leakage, I ( t,l )denotes the vector of the incremental potential caused by integration accumulation, and F ( t,l ) denotesthe vector of the potential reduction caused by spike initiation.
Obviously, the above three vectorsreﬂect the consequences of the three neuronal behaviors.
In Eq.(7)(8)(9), the gating units G α ( · ) , 144 G β ( · ) , and G γ ( · ) are proposed to formulate the three neuronal behaviors, thus computing theconsequent vectors.
Computing L ( t,l ) .
In computing L ( t,l ) , the two primitives τ lin and τ exp are considered.
τ lin is thelinear decay primitive, and τ exp is the exponential decay primitive.
As their names indicate, τ lin andτ exp play the roles of the linear decay and exponential decay on U ( t − 1 ,l ) .
Through the gating unit 149 G α ( · ) , the two primitives are fused together to perform membrane potential leakage.
Following isthe formulation of G α ( · ) :
[Equation]
where α is the gating factor of G α ( · ) , whose value is bounded within (0 , 1) .
When α approaches 0 ,[1 − α (1 − τ exp )] and (1 − α ) tend to 1 , which means GLIF performs linear decay.
As α increases from 0 to 1 , (1 − α )
τ lin decreases from τ lin to 0 , while [1 − α (1 − τ exp )] decreases from 1 to τ exp ,which means GLIF is shifting from linear decay to exponential decay.
Therefore, in Eq.(10), the twoprimitives τ lin and τ exp are dually fused with the help of the gating factor α , which can also balancethe amplitudes of linear decay and exponential decay.
Note that the primitives in the same gating unitare referred to as the dual primitives in the following content, and correspondingly, the bio-featuresthe dual primitives describe are referred to as the dual bio-features .
In the extreme case of α = 0or 1 , G α preserves only single primitive, which means GLIF covers the simplex formulations ofmembrane potential leakage.
Computing I ( t,l ) .
In computing I ( t,l ) , the dual primitives g t and 1 are considered to performintegration accumulation.
g t denotes the conductance primitive, namely, the time-dependent synapticweight, which can be coded with different patterns, such as rank-order based patterns[29] and phasecoding based patterns[31].
So, we can view it as the ﬂexible-coding primitive.
While, the constant1 means g t ≡ 1 , representing the uniform-coding primitive, which is the dominant paradigm andproven doing well in the direct learning of SNNs[45, 12, 11].
Following is the formulation of G β ( · ) :
[Equation]
(11) where β , bounded within (0 , 1) , is the gating factor of G β ( · ) .
Similarly, as β → 0 , the uniform-codingpattern is preferred rather than the ﬂexible-coding, while β → 1 is the opposite case.
When β = 0 orβ = 1 , GLIF supports single spike coding pattern of integration accumulation.
Computing F ( t,l ) .
In computing F ( t,l ) , the soft-reset primitive V re and the hard-reset primitive 0are coupled as the dual primitives to perform spike initiation.
Following is the formulation of G γ ( · ) :
[Equation]
where γ bounded within (0 , 1) is the gating factor of G γ ( · ) .
The term (1 − γ ) V re is the actualpotential decrease caused by soft reset, while the term ( 0 + γ ) L ( t,l ) exp is the actual potential decreasecaused by hard reset.
Note that L ( t,l ) exp is given by [1 − α (1 − τ exp )]
U ( t − 1 ,l ) in Eq.(10), which means
GLIF only performs hard reset on the exponential-decay-related part of the membrane potential.
In asimilar way, as γ → 0 , soft reset is incrementally biased, while γ → 1 , hard reset is incrementallybiased.
When γ = 0 or 1 , GLIF supports the simplex reset formulation of spike initiation.
Therefore, with gating units, the GLIF-based spiking neurons can fuse the dual bio-features in eachone of the three neuronal behaviors.
When gating factors are binary, GLIF transforms into thecorresponding simplex LIFs.
For example, in the case of α, β, γ = 1 , 0 , 1 , GLIF is equivalent to thevanilla LIF.
In a nutshell, GLIF uniﬁes different simplex LIFs by incorporating dual bio-features andexploits gating factors to balance the dual bio-features.
Channel-wise parametric method.
Parametric method is applied to all the membrane-relatedparameters in our GLIF, including primitives ( τ lin , τ exp , V re , g t , and V th ) and gating factors ( α ,β , and γ ).
Given x is one of these membrane-related parameters, we can obtain x by the Sigmoidfunction
[Equation]
(13) where x p ∈ ( −∞ , + ∞ ) is a learnable parameter that changes in every training iteration.
With allthese membrane-related parameters trainable, the adaptivity of the spiking neurons is increased.
Suchthat, GLIF neurons can adjust their neuronal response characteristics in a wide range during training.
Furthermore, different from the commonly adopted layer-wise sharing scheme where spiking neuronsin the same layer share the same values of the membrane-related parameters, the channel-wisesharing, where spiking neurons in the same channel share a unique settings of the membrane-relatedparameters, is employed in our GLIF.
With the channel-wise sharing scheme and the parametricmethod combined, we can greatly increase the neuronal dynamic diversity and heterogeneity ofspiking neurons after training.
Training framework.
In terms of the training framework, we utilize backpropagation through time(BPTT) as the training algorithm for our GLIF SNNs.
As for the non-differentiable Heaviside step
Leak
V th V th
V th
V re V re
V re γ γ γ
Leak
V th V th
V th
V re V re
V re γ γ γ
Leak
Leak α τ lin τ exp
Soft & hard reset
V re (c) (d) (b) (a) α β
Leak
V th V th
V th
V re V re
V re γ γ γ
Leak
V th V th
V th
V re V re
V re γ γ γ function in Eq.(6), we solve it by the surrogate gradient deﬁned as:
[Equation]
Thus, GLIF SNNs can be trained end-to-end.
3.3 Dynamic Analysis
GLIF contains dual bio-features, which brings great diversity to the neuronal dynamics.
Here weuse a simple case to analyze the dynamic properties of GLIF.
Fig.
(2) illustrates the response ofthe spiking neurons to the constant input, which spikes each time step until the membrane potentialreaches the threshold.
As shown in the ﬁgure, GLIF neurons with different parameter conﬁgurationshave completely different dynamic responses to the same input.
Bridging the gap between exponential decay and linear decay.
As shown in Fig.
2(a), comparedwith non-decay, on the one hand, exponential decay brings the neuronal dynamic with stability andastringency, making it increasingly difﬁcult for the membrane potential to climb over the threshold.
However, the membrane potential falls into the saturation zone and fails to reach the potentialthreshold, which indicates exponential decay may impair the spiking responsiveness.
On the otherhand, linear decay operates equally at each time step to avoid saturation, thus maintaining the highspiking responsiveness.
However, if there are no inputs, linear decay leads to the potential decreasingin an unlimited way, which impairs the stability.
While in GLIF, the above two types of decays arethe extreme cases, i.e., GLIF only acts exponential decay if α = 1 and only serves linear decayif α = 0 .
In most cases where α ∈ (0 , 1) , GLIF is eligible for exponential decay’s astringencyand beneﬁts from the high spiking responsiveness linear decay brings about.
To some extent, thegating factor α makes the two decays complementary and determines the proportion of each of thetwo decays, bridging the gap between the neuronal dynamics of exponential decay and linear decayand enlarging the representation space of membrane potential leakage.
Furthermore, since τ exp andτ lin are learnable in GLIF, the gap between exponential and linear decay becomes more ﬂexible.
Therefore, the actual representation space given by τ exp , τ lin , and α could be even larger.
Incorporating time-dependent synaptic weights.
In GLIF, we assign the time-dependent synapticweights g t to the spike train to differentiate spike stimulation levels at different time steps.
Sinceg t is learnable, the ﬂexible-coding patterns of different GLIF neurons may differ signiﬁcantly.
Forthe convenience of analysis, we add constraints to g t , assuming that g t is in the form of cosineoscillation, where β controls the amplitude.
As shown in Fig.
2(b), when β = 0 , which meansthe uniform-coding pattern, each time the spiking neuron receives an input spike, the membranepotential increases by the same amount.
Our GLIF is different.
When β ∈ (0 , 1) , each time the
GLIF receives an input spike, the membrane potential responds differently depending on the speciﬁcstimulation level of the current input spike.
On the one hand, considering that there is no constrainton g t in practice, GLIF is more ﬂexible, and the representation space is further enlarged.
On the otherhand, because the stimulus of input spikes at different times are considered differently, incorporatingtime-dependent synaptic weights also helps to better process temporal features.
Bridging the gap between hard reset and soft reset.
Fig. 2(c) compares the neuronal dynamicsbetween soft reset and hard reset.
As we can see, soft reset reduces the current membrane potential by
V re .
Since V re is not related to the current membrane potential, soft reset may lead to over-resetting orunder-resetting of membrane potential, which is ﬂexible for the spiking neurons.
Hard reset reducesthe current membrane potential to an empirical value, e.g., 0 , restarting the potential trace all over,providing the neuronal dynamics with stability.
While in GLIF, we consider the potential reductioncaused by spike initiation to be both related to V re and the current membrane potential.
Becauseγ ∈ (0 , 1) balances the amplitudes of the two resets, GLIF can reset the membrane potential to anypossible levels between the consequent potentials of hard reset and soft reset as shown in Fig.
2(d),enlarging the representation space of spike initiation.
Experiments
In this section, we demonstrate the effectiveness of our proposed GLIF model.
We ﬁrst compareour results with other existing state-of-the-arts, then carry out ablation studies to evaluate differentaspects of our proposed method.
4.1 Experimental Settings
We perform experiments on two kinds of datasets.
The ﬁrst is image classiﬁcation datasets, including
CIFAR[46] and ImageNet[47].
The other one is the event dataset, i.e. CIFAR10-DVS [48], whichis converted by an popular image recognition dataset CIFAR-10 using a DVS camera.
For a faircomparison with other methods, we choose the commonly used network architectures in existingworks for experiments, including ResNet-family[49, 9, 44], 7B-wideNet[43], and CIFARNet[42,8, 50], and apply tdBN[9].
More implementation details of training those SNNs are revealed in Appendix.
4.2 Comparison with Existing Works
CIFAR.
We test our GLIF on CIFAR-10 and CIFAR-100 datasets and run three times for eachexperiment to report the “mean ± std”.
As shown in Table 1, our GLIF SNNs can outperform existingworks.
Speciﬁcally, on CIFAR-10, we ﬁrst test our GLIF on a smaller network CIFARNet and achievethe superior accuracy over other methods [42, 8, 50], even with fewer time steps.
Since tdBN isemployed in our GLIF SNNs, we then compare GLIF with STBP-tdBN[51] on ResNet-19.
By simplychanging the neurons in STBP-tdBN to GLIF, our SNNs can achieve signiﬁcant improvement acrossall the tested time-steps.
Compared with the state-of-the-art TET[12], our method can still obtainbetter accuracy, and our ResNet-19 with time-step 4 even outperforms theirs with time-step 6 by0 .
On CIFAR-100, our method can also achieve much better performance than STBP-tdBNand TET.
We also compare GLIF ResNet-18 with the highly handcrafted ResNet-18 of Dspike[11].
As listed in Table 1, our GLIF ResNet-18 can outperform Dspike ResNet-18 by at least 0 .
63% and2 .
92% absolute accuracy on CIFAR-10 and CIFAR-100, respectively.
ImageNet.
We apply GLIF to ResNet-34 and MS-ResNet-18[44].
Table 2 shows the results onthe validation set.
We ﬁrst compare our method with STBP-tdBN, which can be viewed as ourbaseline, on ResNet-34, and we can observe that our GLIF SNNs achieve better performance ( 69 . 09%
[Table] [Table]
[Table]
v.s. 63 .
Since parametric technology is employed in our GLIF, we also compare GLIF withthe parametric spiking neuron PLIF[43].
Without changing the model structure, our method canstill achieve higher accuracy than PLIF on ResNet-34 ( 67 . 52% v.s. 67 .
Moreover, comparedwith MS-ResNet[44], whose handcrafted residual block achieves the state-of-the-art performance,we apply our GLIF to MS-ResNet-18 and can also signiﬁcantly increase the accuracy ( 68 . 11% v.s.63 .
These results demonstrate the effectiveness of our proposed GLIF model.
In addition,compared with other SNNs in TET[12] and Dspike[11], GLIF SNNs can also obtain higher accuracy.
CIFAR10-DVS.
To demonstrate the effectiveness of GLIF in processing spatio-temporal features,we conduct experiments on the event-based dataset CIFAR10-DVS.
Since SEW+PLIF[43] is the mostclose to our work for it also exploits parametric methods and tdBN, we compare with existing methods
PLIF[10], SEW-PLIF[43], STBP-tdBN[9], and LIAF[28] on 7B-wideNet.
As shown in Table 3, our
GLIF SNNs can outperform all of those methods.
Speciﬁcally, GLIF outperform SEW-PLIF on thesame network architecture 7B-wideNet, which demonstrates the effectiveness of the proposed GLIFon event-based dataset.
[Table]
[Table] [Table]
4.3 Ablation Study
Effectiveness of different bio-features.
To demonstrate the effectiveness of different bio-featuresthat are incorporated in our GLIF, we evaluate the performance of GLIF SNNs in extreme caseswhere GLIF only acts single bio-feature in each neuronal behavior.
Since the gating factors controlbio-features, we can obtain eight simplex LIFs by combining different gated biometrics, all of whichhave static binary gating factors.
We name those simplex LIFs with the values of their gating factors,e.g., 101 is the GLIF with α, β, γ = 1 , 0 , 1 that only acts exponential decay, the uniform-codingpattern, and hard reset.
On CIFAR-100, we use ResNet-18 and ResNet-19 with time-step 4 forexperiments.
As shown in Tabel 4, our proposed GLIF outperforms all of those simplex LIFs, whichdemonstrates the effectiveness of fusing different bio-features.
Importance of learnable gating factors To demonstrate the effectiveness of our proposed learnable gating factors in fusing bio-features, we compare GLIF with other two GLIF variants: GLIF_s and
GLIF_f, where GLIF_s keeps gating factors static after initialization, GLIF_f removes gating-factorsand simply fuses two different bio-features by addition.
On CIFAR-100, we also use ResNet-18and ResNet-19 with time-step 4.
Table 5 lists the experimental results.
On the one hand, comparedwith GLIF_s, our method can get higher accuracy, which demonstrates that learnable gating factorscan make GLIF better.
On the other hand, compared with GLIF_f, GLIF can also achieve betterperformance, this is because learnable gating factors can balance dual bio-features in each neuronalbehavior.
As as result, learnable gating factors are important in GLIF.
Channel-wise v.s. Layer-wise.
Besides the investigations above, we conduct an ablation studyon GLIF with different parameter-sharing schemes, which are layer-wise sharing and channel-wisesharing.
We use ResNet-19 and ResNet-18 with time-step 2 and 4 on CIFAR-100.
As listed in Table6, GLIF, which leverages channel-wise sharing scheme, consistently improves the performance overthe layer-wise sharing scheme.
Conclusion
In this paper, we propose GLIF, a uniﬁed spiking neuron, that fuses the bio-features in each neuronalbehavior.
In GLIF, our method utilizes the gating factor to bridge the gap between the dual bio-features,enlarging the representation space and enhancing the neuronal dynamic diversity of spiking neurons.
Furthermore, we adopt the channel-wise parametric method to make all membrane-related parameterslearnable, as well as the gating factors, and train those parameters through BPTT, increasing theneuronal adaptivity of spiking neurons.
We validate our GLIF SNNs on various benchmark datasets,showing the state-of-the-art performance compared with other methods.
Additionally, how to apply
GLIF in other training frameworks, e.g., ANN2SNN conversion, remains a worth-studying problem.