E NABLING P ROBABILISTIC I NFERENCE ON L ARGES CALE S PIKING N EURAL N ETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep spiking neural networks have demonstrated success in many machine learning tasks.
However, most previous works ignore inherent noisiness and randomness of neurons and only consider deterministic inference on SNNs.
This work shows how to perform probabilistic inference and learning on large SNNs by membrane noise injection.
Specifically, we introduce Noisy SNNs (NSNNs) built upon the Noisy LIF neuron, which enables probabilistic inference on large-scale
SNNs and provides a theoretical framework for investigating spiking neural models.
By viewing NSNN as a Bayesian Network, we derive a three-factor learning rule called noise-driven learning (NDL) for NSNN synaptic optimization.
The post-synaptic factor in NDL is calculated using the neuronal membrane noise statistics, providing an insightful interpretation for surrogate gradients from the view of random noise.
NDL is backpropagation-compatible, enabling NSNNs to be extended to any SNN architecture through modular replacement 1 .
Evaluations on CIFAR-10/100, DVS-CIFAR, and DVS-Gesture show that the NSNN framework leads to competitive SNN models.
Furthermore, NSNNs exhibit higher robustness against challenging perturbations, including adversarial attacks.
I NTRODUCTION
Spiking Neural Networks (SNNs) (Maass, 1997) have received mounting interest for their high biological plausibility and low power consumption.
Recent works introduce deep learning methods to
SNNs and use large-scale neural network architectures, which are proven to have superior representation abilities (Simonyan & Zisserman, 2014; Szegedy et al., 2015) and thus achieved success on many tasks (Lee et al., 2016; Wu et al., 2018; Deng et al., 2021; Zhang et al., 2021).
Nevertheless, most existing studies consider deterministic SNNs (DSNNs), which ignore the inherent randomness of spiking neurons.
Using the neuron model with a noisy dynamic is an effective way to introduce stochasticity into SNNs.
This method has two advantages: First, it incurs a potential benefit in generalization performance by encouraging the model to learn a representation space that is more fault-tolerant (Liu et al., 2020b; Camuto et al., 2020;
Lim et al., 2021) and preventing overfitting (Bengio et al., 2013; Hinton et al., 2012).
Second, spiking neurons with noise-perturbed dynamics are more biologically realistic because ion channel fluctuations and synaptic transmission randomness give rise to noisy sub-threshold membrane voltages (Verveen & DeFelice, 1974; Kempter et al., 1998; Stein et al., 2005; Faisal et al., 2008).
Existing related research, however, was limited to small scales (Plesser & Gerstner, 2000; Deneve, 2008; Pecevski et al., 2011); while instructive, they have low scalability and are difficult to scale to larger architectures.
This work introduces Noisy Spiking Neural Network (NSNN), which enables probabilistic inference on large SNNs and provides a general theoretical framework for investigating spiking neural models from the perspective of random noise.
To be specific, we (1) build the NSNN upon the discrete Noisy LIF neuron to form a general framework for SNNs; (2) derive a novel three-factor learning rule called noise-driven learning (NDL) for NSNN synaptic optimization by interpreting NSNN as a Bayesian Network; (3) show a mathematical relationship between surrogate gradient learning and NDL, providing an insightful interpretation for surrogate gradients; (4) show that the NSNN framework leads to competitive SNN models, demonstrated by experiments on CIFAR-10/100, DVS-CIFAR and DVS-Gesture datasets; (5) demonstrate that NSNN framework leads to more robust SNN models when facing challenging perturbations (including adversarial attacks).
(6) By NSNN-based neural code analysis, we demonstrate the potential of NSNN as a neural coding framework for computational neuroscience.
Notations We adopt x, u, o to represent neuron input, membrane potential and neuron output, respectively.
Moreover, x tl,m , u tl,m , o tl,m for variables of neuron m in layer l (whose dimension is dim( l ) ) at time t , where m ∈ [1 , dim( l )] , l ∈ [1 , L ] and t ∈ [1 , T ] .
We also use boldface type x , u , o to denote the sets of variables, e.g. , variables of layer l at timestep t are marked as x tl , u tl , o tl .
E [ · ] , P [ · ] , p ( · ) and F ( · ) are, respectively, expectation, probability, probability distribution and CDF.
R ELATED WORKS
Surrogate Gradient Learning The main obstacle during the direct-training of prevailing deterministic SNNs is the almost everywhere zero nature of the gradient of the Heaviside firing function.
As a remedy, surrogate gradient function (SG) (Neftci et al., 2019; Zenke & Vogels, 2021) are adopted, i.e. use a smooth function to replace the derivative of the firing function in the backward pass and still use the firing function in the forward passage.
Surrogate gradient learning (SGL) refers to synaptic optimization using surrogate gradients.
Noisy spiking neural models Gerstein and Mandelbrot proposed the earliest integrate-and-fire (IF) neuron models with stochastic activity (Gerstein & Mandelbrot, 1964).
Following developments (Stein, 1965; Tuckwell, 1989; Plesser & Gerstner, 2000; Di Maio et al., 2004; Burkitt, 2006) have expanded on the diffusion approach by employing stochastic differential equations.
Rao demonstrated that recurrent networks of noisy IF neurons could perform approximate Bayesian inference of dynamic graphical models.
Patel & Kosko introduced the conditions for the noise benefit (Wiesenfeld & Moss, 1995) of additive white noises.
Fiete & Seung proposed an estimator that correlates reinforcement reward signal and synaptic perturbations by introducing white noises and adopting first-order Taylor expansions for noisy neuron learning.
In Bengio et al.
a locally-computed gradient estimator for neurons with stochastic decisions is introduced.
Skatchkovsky et al.
described a Generalized Linear Model variant of the deterministic Spike Response Model.
LIF neuron model The widely-used LIF neuron model includes the following discrete-time dynamics
[Equation]
where x t is the input at time t , τ is the membrane time constant.
ϕ θ denotes a parameterized input transform, and v th is the firing threshold.
To introduce a simple model of neuronal spiking and refractoriness, we assume v th = 1 , τ = 0 .
5 and u reset = 0 throughout this research.
N OISY S PIKING N EURAL N ETWORKS
We begin by introducing the Noisy LIF model as a fundamental unit, which naturally connects spiking networks to probabilistic graphical models and enables NSNN to function as a general theoretical framework for LIF SNNs. Noisy LIF model
The Noisy LIF presented here is based on previous works that use diffusion approximation (Plesser & Gerstner, 2000; Burkitt, 2006), in which the effective current input to the neuron is described by a deterministic and a random noise part.
As a result, an additive noise term is added to the discrete sub-threshold dynamic:
[Equation]
where ϵ are independently drawn from a known distribution and assumed to satisfy E
[ ϵ ] = 0 and ϵ = − ϵ .
As an example, we use Gaussian ϵ ∼ N here.
Expression (2) can also be obtained by discretizing an Itˆo SDE variant of LIF’s ODE form (Patel & Kosko, 2005; 2008).
… … … … … spatial t e m po r a l
B
C m l,m layer neu r on
L
L+1 l dim ( 1 ) predictive head p r ed i c t i v e head
A codes The membrane potentials and spike outputs become random variables as a result of the injection of random noises. Using noise as a medium, we obtain the probability distribution of Noisy LIF firings based on the threshold firing mechanism, as ϵ = − ϵ , we have
[Equation]
The expressions above show how a single neuron encodes for a spike state random variable (Maass, 2014), allowing us to formulate the probabilistic firing of Noisy LIF by probabilistic firing: o t ∼ Ber( P [ o t = 1]) , where P [ o t = 1] = F ϵ ( u t − v th ) .
Specifically, it relates to previous literature in which the difference u − v th governs the neuron firing probabilities (Maass, 1995; Plesser & Gerstner, 2000).
In addition, Noisy LIF employs the same resetting mechanism as the LIF model.
Noisy LIF is a general form of spiking neurons, making NSNN a theoretical framework for SNNs.
If
Var[ ϵ ] → 0 , F ϵ will approach the Heaviside step function; hence, the Noisy LIF model covers the deterministic LIF case.
Further, if we consider ϵ ∼ Logistic , the Noisy LIF describes a sigmoidal neuron (Maass, 2014).
Noisy SNN Let x t 1 denote the input at t -th timestep, using the dynamics of Noisy LIF in (2,4), an
NSNN with L + 1 layers is given by
[Equation]
predictive
[Equation]
[Equation]
The spike output o tl of layer l is a representation vector in S dim( l ) , where we denote the spike state space as S =
The noise vector ⃗ϵ consists of independent random noise with a known distribution (Gaussian here).
The predictive head f θ L +1 ( o tL ) includes a mapping ϕ
θ L +1 ( o tL ) and a loss function
f , denoting the part that decodes predictions from the neural representation o tL and compute the loss value.
ϕ θ l represents a map, such as fully-connected or convolution and is thus differentiable w.r.t.
parameter θ l .
Also, dividing the synaptic parameters by layers, as mentioned above, results in no loss of generality as they can be defined as any differentiable mapping.
For example, to solve classification problems we shall consider the predictive probability model p θ L +1 ( y | o tL )
=
softmax( ϕ θ L +1 ( o tL )) , where the map ϕ θ
L +1 computes the predictive scores using the neural representation o tL .
The function f can be the cross-entropy of the predictive distribution p θ L +1 ( y | o tL ) and the target distribution p tar ( y | x t 1 ) .
Note that f θ L +1 ( o tL ) here computes the instantaneous loss, different from the 1 T (cid:80) t f t , which is computed over the entire time window and ignores potential online learning (Xiao et al., 2022).
Since each neuron codes for a random variable o tl,m , we can describe the NSNN by the Bayesian
Network model and represent the joint distribution of all spike states given the input x
t 1 as
[Equation]
where p θ l ( o tl | o tl − 1 , o
[Equation]
=
[Equation]
3. A N OISE - DRIVEN L EARNING R ULE I NDUCED BY N OISE I NJECTION
It is suggested that noise supports learning from supervise signals of networks of spiking neurons, rather than being a nuisance (Maass, 2014), and previous ANN literature also suggest similar stances (Liu et al., 2020b; Camuto et al., 2020; Lim et al., 2021).
But how exactly is this achieved on spiking neurons?
Within the NSNN framework, we derive a novel noise-driven learning rule (Fig.
2.A) induced by membrane noise injection for synaptic optimization.
To perform NSNN synaptic optimization, the central problem is to estimate the gradient of the expected loss:
[Equation]
As (7) is intractable to compute, we expect an estimation so that the parameters can be tuned using gradient-based routines.
The dimensionality of the spike state space is rather limited (either spike or silence).
Based on this property, we can derive an estimator by conditioning (local marginalization), which performs exact summation over single random variable to reduce variance (Burt Jr & Garman, 1971; Titsias et al., 2015).
We first factorize the joint distribution p θ ( o t 1 ··· L | x t 1 , o t − 1 1 ··· L ) as (cid:0) (cid:81)
i
[Equation]
=
[Equation]
p
[Equation]
ti
[Equation]
[Equation] [Equation]
o
[Equation]
l,m ) .
Hence, (7) becomes g l =
[Equation]
Since P [ o tl,m = 0] = 1 − P [ o tl,m = 1] , we have (cid:88) o tl,m
[Equation]
(9) where we use o t ¯ l,m to denote the new state o tL if o l,m alters.
Together with (cid:80) o tl,m p θ l ( o tl,m ) = 1 and (8, 9), we have
[Equation]
ˆ
[Equation]
] , where
[Equation]
To get an estimate of g l , we can simply sample from p θ ( o t ) and calculate using (10).
However, it is unwise to compute f θ L +1 ( o tL ) − f θ L +
( o t ¯ l,m ) , as it requires a lot of additional computations, and thus cannot scale to large models.
Inspired by Fiete & Seung (2006), we may attribute the change of the loss to the state flip of variable o tl,m .
By doing this, we can approximate the change of the loss when the state of o tl,m alters using a first-order approximation:
[Equation]
Note that, this approximation introduce bias to the gradient estimator, except when the map f is multilinear (Tokui & Sato, 2017).
Substituting (11) into (10), we obtain
[Equation]
Proposition 1.
For a Noisy LIF neuron ( l, m ) in an NSNN, where l ∈ [1 , L ] , m ∈ [1 , dim( l )] , we have ∇ θ l p θ l ( o t l,m | o t l − 1 , o t − 1 l,m ) =
(2 o t l,m − 1) F ′ ϵ
( u t l,m − v th ) ∇ θ l u t l,m .
Surrogate Gradient Learning (Eqn.11) statistics: spike output layerlayer L synapses soma l,m
Noise-driven Learning (Eqn.10) p r ed
i c t i v e head
Lo ss v a l ue
Pre-synaptic factor
Post-synaptic factor
Global signal synaptic weights neuron (l, m) computed via approximate by surrogate gradients: increase
[Figure]
SG type
ERF SG
Sigmoidal SG
Triangular SG
Rectangular SG
Normal distribution
Logistic distribution
Triangular distribution
Uniform distribution
Noise from
A
B   0 0.
A
B   0 0.
[Figure] [Figure]
[Figure]
informative high low # Iterations
P e r f
[Figure]
an c
[Figure]
SG scale tunning as variance selection decrease
Proof.
Proved using (3).
Combining Proposition 1 and (12), we formulate the noise-driven learning rule (Fig.
2.A) as
[Equation]
When using (13), there is not need to calculate an additional gradient generator in the forward pass, and ˆ g l can be computed layer by layer in a single backward passage.
As a result, NDL is easy to implement and can mesh well with modern frameworks of automatic differentiation.
For neuron ( l, m ) , the gradient estimation is performed by a backward pass, in which the post-synaptic factor is acquired from the PDF F ′ ϵ .
Plesser & Gerstner (2000); Shrestha & Orchard (2018) also constructed surrogate gradient function by empirically adding infinitesimal gaussian perturbations to a spiking neuron.
However, these works analyze an isolated neuron, whereas results of this work are derived from the network level.
Since the estimator in (13) is backpropagation-compatible, we can easily optimize NSNNs of any architecture with the BPTT algorithm (Robinson & Fallside, 1987).
Relationship to the Surrogate Gradient Learning.
In the Surrogate Gradient Learning (SGL), the derivative of neuron firing function ∂o/∂u is replaced by a smooth function SG to mesh with the backpropagation scheme.
SGL calculates the gradient g l by
[Equation]
Eqn.
(14) and (13) show a close mathematical relationship between NDL and SGL.
The derivative of firing function, provided by surrogate gradient functions in SGL, corresponds to the membrane potential noise’s PDF F ′ ϵ
= p ϵ of the post-synaptic neuron.
Indeed, when we extend the Gaussian noise in to general stochastic processes with static increments, commonly-used symmetric (subject to the assumptions we used in the derivation) surrogate gradients can be explained by corresponding
PDFs of membrane potential noises ( e.g. , rectangular SG v.s. uniform noise, sigmoidal SG v.s. logistic noise).
Biological interpretation: noise as a resource for learning.
We re-write the NGL estimator in (13) to frame it as a three-factor learning rule (Fr ´ emaux & Gerstner, 2016; Gerstner et al., 2018):
[Equation]
The post-synaptic factor in NDL is calculated by the probability density function of the post synaptic neuron’s membrane potential noise, which computationally validates the idea of “ noise as a resource
[Table]
for learning ” (Maass, 2014).
Also, in NDL, adjusting the noise variance causes a change in the shape of its PDF, which corresponds to tuning the scale parameter for the surrogate gradient function in
SGL (Fig. 2.B).
Thus, the post-synaptic factor of NDL explains the scale tunning (Zenke & Vogels, 2021) in SGL.
The scale tunning of SGs can be viewed as variance selection of membrane noise ϵ : a mild noise plays an essential role in learning (Maass, 2014).
Proper variance is essential to achieve high performance: small variance noise (low entropy) is not informative enough to learn well
[Table]
while high variance noise is also harmful (Fig. 2.B) (Yarom & Hounsgaard, 2011).
In Sec.
4.3, we investigate the effect of noise level on performance in further detail.
E XPERIMENTS
In this section
[Table]
we conduct evaluations to show that: (1) NSNNs show highly competitive performance; (2) NSNNs are more robust against challenging perturbations.
We also conduct experiments to (3) study the effects of membrane noise level on performance and (4) offer a novel take on the neural coding policy in performing different tasks in spiking networks, demonstrating how NSNNs can be used as a tool for computational neuroscience.
We adopt various network architectures including residual nets and VGG nets.
For DSNNs, we use the ERF surrogate gradient SG ERF ( x ) = 1 √ π exp( − x 2 ) for SGL.
All networks were trained using
Adam solvers with the cosine annealing learning rate scheduler.
4. C OMPARISON OF R ECOGNITION T ASK P ERFORMANCE
In this section, we compare the capabilities of NSNNs and DSNNs on static image benchmarks
CIFAR-10/100 (Krizhevsky et al., 2009), dynamic datasets DVS-CIFAR (Li et al., 2017), and DVSGesture (Amir et al., 2017).
The results are reported as mean ± std across three independent runs.
More experimental details are provided in Sec.
A.2 and more results are presented in Tab. 8.
We set
[Figure] [Figure]
[Table]
the standard deviation of membrane noise to 0 .
3 for CIFAR-10/100, DVS-Gesture experiments and 0 .
2 for DVS-CIFAR.
These configurations offer a fair balance between performance and resilience (refer to Fig.
2 and Sec. 4.3)
[Table]
According to results presented in Table 1.
When compared to their deterministic counterparts
[Table]
it can be seen that NSNNs with different combinations of training algorithms and network architectures achieve competitive performances.
Specifically, our NSNNs show consistent merits for the eventstream classification task on the DVS-CIFAR dataset.
We suggest that the intrinsic randomness of the Noisy LIF neurons plays the role of a regularizer (Camuto et al., 2020; Lim et al., 2021), thus alleviating the overfitting to some extent.
Evaluations in this part demonstrate the benefit of NSNNs:
NSNNs can perform stochastic inference on large-scale architectures while achieving comparable or better performance than those deterministic inference ones.
4. R OBUSTNESS E VALUATION
We further evaluate the robustness of DSNNs and NSNNs on CIFAR-10/100 and DVS-CIFAR datasets.
The default simulation timestep for static image datasets is T = 2 .
The models we used for evaluation in this section are trained as described in Section 4.1.
We consider different perturbations for static, dynamic inputs, respectively.
For CIFAR10/100, we consider untargeted adversarial attack to evaluate the model robustness under the “worst case” (Szegedy et al., 2013; Guo et al., 2022).
We construct adversarial examples by two methods (details in A.2.1): (1) Direct Optimization (DO) method and (2) Fast Gradient Sign method (FGSM, Goodfellow et al.).
For DVS-CIFAR, we consider the EventDrop perturbation (Gu et al., 2021), whose basic idea is to randomly drop a proportion of events, with a probability of ρ ∈ [0 , 1] .
In addition, the evaluation under hidden state-level (neuronal spike-level) perturbations are presented in Section A.2.2.
Figure 3 shows the performance dynamics on CIFAR-10/100 datasets against DO/FGSM adversarial attacks.
Our results indicate that NSNNs are highly resilient to these challenging adversarial perturbations, whereas DSNNs’ reliability degrades radically.
Table 2 summarizes the losses and accuracies of three groups of models concerning input-level EventDrop perturbations on the DVSCIFAR dataset.
In most cases, the proposed NSNNs appear to be less sensitive to perturbations than competitors, demonstrating relatively high robustness to various perturbations and adaptability to multiple training algorithms and network architectures.
A
B 1 0 0.2 0.
A cc ( % )
DVS-Gesture 0.
A cc ( % ) 1e 1e- # of Iterations
[Figure] [Figure]
[Figure] [Figure]
CIFAR
[Figure] [Figure]
[Figure] [Figure]
[Figure]
4. E FFECT OF I NTERNAL N OISE L EVEL ON P ERFORMANCE
We further explore the effect of the membrane potential noise level in NSNN on the performance as an extension to the related content in Figure 2.B. We run experiments using the CIFAR-10 and
DVS-Gesture datasets and train identical networks with different standard deviation settings forepochs.
Results are presented by learning curves and the accuracy-standard deviation curves in
Fig.
As the variance of membrane potential noise ϵ increases, the model performance exhibits a dynamic process of increasing and then declining.
In particular, NSNNs achieve high performance near a moderate value (Fig. 4.B), confirming our intuition that moderate noise is essential for high performance.
As shown in Fig.
4.B, changes in std[ ϵ ] within a “moderate noise” range (from 0.2 to 0.5) have no significant effect on final performance.
This gives us a range of internal noise levels to choose from when using NSNNs in practice.
Biological interpretation of the effect of noise level.
As a critical component in NDL, the postsynaptic factor F ′ ϵ ( u − v th ) is calculated by the PDF of membrane noise ϵ during the backward pass.
When the noise variance is very small, the noise distribution converges to a Dirac distribution with minimal information (as measured by entropy), and the post-synaptic factor cannot obtain enough information for synaptic optimization.
In the case of inference, the noise level directly affects the randomness of the neuron firing distribution.
A high variance noise would disrupt the flow of valuable information from the observation in the network, causing NSNN’s performance to deteriorate greatly.
4. NSNN N EURAL C ODE A NALYSIS
The intrinsic randomness of NSNNs results in trial-by-trial variability (Stein et al., 2005), allowing for exploration of neural representations in spiking networks.
In this section, we analyze the neural code embedded in the spike trains in NSNNs.
Also, we consider an additional sinusoidal series forecasting (SSF) task to investigate possible coding strategies of NSNNs when performing different types of tasks (refer to A.2.3 for details).
We estimate the possibility of rate code by measuring the correlation between the neural code (outputs of the penultimate layer) variation and prediction stability.
We use the Fano factor (FF) to numerically measure the neural code variation and cosine similarity to assess prediction stability.
In addition to the firing rate, it has been suggested that correlations between neurons provide an additional channel of information (Alonso et al., 1996;
Hung et al., 2005).
In this section, we use simplified network architectures with 16 neurons in the last ( L -th) spiking layer for the DVS-Gesture and the SSF experiments to enable pairwise firing correlation analyses.
The settings for CIFAR-10/100 and DVS-CIFAR experiments are the same as those in Section 4.1.
The simulation timesteps of CIFAR-10/100, DVS-CIFAR, DVS-Gesture, and
SSF experiments are 2, 10, 16, and 48, respectively.
The results in Figure 5.B show a decreasing monotonic trend between the prediction similarity and the neural code variation (measured by avg.
FF).
The average FF and prediction similarity, in particular, on some experiments ( e.g. , TET+ResNet18), show a strong negative correlation, indicating that these NSNNs are likely to primarily adopt rate code.
It makes sense as the membrane noise injection introduces uncertainty into the firing process, lowering the reliability of the precise spiking time-based coding.
As the same firing rate (represented as firing count in simulation steps here)
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure] [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
can correspond to different spike trains, the rate-based coding can improve the model’s robustness by constructing a representation space with better fault tolerance.
Figure 5.C shows that when performing forecasting tasks, NSNN appears to be less dependent on the rate code.
By measuring the pairwise firing correlation (Fig.
5.D, full version in Fig.
7,8), we also discover that a neuron population exhibits significant co-activation, which was not observed in the DVS-Gesture experiment.
Therefore, NSNN may also utilize the firing correlations to carry important information when performing forecasting tasks, implying that the optimal neural code (neural representation) might be task-dependent (Bredenberg et al., 2020; Xie et al., 2022).
C ONCLUSION
We introduce NSNN in this work.
Based on its Bayesian Network form, we propose a novel threefactor learning rule called noise-driven learning (NDL), which offers an insightful probabilistic interpretation of the surrogate gradient learning.
We demonstrate NSNN’s capability through experiments on various recognition tasks.
Moreover, we conduct experiments with challenging perturbations (such as adversarial attacks) and demonstrate that NSNNs are more robust than their deterministic counterparts.
We investigate the effect of NSNN internal noise level on performance and give a recommended range of standard deviation values (only consider Gaussian noise here).
In addition, we demonstrate the potential of NSNN as a neural coding scheme through NSNN-based neural code analysis.
A A PPENDIX
Roadmap For additional experimental details and results, refer to A.2.
Notations We adopt lower case letters x, u, o to represent neuron input, membrane potential and neuron output respectively.
Moreover, x tl,m , u tl,m , o tl,m for variables of neuron m in layer l (whose dimension is dim( l ) ) at time t , where m ∈ [1 , dim( l )] , l ∈ [1 , L ] and t ∈ [1 , T ] .
Similarly, variables of layer l at timestep t are marked as x tl , u tl , o tl .
We also use boldface type x , u , o to denote the sets of all variables of the network.
E [ · ] stands for expectation, Var[ · ] for variance, P [ · ] for probability, p ( · ) for probability distribution and F ( · ) for CDF.
Notation R denotes real number space R ≜ ( −∞ , + ∞ ) and S stands for the spike state space S ≜ { 0 , 1 } .
Fano Factor Fano factor (Fano, 1947) is a measurement of the spike count variability.
Let us denote the spike count of neuron ( L, m ) as n trial ID L,m , the average value as avg( n L,m ) =
1 # trials (cid:80) k n kL,m .
The deviations from the mean is computed as ∆ n trial ID L,m = n trial ID L,m − n L,m , and the Fano factor is
FF L,m =
Var[ n L,m ] n L,m
A. R ELATING DSNN SGL TO NSNN NDL
Surrogate gradient learning is widely-adopted as an empirically solution to overcome the almosteverywhere-zero problem when computing gradients through step spiking functions.
In this work, we propose to view surrogate gradient learning as a special form of noise-driven learning, and we reveal the close relationship between surrogate gradient function and voltage-level noise distributions in Table 3.
surrogate gradient functions and their corresponding noise distribution p ϵ .
Discarding the biological fact that the ions are subject to Brownian movement that corresponds to the Gaussian case we derived in the main body, we can extend the results in the table to noise to other continuous random distributions with zero-mean and symmetry PDF.
ERF
[Equation]
Sigmoid
Logistic
[Equation]
Rectangular
[Equation]
Triangular Triangular
[Equation]
Arctangent
Atan
[Equation]
[Equation]
DSNN
Surrogate Gradient Learning
NSNN
Noise-driven Learning
Inference
Spiking procedure deterministic (approximate the stochastic inference)
[Equation]
stochastic
[Equation]
[Table]
[Equation]
[Table]
A. E XPERIMENTAL D ETAILS AND A DDITIONAL R ESULTS A.2. E XPERIMENTAL D ETAILS
We conducted most experiments on a workstation with an Intel i5-10400 core, 64 GB memory, and an NVIDIA RTX 3090 card.
Experimental results presented in the form of mean ± std are acquired across three independent trials.
We list experimental details in points as follows.
Training Details We optimize all networks with Adam solvers (Kingma & Ba, 2014) and adopt cosine annealing (decay to zero) learning decay policy (Loshchilov & Hutter, 2016).
We list hyperparameters for our experiments as follows in Table 4, algorithms and network architectures mentioned in Table
[Table] [Table]
N
[Table]
N
[Table]
N tdBN
[Table]
N
[Table] [Table]
[Table] [Table]
Baselines We summarize the baseline methods (algorithms) mentioned in our comparison experiments below.
• Hybrid conversion and spike timing dependent backpropagation (Rathi et al., 2019).
• Conversion-based spiking residual networks (Hu et al., 2018).
• Backpropagation-based temporal spike sequence learning (TSSL) (Zhang & Li, 2020).
• Spatiotemporal backpropagation (STBP) (Wu et al., 2019).
• DIET-SNN (Rathi & Roy, 2020).
• Spatiotemporal backpropagation with temporal dimension batch norm (STBP-tdBN) (Zheng et al., 2021).
• Histograms of averaged time surfaces (HATS descriptor)
(Sironi et al., 2018).
• Distribution-aware retinal transform (DART descriptor)
(Ramesh et al., 2019).
• ANN rollout and conversion (Kugele et al., 2020).
• AER object recognition by segmented probability maximisation (SPA) algorithm
(Liu et al., 2020a).
• Leaky-integrate and analog fire network (LIAF-Net) (Wu et al., 2021).
• Spike-element-wise (SEW) residual networks (Fang et al., 2021b).
• Temporal efficient training (TET) objective function (Deng et al., 2021).
Datasets and Pre-processings We adopt static datasets including static datasets CIFAR-10& (Krizhevsky et al., 2009), dynamic dataset DVS-CIFAR (Li et al., 2017).
• CIFAR dataset includes 50k 32 × 32 images for training and 10k for evaluation.
We adopt random crop, random horizontal flip and AutoAugment (Cubuk et al., 2018) for the training samples.
For both training and evaluation phases, the preprocessed samples are normalized using z-score scaling.
• DVS-CIFAR dataset is a challenging neuromorphic benchmark recorded via a DVS camera using CIFAR-10 images.
We adopt pre-processing pipeline in Samadzadeh et al.
(2020), i.e. , divide the original set into a 9k-sample training set and 1k-sample evaluation set and all event stream files are spatially downsampled to 48 × 48 .
We augment the training samples following Deng et al.
• DVS-Gesture dataset (Amir et al., 2017).
This datasets is recorded using DVS128, it containshand gestures from 29 subjects under 3 illumination conditions.
Perturbation Details We list the details about the perturbations in our experiments as following:
• In the Direct Optimization (DO) method, we construct the adversarial samples by directly solving the constrained optimization problem arg max || ∆ x || 2 = γ ℓ ( f ( x +∆ x ) , y ) , where ∆ x for the adversarial perturbation and x + ∆ x is the adversarial example.
It is implemented using PyTorch and GeoTorch (Lezcano-Casado, 2019) toolkits.
The L-2 norm bounded additive disturbance tensor are first zero-initialized and then optimized by an Adam solver with learning rate 0 .
002 foriterations.
After that, the additive perturbations are used to produce adversarial samples and then fed into the testees to evaluate (attack) the target models (either DSNNs or NSNNs in this work).
• The implementation of FGSM method follows Goodfellow et al.
(2015), the adversarial example is constructed as ˜ x adv = x + γ FGSM × sign[ ∇ x ℓ (NN( x ) , y ))] .
• The input-level EventDrop perturbation for dynamic inputs are constructed by randomly dropping spikes in the raw input spike trains.
The dropping probability is set by a parameter ρ , the strategy of dropping we consider is Random Drop
(Gu et al., 2021), which combines spatial and temporalwise event dropping strategies.
During the evaluation, we first individually perform EventDrop over every samples from the test set, and then fed our testees with the disturbed inputs.
• The hidden state-level noise includes two types of disturbances, the emission state from 1 to(spike to silence) and emission state from 0 to 1 (silence to spike).
To simplify the settings, we use one parameter β to control the probability of both kinds of changes.
Let variable
o denotes a spike state, if o = 1 , we
[Table]
α Membrane noise parameter: standard deviation of the normal distribution N ∆ u β Hidden state (spike train) noise parameter: adding or dropping probability for a spike.
γ DO Static input adversarial perturbation-DO method parameter: L-2 norm constraint of the vector ∆ x .
γ FGSM Static input adversarial perturbation-FGSM parameter:
∆ x updating step size.
ρ Dynamic input EventDrop perturbation parameter: event dropping probability for events (spikes).
[Table]
Used in gaussian
[Table]
N (0 ,
[Table]
s Used in logistic noise ϵ ∼ Logistic(0 , s ) .
a Used in rectangular ϵ ∼ U ( − a, a ) or triangular ϵ ∼ Triang( − a, 0 , a ) noises.
ϕ Used in arctangent noise ϵ ∼ Atan(0 , ϕ ) .
[Table]
A.2. E VALUATION RESULTS UNDER HIDDEN STATE - LEVEL PERTURBATION
We further consider hidden state-level perturbations to directly mimic the spike train variability (Tuckwell et al., 2009; Yarom & Hounsgaard, 2011) aside the spike variance caused by membrane voltage fluctuations.
The hidden state-level perturbation is directly put on the emitted spikes of all spiking neurons (Kasabov, 2010; Zhang et al., 2017)
[Table]
implemented by randomly flipping the state o tl,m of all hidden neurons, the probability of flipping is controlled by a parameter β as P [ 1 to 0 ] =
P [0 to 1] = β .
Results are presented in Tab. 7 and Fig.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
C o rr e l a t i on t i m e s t ep timestep
JPSTH Figure Legend
PS T H
N o r m a li z ed J
PS T H neu r on n t o neu r on n neu r on n t
o neu r on m empty if not valid.
[Figure]
A.2. N EURAL C ODE A NALYSIS D ETAILS
Experimental Details
The sine series forecasting (SSF) uses a NSNN MLP with 32 fc - 16 fc hidden units, simulation timestep is set to 48 .
The sinusoidal series is generated using sin ( x ) , where the step of x is 0 . 1 .
The training loss function is MSE loss.
For the DVS-Gesture experiment in
Section 4.4, we use a 7B-Net (in Tab. 5) and reduce the last layer’s dimension to 16.
The PSTH, normalized JPSTH plots are presented in Fig.
The normalized JPSTH is a two-dim gram whose value at position u, v represents the Pearson correlation of one neuron firing at time u
and the other one firing at time v .
A
A LL COMPARISON RESULTS ON RECOGNITION BENCHMARKS .
We list the results of the full comparison experiment in Tab.
8 due to space constraints.
PS T H
N o r m a li z ed J
PS T H neu r on n t o neu r on n neu r on n t
o neu r on m
Input: sinusoidal sequence 0.
C o rr e l a t i on t i m e s t ep
[Figure]
JPSTH
[Figure] [Table]
[Table]
AR
[Table] [Table]
DV SC I F
AR
[Table]