Original: DTMD: Learning Improvement of Spiking Neural Networks with Dynamic Thresholding Neurons and Moderate Dropout
Revised: LTMD: Learning Improvement of Spiking Neural Networks with Learnable Thresholding Neurons and Moderate Dropout

Original: Abstract
Revised: Abstract

Original: Spiking Neural Networks (SNNs) have shown great promise in processing spatiotemporal data, mimicking biological neuronal mechanisms, and saving computational power.
Revised: Spiking Neural Networks (SNNs) have shown substantial promise in processing spatio-temporal data, mimicking biological neuronal mechanisms, and saving computational power.

Original: However, most SNNs use fixed model regardless of their locationsin the network.
Revised: However, most SNNs use fixed model regardless of their locations in the network.

Original: This limits SNNs’ capability of transmitting precise informationin the network, which becomes worse for deeper SNNs.
Revised: This limits SNNs’ capability of transmitting precise information in the network, which becomes worse for deeper SNNs.

Original: Some researchers try touse specified parametric models in different network layers or regions, but moststill use preset or suboptimal parameters.
Revised: Some researchers try to use specified parametric models in different network layers or regions, but most still use preset or suboptimal parameters.

Original: Inspired by the neuroscience observationthat different neuronal mechanisms exist in disparate brain regions, we propose anew spiking neuronal mechanism, named dynamic thresholding, to address thisissue.
Revised: Inspired by the neuroscience observation that different neuronal mechanisms exist in disparate brain regions, we propose a new spiking neuronal mechanism, named learnable thresholding, to address this issue.

Original: Utilizing learnable threshold values, dynamic thresholding enables flexibleneuronal mechanisms across layers, proper information flow within the network,and fast network convergence.
Revised: Utilizing learnable threshold values, learnable thresholding enables flexible neuronal mechanisms across layers, proper information flow within the network, and fast network convergence.

Original: In addition, we propose a moderate dropout methodto serve as an enhancement technique to minimize inconsistencies between independent dropout runs.
Revised: In addition, we propose a moderate dropout method to serve as an enhancement technique to minimize inconsistencies between independent dropout runs.

Original: Finally, we evaluate the robustness of the proposed dynamicthresholding and moderate dropout for image classification with different initialthresholds for various types of datasets.
Revised: Finally, we evaluate the robustness of the proposed learnable thresholding and moderate dropout for image classification with different initial thresholds for various types of datasets.

Original: Our proposed methods produce superiorresults compared to other approaches for almost all datasets with fewer timesteps. 
Revised: Our proposed methods produce superior results compared to other approaches for almost all datasets with fewer timesteps. Our codes are available at https://github.com/sq117/LTMD.git .

Original:  Introduction
Revised:  Introduction

Original: Modelled after the impulse communication between biological neurons, spiking neural network(SNN) is a new class of neural network with neurons exhibiting a distinctive binary output property.
Revised: Modelled after the impulse communication between biological neurons, spiking neural network (SNN) is a new class of neural network with neurons exhibiting a distinctive binary output property.

Original: SNNs have shown great potential in event-driven data processing, computation reduction and networkbiological plausibility enhancement.
Revised: SNNs have shown great potential in event-driven data processing, computation reduction and network biological plausibility enhancement.

Original: However, unoptimized learning algorithm and complex neuronaldynamics make it challenging to construct high-performance SNN models.
Revised: However, unoptimized learning algorithm and complex neuronal dynamics make it challenging to construct high-performance SNN models.

Original: Current SNN training approaches can generally be divided into two categories: ANN-to-SNNconversion and SNN direct training algorithms based on gradient descent.
Revised: Current SNN training approaches can generally be divided into two categories: ANN-to-SNN conversion and SNN direct training algorithms based on gradient descent.

Original: Both ways only focuson adjusting synaptic connections to minimize the error between a model’s output and target valueswith the same neuronal model being used across the whole network.
Revised: Both ways only focus on adjusting synaptic connections to minimize the error between a model’s output and target values with the same neuronal model being used across the whole network.

Original: However, it has been found from neuroscience observations that biological neurons’ dynamics vary with their relative locationsin the brain especially for primates [1, 2].
Revised: However, it has been found from neuroscience observations that biological neurons’ dynamics vary with their relative locations in the brain especially for primates [1, 2].

Original: Therefore, using the same neuronal model across thewhole SNN may limit the SNN’s expression ability as compared to allowing heterogenous neuronalmodels to be used in the same SNN.
Revised: Therefore, using the same neuronal model across the whole SNN may limit the SNN’s expression ability as compared to allowing heterogenous neuronal models to be used in the same SNN.

Original: Although some researchers have proposed many techniques to balance presynaptic inputs and neuronal behaviors, for example, threshold regularization [3] and spike-based normalization [4], they are either targeting at neuronal state distribution or onlyapplicable for ANN-to-SNN conversion algorithms.
Revised: Although some researchers have proposed many techniques to balance presynaptic inputs and neuronal behaviors, for example, threshold regularization [3] and spike-based normalization [4], they are either targeting at neuronal state distribution or only applicable for ANN-to-SNN conversion algorithms.

Original: Dropout [5, 6] is a commonly used regularization methodology to train neural networks.
Revised: Dropout [5, 6] is a commonly used regularization methodology to train neural networks.

Original: As droppingneurons are randomly selected, each run of network will lead to the input interacting with differentsub-models, which will cause the results in different runs to deviate unpredictably.
Revised: As dropping neurons are randomly selected, each run of network will lead to the input interacting with different sub-models, which will cause the results in different runs to deviate unpredictably.

Original: Consequently,a regularization technique is needed to minimize the inconsistencies between output probabilitydistributions during the training phase to improve the performance.
Revised: Consequently, a regularization technique is needed to minimize the inconsistencies between output probability distributions during the training phase to improve the performance.

Original: In this paper, we propose a new dynamic thresholding mechanism with a moderate dropout methodto enhance the learning of modulated SNNs.
Revised: In this paper, we propose a new learnable thresholding mechanism with a moderate dropout method to enhance the learning of modulated SNNs.

Original: The dynamic thresholding mechanism can be integratedwith the backpropagation based SNN direct training algorithm so that neuronal parameters can bechanged dynamically and self-optimize during training.
Revised: The learnable thresholding mechanism can be integrated with the backpropagation-based SNN direct training algorithm so that neuronal parameters can be changed dynamically and self-optimize during training.

Original: We adapt the DenseNet architecture andmodify the encoding and decoding layers to make the network suitable for diverse datasets.
Revised: We adapt the DenseNet architecture and modify the encoding and decoding layers to make the network suitable for diverse datasets.

Original: A new‘moderate dropout’ technique is developed to minimize inconsistencies between network sub-modelsgenerated in different runs.
Revised: A new ‘moderate dropout’ technique is developed to minimize inconsistencies between network sub-models generated in different runs.

Original: We demonstrate the proposed methods can enhance the SNN’s stabilityand performance through evaluation based on static and neuromorphic datasets.
Revised: We demonstrate the proposed methods can enhance the SNN’s stability and performance through evaluation based on static and neuromorphic datasets.

Original: To the best of ourknowledge, our proposed SNNs achieve the best or comparable accuracies for all the datasets testedcompared to other state-of-the-art SNNs.
Revised: To the best of our knowledge, our proposed SNNs achieve the best or comparable accuracies for all the datasets tested compared to other state-of-the-art SNNs.

Original:  Related Work 
Revised:  Related Work 2.1 ANN-to-SNN Conversion

Original: 2.1 ANN-to-SNN Conversion

Original: ANN-to-SNN conversion [7, 4, 8] aims to convert trained ANNs to SNNs by using rate codingscheme instead of ReLU activation to represent data flowing inside the network [9].
Revised: ANN-to-SNN conversion [7, 4, 8] aims to convert trained ANNs to SNNs by using rate coding scheme instead of ReLU activation to represent data flowing inside the network [9].

Original: An ANN of thesame structure will be trained first, followed by adjusting the trained synapses for spatio-temporaldata.
Revised: An ANN of the same structure will be trained first, followed by adjusting the trained synapses for spatio-temporal data.

Original: Many constraints such as bias term needs to be excluded, and only average pooling should beused, etc. need to be incorporated in the pre-training [10].
Revised: Many constraints such as bias term needs to be excluded, and only average pooling should be used, etc. need to be incorporated in the pre-training [10].

Original: However, this indirect training approachposes a lot of problems, such as temporal information is disregarded during ANN training, longdelay incurred for encoding presynaptic inputs, etc., which seriously limits the network’s applicationspectrum.
Revised: However, this indirect training approach poses a lot of problems, such as temporal information is disregarded during ANN training, long delay incurred for encoding presynaptic inputs, etc., which seriously limits the network’s application spectrum.

Original: Furthermore, this training method is inappropriate for neuromorphic datasets since onlyspatial data can be utilized.
Revised: Furthermore, this training method is inappropriate for neuromorphic datasets since only spatial data can be utilized.

Original: 2.2 Direct-Trained Deep SNNs and Neuronal Model
Revised: 2.2 Direct-Trained Deep SNNs and Neuronal Model

Original: The other method is to train SNNs directly based on gradient descent by backpropagating errors [11]in both spatial and temporal dimensions.
Revised: The other method is to train SNNs directly based on gradient descent by backpropagating errors [11] in both spatial and temporal dimensions.

Original: Various strategies have been proposed to realize gradientvalue calculation for non-differentiable spike activation.
Revised: Various strategies have been proposed to realize gradient value calculation for non-differentiable spike activation.

Original: Some recent works apply pseudo derivativealgorithm to replace the non-differentiable spike triggering part of membrane potential curves withan auxiliary function and report satisfactory results [12, 13, 14].
Revised: Some recent works apply pseudo derivative algorithm to replace the non-differentiable spike triggering part of membrane potential curves with an auxiliary function and report satisfactory results [12, 13, 14].

Original: As both spatial and temporalinformation are used in network training, dataset restrictions will not blight direct training methodas in the ANN-to-SNN conversion approach.
Revised: As both spatial and temporal information are used in network training, dataset restrictions will not blight direct training method as in the ANN-to-SNN conversion approach.

Original: In addition, simulation latency can be significantlyshortened because  data encoding  can be freely used with gradient descent.
Revised: In addition, simulation latency can be significantly shortened because more efficient data encoding schemes can be freely used with gradient descent.

Original: The direct supervised learning methods train SNNs based on event-driven data and gradient descent,which is scalable and thus suitable for deep SNNs.
Revised: The direct supervised learning methods train SNNs based on event-driven data and gradient descent, which is scalable and thus suitable for deep SNNs.

Original: In [15, 10, 16], several approaches to builddeep SNN with direct learning methods are described.
Revised: In [15, 10, 16], several approaches to build deep SNN with direct learning methods are described.

Original: It is pointed out in [9] that informationexpressiveness loss attributes to step functional activation causes deep SNNs to suffer performancedegradation.
Revised: It is pointed out in [9] that information expressiveness loss attributes to step functional activation causes deep SNNs to suffer performance degradation.

Original: Multiple deep network architectures such as VGG [17], ResNet [18, 9], and DenseNet[19] have been leveraged to SNNs so that spiking neurons can maintain sufficient information fortransmission.
Revised: Multiple deep network architectures such as VGG [17], ResNet [18, 9], and DenseNet [19] have been leveraged to SNNs so that spiking neurons can maintain sufficient information for transmission. [Figure]

Original: Spiking neuronal model plays a crucial role in controlling data transmission in SNNs.
Revised: Spiking neuronal model plays a crucial role in controlling data transmission in SNNs.

Original: Inspiredby biological neurons , some researchers add lateral effect to spiking neurons [20, 13].
Revised: Inspired by biological neurons, some researchers add lateral effect to spiking neurons [20, 13].

Original: Some useparametric neuronal models for different tasks by making time constant to be learnable [21, 22, 23].[Figure]
Revised: Some use parametric neuronal models for different tasks by making time constant to be learnable [21, 22, 23].

Original: One group proposes a network which contains neurons of adaptive thresholds [24], and anothergroup makes the potential leaky process to be learnable [25].
Revised: One group proposes a network which contains neurons of adaptive thresholds [24], and another group makes the potential leaky process to be learnable [25].

Original: Better accuracies and higher networkrobustness are demonstrated with these neuronal model improvements.
Revised: Better accuracies and higher network robustness are demonstrated with these neuronal model improvements.

Original: 2.3 Dropout
Revised: 2.3 Dropout

Original: One critical issue in using backpropagation to update synaptic connections is the gradient vanishingand exploding problem especially for deep networks.
Revised: One critical issue in using backpropagation to update synaptic connections is the gradient vanishing and exploding problem especially for deep networks.

Original: To reduce the degradation effect, numerousoptimization techniques such as regularization [3, 6] and normalization [4, 16] have been designed.
Revised: To reduce the degradation effect, numerous optimization techniques such as regularization [3, 6] and normalization [4, 16] have been designed.

Original: Dropout is one of the widely applied regularization methods to prevent overfitting in deep networks.
Revised: Dropout is one of the widely applied regularization methods to prevent overfitting in deep networks.

Original: Neurons will be retained randomly with certain probability and form multiple sub-networks fortraining, after which the trained network will be generated based on all these sub-networks.
Revised: Neurons will be retained randomly with certain probability and form multiple sub-models for training, after which the trained network will be generated based on all these sub-models.

Original: Someresearchers modify dropout to make it suitable for spatio-temporal data and successfully apply iton SNNs.
Revised: Some researchers modify dropout to make it suitable for spatio-temporal data and successfully apply it on SNNs.

Original: However, as spiking neurons have both spatial and temporal connections, this results insub-models to exhibit high level of inconsistencies in different sub-model runs.
Revised: However, as spiking neurons have both spatial and temporal connections, this results in sub-models to exhibit high level of inconsistencies in different sub-model runs.

Original:  Methods
Revised:  Methods

Revised: In this section, we introduce the learnable thresholding mechanism and moderate dropout and expound the proposed SNN.

Original: 3.1 Leaky Integrate-and-Fire Model with Dynamic Thresholding
Revised: 3.1 Leaky Integrate-and-Fire Model with Learnable Thresholding

Original: Spiking neurons are the basic elements for information processing and transmission via alternatingthe membrane states in SNNs.
Revised: Spiking neurons are the basic elements for information processing and transmission via alternating the membrane states in SNNs.

Original: Many neuronal models have been proposed by neuroscientists tosimulate biological neurons’ behaviors on computers.
Revised: Many neuronal models have been proposed by neuroscientists to simulate biological neurons’ behaviors on computers.

Original: Due to the complexity of neuronal dynamics inreal nervous systems, there exists trade-offs between  plausibility and computational cost in computersimulation.
Revised: Due to the complexity of neuronal dynamics in real nervous systems, there exists trade-offs between biological plausibility and computational cost in computer simulation.

Original: The Leaky Integrate-and-Fire (LIF) is a simple and pre-eminent mathematical modelfor modelling such neuronal behaviors as potential update, spike emission, and state reset.
Revised: The Leaky Integrate-and-Fire (LIF) is a simple and pre-eminent mathematical model for modelling such neuronal behaviors as potential update, spike emission, and state reset.

Original: It can bedescribed as
Revised: It can be described as

Original: [Equation]
Revised: [Equation]

Original: where τ is a time constant which represents how fast a neuron’s potential will decay with time, u ( t )and I ( t ) are the membrane potential and integrated presynaptic input at time t , respectively.
Revised: where τ is a time constant which represents how fast a neuron’s potential will decay with time, u ( t ) and I ( t ) are the membrane potential and integrated presynaptic input at time t , respectively.

Original: In thiswork, we set neurons’ initial states and reset potentials to be zero.
Revised: In this work, we set neurons’ initial states and reset potentials to be zero.

Original: Fig.
Revised: Fig.

Original: 1a illustrates a neuron’sbehaviors under different  presynaptic inputs.
Revised: 1a illustrates a neuron’s different behaviors under presynaptic inputs.

Original: To suit computer simulation, we need to temporally discretize the LIF model and spiking neurons’states.
Revised: To suit computer simulation, we need to temporally discretize the LIF model and spiking neurons’ states.

Original: The following numerical representation of a spiking neuron’s potential states can be derivedfrom Eq. 1:
Revised: The following numerical representation of a spiking neuron’s potential states can be derived from Eq. 1:

Original: [Equation]
Revised: [Equation]

Original: where u t,ni is the membrane potential of postsynaptic neuron i located in layer n at time t , κ isthe decay factor, o t,n − 1 j is the output spike from presynaptic neuron j , and l ( n − 1) represents allpresynaptic neurons connected to target i .
Revised: where u t,ni is the membrane potential of postsynaptic neuron i located in layer n at time t , κ is the decay factor, o t,n − 1 j is the output spike from presynaptic neuron j , and l ( n − 1) represents all presynaptic neurons connected to target i .

Original: f ( u ) is the step functional activation to determine if thetarget neuron will fire at current timestep, and f ( u ) = 1 when u ≥ V th , otherwise f ( u ) = 0 .
Revised: f ( u ) is the step functional activation to determine if the target neuron will fire at current timestep, and f ( u ) = 1 when u ≥ V th , otherwise f ( u ) = 0 .

Original: Eq. 2 describes the membrane potential update and Eq.
Revised: Eq.describes the membrane potential update and Eq.

Original: 3 introduces how spiking neurons emit new spikes.
Revised: 3 introduces how spiking neurons emit new spikes.

Original: In the LIF model, a postsynaptic neuron’s state in the previous timestep and presynaptic inputs frompreceding layers spatio-temporally govern the target neuron’s behaviors.
Revised: In the LIF model, a postsynaptic neuron’s state in the previous timestep and presynaptic inputs from preceding layers spatio-temporally govern the target neuron’s behaviors.

Original: As SNNs need to use backpropagation to update their parameters, we need to circumvent the obstacleof nondifferentiable activation function f ( u ) .
Revised: As SNNs need to use backpropagation to update their parameters, we need to circumvent the obstacle of nondifferentiable activation function f ( u ) .

Original: Hence, we adopt the idea of pseudo derivative anddesignate an auxiliary rectangular function to serve as the approximated gradient:
Revised: Hence, we adopt the idea of pseudo derivative and designate an auxiliary rectangular function to serve as the approximated gradient:

Original: [Equation]
Revised: [Equation]

Original: By doing so, our SNNs is enhanced with the parameter optimization capability through backpropagation.
Revised: By doing so, our SNNs is endowed with the parameter optimization capability through backpropagation.

Original: Our dynamic thresholding mechanism is based on the  LIF model, and it is able to consider thediversities among spiking neurons even if they are in the same network.
Revised: Our learnable thresholding mechanism is developed on the basis of LIF model by specifically considering the diversities among spiking neurons even if they are in the same network.

Original: This means a neuron’sresponse depends not only on its internal state but also the threshold level.
Revised: This means a neuron’s response depends not only on its internal state but also the threshold level.

Original: In order to investigate theeffects of different threshold values on neuronal behaviors, we assume that two presynaptic neuronsare connected with one postsynaptic neuron through synapses of weight w 1 and w 2 , respectively.
Revised: In order to investigate the effects of different threshold values on neuronal behaviors, we assume that two presynaptic neurons are connected with one postsynaptic neuron through synapses of weight wand w 2 , respectively.

Original: When the postsynaptic neuron receives input spikes from preceding layers, its internal state willchange accordingly based on the weighted input spike trains and the preset thresholds.
Revised: When the postsynaptic neuron receives input spikes from preceding layers, its internal state will change accordingly based on the weighted input spike trains and the preset thresholds.

Original: As illustratedin Fig.
Revised: As illustrated in Fig.

Original: 1a, two presynaptic neurons are triggered at t = { 40 , 160 } and { 200 , 240 } respectively.
Revised: 1a, two presynaptic neurons are triggered at t = { 40 , 160 } and { 200 , 240 } respectively.

Original: According to Eq. 2, the weighted input to postsynaptic neuron is I ( t ) =
Revised: According to Eq. 2, the weighted input to postsynaptic neuron is I ( t ) =

Original: w 1 (cid:0) δ ( t − 40)
Revised: w 1 (cid:0) δ ( t − 40)

Original: + δ ( t −160) (cid:1) + w 2 (cid:0) δ ( t − 200)
Revised: + δ ( t − 160) (cid:1) + w 2 (cid:0) δ ( t − 200)

Original: + δ ( t − 240) (cid:1) , where δ ( t ) represents the Dirac delta function.
Revised: + δ ( t − 240) (cid:1) , where δ ( t ) represents the Dirac delta function.

Original: Fig.
Revised: Fig.

Original: 1b depictsthe postsynaptic neurons’ responses to the same input I ( t ) when using different threshold values,showing a trend of decreasing output spike number and lower system sensitivity to the externalstimulations as threshold value becomes higher.
Revised: 1b depicts the postsynaptic neurons’ responses to the same input I ( t ) when using different threshold values, showing a trend of decreasing output spike number and lower system sensitivity to the external stimulations as threshold value becomes higher.

Original: The high sensitivity to the input helps the system totrack small and instant input signals while low sensitivity can enhance the system’s noise tolerance.
Revised: The high sensitivity to the input helps the system to track small and instant input signals while low sensitivity can enhance the system’s noise tolerance.

Original: Thus, considering the effects of various threshold levels on spiking neurons’ behaviors, our dynamicthresholding mechanism can achieve optimal sensitivity using the following modified neuron responsefunction:
Revised: Thus, considering the effects of various threshold levels on spiking neurons’ behaviors, our learnable thresholding mechanism can achieve optimal sensitivity using the following modified neuron response function:

Original: [Equation]
Revised: [Equation]

Original: where M n is the threshold value for neurons in the n th layer.
Revised: where M n is the threshold value for neurons in the n th layer.

Original: g ( u, m ) =
Revised: g ( u, m ) =

Original: f ( u − m ) , and f ( x ) is thesame step function with zero valued threshold and follows Eq. 4.
Revised: f ( u − m ) , and f ( x ) is the same step function with zero valued threshold and follows Eq. 4.

Original: The main idea of dynamic thresholding is to analytically find the comprehensive gradient of the lossfunction, ∇ E ( W, M ) =
Revised: The main idea of learnable thresholding is to analytically find the comprehensive gradient of the loss function, ∇L ( W, M ) =

Original: [ ∂E∂W , ∂E∂M ] , and then simultaneously update synaptic connections W andthe neuron’s threshold M until convergence.
Revised: [ ∂ L ∂W , ∂ L ∂M ] , and then simultaneously update synaptic connections W and the neuron’s threshold M until convergence.

Original: From Eq. 2, 4, and 5,  derivatives of the loss function∂E∂W and ∂E∂M can be calculated by
Revised: From Eq. 2, 4, and 5, partial derivatives of the loss function ∂ L ∂W and ∂ L ∂M can be calculated by

Original: [Equation]
Revised: [Equation]

Original: The term ∂o t,n ∂M n in Eq.
Revised: The term ∂o t,n ∂M n in Eq.

Original: 7 represents the derivative of a neuron’s output with respect to its threshold.
Revised: 7 represents the derivative of a neuron’s output with respect to its threshold.

Original: It can be shown that∂o t,n
Revised: It can be shown that

Original: [Equation]
Revised: [Equation]

Original:  = − f ′ ( u t,n − M n ) , whose value is determined by theauxiliary function’s gradient defined in Eq.4.
Revised: [Equation] = − f ′ ( u t,n − M n ) , whose value is determined by the auxiliary function’s gradient defined in Eq.

Original: It can be seen from the above formulae that there is norestriction on the threshold value M .
Revised: It can be seen from the above formulae that there is no restriction on the threshold value M .

Original: If the threshold becomes extremely low or high, a neuron willkeep firing or stay silent when receiving the input from presynaptic neurons, causing difficulties intransmitting information.
Revised: If the threshold becomes extremely low or high, a neuron will keep firing or stay silent when receiving the input from presynaptic neurons, causing difficulties in transmitting information.

Original: To ensure threshold values to stay within an appropriate region, we create anew parameter k to define M using hyperbolic tangent relation, formulated as M = tanh ( k ) .
Revised: To ensure threshold values to stay within an appropriate region, we create a new parameter k to define M using hyperbolic tangent relation, formulated as M = tanh ( k ) .

Original: Thisstabilizes neuronal activity by avoiding having too many or too few neurons to fire due to extremethreshold levels.
Revised: This stabilizes neuronal activity by avoiding having too many or too few neurons to fire due to extreme threshold levels.

Original: With this, Eq. 7 can be expressed as:
Revised: With this, Eq. 7 can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: With this enhancement, a neuron’s threshold M can be trained iteratively using the backpropagationmethod and takes a value within the range ( − 1 , 1) .
Revised: With this enhancement, a neuron’s threshold M can be trained iteratively using the backpropagation method and takes a value within the range ( − 1 , 1) .

Original: Furthermore, the gradient of the hyperbolictangent function becomes very small when k becomes infinitely large or small, ensuring that therewill not be any sudden big change on the threshold value and thus achieves the stabilization effect.
Revised: Furthermore, the gradient of the hyperbolic tangent function becomes very small when k becomes infinitely large or small, ensuring that there will not be any sudden big change on the threshold value and thus achieves the stabilization effect.

Original: Although some recent works have treated threshold level as variable to introduce heterogeneity to
Revised: Although some recent works have treated threshold level as variable to introduce heterogeneity to

Original: SNN, grim prerequisites such as neuronal states transformation [26] or pre-defined firing count [27]are required.
Revised: SNN, grim prerequisites such as neuronal states transformation [26] or pre-defined firing count [27] are required.

Original: Some other STDP-based approaches only work on excitatory neurons [28], leading toslow convergence and heavy computational load to individually update every neuron’s threshold.
Revised: Some other STDP-based approaches only work on excitatory neurons [28], leading to slow convergence and heavy computational load to individually update every neuron’s threshold.

Revised: However, the proposed strategy provides an easy approach to grant neuronal thresholds with learning capability under light computational load and can be freely migrated to other gradient descent-based

Original:  Though setting a lower threshold level may have a similar effect as having higher synaptic weights,changing threshold values has other benefits, for example, the potential decay curve will not bestretched and the SNN will not overly depend on a subset of synapses for optimization.
Revised: SNNs, which is exactly the aim of this paper. Though setting a lower threshold level may have a similar effect as having higher synaptic weights, changing threshold values has other benefits, for example, the potential decay curve will not be stretched and the SNN will not overly depend on a subset of synapses for optimization.

Original: In this work,neurons in the same layer share the same threshold to reduce  learnable parameters.
Revised: In this work, we apply the learnable thresholding mechanism to all neurons and let neurons in the same layer share the same threshold to reduce total learnable parameters incurred.

Original: 3.2 Input Encoding and Network Architecture
Revised: 3.2 Input Encoding and Network Architecture

Original: Since SNNs are inherently suitable for processing event-driven data, spike trains in neuromorphicdatasets can be directly fed into SNNs without barrier.
Revised: Since SNNs are inherently suitable for processing event-driven data, spike trains in neuromorphic datasets can be directly fed into SNNs without barrier.

Original: However, the situation becomes challengingfor static datasets due to lack of temporal information.
Revised: However, the situation becomes challenging for static datasets due to lack of temporal information.

Original: Rate coding scheme is widely applied in
Revised: Rate coding scheme is widely applied in

Original: ANN-to-SNN conversion and some recent works [29, 6] aim to encode input into serial spikesgenerated with certain firing probability that is proportional to the original value.
Revised: ANN-to-SNN conversion and some recent works [29, 6] aim to encode input into serial spikes generated with certain firing probability that is proportional to the original value.

Original: One spikingneuron can encode maximally T + 1 values into distinguishable spike trains of firing probabilities{ 0 , 1 T , . . 
Revised: One spiking neuron can encode maximally T + 1 values into distinguishable spike trains of firing probabilities { 0 , 1 T , . . .

Original: , 1 } within simulation window T .
Revised: , 1 } within simulation window T .

Original: Therefore, the rate coding scheme suffers long latencydue to the inevitably long simulation period for maintaining high precision among inputs of a widerange.
Revised: Therefore, the rate coding scheme suffers long latency due to the inevitably long simulation period for maintaining high precision among inputs of a wide range.

Original: In this work, the first layer serves as the encoding layer that directly receives input data andthen, converts data into spike signals before transmitting to the second layer.
Revised: In this work, the first layer serves as the encoding layer that directly receives input data and then, converts data into spike signals before transmitting to the second layer.

Original: In other words, there isno value-to-spike pre-processing step, inputs from dataset can be liberally fed into the network evenif they are not in spike forms.
Revised: In other words, there is no value-to-spike pre-processing step, inputs from dataset can be liberally fed into the network even if they are not in spike forms.

Original: In this way, we can significantly shorten the simulation window, whichmakes it possible to apply our SNN for real-time analysis.
Revised: In this way, we can significantly shorten the simulation window, which makes it possible to apply our SNN for analysis within several timesteps.

Original: DenseNet is proposed in [19] where network layers are densely connected so that data from precedinglayers can be transmitted to subsequent layers directly through dense connections between neurons.
Revised: DenseNet is proposed in [19] where network layers are densely connected so that data from preceding layers can be transmitted to subsequent layers directly through dense connections between neurons.

Original: In this work, we leverage the idea and build our SNNs with DenseNet structure to retain informationfrom degradation caused by the step activation.
Revised: In this work, we leverage the idea and build our SNNs with DenseNet structure to retain information from degradation caused by the step activation.

Original: Our proposed dynamic thresholding method is appliedto all neurons in the network and the normal dropout in fully-connected layers is replaced by ourmoderate dropout.
Revised: Our proposed learnable thresholding method is applied to all neurons in the network and the normal dropout in fully-connected layers is replaced by our moderate dropout.

Original: A detailed structural illustration of our network is shown in Fig.
Revised: A detailed structural illustration of our network is shown in Fig.

Original: 3.3 Output Decoding and Lateral Interaction
Revised: 3.3 Output Decoding and Lateral Interaction

Original: In some works, the first firing neuron in the output layer indicates the predicted result of thenetwork.
Revised: In some works, the first firing neuron in the output layer indicates the predicted result of the network.

Original: However, as neurons’ states are temporally discretized in simulation, there exists asituation in which more than one output neuronswill fire together at the same timestep.
Revised: However, as neurons’ states are temporally discretized in simulation, there exists a situation in which more than one output neurons will fire together at the same timestep.

Original: To avoidsuch a situation, we choose to use a neuron’s potential value instead of firing state for prediction.
Revised: To avoid such a situation, we choose to use a neuron’s potential value instead of firing state for prediction.

Original: In this manner, a network will choose the neuronwith the highest internal state as output even ifmultiple neurons are triggered simultaneously.
Revised: In this manner, a network will choose the neuron with the highest internal state as output even if multiple neurons are triggered simultaneously.

Original: A neuron’s internal state is determined by bothenvironmental stimulus and local effects of itsneighboring neurons.
Revised: A neuron’s internal state is determined by both environmental stimulus and local effects of its neighboring neurons.

Original: Excitatory and inhibitoryinteractions within the receptive field are termedas the lateral interaction property of neurons.
Revised: Excitatory and inhibitory interactions within the receptive field are termed as the lateral interaction property of neurons.

Original: Lateral interaction was first introduced to explain an optical phenomenon that color contrast willbe exaggerated to form simultaneous contrast by visual systems when two color blocks of slightlydifferent gray levels start to connect with each other.
Revised: Lateral interaction was first introduced to explain an optical phenomenon that color contrast will be exaggerated to form simultaneous contrast by visual systems when two color blocks of slightly different gray levels start to connect with each other.

Original: Lateral interactions can avoid informationoverload by dampening input from some neurons and enhancing input from others.
Revised: Lateral interactions can avoid information overload by dampening input from some neurons and enhancing input from others.

Original: In this respect, itis used to sharpen the image and enhance system noise tolerance capability.
Revised: In this respect, it is used to sharpen the image and enhance system noise tolerance capability.

Original: In this work, we considerthe lateral interactions among neurons in fully-connected layers to enhance system robustness andplausibility.
Revised: In this work, we consider the lateral interactions among neurons in fully-connected layers to enhance system robustness and plausibility.

Original: A neuron’s state is dominated by two components: potential update triggered by previousstate or preceding neurons, and interaction with its neighboring neurons in the same layer.
Revised: A neuron’s state is dominated by two components: potential update triggered by previous state or preceding neurons, and interaction with its neighboring neurons in the same layer.

Original: Membranepotential of spiking neurons in fully-connected layers and neuron’s output can be derived from Eq. 2and 3 as follows:
Revised: Membrane potential of spiking neurons in fully-connected layers and neuron’s output can be derived from Eq. 2 and 3 as follows:

Original: [Equation]
Revised: [Equation]

Original: where the term  o t − 1 ,n rf· W rf represents lateral interaction between the target neuron and its surroundingneurons, and l fc ( n − 1) is a set of neurons connected to the target neuron i in fully-connected layer( n − 1) .
Revised: where the term W rf o t − 1 ,n rf represents lateral interaction between the target neuron and its surrounding neurons, and l fc ( n − 1) is a set of neurons connected to the target neuron i in fully-connected layer ( n − 1) .

Original: W rf is a matrix of lateral interaction learnable weights, whose size defines the width ofthe receptive field.
Revised: W rf is a matrix of lateral interaction learnable weights, whose size defines the width of the receptive field.

Original: As aforementioned, the proposed network uses potential or spike as output indifferent layers, following Eq. 10.
Revised: As aforementioned, the proposed network uses potential or spike as output in different layers, following Eq. 10.

Original: By integrating the lateral interaction mechanism, the robustness ofthe network will be ameliorated because the noise effect vanishes when neuron interactions in a largerarea are considered, especially for neuromorphic datasets that suffer from greater noise disturbances.
Revised: By integrating the lateral interaction mechanism, the robustness of the network will be ameliorated because the noise effect vanishes when neuron interactions in a larger area are considered, especially for neuromorphic datasets that suffer from greater noise disturbances.

Original: 3.4 Moderate Dropout
Revised: 3.4 Moderate Dropout

Original: Noting that highly deviatory sub-networks generated by dropout will be integrated and scaled, whichmay inhibit the constructed SNN’s performance.
Revised: Noting that highly deviatory sub-models generated by dropout will be integrated and scaled, which may inhibit the constructed SNN’s performance.

Original: In our SNN, the last-layer neurons’ time-averagedpotentials can be treated as the probabilities of classifying input pattern to each potential class.
Revised: In our SNN, the last-layer neurons’ time-averaged potentials can be treated as the probabilities of classifying input pattern to each potential class.

Original: Assuch, we propose the moderate dropout that aims to minimize inconsistencies between differentsub-networks’ output probability distributions.
Revised: As such, we propose the moderate dropout that aims to minimize inconsistencies between different sub-models’ output probability distributions.

Original: The Kullback-Leibler (KL) divergence score is usedto measure the inconsistence in the output probability distributions of different network runs asillustrated in Fig.
Revised: The Kullback-Leibler (KL) divergence score is used to measure the inconsistence in the output probability distributions of different network runs as illustrated in Fig.

Original: With P and Q being the averaged probability distributions over the entiresimulation period on the same probability space, KL divergence is defined as
Revised: With P and Q being the averaged probability distributions over the entire simulation period on the same probability space, KL divergence is defined as

Original: [Equation]
Revised: [Equation]

Original: We compare sub-networks’ output distributions and incorporate the divergence into the overall lossof the network to minimize the inconsistencies between output distributions.
Revised: We compare sub-models’ output distributions and incorporate the divergence into the overall loss of the network to minimize the inconsistencies between output distributions.

Original: Therefore, the overallloss function comprises the loss of actual output distributions against the target labels and the KLdivergence loss between different distributions output by the same model with moderate dropout.
Revised: Therefore, the overall loss function comprises the loss of actual output distributions against the target labels and the KL divergence loss between different distributions output by the same model with moderate dropout.

Original: It can be observed from Eq.
Revised: It can be observed from Eq.

Original: 11 that the KL divergence, depicting the relative entropy of P with respect
Revised: 11 that the KL divergence, depicting the relative entropy of P with respect

Original: Q , is asymmetric in nature, such that D KL ( P ∥ Q ) ̸
Revised: Q , is asymmetric in nature, such that D KL ( P ∥ Q ) ̸

Original: = D KL ( Q ∥ P ) .
Revised: = D KL ( Q ∥ P ) .

Original: In this respect, we involveboth divergences in the network loss calculation.
Revised: In this respect, we involve both divergences in the network loss calculation.

Original: Furthermore, we use one additional parameter γ tocontrol the impact of moderate dropout in the overall loss function as follows:
Revised: Furthermore, we use one additional parameter γ to control the impact of moderate dropout in the overall loss function as follows:

Original: [Equation]
Revised: [Equation]

Original: From the discussion on dropout, it can be seen that the probability p in which neurons are obscuredfrom the fully-connected layers are stipulated before training and will be kept fixed throughoutthe training phase when drawing from the Bernoulli distribution.
Revised: From the discussion on dropout, it can be seen that the probability p in which neurons are obscured from the fully-connected layers are stipulated before training and will be kept fixed throughout the training phase when drawing from the Bernoulli distribution.

Original: However, this does not take intoaccount the membrane potential values of the neurons before deciding to drop from the network,leading to possible scenarios where essential neurons with high potential are temporarily eliminatedfrom the network.
Revised: However, this does not take into account the membrane potential values of the neurons before deciding to drop from the network, leading to possible scenarios where essential neurons with high potential are temporarily eliminated from the network.

Original: Especially in our SNNs, the potential values are directly used to represent the neurons’ states in fully-connected layers.
Revised: Especially in our SNNs, the potential values are directly used to represent the neurons’ states in fully-connected layers.

Original: The dropping probability of neurons should be accordedsome flexibility for alteration taking into consideration the neuronal membrane potential valuesso that neurons processing larger membrane values can have smaller dropping probabilities.
Revised: The dropping probability of neurons should be accorded some flexibility for alteration taking into consideration the neuronal membrane potential values so that neurons processing larger membrane values can have smaller dropping probabilities.

Original: If aneuron i in fully-connected layer n has a membrane value of u ni , and u ni lies in a distribution withu ni ∈ ( u nmin , u nmax ) , where u nmin and u nmax correspond to the minimum and maximum potentialvalues of neurons in that layer respectively, we can derive the following equation to transform andscale up the potential value u ni and use it to compute the corresponding dropping probability p ni .
Revised: If a neuron i in fully-connected layer n has a membrane value of u ni , and u ni lies in a distribution with u ni ∈ ( u nmin , u n max ) , where u nmin and u n max correspond to the minimum and maximum potential values of neurons in that layer respectively, we can derive the following equation to transform and scale up the potential value u ni and use it to compute the corresponding dropping probability p ni .

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 13 ensures that a neuron’s transformed dropping probability is inversely proportional to itsmembrane potential and the average dropping probability of all neurons in the same layer equalsto the drop rate p drop , which has been defined before training.
Revised: 13 ensures that a neuron’s transformed dropping probability is inversely proportional to its membrane potential and the average dropping probability of all neurons in the same layer equals to the drop rate p drop , which has been defined before training.

Original: In this regard, neuron droppingprobabilities can be altered throughout the training phase, with larger potentials denoting smallerprobabilities, rather than being preset before training in conventional dropout.
Revised: In this regard, neuron dropping probabilities can be altered throughout the training phase, with larger potentials denoting smaller probabilities, rather than being preset before training in conventional dropout.

Original: With the creation ofpotential-dependent probability tensor, this can be subsequently placed into the Bernoulli distributionto generate binary masks for different sub-networks.
Revised: With the creation of potential-dependent probability tensor, this can be subsequently placed into the Bernoulli distribution to generate binary masks for different sub-models.

Original: KL divergence between the different outputdistributions from sub-network runs is incorporated into the loss function according to Eq.
Revised: KL divergence between the different output distributions from sub-model runs is incorporated into the loss function according to Eq.

Original: 12, withthe overall objective being to minimize the loss, leading to increased consistencies among differentruns of the network. 
Revised: 12, with the overall objective being to minimize the loss, leading to increased consistencies among different runs of the network. [Table] [Table]

Original:  Experiment
Revised:  Experiment

Original: [Figure] We evaluate our proposed dynamic thresholding and moderate dropout by testing our deep SNNs of DenseNet architecture for classification tasks using both static and neuromorphic datasets.
Revised: We evaluate our proposed learnable thresholding and moderate dropout by testing our deep SNNs of DenseNet architecture for classification tasks using both static and neuromorphic datasets.

Original: We demonstrate the superiority of our methods in terms of inference accuracies and simulation timesteps by comparing it with other state-ofthe-art SNN approaches.
Revised: We demonstrate the superiority of our methods in terms of inference accuracies and simulation timesteps by comparing it with other state-of-the-art SNN approaches.

Original: 4.1 Empirical Evaluation
Revised: 4.1 Empirical Evaluation

Original: Due to the simplicity of MNIST, we specially create an 8-layer fully-connected SNN( in-conv1-pool1-conv2-pool2-conv3-pool3-fc1fc2-out ) for this dataset.
Revised: Due to the simplicity of MNIST, we specially create an 5-layer fully-connected SNN ( input-conv1pool1-conv2-pool2-conv3-pool3-fc1-fc2-output ) for this dataset.

Original: All the other datasetsare tested with deep-layer SNNs of DenseNetarchitecture.
Revised: All the other datasets are tested with deep-layer SNNs of DenseNet architecture.

Original: Details of the DenseNet SNNs webuild are listed in Tab. 1.
Revised: Detailed structures of the DenseNet SNNs we build are listed in Tab. 1.

Original: We implement the proposed methods in our deep SNNs and test their accuracies for classificationtasks on both static MNIST, CIFAR10 datasets, and neuromorphic N-MNIST, DVS-CIFAR10 datasets.
Revised: We implement the proposed methods in our deep SNNs and test their accuracies for classification tasks on both static MNIST, CIFAR10 datasets, and neuromorphic N-MNIST, DVS-CIFAR10 datasets.

Original: We set the initial threshold parameter k = 0 .
Revised: We set the initial threshold parameter k = 0 .

Original: 5 and moderate dropout impact factor γ = 0 .
Revised: 5 and moderate dropout impact factor γ = 0 .

Original: 5 for all thedatasets.
Revised: 5 for all the datasets.

Original: Dataset description, network parameters, and setting details are given in Appendix.
Revised: Dataset description, network parameters, and setting details are given in Appendix.

Original: Theresults we obtained are shown in Tab. 2, it can be seen that our SNNs outperforms other state-ofthe-art SNNs for all datasets except MNIST.
Revised: The results we obtained are shown in Tab. 2, it can be seen that our SNNs outperforms other state-ofthe-art SNNs for all datasets except MNIST.

Original: Although our inference accuracy for MNIST is slightlylower than [42], our SNN achieves a comparable result with much fewer timesteps needed.
Revised: Although our inference accuracy for MNIST is slightly lower than [30], our SNN achieves a comparable result with much fewer timesteps needed.

Original: Thismakes our proposed SNN suitable for processing real-time data.
Revised: This makes our proposed SNN suitable for fast data analysis. [Figure]

Original: The number of timesteps usedin previous works and ours are compared in Tab. 2.
Revised: The number of timesteps used in previous works and ours are compared in Tab. 2.

Original: It can be seen from the table that our SNNsuse much fewer timesteps in training and inferencing compared to otherSNNs (whether based on
Revised: It can be seen from the table that our SNNs use much fewer timesteps in training and inferencing compared to other

Original:  ANN-to-SNN conversion or backpropagation) and our proposed methods can significantly improve [Table] Direct training93.16% ( 6 ) - 67.80% ( 10 ) [6] Direct training 99.59% ( 50 ) 90.95% ( 100 ) 99.09% ( 100 ) - [13]
Revised: SNNs (whether based on ANN-to-SNN conversion or backpropagation) and our proposed methods can significantly improve the performance of SNNs in many few timesteps.

Original: Direct training 99.50% ( 20 ) - 99.45% ( 20 ) - [43] Direct training 99.46% ( 25 ) - 99.39% ( 25 ) - [44] Direct training - - 96.30% ( 120 ) 32.2% ( 80 ) [9] Direct training - - - 70.2% ( 8 )
Revised: As a result, our proposed

Original: DTMD

Original: Direct training

Original: [Figure]

Original: [Figure]

Original: [Figure] [Figure] are used the performance of SNNs in many few timesteps. As a result, our proposed SNNs not only have afaster respond time but also need less memory space due to  reduced network computations.
Revised: SNNs not only have a lower respond latency but also need less memory space due to the reduced temporal states of neurons.

Original: 4.2 Simulation Study
Revised: 4.2 Simulation Study

Original: Dataset
Revised: In this section

Original: [Table]

Original: [Table]
Revised: [Figure]

Original: In this section, we conduct a series of comprehensivesimulation studies to evaluate the effects of the proposed dynamic thresholding and moderate dropout.
Revised: we conduct a series of comprehensive simulation studies to evaluate the effects of the proposed learnable thresholding and moderate dropout.

Original: We train SNNs with LIF and dynamic thresholdingneurons respectively, followed by integrating moderate dropout.
Revised: We train SNNs with LIF and learnable thresholding neurons respectively, followed by integrating moderate dropout.

Original: Then, we test and compare the proposed SNNs’ inference capabilities.
Revised: Then, we test and compare the proposed SNNs’ inference capabilities.

Original: As can be observed from Tab. 3 and Fig. 4, accuracies and convergencespeed improve after using dynamic thresholding, and are further enhanced with moderate dropout.[Figure]
Revised: As can be observed from Tab. 3 and Fig. 4, accuracies and convergence speed improve after using learnable thresholding, and are further enhanced with moderate dropout.

Original: We show the change in network inference result during training to evaluate the effect of initialthreshold value on our network performance.
Revised: We show the change in network inference result during training to evaluate the effect of different initial threshold values on our network performance.

Original: Fig.
Revised: Fig.

Original: 5 shows the high robustness of our proposed SNNsthat comprise dynamic thresholding neurons.
Revised: 5 shows the high robustness of our proposed SNNs that comprise learnable thresholding neurons.

Original: [Table]

Original: To analyze how thresholds change duringnetwork convergence, in Fig.
Revised: To analyze how thresholds change during network convergence, in Fig.

Original: 6, we use an8-layer fully-connected SNN with 4 different initial thresholds to classify CIFAR10dataset and plot the threshold update curvesof neurons in different layers during training.
Revised: 6, we use an 8-layer fullyconnected SNN with 4 different initial thresholds to classify CIFAR10 dataset and plot the threshold update curves of neurons in different layers during training.

Original: It can be seen from the figure thatalthough different initial threshold values are used, the neurons’ threshold curves tend to convergeafter training with dynamic thresholding.
Revised: It can be seen from the figure that although different initial threshold values are used, the neurons’ threshold curves tend to converge after training with learnable thresholding.

Original: This shows the robustness of SNNs with dynamic thresholding implemented to the initial threshold setting. 
Revised: This shows the robustness of SNNs with learnable thresholding implemented to the initial threshold setting. Additionally, it can be seen that threshold [Figure] [Figure]

Original: Additionally, it can be seen that threshold valuesincrease as the network goes deeper, indicating that information can pass through shallow-layerneurons to maintain sufficient data flow and be filtered by deep-layer neurons to extract usefulinformation for classification. 
Revised: [Table] values increase as the network goes deeper, indicating that information can pass through shallowlayer neurons to maintain sufficient data flow and be filtered by deep-layer neurons to extract useful information for classification. The proposed learnable thresholding scheme is based on gradient descent method to update neuronal thresholds, which is always functionable when the activation gradient exists among network layers.

Original: [Figure]
Revised: And the moderate dropout scheme is a network stability enhancement strategy to be implemented to replace the original normal dropout in the model. Consequently, these proposed methods are not restricted to SNNs based on DenseNet architecture.

Original: Conclusion
Revised: We further conduct ablation study to evaluate effect of the proposed LTMD on network performance with ResNet and VGG architectures. The experimental results are shown in Tab. 4. Consistent performance improvement triggered by the proposed methods can be observed from this table, which verifies the generalized effectiveness of LTMD.

Original: In this work, we propose dynamic thresholding to endow spiking neurons with the ability of selfoptimizing their threshold values during training and moderate dropout to enhance model stability byminimizing inconsistencies between output probability distributions in different  runs.
Revised: In this work, we propose learnable thresholding to endow spiking neurons with the ability of selfoptimizing their threshold values during training and moderate dropout to enhance model stability by minimizing inconsistencies between output probability distributions in different sub-model runs.

Original: We constructdeep SNNs based on DenseNet architecture and incorporate the aforementioned methods and obtaining promising results.
Revised: We construct deep SNNs based on DenseNet architecture and incorporate the aforementioned methods and obtaining promising results.

Original: Our proposed SNNs achieve higher accuracies for almost all the test datasetswith significantly fewer timesteps required, which reflect their excellent classification capability andlow system latency.
Revised: Our proposed SNNs achieve higher accuracies for almost all the test datasets with significantly fewer timesteps required, which reflect their excellent classification capability and low system latency.

Original: We demonstrate higher robustness and faster convergence to the initial thresholdfor the dynamic thresholding mechanism compared to using normal LIF neurons. 
Revised: We demonstrate higher robustness and faster convergence to the initial threshold for the learnable thresholding mechanism compared to using normal LIF neurons. The proposed methods’ generalized effectiveness is also illustrated.

Original: Last but not least,we show that SNNs with dynamic thresholding neurons have higher thresholds in deep layers toconstraint the number of firing neurons, which  helps to reduce the amount of computation.
Revised: Last but not least, we show that SNNs with learnable thresholding neurons have higher thresholds in deep layers after training to constraint the number of firing neurons, which proves the neuronal heterogeneity and helps to reduce the amount of computation.

Original: Checklist
Revised: A Appendix A.1 Algorithms

Original: For all authors... (a) Do the main claims made in the abstract andintroduction accurately reflect the paper’s contributions and scope?
Revised: In this section, we provide the pseudo code of the potential-dependent dropping scheme (Algorithm 1) and the overall training procedures (Algorithm 2) of SNNs with our proposed methods.

Original: [Yes]
Revised: Training: 1: u max ← max ( X ) 2: u mean ← average ( X ) 3: for i = 1 to L ( n ) do 4: u i ← average ( X i ( t )) 5: p i ← p drop · u max − u i u max − u mean 6: mask i = Bernoulli ( p i ) 7: for t = 1 to T do 8: o i ( t ) ← mask i ·

Original: (b) Did you describe the limitations of your work?
Revised: X i ( t ) 9: end for 10: end for 11: return o

Original: [Yes] Our methods will induce a bitmore computational load, please see Appendix A.
Revised: A.2 Details of Datasets and Training Settings A.2.1 MNIST

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: The MNIST dataset contains 60000 images for training and 10000 for testing.

Original: [No] There isno potential negative societal impacts of this work(d)
Revised: Each sample in MNIST is a gray-scale handwritten digit in size of 28 × 28 pixels. A.2.2 CIFAR  The CIFAR10 is a collection of 60000 color images, divided into 50000 images for training and 10000 images for testing.

Original: Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: All images are equally distributed and labelled as 10 classes. Samples in CIFAR10 are color images of size 32 × 32 and contain 3 RGB channels. A.2.3 N-MNIST

Original: [Yes]
Revised: It is the neuromorphic transformed version of MNIST dataset, composed of event data stored inspike trains per pixel. A saccade moving Dynamic Version Sensor (DVS) is used to capture intensity change of MNIST pixels displayed on an LCD screen. Intensity increment and decrement are saved as ON and OFF events in the dataset. Due to the sensor saccade moving, images recorded are slightly larger than the original MNIST images, in size of 34 × 34 × 2 .

Original: If you are including theoretical results...
Revised: Each spike train in N-MNIST lasts 300000 µs , and we first reduce the resolution by 1000 times and then divide it into 15 steps. Each step contains the accumulated events within a period of 20 ms in our experiment. A.2.4 DVS-CIFAR  DVS-CIFAR10 is converted from its static version CIFAR10.

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: There are 10 classes with 1000 images each in DVS-CIFAR10. The dataset image consists of 128 × 128 × 2 spike trains and very noisy background, make it challenging for recognition tasks. We reduce resolution of DVS-CIFAR10 by 1000 times and accumulate events every 20 ms . A.2.5 Loss Function and Optimizer

Original: [Yes] See Section 3.(b)
Revised: For all the experiments except N-MNIST in this work, we use cross entropy and stochastic gradient descent (SGD) optimizer with momentum of 0.9 to calculate the error and iteratively update the synapses. Specially for N-MNIST dataset, Adam and mean squared error (MSE) are used. Algorithm Input: input X , target label vector Y .

Original: Did you include complete proofs of all theoretical results?
Revised: Output: layer i synaptic weights W i , threshold M i , prediction output P , Q .

Original: [Yes] See Section 3.1, 3.3,and 3.4.3. If you ran experiments...
Revised: Parameter: function NeuronUpdate ( u, X, i ) 1: for t = 1 to T do 2: u ( t ) = κu ( t − 1)

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See the pseudocode provided in Appendix A.1.
Revised: + W

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: i X ( t ) 3: end for 4: if u ( t ) ≥ M i then 5: o ( t ) =

Original: [Yes] Hyperparameters settings are provided in Appendix A.2.5, A.2.6,and A.2.7.
Revised: 6: u ( t ) =7: else 8: o ( t )

Original: Full details can be found in code.(c) Did you report error bars (e
Revised: =9: end if 10: return o function FCUpdate ( u, X, i ) 11: for t = 1 to T do 12: u ( t )

Original: g., with respect to the random seed after running experiments multiple times)?
Revised: =

Original: [Yes] We ran experiments for multiple times and please seeTab.
Revised: κu ( t − 1) + W i X ( t ) + W rf

Original: 5 in Appendix A.3 for details. (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes] Number and type of GPUs used inthis work are listed in Tab. 4 in Appendix A.2.8. If
Revised: o i ( t ) 13: end for 14: if u ( t ) ≥ M i

Original: you are using existing assets (e.g., code, data, models) or curating/releasing new assets...(
Revised: then 15: o ( t ) = u ( t ) 16: u ( t ) =17: else 18: o ( t ) =19: end if 20: return o Training: 21: Forward: 22: o 1 ← NeuronUpdate ( u 1 , X, i = 1) 23: for i = 2 to N − 2 do 24: o i ← NeuronUpdate ( u i , o i − 1 , i ) 25: end for 26:

Original: a)
Revised: o N − 2 ← concatenate ( o N − 2 ) 27: o N − 1 ← FCUpdate ( u N − 1 , o N − 2 , N − 1) 28: o N − 1 ← PotentialDependentDrop ( p drop , o N − 1 ) 29: o N ← FCUpdate ( u N , o N − 1 , N ) 30: P , Q ← average ( deconcatenate ( o N ( t ))) 31: E ← Loss ( P, Q, Y ) 32: Backward: 33: ∂ L ∂W ,

Original: If
Revised: ∂ L ∂k ← Autograd Inference: 34: o 1 ← NeuronUpdate ( u 1 , X, i = 1) 35: for i = 2 to N − 2 do 36: o i ← NeuronUpdate ( u i , o i − 1 , i ) 37: end for 38: for i = N − 1 to N do 39: o i ← FCUpdate ( u i , o i − 1 , i ) 40: end for 41: P ← average ( deconcatenate ( o N ( t )))

Revised: A.2.6 Neuron Setting

Original: your work 
Revised: Neurons in this work share the same decay factor κ = 0 . 25 and have reset state of 0 . As events in neuromorphic datasets are sparse, we set the overall dropping rate of moderate dropout p drop = 0 .for all the experiments so that sufficient spiking neurons can be retained for information processing.

Original: uses existing assets, did
Revised: Down sampling layers are used to reduce the feature map size and set with initial output channel = 256 for static datasets and 64 for neuromorphic datasets. Conv pooling layer with kernel size =and stride = 2 is used to reduce the feature map size while maintaining sufficient information of the original samples. One more Avg pooling layer is used for DVS-CIFAR10 dataset due to its large sample size.

Revised: All Conv in dense blocks and transition layers have stride = 1 , padding = 1 , and corresponding kernel sizes indicated in Fig.

Original: you
Revised: Avg pooling layers in the SNN are configured to have kernel size = 2 and stride = 2 for reducing feature map sizes. Adaptive avg pooling layer can ensure output feature is of size 1 × 1 to make it manageable for computation. Two FC layers further reduce the ‘ in feature ’ to 256 and eventually C , where C is the total number of candidate classes. Lateral interaction effect with receptive field width d rf = 5 is considered in both FC layers.

Original: cite the 
Revised: Our proposed moderate dropout is placed in between the FC layers. In the end, the last output decoding layer averages the accumulated output potentials followed by the softmax function to give the final prediction. A.2.8 Computational Resource Network models in this work are programmed in Pytroch.

Original: creators [N/A] b) Did{+All the experiments were conducted on NVIDIA GeForce RTX 
+}
Revised: {+All the experiments were conducted on NVIDIA GeForce RTX 
+}

Original: you
Revised: Ti.

Original: mention the
Revised: Some models were trained on multiple GPUs. Detailed information about number of GPUs we used and batch size for each experiment trial are listed in Tab. 5. [Table] A.3 Multiple-trial Experiment Results

Original: license
Revised: Multiple-trial experiments have been conducted and results are reported in Tab. 6, showing the consistent high performance of our model. [Table] A.4 Computational Load of SNN

Original: of the assets?
Revised: In Tab.

Original: [N/A]
Revised: [Table]

Original: (c) Did you include any new assets either in the supplemental
Revised: we analyze the training time, number of additions, number of multiplications

Original: material
Revised: [Table]

Original: or as a URL?
Revised: and number of learnable parameters of our SNN on CIFAR10 using 4 timesteps. For ANN, an addition and a multiplication are combined as multiply-accumulate (MAC) operation and we also provide these counts in the table. In SNNs, neurons’ outputs are binarized and only one addition operation is needed if neuron fires. The additions count for SNN is calculated by r × T × A , where r is the average firing rate, T is the total timesteps

Original: [N/A]
Revised: [Table]

Original: (d) Did you discuss whether and how
Revised: and A is the addition count in ANN. Average firing rates of neurons in each layer of SNN with LTMD applied are shown in Fig.

Original: consent was obtained from
Revised: It should be noted that our SNN

Original: people
Revised: Layer 0.

Original: whose data you’re
Revised: A v e r a g e

Original: using
Revised: f i r i

Original: /curating?
Revised: ng r a t e need multiplications because the encoding layer encodes real-value input into spikes and decoding layer uses neuronal potentials to provide network predictions.

Original: [N/A e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content? [N/A]5.
Revised: The comparison to ANNs indicates that event-driven SNNs can have significant operation reduction.

Original: If you used crowdsourcing or conducted research with human subjects... (a Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: And the proposed LTMD will only induce affordable computational load to the system but effectively enhance the performance.

Original: [N/A](
Revised: [Table]

Original: b) Did you describe any potential participant risks, with links to Institutional Review
Revised: A.5 Noise Tolerance of Lateral Interactions

Original: Board (IRB) approvals, if applicable?
Revised: To investigate system robustness

Original: [N/A]
Revised: [Table]

Original: (c) Did
Revised: we measure normal output decoding (without lateral interaction) and our model’s performance on test dataset that is appended with Gaussian noise.

Original: you include the estimated hourly wage paid to participants and the total amountspent on participant compensation
Revised: Tab. 8 shows the inference accuracies on MNIST test datasets with different level of noise, mean value ranges fromto 0.5 with an interval of 0.1.

Original: [N/A]
Revised: [Table]

Revised: From the table we can conclude that our decoding layer with lateral interaction outperforms the one without lateral interaction at all noise level situations, especially when noise interference becomes large.