Original: LTMD: Learning Improvement of Spiking Neural Networks with Learnable Thresholding Neurons and Moderate Dropout
Revised: LTMD: Learning Improvement of Spiking Neural Networks with Learnable Thresholding Neurons and Moderate Dropout

Original: Abstract
Revised: Abstract

Original: Spiking Neural Networks (SNNs) have shown substantial promise in processing spatio-temporal data, mimicking biological neuronal mechanisms, and saving computational power.
Revised: Spiking Neural Networks (SNNs) have shown substantial promise in processing spatio-temporal data, mimicking biological neuronal mechanisms, and saving computational power.

Original: However, most SNNs use fixed model regardless of their locations in the network.
Revised: However, most SNNs use fixed model regardless of their locations in the network.

Original: This limits SNNs’ capability of transmitting precise information in the network, which becomes worse for deeper SNNs.
Revised: This limits SNNs’ capability of transmitting precise information in the network, which becomes worse for deeper SNNs.

Original: Some researchers try to use specified parametric models in different network layers or regions, but most still use preset or suboptimal parameters.
Revised: Some researchers try to use specified parametric models in different network layers or regions, but most still use preset or suboptimal parameters.

Original: Inspired by the neuroscience observation that different neuronal mechanisms exist in disparate brain regions, we propose a new spiking neuronal mechanism, named learnable thresholding, to address this issue.
Revised: Inspired by the neuroscience observation that different neuronal mechanisms exist in disparate brain regions, we propose a new spiking neuronal mechanism, named learnable thresholding, to address this issue.

Original: Utilizing learnable threshold values, learnable thresholding enables flexible neuronal mechanisms across layers, proper information flow within the network, and fast network convergence.
Revised: Utilizing learnable threshold values, learnable thresholding enables flexible neuronal mechanisms across layers, proper information flow within the network, and fast network convergence.

Original: In addition, we propose a moderate dropout method to serve as an enhancement technique to minimize inconsistencies between independent dropout runs.
Revised: In addition, we propose a moderate dropout method to serve as an enhancement technique to minimize inconsistencies between independent dropout runs.

Original: Finally, we evaluate the robustness of the proposed learnable thresholding and moderate dropout for image classification with different initial thresholds for various types of datasets.
Revised: Finally, we evaluate the robustness of the proposed learnable thresholding and moderate dropout for image classification with different initial thresholds for various types of datasets.

Original: Our proposed methods produce superior results compared to other approaches for almost all datasets with fewer timesteps.
Revised: Our proposed methods produce superior results compared to other approaches for almost all datasets with fewer timesteps.

Original: Our codes are available at https://github.com/sq117/LTMD.git .
Revised: Our codes are available at https://github.com/sq117/LTMD.git .

Original:  Introduction
Revised:  Introduction

Original: Modelled after the impulse communication between biological neurons, spiking neural network (SNN) is a new class of neural network with neurons exhibiting a distinctive binary output property.
Revised: Modelled after the impulse communication between biological neurons, spiking neural network (SNN) is a new class of neural network with neurons exhibiting a distinctive binary output property.

Original: SNNs have shown great potential in event-driven data processing, computation reduction and network biological plausibility enhancement.
Revised: SNNs have shown great potential in event-driven data processing, computation reduction and network biological plausibility enhancement.

Original: However, unoptimized learning algorithm and complex neuronal dynamics make it challenging to construct high-performance SNN models.
Revised: However, unoptimized learning algorithm and complex neuronal dynamics make it challenging to construct high-performance SNN models.

Original: Current SNN training approaches can generally be divided into two categories: ANN-to-SNN conversion and SNN direct training algorithms based on gradient descent.
Revised: Current SNN training approaches can generally be divided into two categories: ANN-to-SNN conversion and SNN direct training algorithms based on gradient descent.

Original: Both ways only focus on adjusting synaptic connections to minimize the error between a model’s output and target values with the same neuronal model being used across the whole network.
Revised: Both ways only focus on adjusting synaptic connections to minimize the error between a model’s output and target values with the same neuronal model being used across the whole network.

Original: However, it has been found from neuroscience observations that biological neurons’ dynamics vary with their relative locations in the brain especially for primates [1, 2].
Revised: However, it has been found from neuroscience observations that biological neurons’ dynamics vary with their relative locations in the brain especially for primates [1, 2].

Original: Therefore, using the same neuronal model across the whole SNN may limit the SNN’s expression ability as compared to allowing heterogenous neuronal models to be used in the same SNN.
Revised: Therefore, using the same neuronal model across the whole SNN may limit the SNN’s expression ability as compared to allowing heterogenous neuronal models to be used in the same SNN.

Original: Although some researchers have proposed many techniques to balance presynaptic inputs and neuronal behaviors, for example, threshold regularization [3] and spike-based normalization [4], they are either targeting at neuronal state distribution or only applicable for ANN-to-SNN conversion algorithms.
Revised: Although some researchers have proposed many techniques to balance presynaptic inputs and neuronal behaviors, for example, threshold regularization [3] and spike-based normalization [4], they are either targeting at neuronal state distribution or only applicable for ANN-to-SNN conversion algorithms.

Original: Dropout [5, 6] is a commonly used regularization methodology to train neural networks.
Revised: Dropout [5, 6] is a commonly used regularization methodology to train neural networks.

Original: As dropping neurons are randomly selected, each run of network will lead to the input interacting with different sub-models, which will cause the results in different runs to deviate unpredictably.
Revised: As dropping neurons are randomly selected, each run of network will lead to the input interacting with different sub-models, which will cause the results in different runs to deviate unpredictably.

Original: Consequently, a regularization technique is needed to minimize the inconsistencies between output probability distributions during the training phase to improve the performance.
Revised: Consequently, a regularization technique is needed to minimize the inconsistencies between output probability distributions during the training phase to improve the performance.

Original: In this paper, we propose a new learnable thresholding mechanism with a moderate dropout method to enhance the learning of modulated SNNs.
Revised: In this paper, we propose a new learnable thresholding mechanism with a moderate dropout method to enhance the learning of modulated SNNs.

Original: The learnable thresholding mechanism can be integrated with the backpropagation-based SNN direct training algorithm so that neuronal parameters can be changed dynamically and self-optimize during training.
Revised: The learnable thresholding mechanism can be integrated with the backpropagation-based SNN direct training algorithm so that neuronal parameters can be changed dynamically and self-optimize during training.

Original: We adapt the DenseNet architecture and modify the encoding and decoding layers to make the network suitable for diverse datasets.
Revised: We adapt the DenseNet architecture and modify the encoding and decoding layers to make the network suitable for diverse datasets.

Original: A new ‘moderate dropout’ technique is developed to minimize inconsistencies between network sub-models generated in different runs.
Revised: A new ‘moderate dropout’ technique is developed to minimize inconsistencies between network sub-models generated in different runs.

Original: We demonstrate the proposed methods can enhance the SNN’s stability and performance through evaluation based on static and neuromorphic datasets.
Revised: We demonstrate the proposed methods can enhance the SNN’s stability and performance through evaluation based on static and neuromorphic datasets.

Original: To the best of our knowledge, our proposed SNNs achieve the best or comparable accuracies for all the datasets tested compared to other state-of-the-art SNNs.
Revised: To the best of our knowledge, our proposed SNNs achieve the best or comparable accuracies for all the datasets tested compared to other state-of-the-art SNNs.

Original:  Related Work 2.1 ANN-to-SNN Conversion
Revised:  Related Work 2.1 ANN-to-SNN Conversion

Original: ANN-to-SNN conversion [7, 4, 8] aims to convert trained ANNs to SNNs by using rate coding scheme instead of ReLU activation to represent data flowing inside the network [9].
Revised: ANN-to-SNN conversion [7, 4, 8] aims to convert trained ANNs to SNNs by using rate coding scheme instead of ReLU activation to represent data flowing inside the network [9].

Original: An ANN of the same structure will be trained first, followed by adjusting the trained synapses for spatio-temporal data.
Revised: An ANN of the same structure will be trained first, followed by adjusting the trained synapses for spatio-temporal data.

Original: Many constraints such as bias term needs to be excluded, and only average pooling should be used, etc. need to be incorporated in the pre-training [10].
Revised: Many constraints such as bias term needs to be excluded, and only average pooling should be used, etc. need to be incorporated in the pre-training [10].

Original: However, this indirect training approach poses a lot of problems, such as temporal information is disregarded during ANN training, long delay incurred for encoding presynaptic inputs, etc., which seriously limits the network’s application spectrum.
Revised: However, this indirect training approach poses a lot of problems, such as temporal information is disregarded during ANN training, long delay incurred for encoding presynaptic inputs, etc., which seriously limits the network’s application spectrum.

Original: Furthermore, this training method is inappropriate for neuromorphic datasets since only spatial data can be utilized.
Revised: Furthermore, this training method is inappropriate for neuromorphic datasets since only spatial data can be utilized.

Original: 2.2 Direct-Trained Deep SNNs and Neuronal Model
Revised: 2.2 Direct-Trained Deep SNNs and Neuronal Model

Original: The other method is to train SNNs directly based on gradient descent by backpropagating errors [11] in both spatial and temporal dimensions.
Revised: The other method is to train SNNs directly based on gradient descent by backpropagating errors [11] in both spatial and temporal dimensions.

Original: Various strategies have been proposed to realize gradient value calculation for non-differentiable spike activation.
Revised: Various strategies have been proposed to realize gradient value calculation for non-differentiable spike activation.

Original: Some recent works apply pseudo derivative algorithm to replace the non-differentiable spike triggering part of membrane potential curves with an auxiliary function and report satisfactory results [12, 13, 14].
Revised: Some recent works apply pseudo derivative algorithm to replace the non-differentiable spike triggering part of membrane potential curves with an auxiliary function and report satisfactory results [12, 13, 14].

Original: As both spatial and temporal information are used in network training, dataset restrictions will not blight direct training method as in the ANN-to-SNN conversion approach.
Revised: As both spatial and temporal information are used in network training, dataset restrictions will not blight direct training method as in the ANN-to-SNN conversion approach.

Original: In addition, simulation latency can be significantly shortened because more efficient data encoding schemes can be freely used with gradient descent.
Revised: In addition, simulation latency can be significantly shortened because more efficient data encoding schemes can be freely used with gradient descent.

Original: The direct supervised learning methods train SNNs based on event-driven data and gradient descent, which is scalable and thus suitable for deep SNNs.
Revised: The direct supervised learning methods train SNNs based on event-driven data and gradient descent, which is scalable and thus suitable for deep SNNs.

Original: In [15, 10, 16], several approaches to build deep SNN with direct learning methods are described.
Revised: In [15, 10, 16], several approaches to build deep SNN with direct learning methods are described.

Original: It is pointed out in [9] that information expressiveness loss attributes to step functional activation causes deep SNNs to suffer performance degradation.
Revised: It is pointed out in [9] that information expressiveness loss attributes to step functional activation causes deep SNNs to suffer performance degradation.

Original: Multiple deep network architectures such as VGG [17], ResNet [18, 9], and DenseNet [19] have been leveraged to SNNs so that spiking neurons can maintain sufficient information for transmission.
Revised: Multiple deep network architectures such as VGG [17], ResNet [18, 9], and DenseNet [19] have been leveraged to SNNs so that spiking neurons can maintain sufficient information for transmission.

Original: [Figure]
Revised: [Figure]

Original: Spiking neuronal model plays a crucial role in controlling data transmission in SNNs.
Revised: Spiking neuronal model plays a crucial role in controlling data transmission in SNNs.

Original: Inspired by biological neurons, some researchers add lateral effect to spiking neurons [20, 13].
Revised: Inspired by biological neurons, some researchers add lateral effect to spiking neurons [20, 13].

Original: Some use parametric neuronal models for different tasks by making time constant to be learnable [21, 22, 23].
Revised: Some use parametric neuronal models for different tasks by making time constant to be learnable [21, 22, 23].

Original: One group proposes a network which contains neurons of adaptive thresholds [24], and another group makes the potential leaky process to be learnable [25].
Revised: One group proposes a network which contains neurons of adaptive thresholds [24], and another group makes the potential leaky process to be learnable [25].

Original: Better accuracies and higher network robustness are demonstrated with these neuronal model improvements.
Revised: Better accuracies and higher network robustness are demonstrated with these neuronal model improvements.

Original: 2.3 Dropout
Revised: 2.3 Dropout

Original: One critical issue in using backpropagation to update synaptic connections is the gradient vanishing and exploding problem especially for deep networks.
Revised: One critical issue in using backpropagation to update synaptic connections is the gradient vanishing and exploding problem especially for deep networks.

Original: To reduce the degradation effect, numerous optimization techniques such as regularization [3, 6] and normalization [4, 16] have been designed.
Revised: To reduce the degradation effect, numerous optimization techniques such as regularization [3, 6] and normalization [4, 16] have been designed.

Original: Dropout is one of the widely applied regularization methods to prevent overfitting in deep networks.
Revised: Dropout is one of the widely applied regularization methods to prevent overfitting in deep networks.

Original: Neurons will be retained randomly with certain probability and form multiple sub-models for training, after which the trained network will be generated based on all these sub-models.
Revised: Neurons will be retained randomly with certain probability and form multiple sub-models for training, after which the trained network will be generated based on all these sub-models.

Original: Some researchers modify dropout to make it suitable for spatio-temporal data and successfully apply it on SNNs.
Revised: Some researchers modify dropout to make it suitable for spatio-temporal data and successfully apply it on SNNs.

Original: However, as spiking neurons have both spatial and temporal connections, this results in sub-models to exhibit high level of inconsistencies in different sub-model runs.
Revised: However, as spiking neurons have both spatial and temporal connections, this results in sub-models to exhibit high level of inconsistencies in different sub-model runs.

Original:  Methods
Revised:  Methods

Original: In this section, we introduce the learnable thresholding mechanism and moderate dropout and expound the proposed SNN.
Revised: In this section, we introduce the learnable thresholding mechanism and moderate dropout and expound the proposed SNN.

Original: 3.1 Leaky Integrate-and-Fire Model with Learnable Thresholding
Revised: 3.1 Leaky Integrate-and-Fire Model with Learnable Thresholding

Original: Spiking neurons are the basic elements for information processing and transmission via alternating the membrane states in SNNs.
Revised: Spiking neurons are the basic elements for information processing and transmission via alternating the membrane states in SNNs.

Original: Many neuronal models have been proposed by neuroscientists to simulate biological neurons’ behaviors on computers.
Revised: Many neuronal models have been proposed by neuroscientists to simulate biological neurons’ behaviors on computers.

Original: Due to the complexity of neuronal dynamics in real nervous systems, there exists trade-offs between biological plausibility and computational cost in computer simulation.
Revised: Due to the complexity of neuronal dynamics in real nervous systems, there exists trade-offs between biological plausibility and computational cost in computer simulation.

Original: The Leaky Integrate-and-Fire (LIF) is a simple and pre-eminent mathematical model for modelling such neuronal behaviors as potential update, spike emission, and state reset.
Revised: The Leaky Integrate-and-Fire (LIF) is a simple and pre-eminent mathematical model for modelling such neuronal behaviors as potential update, spike emission, and state reset.

Original: It can be described as
Revised: It can be described as

Original: [Equation]
Revised: [Equation]

Original: where τ is a time constant which represents how fast a neuron’s potential will decay with time, u ( t ) and I ( t ) are the membrane potential and integrated presynaptic input at time t , respectively.
Revised: where τ is a time constant which represents how fast a neuron’s potential will decay with time, u ( t ) and I ( t ) are the membrane potential and integrated presynaptic input at time t , respectively.

Original: In this work, we set neurons’ initial states and reset potentials to be zero.
Revised: In this work, we set neurons’ initial states and reset potentials to be zero.

Original: Fig.
Revised: Fig.

Original: 1a illustrates a neuron’s different behaviors under presynaptic inputs.
Revised: 1a illustrates a neuron’s different behaviors under presynaptic inputs.

Original: To suit computer simulation, we need to temporally discretize the LIF model and spiking neurons’ states.
Revised: To suit computer simulation, we need to temporally discretize the LIF model and spiking neurons’ states.

Original: The following numerical representation of a spiking neuron’s potential states can be derived from Eq. 1:
Revised: The following numerical representation of a spiking neuron’s potential states can be derived from Eq. 1:

Original: [Equation]
Revised: [Equation]

Original: where u t,ni is the membrane potential of postsynaptic neuron i located in layer n at time t , κ is the decay factor, o t,n − 1 j is the output spike from presynaptic neuron j , and l ( n − 1) represents all presynaptic neurons connected to target i .
Revised: where u t,ni is the membrane potential of postsynaptic neuron i located in layer n at time t , κ is the decay factor, o t,n − 1 j is the output spike from presynaptic neuron j , and l ( n − 1) represents all presynaptic neurons connected to target i .

Original: f ( u ) is the step functional activation to determine if the target neuron will fire at current timestep, and f ( u ) = 1 when u ≥ V th , otherwise f ( u ) = 0 .
Revised: f ( u ) is the step functional activation to determine if the target neuron will fire at current timestep, and f ( u ) = 1 when u ≥ V th , otherwise f ( u ) = 0 .

Original: Eq.describes the membrane potential update and Eq.
Revised: Eq.describes the membrane potential update and Eq.

Original: 3 introduces how spiking neurons emit new spikes.
Revised: 3 introduces how spiking neurons emit new spikes.

Original: In the LIF model, a postsynaptic neuron’s state in the previous timestep and presynaptic inputs from preceding layers spatio-temporally govern the target neuron’s behaviors.
Revised: In the LIF model, a postsynaptic neuron’s state in the previous timestep and presynaptic inputs from preceding layers spatio-temporally govern the target neuron’s behaviors.

Original: As SNNs need to use backpropagation to update their parameters, we need to circumvent the obstacle of nondifferentiable activation function f ( u ) .
Revised: As SNNs need to use backpropagation to update their parameters, we need to circumvent the obstacle of nondifferentiable activation function f ( u ) .

Original: Hence, we adopt the idea of pseudo derivative and designate an auxiliary rectangular function to serve as the approximated gradient:
Revised: Hence, we adopt the idea of pseudo derivative and designate an auxiliary rectangular function to serve as the approximated gradient:

Original: [Equation]
Revised: [Equation]

Original: By doing so, our SNNs is endowed with the parameter optimization capability through backpropagation.
Revised: By doing so, our SNNs is endowed with the parameter optimization capability through backpropagation.

Original: Our learnable thresholding mechanism is developed on the basis of LIF model by specifically considering the diversities among spiking neurons even if they are in the same network.
Revised: Our learnable thresholding mechanism is developed on the basis of LIF model by specifically considering the diversities among spiking neurons even if they are in the same network.

Original: This means a neuron’s response depends not only on its internal state but also the threshold level.
Revised: This means a neuron’s response depends not only on its internal state but also the threshold level.

Original: In order to investigate the effects of different threshold values on neuronal behaviors, we assume that two presynaptic neurons are connected with one postsynaptic neuron through synapses of weight wand w 2 , respectively.
Revised: In order to investigate the effects of different threshold values on neuronal behaviors, we assume that two presynaptic neurons are connected with one postsynaptic neuron through synapses of weight wand w 2 , respectively.

Original: When the postsynaptic neuron receives input spikes from preceding layers, its internal state will change accordingly based on the weighted input spike trains and the preset thresholds.
Revised: When the postsynaptic neuron receives input spikes from preceding layers, its internal state will change accordingly based on the weighted input spike trains and the preset thresholds.

Original: As illustrated in Fig.
Revised: As illustrated in Fig.

Original: 1a, two presynaptic neurons are triggered at t = { 40 , 160 } and { 200 , 240 } respectively.
Revised: 1a, two presynaptic neurons are triggered at t = { 40 , 160 } and { 200 , 240 } respectively.

Original: According to Eq. 2, the weighted input to postsynaptic neuron is
Revised: According to Eq. 2, the weighted input to postsynaptic neuron is

Original: I ( t ) =
Revised: I ( t ) =

Original: w 1 (cid:0) δ ( t − 40)
Revised: w 1 (cid:0) δ ( t − 40)

Original: + δ ( t − 160) (cid:1) + w 2 (cid:0) δ ( t − 200)
Revised: + δ ( t − 160) (cid:1) + w 2 (cid:0) δ ( t − 200)

Original: + δ ( t − 240) (cid:1) , where δ ( t ) represents the Dirac delta function.
Revised: + δ ( t − 240) (cid:1) , where δ ( t ) represents the Dirac delta function.

Original: Fig.
Revised: Fig.

Original: 1b depicts the postsynaptic neurons’ responses to the same input I ( t ) when using different threshold values, showing a trend of decreasing output spike number and lower system sensitivity to the external stimulations as threshold value becomes higher.
Revised: 1b depicts the postsynaptic neurons’ responses to the same input I ( t ) when using different threshold values, showing a trend of decreasing output spike number and lower system sensitivity to the external stimulations as threshold value becomes higher.

Original: The high sensitivity to the input helps the system to track small and instant input signals while low sensitivity can enhance the system’s noise tolerance.
Revised: The high sensitivity to the input helps the system to track small and instant input signals while low sensitivity can enhance the system’s noise tolerance.

Original: Thus, considering the effects of various threshold levels on spiking neurons’ behaviors, our learnable thresholding mechanism can achieve optimal sensitivity using the following modified neuron response function:
Revised: Thus, considering the effects of various threshold levels on spiking neurons’ behaviors, our learnable thresholding mechanism can achieve optimal sensitivity using the following modified neuron response function:

Original: [Equation]
Revised: [Equation]

Original: where M n is the threshold value for neurons in the n th layer.
Revised: where M n is the threshold value for neurons in the n th layer.

Original: g ( u, m ) =
Revised: g ( u, m ) =

Original: f ( u − m ) , and f ( x ) is the same step function with zero valued threshold and follows Eq. 4.
Revised: f ( u − m ) , and f ( x ) is the same step function with zero valued threshold and follows Eq. 4.

Original: The main idea of learnable thresholding is to analytically find the comprehensive gradient of the loss function, ∇L ( W, M ) =
Revised: The main idea of learnable thresholding is to analytically find the comprehensive gradient of the loss function, ∇L ( W, M ) =

Original: [ ∂ L ∂W , ∂ L ∂M ] , and then simultaneously update synaptic connections W and the neuron’s threshold M until convergence.
Revised: [ ∂ L ∂W , ∂ L ∂M ] , and then simultaneously update synaptic connections W and the neuron’s threshold M until convergence.

Original: From Eq. 2, 4, and 5, partial derivatives of the loss function ∂ L ∂W and ∂ L ∂M can be calculated by
Revised: From Eq. 2, 4, and 5, partial derivatives of the loss function ∂ L ∂W and ∂ L ∂M can be calculated by

Original: [Equation]
Revised: [Equation]

Original: The term ∂o t,n ∂M n in Eq.
Revised: The term ∂o t,n ∂M n in Eq.

Original: 7 represents the derivative of a neuron’s output with respect to its threshold.
Revised: 7 represents the derivative of a neuron’s output with respect to its threshold.

Original: It can be shown that
Revised: It can be shown that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: = − f ′ ( u t,n − M n ) , whose value is determined by the auxiliary function’s gradient defined in Eq.
Revised: = − f ′ ( u t,n − M n ) , whose value is determined by the auxiliary function’s gradient defined in Eq.

Original: It can be seen from the above formulae that there is no restriction on the threshold value M .
Revised: It can be seen from the above formulae that there is no restriction on the threshold value M .

Original: If the threshold becomes extremely low or high, a neuron will keep firing or stay silent when receiving the input from presynaptic neurons, causing difficulties in transmitting information.
Revised: If the threshold becomes extremely low or high, a neuron will keep firing or stay silent when receiving the input from presynaptic neurons, causing difficulties in transmitting information.

Original: To ensure threshold values to stay within an appropriate region, we create a new parameter k to define M using hyperbolic tangent relation, formulated as M = tanh ( k ) .
Revised: To ensure threshold values to stay within an appropriate region, we create a new parameter k to define M using hyperbolic tangent relation, formulated as M = tanh ( k ) .

Original: This stabilizes neuronal activity by avoiding having too many or too few neurons to fire due to extreme threshold levels.
Revised: This stabilizes neuronal activity by avoiding having too many or too few neurons to fire due to extreme threshold levels.

Original: With this, Eq. 7 can be expressed as:
Revised: With this, Eq. 7 can be expressed as:

Original: [Equation]
Revised: [Equation]

Original: With this enhancement, a neuron’s threshold M can be trained iteratively using the backpropagation method and takes a value within the range ( − 1 , 1) .
Revised: With this enhancement, a neuron’s threshold M can be trained iteratively using the backpropagation method and takes a value within the range ( − 1 , 1) .

Original: Furthermore, the gradient of the hyperbolic tangent function becomes very small when k becomes infinitely large or small, ensuring that there will not be any sudden big change on the threshold value and thus achieves the stabilization effect.
Revised: Furthermore, the gradient of the hyperbolic tangent function becomes very small when k becomes infinitely large or small, ensuring that there will not be any sudden big change on the threshold value and thus achieves the stabilization effect.

Original: Although some recent works have treated threshold level as variable to introduce heterogeneity to
Revised: Although some recent works have treated threshold level as variable to introduce heterogeneity to

Original: SNN, grim prerequisites such as neuronal states transformation [26] or pre-defined firing count [27] are required.
Revised: SNN, grim prerequisites such as neuronal states transformation [26] or pre-defined firing count [27] are required.

Original: Some other STDP-based approaches only work on excitatory neurons [28], leading to slow convergence and heavy computational load to individually update every neuron’s threshold.
Revised: Some other STDP-based approaches only work on excitatory neurons [28], leading to slow convergence and heavy computational load to individually update every neuron’s threshold.

Original: However, the proposed strategy provides an easy approach to grant neuronal thresholds with learning capability under light computational load and can be freely migrated to other gradient descent-based
Revised: However, the proposed strategy provides an easy approach to grant neuronal thresholds with learning capability under light computational load and can be freely migrated to other gradient descent-based

Original: SNNs, which is exactly the aim of this paper.
Revised: SNNs, which is exactly the aim of this paper.

Original: Though setting a lower threshold level may have a similar effect as having higher synaptic weights, changing threshold values has other benefits, for example, the potential decay curve will not be stretched and the SNN will not overly depend on a subset of synapses for optimization.
Revised: Though setting a lower threshold level may have a similar effect as having higher synaptic weights, changing threshold values has other benefits, for example, the potential decay curve will not be stretched and the SNN will not overly depend on a subset of synapses for optimization.

Original: In this work, we apply the learnable thresholding mechanism to all neurons and let neurons in the same layer share the same threshold to reduce total learnable parameters incurred.
Revised: In this work, we apply the learnable thresholding mechanism to all neurons and let neurons in the same layer share the same threshold to reduce total learnable parameters incurred.

Original: 3.2 Input Encoding and Network Architecture
Revised: 3.2 Input Encoding and Network Architecture

Original: Since SNNs are inherently suitable for processing event-driven data, spike trains in neuromorphic datasets can be directly fed into SNNs without barrier.
Revised: Since SNNs are inherently suitable for processing event-driven data, spike trains in neuromorphic datasets can be directly fed into SNNs without barrier.

Original: However, the situation becomes challenging for static datasets due to lack of temporal information.
Revised: However, the situation becomes challenging for static datasets due to lack of temporal information.

Original: Rate coding scheme is widely applied in
Revised: Rate coding scheme is widely applied in

Original: ANN-to-SNN conversion and some recent works [29, 6] aim to encode input into serial spikes generated with certain firing probability that is proportional to the original value.
Revised: ANN-to-SNN conversion and some recent works [29, 6] aim to encode input into serial spikes generated with certain firing probability that is proportional to the original value.

Original: One spiking neuron can encode maximally T + 1 values into distinguishable spike trains of firing probabilities { 0 , 1 T , . . .
Revised: One spiking neuron can encode maximally T + 1 values into distinguishable spike trains of firing probabilities { 0 , 1 T , . . .

Original: , 1 } within simulation window T .
Revised: , 1 } within simulation window T .

Original: Therefore, the rate coding scheme suffers long latency due to the inevitably long simulation period for maintaining high precision among inputs of a wide range.
Revised: Therefore, the rate coding scheme suffers long latency due to the inevitably long simulation period for maintaining high precision among inputs of a wide range.

Original: In this work, the first layer serves as the encoding layer that directly receives input data and then, converts data into spike signals before transmitting to the second layer.
Revised: In this work, the first layer serves as the encoding layer that directly receives input data and then, converts data into spike signals before transmitting to the second layer.

Original: In other words, there is no value-to-spike pre-processing step, inputs from dataset can be liberally fed into the network even if they are not in spike forms.
Revised: In other words, there is no value-to-spike pre-processing step, inputs from dataset can be liberally fed into the network even if they are not in spike forms.

Original: In this way, we can significantly shorten the simulation window, which makes it possible to apply our SNN for analysis within several timesteps.
Revised: In this way, we can significantly shorten the simulation window, which makes it possible to apply our SNN for analysis within several timesteps.

Original: DenseNet is proposed in [19] where network layers are densely connected so that data from preceding layers can be transmitted to subsequent layers directly through dense connections between neurons.
Revised: DenseNet is proposed in [19] where network layers are densely connected so that data from preceding layers can be transmitted to subsequent layers directly through dense connections between neurons.

Original: In this work, we leverage the idea and build our SNNs with DenseNet structure to retain information from degradation caused by the step activation.
Revised: In this work, we leverage the idea and build our SNNs with DenseNet structure to retain information from degradation caused by the step activation.

Original: Our proposed learnable thresholding method is applied to all neurons in the network and the normal dropout in fully-connected layers is replaced by our moderate dropout.
Revised: Our proposed learnable thresholding method is applied to all neurons in the network and the normal dropout in fully-connected layers is replaced by our moderate dropout.

Original: A detailed structural illustration of our network is shown in Fig.
Revised: A detailed structural illustration of our network is shown in Fig.

Original: 3.3 Output Decoding and Lateral Interaction
Revised: 3.3 Output Decoding and Lateral Interaction

Original: In some works, the first firing neuron in the output layer indicates the predicted result of the network.
Revised: In some works, the first firing neuron in the output layer indicates the predicted result of the network.

Original: However, as neurons’ states are temporally discretized in simulation, there exists a situation in which more than one output neurons will fire together at the same timestep.
Revised: However, as neurons’ states are temporally discretized in simulation, there exists a situation in which more than one output neurons will fire together at the same timestep.

Original: To avoid such a situation, we choose to use a neuron’s potential value instead of firing state for prediction.
Revised: To avoid such a situation, we choose to use a neuron’s potential value instead of firing state for prediction.

Original: In this manner, a network will choose the neuron with the highest internal state as output even if multiple neurons are triggered simultaneously.
Revised: In this manner, a network will choose the neuron with the highest internal state as output even if multiple neurons are triggered simultaneously.

Original: A neuron’s internal state is determined by both environmental stimulus and local effects of its neighboring neurons.
Revised: A neuron’s internal state is determined by both environmental stimulus and local effects of its neighboring neurons.

Original: Excitatory and inhibitory interactions within the receptive field are termed as the lateral interaction property of neurons.
Revised: Excitatory and inhibitory interactions within the receptive field are termed as the lateral interaction property of neurons.

Original: Lateral interaction was first introduced to explain an optical phenomenon that color contrast will be exaggerated to form simultaneous contrast by visual systems when two color blocks of slightly different gray levels start to connect with each other.
Revised: Lateral interaction was first introduced to explain an optical phenomenon that color contrast will be exaggerated to form simultaneous contrast by visual systems when two color blocks of slightly different gray levels start to connect with each other.

Original: Lateral interactions can avoid information overload by dampening input from some neurons and enhancing input from others.
Revised: Lateral interactions can avoid information overload by dampening input from some neurons and enhancing input from others.

Original: In this respect, it is used to sharpen the image and enhance system noise tolerance capability.
Revised: In this respect, it is used to sharpen the image and enhance system noise tolerance capability.

Original: In this work, we consider the lateral interactions among neurons in fully-connected layers to enhance system robustness and plausibility.
Revised: In this work, we consider the lateral interactions among neurons in fully-connected layers to enhance system robustness and plausibility.

Original: A neuron’s state is dominated by two components: potential update triggered by previous state or preceding neurons, and interaction with its neighboring neurons in the same layer.
Revised: A neuron’s state is dominated by two components: potential update triggered by previous state or preceding neurons, and interaction with its neighboring neurons in the same layer.

Original: Membrane potential of spiking neurons in fully-connected layers and neuron’s output can be derived from Eq. 2 and 3 as follows:
Revised: Membrane potential of spiking neurons in fully-connected layers and neuron’s output can be derived from Eq. 2 and 3 as follows:

Original: [Equation]
Revised: [Equation]

Original: where the term W rf o t − 1 ,n rf represents lateral interaction between the target neuron and its surrounding neurons, and l fc ( n − 1) is a set of neurons connected to the target neuron i in fully-connected layer ( n − 1) .
Revised: where the term W rf o t − 1 ,n rf represents lateral interaction between the target neuron and its surrounding neurons, and l fc ( n − 1) is a set of neurons connected to the target neuron i in fully-connected layer ( n − 1) .

Original: W rf is a matrix of lateral interaction learnable weights, whose size defines the width of the receptive field.
Revised: W rf is a matrix of lateral interaction learnable weights, whose size defines the width of the receptive field.

Original: As aforementioned, the proposed network uses potential or spike as output in different layers, following Eq. 10.
Revised: As aforementioned, the proposed network uses potential or spike as output in different layers, following Eq. 10.

Original: By integrating the lateral interaction mechanism, the robustness of the network will be ameliorated because the noise effect vanishes when neuron interactions in a larger area are considered, especially for neuromorphic datasets that suffer from greater noise disturbances.
Revised: By integrating the lateral interaction mechanism, the robustness of the network will be ameliorated because the noise effect vanishes when neuron interactions in a larger area are considered, especially for neuromorphic datasets that suffer from greater noise disturbances.

Original: 3.4 Moderate Dropout
Revised: 3.4 Moderate Dropout

Original: Noting that highly deviatory sub-models generated by dropout will be integrated and scaled, which may inhibit the constructed SNN’s performance.
Revised: Noting that highly deviatory sub-models generated by dropout will be integrated and scaled, which may inhibit the constructed SNN’s performance.

Original: In our SNN, the last-layer neurons’ time-averaged potentials can be treated as the probabilities of classifying input pattern to each potential class.
Revised: In our SNN, the last-layer neurons’ time-averaged potentials can be treated as the probabilities of classifying input pattern to each potential class.

Original: As such, we propose the moderate dropout that aims to minimize inconsistencies between different sub-models’ output probability distributions.
Revised: As such, we propose the moderate dropout that aims to minimize inconsistencies between different sub-models’ output probability distributions.

Original: The Kullback-Leibler (KL) divergence score is used to measure the inconsistence in the output probability distributions of different network runs as illustrated in Fig.
Revised: The Kullback-Leibler (KL) divergence score is used to measure the inconsistence in the output probability distributions of different network runs as illustrated in Fig.

Original: With P and Q being the averaged probability distributions over the entire simulation period on the same probability space, KL divergence is defined as
Revised: With P and Q being the averaged probability distributions over the entire simulation period on the same probability space, KL divergence is defined as

Original: [Equation]
Revised: [Equation]

Original: We compare sub-models’ output distributions and incorporate the divergence into the overall loss of the network to minimize the inconsistencies between output distributions.
Revised: We compare sub-models’ output distributions and incorporate the divergence into the overall loss of the network to minimize the inconsistencies between output distributions.

Original: Therefore, the overall loss function comprises the loss of actual output distributions against the target labels and the KL divergence loss between different distributions output by the same model with moderate dropout.
Revised: Therefore, the overall loss function comprises the loss of actual output distributions against the target labels and the KL divergence loss between different distributions output by the same model with moderate dropout.

Original: It can be observed from Eq.
Revised: It can be observed from Eq.

Original: 11 that the KL divergence, depicting the relative entropy of P with respect
Revised: 11 that the KL divergence, depicting the relative entropy of P with respect

Original: Q , is asymmetric in nature, such that D KL ( P ∥ Q ) ̸
Revised: Q , is asymmetric in nature, such that D KL ( P ∥ Q ) ̸

Original: = D KL ( Q ∥ P ) .
Revised: = D KL ( Q ∥ P ) .

Original: In this respect, we involve both divergences in the network loss calculation.
Revised: In this respect, we involve both divergences in the network loss calculation.

Original: Furthermore, we use one additional parameter γ to control the impact of moderate dropout in the overall loss function as follows:
Revised: Furthermore, we use one additional parameter γ to control the impact of moderate dropout in the overall loss function as follows:

Original: [Equation]
Revised: [Equation]

Original: From the discussion on dropout, it can be seen that the probability p in which neurons are obscured from the fully-connected layers are stipulated before training and will be kept fixed throughout the training phase when drawing from the Bernoulli distribution.
Revised: From the discussion on dropout, it can be seen that the probability p in which neurons are obscured from the fully-connected layers are stipulated before training and will be kept fixed throughout the training phase when drawing from the Bernoulli distribution.

Original: However, this does not take into account the membrane potential values of the neurons before deciding to drop from the network, leading to possible scenarios where essential neurons with high potential are temporarily eliminated from the network.
Revised: However, this does not take into account the membrane potential values of the neurons before deciding to drop from the network, leading to possible scenarios where essential neurons with high potential are temporarily eliminated from the network.

Original: Especially in our SNNs, the potential values are directly used to represent the neurons’ states in fully-connected layers.
Revised: Especially in our SNNs, the potential values are directly used to represent the neurons’ states in fully-connected layers.

Original: The dropping probability of neurons should be accorded some flexibility for alteration taking into consideration the neuronal membrane potential values so that neurons processing larger membrane values can have smaller dropping probabilities.
Revised: The dropping probability of neurons should be accorded some flexibility for alteration taking into consideration the neuronal membrane potential values so that neurons processing larger membrane values can have smaller dropping probabilities.

Original: If a neuron i in fully-connected layer n has a membrane value of u ni , and u ni lies in a distribution with u ni ∈ ( u nmin , u n max ) , where u nmin and u n max correspond to the minimum and maximum potential values of neurons in that layer respectively, we can derive the following equation to transform and scale up the potential value u ni and use it to compute the corresponding dropping probability p ni .
Revised: If a neuron i in fully-connected layer n has a membrane value of u ni , and u ni lies in a distribution with u ni ∈ ( u nmin , u n max ) , where u nmin and u n max correspond to the minimum and maximum potential values of neurons in that layer respectively, we can derive the following equation to transform and scale up the potential value u ni and use it to compute the corresponding dropping probability p ni .

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 13 ensures that a neuron’s transformed dropping probability is inversely proportional to its membrane potential and the average dropping probability of all neurons in the same layer equals to the drop rate p drop , which has been defined before training.
Revised: 13 ensures that a neuron’s transformed dropping probability is inversely proportional to its membrane potential and the average dropping probability of all neurons in the same layer equals to the drop rate p drop , which has been defined before training.

Original: In this regard, neuron dropping probabilities can be altered throughout the training phase, with larger potentials denoting smaller probabilities, rather than being preset before training in conventional dropout.
Revised: In this regard, neuron dropping probabilities can be altered throughout the training phase, with larger potentials denoting smaller probabilities, rather than being preset before training in conventional dropout.

Original: With the creation of potential-dependent probability tensor, this can be subsequently placed into the Bernoulli distribution to generate binary masks for different sub-models.
Revised: With the creation of potential-dependent probability tensor, this can be subsequently placed into the Bernoulli distribution to generate binary masks for different sub-models.

Original: KL divergence between the different output distributions from sub-model runs is incorporated into the loss function according to Eq.
Revised: KL divergence between the different output distributions from sub-model runs is incorporated into the loss function according to Eq.

Original: 12, with the overall objective being to minimize the loss, leading to increased consistencies among different runs of the network.
Revised: 12, with the overall objective being to minimize the loss, leading to increased consistencies among different runs of the network.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  Experiment
Revised:  Experiment

Original: We evaluate our proposed learnable thresholding and moderate dropout by testing our deep SNNs of DenseNet architecture for classification tasks using both static and neuromorphic datasets.
Revised: We evaluate our proposed learnable thresholding and moderate dropout by testing our deep SNNs of DenseNet architecture for classification tasks using both static and neuromorphic datasets.

Original: We demonstrate the superiority of our methods in terms of inference accuracies and simulation timesteps by comparing it with other state-of-the-art SNN approaches.
Revised: We demonstrate the superiority of our methods in terms of inference accuracies and simulation timesteps by comparing it with other state-of-the-art SNN approaches.

Original: 4.1 Empirical Evaluation
Revised: 4.1 Empirical Evaluation

Original: Due to the simplicity of MNIST, we specially create an 5-layer fully-connected SNN ( input-conv1pool1-conv2-pool2-conv3-pool3-fc1-fc2-output ) for this dataset.
Revised: Due to the simplicity of MNIST, we specially create an 5-layer fully-connected SNN ( input-conv1pool1-conv2-pool2-conv3-pool3-fc1-fc2-output ) for this dataset.

Original: All the other datasets are tested with deep-layer SNNs of DenseNet architecture.
Revised: All the other datasets are tested with deep-layer SNNs of DenseNet architecture.

Original: Detailed structures of the DenseNet SNNs we build are listed in Tab. 1.
Revised: Detailed structures of the DenseNet SNNs we build are listed in Tab. 1.

Original: We implement the proposed methods in our deep SNNs and test their accuracies for classification tasks on both static MNIST, CIFAR10 datasets, and neuromorphic N-MNIST, DVS-CIFAR10 datasets.
Revised: We implement the proposed methods in our deep SNNs and test their accuracies for classification tasks on both static MNIST, CIFAR10 datasets, and neuromorphic N-MNIST, DVS-CIFAR10 datasets.

Original: We set the initial threshold parameter k = 0 .
Revised: We set the initial threshold parameter k = 0 .

Original: 5 and moderate dropout impact factor γ = 0 .
Revised: 5 and moderate dropout impact factor γ = 0 .

Original: 5 for all the datasets.
Revised: 5 for all the datasets.

Original: Dataset description, network parameters, and setting details are given in Appendix.
Revised: Dataset description, network parameters, and setting details are given in Appendix.

Original: The results we obtained are shown in Tab. 2, it can be seen that our SNNs outperforms other state-ofthe-art SNNs for all datasets except MNIST.
Revised: The results we obtained are shown in Tab. 2, it can be seen that our SNNs outperforms other state-ofthe-art SNNs for all datasets except MNIST.

Original: Although our inference accuracy for MNIST is slightly lower than [30], our SNN achieves a comparable result with much fewer timesteps needed.
Revised: Although our inference accuracy for MNIST is slightly lower than [30], our SNN achieves a comparable result with much fewer timesteps needed.

Original: This makes our proposed SNN suitable for fast data analysis.
Revised: This makes our proposed SNN suitable for fast data analysis.

Original: [Figure]
Revised: [Figure]

Original: The number of timesteps used in previous works and ours are compared in Tab. 2.
Revised: The number of timesteps used in previous works and ours are compared in Tab. 2.

Original: It can be seen from the table that our SNNs use much fewer timesteps in training and inferencing compared to other
Revised: It can be seen from the table that our SNNs use much fewer timesteps in training and inferencing compared to other

Original: SNNs (whether based on ANN-to-SNN conversion or backpropagation) and our proposed methods can significantly improve the performance of SNNs in many few timesteps.
Revised: SNNs (whether based on ANN-to-SNN conversion or backpropagation) and our proposed methods can significantly improve the performance of SNNs in many few timesteps.

Original: As a result, our proposed
Revised: As a result, our proposed

Original: SNNs not only have a lower respond latency but also need less memory space due to the reduced temporal states of neurons.
Revised: SNNs not only have a lower respond latency but also need less memory space due to the reduced temporal states of neurons.

Original: 4.2 Simulation Study
Revised: 4.2 Simulation Study

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we conduct a series of comprehensive simulation studies to evaluate the effects of the proposed learnable thresholding and moderate dropout.
Revised: we conduct a series of comprehensive simulation studies to evaluate the effects of the proposed learnable thresholding and moderate dropout.

Original: We train SNNs with LIF and learnable thresholding neurons respectively, followed by integrating moderate dropout.
Revised: We train SNNs with LIF and learnable thresholding neurons respectively, followed by integrating moderate dropout.

Original: Then, we test and compare the proposed SNNs’ inference capabilities.
Revised: Then, we test and compare the proposed SNNs’ inference capabilities.

Original: As can be observed from Tab. 3 and Fig. 4, accuracies and convergence speed improve after using learnable thresholding, and are further enhanced with moderate dropout.
Revised: As can be observed from Tab. 3 and Fig. 4, accuracies and convergence speed improve after using learnable thresholding, and are further enhanced with moderate dropout.

Original: We show the change in network inference result during training to evaluate the effect of different initial threshold values on our network performance.
Revised: We show the change in network inference result during training to evaluate the effect of different initial threshold values on our network performance.

Original: Fig.
Revised: Fig.

Original: 5 shows the high robustness of our proposed
Revised: 5 shows the high robustness of our proposed

Original: SNNs that comprise learnable thresholding neurons.
Revised: SNNs that comprise learnable thresholding neurons.

Original: To analyze how thresholds change during network convergence, in Fig.
Revised: To analyze how thresholds change during network convergence, in Fig.

Original: 6, we use an 8-layer fullyconnected SNN with 4 different initial thresholds to classify CIFAR10 dataset and plot the threshold update curves of neurons in different layers during training.
Revised: 6, we use an 8-layer fullyconnected SNN with 4 different initial thresholds to classify CIFAR10 dataset and plot the threshold update curves of neurons in different layers during training.

Original: It can be seen from the figure that although different initial threshold values are used, the neurons’ threshold curves tend to converge after training with learnable thresholding.
Revised: It can be seen from the figure that although different initial threshold values are used, the neurons’ threshold curves tend to converge after training with learnable thresholding.

Original: This shows the robustness of SNNs with learnable thresholding implemented to the initial threshold setting.
Revised: This shows the robustness of SNNs with learnable thresholding implemented to the initial threshold setting.

Original: Additionally, it can be seen that threshold
Revised: Additionally, it can be seen that threshold

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: values increase as the network goes deeper, indicating that information can pass through shallowlayer neurons to maintain sufficient data flow and be filtered by deep-layer neurons to extract useful information for classification.
Revised: values increase as the network goes deeper, indicating that information can pass through shallowlayer neurons to maintain sufficient data flow and be filtered by deep-layer neurons to extract useful information for classification.

Original: The proposed learnable thresholding scheme is based on gradient descent method to update neuronal thresholds, which is always functionable when the activation gradient exists among network layers.
Revised: The proposed learnable thresholding scheme is based on gradient descent method to update neuronal thresholds, which is always functionable when the activation gradient exists among network layers.

Original: And the moderate dropout scheme is a network stability enhancement strategy to be implemented to replace the original normal dropout in the model.
Revised: And the moderate dropout scheme is a network stability enhancement strategy to be implemented to replace the original normal dropout in the model.

Original: Consequently, these proposed methods are not restricted to SNNs based on DenseNet architecture.
Revised: Consequently, these proposed methods are not restricted to SNNs based on DenseNet architecture.

Original: We further conduct ablation study to evaluate effect of the proposed LTMD on network performance with ResNet and VGG architectures.
Revised: We further conduct ablation study to evaluate effect of the proposed LTMD on network performance with ResNet and VGG architectures.

Original: The experimental results are shown in Tab. 4.
Revised: The experimental results are shown in Tab. 4.

Original: Consistent performance improvement triggered by the proposed methods can be observed from this table, which verifies the generalized effectiveness of
Revised: Consistent performance improvement triggered by the proposed methods can be observed from this table, which verifies the generalized effectiveness of

Original: LTMD.
Revised: LTMD.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we propose learnable thresholding to endow spiking neurons with the ability of selfoptimizing their threshold values during training and moderate dropout to enhance model stability by minimizing inconsistencies between output probability distributions in different sub-model runs.
Revised: In this work, we propose learnable thresholding to endow spiking neurons with the ability of selfoptimizing their threshold values during training and moderate dropout to enhance model stability by minimizing inconsistencies between output probability distributions in different sub-model runs.

Original: We construct deep SNNs based on DenseNet architecture and incorporate the aforementioned methods and obtaining promising results.
Revised: We construct deep SNNs based on DenseNet architecture and incorporate the aforementioned methods and obtaining promising results.

Original: Our proposed SNNs achieve higher accuracies for almost all the test datasets with significantly fewer timesteps required, which reflect their excellent classification capability and low system latency.
Revised: Our proposed SNNs achieve higher accuracies for almost all the test datasets with significantly fewer timesteps required, which reflect their excellent classification capability and low system latency.

Original: We demonstrate higher robustness and faster convergence to the initial threshold for the learnable thresholding mechanism compared to using normal LIF neurons.
Revised: We demonstrate higher robustness and faster convergence to the initial threshold for the learnable thresholding mechanism compared to using normal LIF neurons.

Original: The proposed methods’ generalized effectiveness is also illustrated.
Revised: The proposed methods’ generalized effectiveness is also illustrated.

Original: Last but not least, we show that
Revised: Last but not least, we show that

Original: SNNs with learnable thresholding neurons have higher thresholds in deep layers after training to constraint the number of firing neurons, which proves the neuronal heterogeneity and helps to reduce the amount of computation.

Original: A Appendix A.1 Algorithms

Original: In this section, we provide the pseudo code of the potential-dependent dropping scheme (Algorithm 1) and the overall training procedures (Algorithm 2) of SNNs with our proposed methods.

Original: Training: 1: u max ← max ( X ) 2: u mean ← average ( X ) 3: for i = 1 to L ( n ) do 4: u i ← average ( X i ( t )) 5: p i ← p drop · u max − u i u max − u mean 6: mask i =

Original: Bernoulli ( p i ) 7: for t = 1 to T do 8: o i ( t ) ← mask i ·

Original: X i ( t ) 9: end for 10: end for 11: return o

Original: A.2 Details of Datasets and Training Settings A.2.1 MNIST

Original: The MNIST dataset contains 60000 images for training and 10000 for testing.

Original: Each sample in MNIST is a gray-scale handwritten digit in size of 28 × 28 pixels.

Original: A.2.2 CIFAR

Original: The CIFAR10 is a collection of 60000 color images, divided into 50000 images for training and 10000 images for testing.

Original: All images are equally distributed and labelled as 10 classes.

Original: Samples in

Original: CIFAR10 are color images of size 32 × 32 and contain 3 RGB channels.

Original: A.2.3 N-MNIST

Original: It is the neuromorphic transformed version of MNIST dataset, composed of event data stored inspike trains per pixel.

Original: A saccade moving Dynamic Version Sensor (DVS) is used to capture intensity change of MNIST pixels displayed on an LCD screen.

Original: Intensity increment and decrement are saved as ON and OFF events in the dataset.

Original: Due to the sensor saccade moving, images recorded are slightly larger than the original MNIST images, in size of 34 × 34 × 2 .

Original: Each spike train in N-MNIST lasts 300000 µs , and we first reduce the resolution by 1000 times and then divide it into 15 steps.

Original: Each step contains the accumulated events within a period of 20 ms in our experiment.

Original: A.2.4 DVS-CIFAR

Original: DVS-CIFAR10 is converted from its static version CIFAR10.

Original: There are 10 classes with 1000 images each in DVS-CIFAR10.

Original: The dataset image consists of 128 × 128 × 2 spike trains and very noisy background, make it challenging for recognition tasks.

Original: We reduce resolution of DVS-CIFAR10 by 1000 times and accumulate events every 20 ms .

Original: A.2.5 Loss Function and Optimizer

Original: For all the experiments except N-MNIST in this work, we use cross entropy and stochastic gradient descent (SGD) optimizer with momentum of 0.9 to calculate the error and iteratively update the synapses.

Original: Specially for N-MNIST dataset, Adam and mean squared error (MSE) are used.

Original: Algorithm

Original: Input: input X , target label vector Y .

Original: Output: layer i synaptic weights W i , threshold M i , prediction output P , Q .

Original: Parameter: function NeuronUpdate ( u, X, i ) 1: for t = 1 to T do 2: u ( t ) =

Original: κu ( t − 1)

Original: + W

Original: i X ( t ) 3: end for 4: if u ( t ) ≥ M i then 5: o ( t ) =

Original: 6: u ( t ) =7: else 8: o ( t )

Original: =9: end if 10: return

Original: o function FCUpdate ( u, X, i ) 11: for t = 1 to T do 12: u ( t )

Original: =

Original: κu ( t − 1)

Original: + W i

Original: X ( t )

Original: + W rf

Original: o i ( t ) 13: end for 14: if u ( t ) ≥ M i

Original: then 15: o ( t ) =

Original: u ( t ) 16: u ( t ) =17: else 18: o ( t ) =19: end if 20: return o

Original: Training: 21: Forward: 22: o 1 ← NeuronUpdate ( u 1 , X, i = 1) 23: for i = 2 to N − 2 do 24:

Original: o i ← NeuronUpdate ( u i , o i − 1 , i ) 25: end for 26:

Original: o N − 2 ← concatenate ( o N − 2 ) 27:

Original: o N − 1 ← FCUpdate ( u N − 1 , o N − 2 , N − 1) 28:

Original: o N − 1 ← PotentialDependentDrop ( p drop , o N − 1 ) 29:

Original: o N ← FCUpdate ( u N , o N − 1 , N ) 30: P , Q ← average ( deconcatenate ( o N ( t ))) 31: E ← Loss ( P, Q, Y ) 32: Backward: 33: ∂ L ∂W ,

Original: ∂

Original: L ∂k ← Autograd

Original: Inference: 34: o 1 ← NeuronUpdate ( u 1 , X, i = 1) 35: for i = 2 to N − 2 do 36:

Original: o i ← NeuronUpdate ( u i , o i − 1 , i ) 37: end for 38: for i = N − 1 to N do 39: o i ← FCUpdate ( u i , o i − 1 , i ) 40: end for 41: P ← average ( deconcatenate ( o N ( t )))

Original: A.2.6 Neuron Setting

Original: Neurons in this work share the same decay factor κ = 0 .

Original: 25 and have reset state of 0 .

Original: As events in neuromorphic datasets are sparse, we set the overall dropping rate of moderate dropout p drop = 0 .for

Original: all the experiments so that sufficient spiking neurons can be retained for information processing.

Original: A.2.7 Network Setting

Original: Down sampling layers are used to reduce the feature map size and set with initial output channel = 256 for static datasets and 64 for neuromorphic datasets.

Original: Conv pooling layer with kernel size =and stride = 2 is used to reduce the feature map size while maintaining sufficient information of the original samples.

Original: One more Avg pooling layer is used for DVS-CIFAR10 dataset due to its large sample size.

Original: All Conv in dense blocks and transition layers have stride = 1 , padding = 1 , and corresponding kernel sizes indicated in Fig.

Original: Avg pooling layers in the SNN are configured to have kernel size = 2 and stride = 2 for reducing feature map sizes.

Original: Adaptive avg pooling layer can ensure output feature is of size 1 × 1 to make it manageable for computation.

Original: Two FC layers further reduce the ‘ in feature ’ to 256 and eventually C , where C is the total number of candidate classes.

Original: Lateral interaction effect with receptive field width d rf = 5 is considered in both FC layers.

Original: Our proposed moderate dropout is placed in between the FC layers.

Original: In the end, the last output decoding layer averages the accumulated output potentials followed by the softmax function to give the final prediction.

Original: A.2.8 Computational Resource

Original: Network models in this work are programmed in Pytroch.

Original: All the experiments were conducted on

Original: [-NVIDIA GeForce RTX 
-]
Revised: [-NVIDIA GeForce RTX 
-]

Original: Ti.

Original: Some models were trained on multiple GPUs.

Original: Detailed information about number of GPUs we used and batch size for each experiment trial are listed in Tab. 5.

Original: [Table]

Original: A.3 Multiple-trial Experiment Results

Original: Multiple-trial experiments have been conducted and results are reported in Tab. 6, showing the consistent high performance of our model.

Original: [Table]

Original: A.4 Computational Load of SNN

Original: In Tab.

Original: [Table]

Original: we analyze the training time, number of additions, number of multiplications

Original: [Table]

Original: and number of learnable parameters of our SNN on CIFAR10 using 4 timesteps.

Original: For ANN, an addition and a multiplication are combined as multiply-accumulate (MAC) operation and we also provide these counts in the table.

Original: In SNNs, neurons’ outputs are binarized and only one addition operation is needed if neuron fires.

Original: The additions count for SNN is calculated by r × T × A , where r is the average firing rate, T is the total timesteps

Original: [Table]

Original: and A is the addition count in ANN.

Original: Average firing rates of neurons in each layer of SNN with LTMD applied are shown in Fig.

Original: It should be noted that our SNN

Original: Layer 0.

Original: A v e r a g e

Original: f i r i ng r a t e need multiplications because theencoding layer encodes real-value input into spikes and decoding layer uses neuronal potentials to provide network predictions. The comparison to ANNs indicates that event-driven SNNs can have significant operation reduction.
Revised: SNNs with learnable thresholding neurons have higher thresholds in deep layers after training to constraint the number of firing neurons, which proves the neuronal heterogeneity and helps to reduce the amount of computation.

Original: And the proposed LTMD will only induce affordable computational load to the system but effectively enhance the performance.

Original: [Table]

Original: A.5 Noise Tolerance of Lateral Interactions

Original: To investigate system robustness

Original: [Table]

Original: we measure normal output decoding (without lateral interaction) and our model’s performance on test dataset that is appended with Gaussian noise.

Original: Tab. 8 shows the inference accuracies on MNIST test datasets with different level of noise, mean value ranges fromto 0.5 with an interval of 0.1.

Original: [Table]

Original: From the table we can conclude that our decoding layer with lateral interaction outperforms the one without lateral interaction at all noise level situations, especially when noise interference becomes large.