Original: IM-Loss: Information Maximization Loss for Spiking Neural Networks
Revised: IM-Loss: Information Maximization Loss for Spiking Neural Networks

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Spiking Neural Network (SNN), recognized as a type of biologically plausiblearchitecture, has recently drawn much research attention.
Revised: Spiking Neural Network (SNN), recognized as a type of biologically plausible architecture, has recently drawn much research attention.

Original: It transmits informationby 0 / 1 spikes.
Revised: It transmits information by 0 / 1 spikes.

Original: This bio-mimetic mechanism of SNN demonstrates extreme energyefficiency since it avoids any multiplications on neuromorphic hardware.
Revised: This bio-mimetic mechanism of SNN demonstrates extreme energy efficiency since it avoids any multiplications on neuromorphic hardware.

Original: However,the forward-passing 0 / 1 spike quantization will cause information loss and accuracy degradation.
Revised: However, the forward-passing 0 / 1 spike quantization will cause information loss and accuracy degradation.

Original: To deal with this problem, the Information maximization loss(IM-Loss) that aims at maximizing the information flow in the SNN is proposed inthe paper.
Revised: To deal with this problem, the Information maximization loss (IM-Loss) that aims at maximizing the information flow in the SNN is proposed in the paper.

Original: The IM-Loss not only enhances the information expressiveness of an
Revised: The IM-Loss not only enhances the information expressiveness of an

Original: SNN directly but also plays a part of the role of normalization without introducingany additional operations ( e.g. , bias and scaling) in the inference phase.
Revised: SNN directly but also plays a part of the role of normalization without introducing any additional operations ( e.g. , bias and scaling) in the inference phase.

Original: Additionally, we introduce a novel differentiable spike activity estimation, Evolutionary
Revised: Additionally, we introduce a novel differentiable spike activity estimation, Evolutionary

Original: Surrogate Gradients (ESG) in SNNs.
Revised: Surrogate Gradients (ESG) in SNNs.

Original: By appointing automatic evolvable surrogategradients for spike activity function, ESG can ensure sufficient model updates atthe beginning and accurate gradients at the end of the training, resulting in botheasy convergence and high task performance.
Revised: By appointing automatic evolvable surrogate gradients for spike activity function, ESG can ensure sufficient model updates at the beginning and accurate gradients at the end of the training, resulting in both easy convergence and high task performance.

Original: Experimental results on both popularnon-spiking static and neuromorphic datasets show that the SNN models trainedby our method outperform the current state-of-the-art algorithms.
Revised: Experimental results on both popular non-spiking static and neuromorphic datasets show that the SNN models trained by our method outperform the current state-of-the-art algorithms.

Original:  Introduction
Revised:  Introduction

Original: Recent progress in Artificial Neural Networks (ANNs) has been successfully applied in many machineintelligence fields.
Revised: Recent progress in Artificial Neural Networks (ANNs) has been successfully applied in many machine intelligence fields.

Original: Although ANNs imitate the structure of the brain to a certain extent, their signalprocessing mechanism using operations on real, dense valued tensors, still differs from how the brainworks based on sparse spike events.
Revised: Although ANNs imitate the structure of the brain to a certain extent, their signal processing mechanism using operations on real, dense valued tensors, still differs from how the brain works based on sparse spike events.

Original: Being another sub-category of brain-inspired learning models,the Spiking Neural Networks (SNNs) utilize 0 / 1 binary spike signals to transmit information betweenunits and spatio-temporal dynamics to mimic brain behaviors.
Revised: Being another sub-category of brain-inspired learning models, the Spiking Neural Networks (SNNs) utilize 0 / 1 binary spike signals to transmit information between units and spatio-temporal dynamics to mimic brain behaviors.

Original: Unlike the ANN, where all neurons inthe same layer need to be evaluated before passing information to the next layer, the SNN processesinformation asynchronously and sparsely based on sparse spike events.
Revised: Unlike the ANN, where all neurons in the same layer need to be evaluated before passing information to the next layer, the SNN processes information asynchronously and sparsely based on sparse spike events.

Original: Benefitting from such aninformation processing paradigm, the SNN can efficiently run on specialized neuromorphic hardware( e.g. , Loihi [4]) and has achieved substantial progress in recent years.
Revised: Benefitting from such an information processing paradigm, the SNN can efficiently run on specialized neuromorphic hardware ( e.g. , Loihi [4]) and has achieved substantial progress in recent years.

Original: Although the SNN is more energy-efficient compared with the ANN, it still has some limitationsto achieve satisfactory task performance.
Revised: Although the SNN is more energy-efficient compared with the ANN, it still has some limitations to achieve satisfactory task performance.

Original: Due to the usage of spikes for information receiving andtransmitting, the SNN needs to quantize real analog values into 0 / 1 -formed spikes in implementation.
Revised: Due to the usage of spikes for information receiving and transmitting, the SNN needs to quantize real analog values into 0 / 1 -formed spikes in implementation.

Original: This quantization causes information loss in forward propagation.
Revised: This quantization causes information loss in forward propagation.

Original: There are some remarkable existingworks that alleviate the problem of information loss of SNNs by adjusting the spike rate by learningmore parameters.
Revised: There are some remarkable existing works that alleviate the problem of information loss of SNNs by adjusting the spike rate by learning more parameters.

Original: In [19] and [31], a method that aims to learn the correct membrane leak-and-firingthreshold for each layer of the SNN was proposed.
Revised: In [22] and [34], a method that aims to learn the correct membrane leak-and-firing threshold for each layer of the SNN was proposed.

Original: Results showed that learning the correct thresholdcan help increase the information capacity of the SNN.
Revised: Results showed that learning the correct threshold

Original: A wise choice of the firing threshold must besmall enough that sufficient spikes are fired while also being large enough that avoids all neurons
Revised: Pre

Original: [Figure]
Revised: [Figure]

Original: Post
Revised: Post

Original: [Figure]
Revised: [Figure]

Original: Leaky Integrate-Fire
Revised: Leaky Integrate-Fire

Original: [Figure]
Revised: [Figure]

Original:  fired at the same time.
Revised: can help increase the information capacity of the SNN. A wise choice of the firing threshold must be small enough that sufficient spikes are fired while also being large enough that avoids all neurons fired at the same time.

Original: Similarly, by incorporating the learnable membrane time parameter intothe SNN, the parametric neuron [6; 26] can not only increase the diversity of neurons but also theinformation expressiveness of the network.
Revised: Similarly, by incorporating the learnable membrane time parameter into the SNN, the parametric neuron [8; 29] can not only increase the diversity of neurons but also the information expressiveness of the network.

Original: However, all these methods require more parameters andattempt to increase the information expressiveness of the SNN in an indirect manner.
Revised: However, all these methods require more parameters and attempt to increase the information expressiveness of the SNN in an indirect manner.

Original: Meanwhile,the non-differentiable spike activities also prevent the SNN from producing accurate gradients inbackward propagation, which increase the difficulty of the model optimization.
Revised: Meanwhile, the non-differentiable spike activities also prevent the SNN from producing accurate gradients in backward propagation, which increase the difficulty of the model optimization.

Original: In this work, we propose information maximization (IM) loss to deal with the information loss in
Revised: In this work, we propose information maximization (IM) loss to deal with the information loss in

Original: SNNs.
Revised: SNNs.

Original: More specifically, we first analyze the information expressiveness ability of the SNN from theview of information entropy of the spikes.
Revised: More specifically, we first analyze the information expressiveness ability of the SNN from the view of information entropy of the spikes.

Original: Then a spike distribution loss is introduced for calculatingthe information loss, i.e. , IM-Loss, of the SNN, thus the information expressiveness of the SNN canbe maximized directly by minimizing this information loss, which benefits the accuracy improvementof the SNN.
Revised: Then a spike distribution loss is introduced for calculating the information loss, i.e. , IM-Loss, of the SNN, thus the information expressiveness of the SNN can be maximized directly by minimizing this information loss, which benefits the accuracy improvement of the SNN.

Original: To overcome the non-differentiable spike activities during the model optimization in
Revised: To overcome the non-differentiable spike activities during the model optimization in

Original: SNNs, we also introduce Evolutionary Surrogate Gradients (ESG) in the paper.
Revised: SNNs, we also introduce Evolutionary Surrogate Gradients (ESG) in the paper.

Original: The ESG can evolveautomatically during training and provides a pathway to bridge the gap between the pseudo derivativeand appropriate derivative.
Revised: The ESG can evolve automatically during training and provides a pathway to bridge the gap between the pseudo derivative and appropriate derivative.

Original: With evolving, it not only ensures sufficient updating at the beginningbut also accurate gradients calculation at the end of the training, resulting in networks with betterperformance.
Revised: With evolving, it not only ensures sufficient updating at the beginning but also accurate gradients calculation at the end of the training, resulting in networks with better performance.

Original: The overall workflow of our algorithm can be seen in Fig.1.
Revised: The overall workflow of our algorithm can be seen in Fig.1.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We propose the IM-Loss to preserve the information in SNNs.
Revised: • We propose the IM-Loss to preserve the information in SNNs.

Original: It presents a new perspectiveto understand the ability of the SNN to express information.
Revised: It presents a new perspective to understand the ability of the SNN to express information.

Original: IM-Loss also plays a part

Original: of the role of normalization but without introducing any multiply-and-accumulate (MAC)

Original: operations in the inference phase.

Original: With the IM-Loss, we can train a more accurate and deeper SNN.
Revised: With the IM-Loss, we can train a more accurate and deeper SNN.

Original: • We also introduce the ESG to effectively approximate the gradients of the spike activityfunction.
Revised: • We also introduce the ESG to effectively approximate the gradients of the spike activity function.

Original: The ESG can ensure both sufficient updating of weights at the beginning andaccurate gradients at the end of the training in an evolutionary way so that the SNN wouldbe more accurate and easier to converge.
Revised: The ESG can ensure both sufficient updating of weights at the beginning and accurate gradients at the end of the training in an evolutionary way so that the SNN would be more accurate and easier to converge.

Original: • Our method is evaluated on both static and dynamic datasets.
Revised: • Our method is evaluated on both static and dynamic datasets.

Original: Extensive experimental resultsshow that our algorithm performs remarkably well across various network structures.
Revised: Extensive experimental results show that our algorithm performs remarkably well across various network structures.

Original: To our best knowledge, this is the first work that can directly train deep SNNs from scratch without normalization or warm-start techniques.
Revised: To our best knowledge, this is the first work that can directly train deep SNNs from scratch without normalization or warm-start techniques.

Original:  Related Work
Revised:  Related Work

Original: In this section, we first briefly review the learning methods of SNNs to provide readers an overallunderstanding.
Revised: In this section, we first briefly review the learning methods of SNNs to provide readers an overall understanding.

Original: Then we introduce the information theory used in DNNs which gives us insight todesign our method for handling information loss in SNNs.
Revised: Then we introduce the information theory used in DNNs which gives us insight to design our method for handling information loss in SNNs.

Original: 2.1 Learning Methods of Spiking Neural Networks
Revised: 2.1 Learning Methods of Spiking Neural Networks

Original: The learning algorithms for SNNs can be divided into three categories: (i) converting ANN to SNN(ANN2SNN)
Revised: The learning algorithms for SNNs can be divided into three categories: (i) converting ANN to SNN (ANN2SNN)

Original: [21]; (ii) unsupervised learning [17]; and (iii) supervised learning [16].
Revised: [24; 16]; (ii) unsupervised learning [6]; and (iii) supervised learning [20; 22; 28; 17; 10].

Original: ANN2SNNconverts a special trained ANN to an SNN that yields the same input-output mapping for a giventask.
Revised: ANN2SNN converts a special trained ANN to an SNN that yields the same input-output mapping for a given task.

Original: Since the research of the ANN has been very successful, ANN2SNN can gain competitiveaccuracy.
Revised: Since the research of the ANN has been very successful, ANN2SNN can gain competitive accuracy.

Original: Some recent conversion methods have achieved nearly loss-less accuracy with VGG and
Revised: Some recent conversion methods have achieved nearly loss-less accuracy with VGG and

Original: ResNet as original ANN models [13].
Revised: ResNet as original ANN models [16].

Original: However, this kind of method is restricted to rate-codingand can only handle static datasets.
Revised: However, this kind of method is restricted to rate-coding and can only handle static datasets.

Original: Moreover, ANN2SNN requires long timesteps (usually needshundreds of timesteps) to converge and cannot exploit the spatio-temporal features during training forsolving temporal tasks.
Revised: Moreover, ANN2SNN requires long timesteps (usually needs hundreds of timesteps) to converge and cannot exploit the spatio-temporal features during training for solving temporal tasks.

Original: Unsupervised learning is based on biological plausible local learning rulesand is usually considered more biologically plausible than other categories.
Revised: Unsupervised learning is based on biological plausible local learning rules and is usually considered more biologically plausible than other categories.

Original: However, unsupervisedlearning can only train shallow SNNs and is hard to achieve desirable performance.
Revised: However, unsupervised learning can only train shallow SNNs and is hard to achieve desirable performance.

Original: Supervisedlearning uses the derivable approximation to overcome the non-differentiability of the spike activities.
Revised: Supervised learning uses the derivable approximation to overcome the non-differentiability of the spike activities.

Original: Hence, the SNN can be optimized with gradient descent algorithms as the ANN and achieves highperformance.
Revised: Hence, the SNN can be optimized with gradient descent algorithms as the ANN and achieves high performance.

Original: Moreover, Supervised learning can greatly reduce the number of timesteps and handledynamic datasets.
Revised: Moreover, Supervised learning can greatly reduce the number of timesteps and handle dynamic datasets.

Original: Thus, supervised learning has increasingly aroused researchers’ great interest inrecent years.
Revised: Thus, supervised learning has increasingly aroused researchers’ great interest in recent years.

Original: We focus on providing some insights to improve the performance of the supervisedlearning-based SNNs.
Revised: We focus on providing some insights to improve the performance of the supervised learning-based SNNs.

Original: 2.2 Information Theory in Deep Neural Networks
Revised: 2.2 Information Theory in Deep Neural Networks

Original: Information theory is widely used in DNNs, ranging from theoretical investigations ( e.g. , the generalization bound induced by mutual information [23]) to practical applications ( e.g. , the variationalinformation bottleneck in representation learning [2]).
Revised: Information theory is widely used in DNNs, ranging from theoretical investigations ( e.g. , the generalization bound induced by mutual information [26]) to practical applications ( e.g. , the variational information bottleneck in representation learning [2]).

Original: It provides a useful way to understand theunderlying behavior of random variables(or processes) and this is a key factor in developing and analyzing deep models.
Revised: It provides a useful way to understand the underlying behavior of random variables and this is a key factor in developing and analyzing deep models.

Original: There are many popular information-theoretic principles that have demonstratedgreat potential in DNNs [28].
Revised: There are many popular information-theoretic principles that have demonstrated great potential in DNNs [31].

Original: In [27], deep energy-based models are trained with the f -divergence(a generalization of KL divergence) minimization.
Revised: In [30], deep energy-based models are trained with the f -divergence (a generalization of KL divergence) minimization.

Original: In reinforcement learning, the maximum entropyregularization is adopted to encourage exploration and avoid getting stuck in a local optima [1].
Revised: In reinforcement learning, the maximum entropy regularization is adopted to encourage exploration and avoid getting stuck in a local optima [1].

Original: Mutual information maximization is used in [18] to improve the expressiveness of a model by shiftingthe distribution of weights to zero-mean in every training iteration.
Revised: Mutual information maximization is used in [21] to improve the expressiveness of a model by shifting the distribution of weights to zero-mean in every training iteration.

Original: In [3], training DNNs for classification is investigated via minimizing the information bottleneck(IB) functional.
Revised: In [3], training DNNs for classification is investigated via minimizing the information bottleneck functional.

Original: Despite the greatsuccess of information theory in DNNs, we find no related works in supervised learning of SNNs.
Revised: Despite the great success of information theory in DNNs, we find no related works in supervised learning of SNNs.

Original: In the paper, we propose the IM-Loss, which is based on the information maximization principle,to improve SNNs.
Revised: In the paper, we propose the IM-Loss, which is based on the information maximization principle, to improve SNNs.

Original: Furthermore, our implementation and idea of the information maximization arevery different from these existing works, which may also provide a new idea to further improve thesemethods.
Revised: Our implementation and idea of the information maximization are very different from these existing works, which may also provide a new idea to further improve these methods.

Original:  Preliminary
Revised:  Preliminary

Original: For better expression, we will first introduce the iterative Leaky Integrate-and-Fire (LIF) model andsurrogate gradients of SNNs in detail, which are also adopted in our method.
Revised: For better expression, we will first introduce the iterative Leaky Integrate-and-Fire (LIF) model and surrogate gradients of SNNs in detail, which are also adopted in our method.

Original: 3.1 Explicitly Iterative LIF Model
Revised: 3.1 Explicitly Iterative LIF Model

Original: The LIF neuron is a commonly used mathematical model to describe the behavior of neural activities,which includes spike firing and update of membrane potential.
Revised: The LIF neuron is a commonly used mathematical model to describe the behavior of neural activities, which includes spike firing and update of membrane potential.

Original: Formally, a LIF neuron in layer l ofan SNN with index i can be given by[Equation]
Revised: Formally, a LIF neuron in layer l of an SNN with index i can be given by

Original: [Equation]
Revised: [Equation]

Original: where u li is the membrane potential, u rest is the membrane resting potential, τ is the membrane timeconstant to describe the membrane potential decaying, I li is the pre-synaptic input, and V th is a givenfiring threshold.
Revised: where u li is the membrane potential, u rest is the membrane resting potential, τ is the membrane time constant to describe the membrane potential decaying, I li is the pre-synaptic input, and V th is a given firing threshold.

Original: From Eq.
Revised: From Eq.

Original: 1, we can see that u li actually acts as a leaky integrator of the pre-synaptic input I li .
Revised: 1, we can see that u li actually acts as a leaky integrator of the pre-synaptic input I li .

Original: And Eq.
Revised: And Eq.

Original: 2 shows that when a neuron membrane potential reaches the firing threshold V th , itemits a spike to transfer its information to other neurons and reset its membrane potential to u rest .
Revised: 2 shows that when a neuron membrane potential reaches the firing threshold V th , it emits a spike to transfer its information to other neurons and reset its membrane potential to u rest .

Original: The above differential expressions described in the continuous domain are difficult to be implementedon mainstream machine learning frameworks ( e.g. , Pytorch, Tensorflow), where the embeddedautomatic differentiation mechanism executes discretely.
Revised: The above differential expressions described in the continuous domain are difficult to be implemented on mainstream machine learning frameworks ( e.g. , Pytorch, Tensorflow), where the embedded automatic differentiation mechanism executes discretely.

Original: Alternatively, the iterative LIF model [25]used the Euler method to solve the first-order differential equation of the LIF model.
Revised: Alternatively, the iterative LIF model [28] used the Euler method to solve the first-order differential equation of the LIF model.

Original: The iterativeexpression of the LIF model can be governed by
Revised: The iterative expression of the LIF model can be governed by

Original: [Equation]
Revised: [Equation]

Original: [Equation] where τ decay is a constant to describe membrane potential decaying, u li ( t ) is the membrane potentialin timestep t , and I li ( t ) is the pre-synaptic input.
Revised: where τ decay is a constant to describe membrane potential decaying, u li ( t ) is the membrane potential in timestep t , and I li ( t ) is the pre-synaptic input.

Original: Since I li ( t ) is the accumulation of spikes from theneurons of layer l − 1 connected to the current neuron, it can be described as I li ( t )
Revised: Since I li ( t ) is the accumulation of spikes from the neurons of layer l − 1 connected to the current neuron, it can be described as I li ( t )

Original: = (cid:80) j w ij o j ( t ) .
Revised: = (cid:80) j w ij o j ( t ) .

Original: Where w ij is the weight of j -th pre-synaptic of the current neuron, o j ( t ) is the output of the neuronfrom the previous layer, which interacts on the current neuron’s j -th pre-synaptic at the moment oft .
Revised: Where w ij is the weight of j -th pre-synaptic of the current neuron, o j ( t ) is the output of the neuron from the previous layer, which interacts on the current neuron’s j -th pre-synaptic at the moment of t .

Original: If we set u reset = 0 , the whole iterative LIF model in both spatial and temporal domain can bedetermined by[Equation]
Revised: If we set u reset = 0 , the whole iterative LIF model in both spatial and temporal domain can be determined by

Original: [Equation]
Revised: [Equation]

Original: The iterative LIF model is friendly to be implemented on general machine learning programmingframeworks.
Revised: The iterative LIF model is friendly to be implemented on general machine learning programming frameworks.

Original: 3.2 Surrogate Gradients of SNNs
Revised: 3.2 Surrogate Gradients of SNNs

Original: The spike activity function shown as Eq.6 can be viewed as a variant of the sign function.
Revised: The spike activity function shown as Eq.6 can be viewed as a variant of the sign function.

Original: Its derivativeis 0 everywhere except for the infinity value at V th .
Revised: Its derivative is 0 everywhere except for the infinity value at V th .

Original: Since the non-differentiability of spike activityfunction, the back-propagation cannot be applied directly for training SNN.
Revised: Since the non-differentiability of spike activity function, the back-propagation cannot be applied directly for training SNN.

Original: To demonstrate thischallenge, we show the expression to compute gradients of weights obtained by the spatial-temporalback-propagation (STBP) as follows,
Revised: To demonstrate this challenge, we show the expression to compute gradients of weights obtained by the spatial-temporal back-propagation (STBP) as follows,

Original: [Equation]
Revised: [Equation]

Original: where y denotes the target output vector, W is the matrix of weights of the SNN, L is the lossfunction, and ∂ y t
Revised: where y denotes the target output vector, W is the matrix of weights of the SNN, L is the loss function, and ∂ y t

Original: ∂ u t is the gradient of the spike activity function.
Revised: ∂ u t is the gradient of the spike activity function.

Original: The weight update is computed as
Revised: The weight update is computed as

Original: [Equation]
Revised: [Equation]

Original: where η is the learning rate.
Revised: where η is the learning rate.

Original: Considering that, ∂ y t ∂ u t is either 0 almost everywhere or infinity at V th ,∆ W is either 0 everywhere or a very large value in a rare case, hence weights would likely not bechanged or updated to a large value some times.
Revised: Considering that, ∂ y t ∂ u t is either 0 almost everywhere or infinity at V th , ∆ W is either 0 everywhere or a very large value in a rare case, hence weights would likely not be changed or updated to a large value some times.

Original: Most prior works adopted surrogate gradients (SG) to overcome the non-differentiability challenge [30; 11; 24; 15; 22].
Revised: Most prior works adopted surrogate gradients (SG) to overcome the non-differentiability challenge [33; 14; 27; 19; 25].

Original: However, the SGs in existing methods are fixed during training, such astrategy would limit the learning capacity of the network.
Revised: However, the SGs in existing methods are fixed during training, such a strategy would limit the learning capacity of the network.

Original: We notice that the SG methods are alsoadopted in Quantization Neural Networks (QNNs) which also suffer from non-differentiability.
Revised: We notice that the SG methods are also adopted in Quantization Neural Networks (QNNs) which also suffer from non-differentiability.

Original: Somechangeable surrogate gradient methods that can achieve better performance by changing SG in thetraining are proposed in QNNs [18; 7].
Revised: Some changeable surrogate gradient methods that can achieve better performance by changing SG in the training are proposed in QNNs [21; 9].

Original: Combining characteristics of SNNs, an evolutionary SG thatcan maintain strong updating ability in the early training phase and approach the accurate gradientsgradually at the end of the training is introduced in the paper.
Revised: Combining characteristics of SNNs, an evolutionary SG that can maintain strong updating ability in the early training phase and approach the accurate gradients gradually at the end of the training is introduced in the paper.

Original:  Methodology
Revised:  Methodology

Original: In this paper, we argue that the challenge of training highly accurate spiking neural networks ismainly limited by the severe information loss caused by the spike activity function.
Revised: In this paper, we argue that the challenge of training highly accurate spiking neural networks is mainly limited by the severe information loss caused by the spike activity function.

Original: To mitigate theproblem, the IM-Loss is proposed.
Revised: To mitigate the problem, the IM-Loss is proposed.

Original: Furthermore, we also introduce the ESG to obtain the appropriatesurrogate gradient for the non-differentiable spike activity in backward propagation.
Revised: Furthermore, we also introduce the ESG to obtain the appropriate surrogate gradient for the non-differentiable spike activity in backward propagation.

Original: 4.1 Information Maximization Loss
Revised: 4.1 Information Maximization Loss

Original: In the forward propagation, the membrane potential to 0 / 1 spikes quantization of the spike activityfunction will cause information loss unquestionably.
Revised: In the forward propagation, the membrane potential to 0 / 1 spikes quantization of the spike activity function will cause information loss unquestionably.

Original: This is one of the main reasons why an SNNmodel cannot achieve as satisfactory performance as its ANN counterpart.
Revised: This is one of the main reasons why an SNN model cannot achieve as satisfactory performance as its ANN counterpart.

Original: Ideally, the spike tensor Oshould reflect the information of the membrane potential tensor U as much as possible.
Revised: Ideally, the spike tensor O should reflect the information of the membrane potential tensor U as much as possible.

Original: In the view ofthe information flow, to maximize the information flow from the full-precision tensor U to the binarytensor O , the mutual information I ( U ; O ) of the random variables U
Revised: In the view of the information flow, to maximize the information flow from the full-precision tensor U to the binary tensor O , the mutual information I ( U ; O ) of the random variables U

Original: and O should be maximized asfollows,
Revised: and O should be maximized as follows,

Original: [Equation]
Revised: [Equation]

Original: where H ( O ) is the entropy of O and H ( O | U ) is the conditional entropy of O given U .
Revised: where H ( O ) is the entropy of O and H ( O | U ) is the conditional entropy of O given U .

Original: It can beproved that, maximizing the mutual information
Revised: It can be proved that, maximizing the mutual information

Original: I ( U ; O ) is equivalent to maximize the informationentropy H ( O ) for SNNs (see Appendix A.1 for detailed proofs) as follows,
Revised: I ( U ; O ) is equivalent to maximize the information entropy H ( O ) for SNNs (see Appendix A.1 for detailed proofs) as follows,

Original: [Equation]
Revised: [Equation]

Original: Moreover, by studying the correlation between the entropy H ( O ) and the random variables U , wenotice that the spike activity function has a fixed firing threshold and decision levels, thus Proposition1 about H ( O ) and the distribution of U can be obtained.

Original: Proposition 1. When the distribution of U satisfies p (0) = p (1) = 0 .
Revised: Obviously, when the distribution of O satisfies p (0) = p (1) = 0 .

Original: 5, the entropy H ( O )ismaximized. The proof of Proposition 1 is shown in Appendix A.2.
Revised: 5 the entropy H ( O ) is maximized, where the p is the marginal probability mass functions of the discrete variables O .

Original: This proposition reveals that to maximize theinformation flow from the full-precision tensor U to the spike tensor O , the spike values should beevenly distributed, i.e. , (cid:80) u<V th p ( u ) =
Revised: This means that to maximize the information flow from the full-precision tensor U to the spike tensor O , the spike values should be evenly distributed, i.e. , (cid:80) u<V th p ( u ) =

Original: (cid:80) u ≥ V th p ( u ) = 0 .
Revised: (cid:80) u ≥ V th p ( u ) = 0 .

Original: We set U ( q ) to denotes the q -quantileof U ’s elements in ascend order where 0 ≤ q ≤ 1 .
Revised: We set U ( q ) to denotes the q -quantile of U ’s elements in ascend order where 0 ≤ q ≤ 1 .

Original: To maximize the H ( O ) , we should have U (0 . 5 − ) <
Revised: To maximize the H ( O ) , we should have U (0 . 5 − ) <

Original: V th and U (0 .
Revised: V th and U (0 .

Original: 5 + ) ≥ V th .
Revised: 5 + ) ≥ V th .

Original: Since the U follows a Gaussian distribution [30], U (0 . 5) is almostequal to the mean value ¯ U .
Revised: Since the U follows a Gaussian distribution [33; 10], U (0 . 5) is almost equal to the mean value ¯ U .

Original: Then ( ¯ U − V th ) 2 can be regarded as a metric of the information-losscaused by the quantization from the full-precision tensor U to the spike tensor O .
Revised: Then ( ¯ U − V th ) 2 can be regarded as a metric of the information-loss caused by the quantization from the full-precision tensor U to the spike tensor O .

Original: Hereby, we definethe proposed IM-Loss as:
Revised: Hereby, we define the proposed IM-Loss as:

Original: [Equation]
Revised: [Equation]

Original: where ¯ U l is the average membrane potential at all timesteps of l -th layer, and L is the total number oflayers.
Revised: where ¯ U l is the average membrane potential at all timesteps of l -th layer, and L is the total number of layers.

Original: Then the overall loss can be calculated as follows,
Revised: Then the overall loss can be calculated as follows,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where L CE is the cross-entropy loss, and λ is a coefficient to balance the classification loss and theinformation loss.
Revised: where L CE is the cross-entropy loss, and λ is a coefficient to balance the classification loss and the information loss.

Original: We set λ as 2 in this paper.
Revised: We set λ as 2 in this paper.

Original: 4.2 The Effect of IM-Loss
Revised: 4.2 The Effect of IM-Loss

Original: SNNs utilize 0 / 1 spikes to communicate between neurons.
Revised: SNNs utilize 0 / 1 spikes to communicate between neurons.

Original: Benefitting from this informationprocessing paradigm, SNNs run efficiently.
Revised: Benefitting from this information processing paradigm, SNNs run efficiently.

Original: However, since the spike rate of the network with randomweights decreases sharply through layers, an SNN is difficult to train.
Revised: However, since the spike rate of the network with random weights decreases sharply through layers, an SNN is difficult to train.

Original: In other words, the spikerate will usually soon become 0 after several layers when training models from scratch, resulting ininformation flow disappearing and difficulty for training large and deep SNNs.
Revised: In other words, the spike rate will usually soon become 0 after several layers when training models from scratch, resulting in information flow disappearing and difficulty for training large and deep SNNs.

Original: For better understanding, the layer-wise average spike rates of spiking CIFARNet and ResNet-19initialized with random weights on the CIFAR10 test set are visualized in Fig.
Revised: For better understanding, the layer-wise average spike rates of spiking CIFARNet and ResNetinitialized with random weights on the CIFAR10 test set are visualized in Fig.

Original: Both networks[Figure] were trained with a timestep of 4 and without normalization.
Revised: Both networks were trained with a timestep of 4 and without normalization.

Original: When epoch is 0, i.e. , the untrainednetworks, spike rates are becoming 0 in the deeper layers, resulting in untrainable networks.
Revised: When epoch is 0, i.e. , the untrained networks, spike rates are becoming 0 in the deeper layers, resulting in untrainable networks.

Original: Thismeans deep SNNs are not easy to train without any special techniques.
Revised: This means deep SNNs are not easy to train without any special techniques.

Original: Some works attempted toalleviate this phenomenon by using normalization techniques.
Revised: Some works attempted to alleviate this phenomenon by using normalization techniques.

Original: However, normalization requires otheradditional operations.
Revised: However, normalization requires other additional operations.

Original: For instance, the bias operation consumes extra computations in the inferencephase when implementing models on some specialized neuromorphic hardware.
Revised: For instance, the bias operation consumes extra computations in the inference phase when implementing models on some specialized neuromorphic hardware.

Original: While for the caseof epoch=1, where the proposed IM-Loss was applied, the spike rate in each layer can be restricted toan appropriate range ( i.e. , about 50% ), which enables the learning algorithm to train a large and deep
Revised: While for the case of epoch=1, where the proposed IM-Loss was applied, the spike rate in each layer can be restricted to an appropriate range ( i.e. , about 50% ), which enables the learning algorithm to train a large and deep

Original: SNN.
Revised: SNN.

Original: By maintaining an appropriate range for the spike rate, IM-Loss can effectively update theweights of the two deep SNNs even with random initialization.
Revised: By maintaining an appropriate range for the spike rate, IM-Loss can effectively update the weights of the two deep SNNs even with random initialization.

Original: In this sense, IM-Loss also possessesthe effect of normalization but without other additional operations or well-designed initializationstrategies.
Revised: In this sense, IM-Loss also possesses the effect of normalization but without other additional operations or well-designed initialization strategies. [Figure] [Figure]

Original: To further demonstrate the information retention ability of IM-Loss, the case of epoch=1000 in Fig.2 shows the spike rate of the SNNs at the end of training.
Revised: To further demonstrate the information retention ability of IM-Loss [Figure] the case of epoch=1000 in Fig. 2 shows the spike rate of the SNNs at the end of training.

Original: For ResNet-19, IM-Loss did not restrictthe layers inside the residual blocks, since these layers tend to learn small perturbations which areincompatible with the restriction by IM-Loss.
Revised: For ResNet-19, IM-Loss did not restrict the layers inside the residual blocks, since these layers tend to learn small perturbations which are incompatible with the restriction by IM-Loss.

Original: It can be seen that the spike rate of each layer restrictedby IM-Loss is near 0.5, meaning the information flow can be maximized by IM-Loss as analyzedabove.
Revised: It can be seen that the spike rate of each layer restricted by IM-Loss is near 0.5, meaning the information flow can be maximized by IM-Loss as analyzed above [Figure]

Original: 4.3 Evolutionary Surrogate Gradients
Revised: 4.3 Evolutionary Surrogate Gradients

Original: The zero-derivative of spike activity function almost everywhere extremely prevents SNNs fromtraining adequately.
Revised: The zero-derivative of spike activity function almost everywhere extremely prevents SNNs from training adequately.

Original: To our best knowledge, nearly all the prior works attempted to alleviate thisproblem by following the principle of replacing the gradient of the spike activity function with afixed SG in SNNs.
Revised: To our best knowledge, nearly all the prior works attempted to alleviate this problem by following the principle of replacing the gradient of the spike activity function with a fixed SG in SNNs.

Original: As aforementioned, changeable surrogate gradient (CSG) methods in QNNs canachieve better performance.
Revised: As aforementioned, changeable surrogate gradient (CSG) methods in QNNs can achieve better performance.

Original: We guess that designing a suitable CSG can also increase accuracy in
Revised: We guess that designing a suitable CSG can also increase accuracy in

Original: SNNs.
Revised: SNNs.

Original: To this end, we first introduce a differentiable asymptotic function to approximate the spikeactivity function as follows,[Equation]
Revised: To this end, we first introduce a differentiable asymptotic function to approximate the spike activity function as follows,

Original: [Equation]
Revised: [Equation]

Original: where the coefficient k determines the shape of the asymptotic function.
Revised: where the coefficient k determines the shape of the asymptotic function.

Original: The gradient of theasymptotic function, φ ( · ) is used to replace the gradient of the spike activity function.
Revised: The gradient of the asymptotic function, φ ( · ) is used to replace the gradient of the spike activity function.

Original: For the sake of intuitive explanation, Fig. 3 illustrates the influences of k on the asymptotic function(left) and its corresponding gradient (right).
Revised: For the sake of intuitive explanation, Fig. 3 illustrates the influences of k on the asymptotic function (left) and its corresponding gradient (right).

Original: It can be seen that, when setting k with a large value, amore accurate gradient in backward can be obtained, so as to increase the difficulty of the training.
Revised: It can be seen that, when setting k with a large value, a more accurate gradient in backward can be obtained, so as to increase the difficulty of the training.

Original: Conversely, when k becomes smaller, the gradient of the asymptotic function will tend to be uniformand keep some relatively large value, thus the strong weight updating ability can be obtained. 
Revised: Conversely, when k becomes smaller, the gradient of the asymptotic function will tend to be uniform and keep some relatively large value, thus the strong weight updating ability can be obtained. [Figure]

Original: Then we design the ev olutionary a symptotic f unction (EvAF) as follows,
Revised: Then we design the ev olutionary a symptotic f unction (EvAF) as follows,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1.7 2.2 x 0. ( x ) k =1 k = k = 1.2 0. 1.7 2.2 x     d ( x ) d x k =1 k = k = where φ ( x ) is the backward approximation substitute for the spike activity function and K ( i ) is thecoefficient that changes dynamically with the training epoch as follows,
Revised: where φ ( x ) is the backward approximation substitute for the spike activity function and K ( i ) is the coefficient that changes dynamically with the training epoch as follows,

Original: [Equation]
Revised: [Equation]

Original:  where K min and K max denotes the lower bound and the upper bound of the value range of K ,respectively, and i is the index of epoch starting from 0 to N − 1 .
Revised: [Equation] [Equation] where K min and K max denotes the lower bound and the upper bound of the value range of K , respectively, and i is the index of epoch starting from 0 to N − 1 .

Original: In practice, we set K min = 10 0and K max = 10 1 .
Revised: In practice, we set K min = 10and K max = 10 1 .

Original: Without using any additional hyper-parameter, K ( i ) will increase with theaccumulation of training epochs.
Revised: Without using any additional hyper-parameter, K ( i ) will increase with the accumulation of training epochs.

Original: Driven by K ( i ) , EvAF can gradually evolve to the spike activityfunction, and a more accurate gradient will be obtained, correspondingly.
Revised: Driven by K ( i ) , EvAF can gradually evolve to the spike activity function, and a more accurate gradient will be obtained, correspondingly.

Original: Such dynamically changinggradient will be used as the SG in our method, namely Evolutionary Surrogate Gradient (ESG).
Revised: Such dynamically changing gradient will be used as the SG in our method, namely Evolutionary Surrogate Gradient (ESG).

Original: Byintroducing the modified asymptotic function, EvAF into the SNNs, both weight updating ability inthe early stage of training and more accurate backward gradient at the end of training can be fulfilledby the proposed ESG.
Revised: By introducing the modified asymptotic function, EvAF into the SNNs, both weight updating ability in the early stage of training and more accurate backward gradient at the end of training can be fulfilled by the proposed ESG.

Original: The rule for designing K ( i ) and its effect is detailed in Appendix A.3.

Original: 4.4 The Effect of ESG
Revised: 4.4 The Effect of ESG

Original: To demonstrate the effect of ESG, we compared the accuracy curves of spiking CIFARNet and
Revised: To demonstrate the effect of ESG, we compared the accuracy curves of spiking CIFARNet and

Original: ResNet-19 (with timestep = 4) on CIFAR10 using the gradient of the asymptotic function withk = 10 0 , k = 10 1 and ESG derived from EvAF with K ∈ [10 0 , 10 1 ] in backward.
Revised: ResNet-19 (with timestep = 4) on CIFAR10 using the gradient of the asymptotic function with k = 10 0 , k = 10 1 and ESG derived from EvAF with K ∈ [10 0 , 10 1 ] in backward.

Original: From thevisualized comparison results shown in Fig.
Revised: From the visualized comparison results shown in Fig.

Original: 5, we can see that the SNNs with a small-valued k havestronger updating ability and are easier to converge compared to the SNNs with a large k .
Revised: 4, we can see that the SNNs with a small-valued k have stronger updating ability and are easier to converge compared to the SNNs with a large k .

Original: The SGs of
Revised: The SGs of

Original: SNNs with a large k are very close to the gradients of the spike activities, in such case, most weightsof these SNNs cannot be updated sufficiently.
Revised: SNNs with a large k are very close to the gradients of the spike activities, in such case, most weights of these SNNs cannot be updated sufficiently.

Original: Hence these SNNs will suffer slow convergence andface severe performance degradation.
Revised: Hence these SNNs will suffer slow convergence and face severe performance degradation.

Original: Our ESG starts with K min at the early stage of training, so thatthe strong updating ability can make the SNN converge easily; then it ends with K max at the end oftraining thus the backward gradient will be more accurate resulting in a better-performed SNN.
Revised: Our ESG starts with K min at the early stage of training, so that the strong updating ability can make the SNN converge easily; then it ends with K max at the end of training thus the backward gradient will be more accurate resulting in a better-performed SNN.

Original: Ingeneral, by using a dynamically changing coefficient, our ESG encourages the SNN with both easierconvergence and higher accuracy.
Revised: In general, by using a dynamically changing coefficient, our ESG encourages the SNN with both easier convergence and higher accuracy.

Original:  Experiments
Revised:  Experiments

Original: In this section, extensive experiments were conducted to demonstrate the effectiveness of the proposedmethods.
Revised: In this section, extensive experiments were conducted to demonstrate the effectiveness of the proposed methods.

Original: The algorithm of the training process of our method is presented in AppendixA.4.
Revised: The algorithm of the training process of our method is presented in Appendix

Original: Itscomplexity analysis is in Appendix A.5.
Revised: A.2.

Original: The experiments include widely-used network structuresincluding spiking CIFARNet [25], ResNet-19 [30], modified VGG-16 [19], and ResNet-34 [8] forboth popular non-spiking static and neuromorphic datasets: CIFAR10/100, ImageNet (ILSVRC12)and CIFAR10-DVS [12].
Revised: The experiments include widely-used network structures including spiking CIFARNet [28], ResNet-19 [33], modified VGG-16 [22], and ResNet-34 [11] for both popular non-spiking static and neuromorphic datasets: CIFAR10/100, ImageNet (ILSVRC12) and CIFAR10-DVS [15].

Original: For being friendly with neuromorphic hardware, the max-pooling layerwas replaced with the average-pooling layer in the used network architectures.
Revised: For being friendly with neuromorphic hardware, the max-pooling layer was replaced with the average-pooling layer in the used network architectures.

Original: The details of datasetdescriptions, data pre-processing, hyper-parameter configurations, and training setups are in Appendix

Original: Appendix A.6.

Original: We set λ as 2.

Original: The ablation study for λ is shown in Appendix A.7.

Original: Besides, we givethe special reason for choosing L 2 -formed loss as information loss in Appendix A.8.

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: 5.1 Ablation Study
Revised: 5.1 Ablation Study

Original: To understand how our method works in practice [Table] the ablation experiments for IM-Loss and ESGwith spiking CIFARNet and ResNet-19 (timestep = 4) were first conducted on CIFAR10.
Revised: To understand how our method works in practice, the ablation experiments for IM-Loss and ESG with spiking CIFARNet and ResNet-19 (timestep = 4) were first conducted on CIFAR10.

Original: To make afair comparison with tdBN, the rectangular surrogate function was used for the model without ESGto compute gradients, as in tdBN.
Revised: To make a fair comparison with tdBN, the rectangular surrogate function was used for the model without ESG to compute gradients, as in tdBN.

Original: The performances of the networks with IM-Loss and ESG aloneand their combinations are shown in Table 1.
Revised: The performances of the networks with IM-Loss and ESG alone and their combinations are shown in Table 1.

Original: For the networks without normalization or IM-Loss,there is a certain probability that they might not be trainable from scratch with inappropriate randominitialization as aforementioned.
Revised: For the networks without normalization or IM-Loss, there is a certain probability that they might not be trainable from scratch with inappropriate random initialization as aforementioned.

Original: Therefore, in this series of experiments, the models trained by
Revised: Therefore, in this series of experiments, the models trained by

Original: IM-Loss after several iterations (training was stopped when the spike rates of the last layer were not0) as the initialization, which can also be regarded as a warm-start for the networks.
Revised: IM-Loss after several iterations (training was stopped when the spike rates of the last layer were not 0) as the initialization, which can also be regarded as a warm-start for the networks.

Original: Results in table 1show that both IM-Loss and ESG can improve the networks’ performances alone.
Revised: Results in tableshow that both IM-Loss and ESG can improve the networks’ performances alone.

Original: And by using thecombination of these two methods, the performances can be further improved.
Revised: And by using the combination of these two methods, the performances can be further improved.

Original: Moreover, our methodis also compatible with other advanced technologies, such as tdBN [30], hence it can be applied in aplug-in manner to enhance the learning capacity of SNNs together with other techniques.
Revised: Moreover, our method is also compatible with other advanced technologies, such as tdBN [33], hence it can be applied in a plug-in manner to enhance the learning capacity of SNNs together with other techniques.

Original: 5.2 Comparisons with State-of-the-art Methods
Revised: 5.2 Comparisons with State-of-the-art Methods

Original: As reported in the ablation study, our algorithm combined with tdBN can achieve better results, hencethis training technique was also adopted in our method.
Revised: As reported in the ablation study, our algorithm combined with tdBN can achieve better results, hence this training technique was also adopted in our method.

Original: The results are listed in table 2.
Revised: The results are listed in table 2.

Original: On CIFAR10, our models achieve better performances than the other state-of-the-art methods (92.20%top-1 accuracy with only 4 timesteps by CIFARNet, 93.85% top-1 accuracy with only 5 timesteps by VGG-16, and 95.49% top-1 accuracy with only 6 timesteps by ResNet-19).
Revised: On CIFAR10, our models achieve better performances than the other state-of-the-art methods (92.20% top-1 accuracy with only 4 timesteps by CIFARNet, 93.85% top-1 accuracy with only 5 timesteps by VGG-16, and 95.49% top-1 accuracy with only 6 timesteps by ResNet-19).

Original: Moreover, with only 2timesteps, our method using ResNet-19 as backbone still outperforms theother ResNet-19-basedmodel, i.e. , STBP-tdBN with 6 timesteps by 0.69% accuracy.
Revised: Moreover, with only 2 timesteps, our method using ResNet-19 as backbone still outperforms the STBP-tdBN with 6 timesteps by 0.69% accuracy.

Original: And our CIFARNet-based model withonly 4 timesteps can also outperform the CIFARNet in [25] with 12 timesteps and the one in [29] [Table] -DVS [Table]
Revised: And our CIFARNet-based model with only 4 timesteps can also outperform the CIFARNet in [28] with 12 timesteps and the one in [32] with 5 timesteps by 1.67% and 0.79% accuracy, respectively.

Original: Streaming

Original: [Table]

Original: STBP-

Original: [Table]

Original: [Table]

Original: Our method

Original: SNN training

Original: [-ResNet72.60% ± 
-]
Revised: [-ResNet72.60% ± 
-]

Original: .with 5 timesteps by 1.67% and 0.79% accuracy, respectively.

Original: These comparison results clearly showthe effective latency reducing the ability of our method.
Revised: These comparison results clearly show the effective latency reducing the ability of our method.

Original: And the same situation can also be seen fromthe results on a more complex dataset, CIFAR100.
Revised: And the same situation can also be seen from the results on a more complex dataset, CIFAR100.

Original: ResNet-34 and VGG-16 were used as backbones on ImageNet.
Revised: ResNet-34 and VGG-16 were used as backbones on ImageNet.

Original: Our ResNet-34 and VGG-16achieve 67.43% and 70.03%
Revised: Our ResNet-34 and VGGachieve 67.43% and 70.03%

Original: top-1 accuracy with only 6 and 5 timesteps, respectively.
Revised: top-1 accuracy with only 6 and 5 timesteps, respectively.

Original: Comparedto STBP-tdBN and Diet-SNN [19], our ResNet-34 and VGG-16 obtain 3.71% and 1.65% accuracyimprovement, respectively. 
Revised: Compared to STBP-tdBN and TET, our ResNet-34 obtains 3.71% and 2.64% accuracy improvement, respectively. Compared to Diet-SNN [22] , our VGG-16 obtains 1.65% accuracy improvement.

Original: These results show the potential of our method  to train more accuratenetworks on a large-scale dataset than the other state-of-the-art methods.
Revised: The Dspike is some better than our method. However, the Dspike is very time-consuming than our method since it is computed by finite difference manner. Evaluating finite difference could be time-consuming, where for each single weight, the model should run two times to evaluate the difference of the loss, and a model can have more than ten million parameters.

Original: To further verify the effectiveness of our methods, the neuromorphic dataset, CIFAR10-DVS, wasalso used to evaluate the proposed method.
Revised: To further verify the effectiveness of our methods, the neuromorphic dataset, CIFAR10-DVS, was also used to evaluate the proposed method.

Original: With ResNet-19 as the backbone, it achieves the bestperformance with 72.60% accuracy in 10 timesteps, which is 4.80% higher than STBP-tdBN.
Revised: With ResNet-19 as the backbone, it achieves the best performance with 72.60% accuracy in 10 timesteps, which is 4.80% higher than STBP [Table]

Original:  Conclusion
Revised:  Conclusion

Original: This work aims at addressing two main problems for training accurate and deep SNNs: (i) theinformation flow loss caused by the 0/1 spike quantization; (ii) the training obstacle caused by thenon-differentiable spike activity function.
Revised: This work aims at addressing two main problems for training accurate and deep SNNs: (i) the information flow loss caused by the 0/1 spike quantization; (ii) the training obstacle caused by the non-differentiable spike activity function.

Original: Inspired by the correlation between membrane potentialsand binary spike signals from the view of information entropy, we design a novel loss, IM-Loss,which can directly maximize the information expressiveness of an SNN, so that a well-performed
Revised: Inspired by the correlation between membrane potentials and binary spike signals from the view of information entropy, we design a novel loss, IM-Loss, which can directly maximize the information expressiveness of an SNN, so that a well-performed

Original: SNN can be obtained.
Revised: SNN can be obtained.

Original: It also plays part of the normalization’s role but does not require additionaloperations or well-designed initialization strategies.
Revised: It also plays part of the normalization’s role but does not require additional operations or well-designed initialization strategies.

Original: Then, by incorporating a dynamically changingcoefficient into a differentiable asymptotic function, we introduce the ESG method for training SNNin a more appropriate way.
Revised: Then, by incorporating a dynamically changing coefficient into a differentiable asymptotic function, we introduce the ESG method for training SNN in a more appropriate way.

Original: ESG encourages the SNN to enjoy both easy convergence and highaccuracy by adjusting the weight updating ability and the accuracy of the gradients.
Revised: ESG encourages the SNN to enjoy both easy convergence and high accuracy by adjusting the weight updating ability and the accuracy of the gradients.

Original: Combining thesetwo methods, our method outperforms the state-of-the-art methods.Checklist
Revised: Combining these two methods, our method outperforms the state-of-the-art methods.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?

Original: [No](c) Did you discuss any potential negative societal impacts of your work?

Original: [No]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] The proof is available

Original: in appendix

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes] Our code willbe available in the supplemental.

Original: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] All implementations are described in the experiments section and

Original: appendix

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We report the mean as well as the standard deviationaccuracy.

Original: (d) Did you include the total amount of compute and the type of resources used (e

Original: g.,type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] The computation resources

Original: description is provided in the appendix.

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a)

Original: If your work uses existing assets, did you cite the creators?

Original: [Yes] (b) Did you mention license of assets? No public datasets.
Revised: Acknowledgment

Original: ( you include assets material No] (d) Did you discuss whether and how consent was obtained from people whose
Revised: This work is supported by grants from the National Natural Science Foundation of China under contracts No.12202412 and No.12202413.

Original: data

Original: you’re

Original: using/

Original: curating?

Original: No](e)

Original: Did you

Original: discuss

Original: whether the data you are using/curating contains personally identifiableinformation or offensive content

Original: No]

Original: If you used crowdsourcing or conducted research with human subjects..

Original: .(a)

Original: Did you include

Original: the full text

Original: of instructions

Original: given to participants and screenshots, if

Original: applicable?

Original: [N/A

Original: ](b) Did you describe

Original: any potential participant risks, with

Original: links

Original: to Institutional

Original: Review

Original: Board (IRB) approvals

Original: , if applicable?

Original: [N/A

Original: ](c) Did you include

Original: the estimated

Original: hourly wage paid

Original: to

Original: participants

Original: and

Original: the

Original: total

Original: amountspent

Original: on participant compensation?

Original: [N/A

Original: A Appendix

Original: Here, we provide the membrane potential distribution of the first layer of the second block in

Original: ResNet20 on CIFAR-10 test sets before and after being trained with IM-Loss.

Original: It can be seen that themembrane potential distribution truely follow the Gaussian-alike distribution with free mean value inour experiments.

Original: [Figure]