Original: IM-Loss: Information Maximization Loss for Spiking Neural Networks
Revised: IM-Loss: Information Maximization Loss for Spiking Neural Networks

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Spiking Neural Network (SNN), recognized as a type of biologically plausiblearchitecture, has recently drawn much research attention.
Revised: Spiking Neural Network (SNN), recognized as a type of biologically plausiblearchitecture, has recently drawn much research attention.

Original: It transmits informationby 0 / 1 spikes.
Revised: It transmits informationby 0 / 1 spikes.

Original: This bio-mimetic mechanism of SNN demonstrates extreme energyefficiency since it avoids any multiplications on neuromorphic hardware.
Revised: This bio-mimetic mechanism of SNN demonstrates extreme energyefficiency since it avoids any multiplications on neuromorphic hardware.

Original: However,the forward-passing 0 / 1 spike quantization will cause information loss and accuracy degradation.
Revised: However,the forward-passing 0 / 1 spike quantization will cause information loss and accuracy degradation.

Original: To deal with this problem, the Information maximization loss(IM-Loss) that aims at maximizing the information flow in the SNN is proposed inthe paper.
Revised: To deal with this problem, the Information maximization loss(IM-Loss) that aims at maximizing the information flow in the SNN is proposed inthe paper.

Original: The IM-Loss not only enhances the information expressiveness of an
Revised: The IM-Loss not only enhances the information expressiveness of an

Original: SNN directly but also plays a part of the role of normalization without introducingany additional operations ( e.g. , bias and scaling) in the inference phase.
Revised: SNN directly but also plays a part of the role of normalization without introducingany additional operations ( e.g. , bias and scaling) in the inference phase.

Original: Additionally, we introduce a novel differentiable spike activity estimation, Evolutionary
Revised: Additionally, we introduce a novel differentiable spike activity estimation, Evolutionary

Original: Surrogate Gradients (ESG) in SNNs.
Revised: Surrogate Gradients (ESG) in SNNs.

Original: By appointing automatic evolvable surrogategradients for spike activity function, ESG can ensure sufficient model updates atthe beginning and accurate gradients at the end of the training, resulting in botheasy convergence and high task performance.
Revised: By appointing automatic evolvable surrogategradients for spike activity function, ESG can ensure sufficient model updates atthe beginning and accurate gradients at the end of the training, resulting in botheasy convergence and high task performance.

Original: Experimental results on both popularnon-spiking static and neuromorphic datasets show that the SNN models trainedby our method outperform the current state-of-the-art algorithms.
Revised: Experimental results on both popularnon-spiking static and neuromorphic datasets show that the SNN models trainedby our method outperform the current state-of-the-art algorithms.

Original:  Introduction
Revised:  Introduction

Original: Recent progress in Artificial Neural Networks (ANNs) has been successfully applied in many machineintelligence fields.
Revised: Recent progress in Artificial Neural Networks (ANNs) has been successfully applied in many machineintelligence fields.

Original: Although ANNs imitate the structure of the brain to a certain extent, their signalprocessing mechanism using operations on real, dense valued tensors, still differs from how the brainworks based on sparse spike events.
Revised: Although ANNs imitate the structure of the brain to a certain extent, their signalprocessing mechanism using operations on real, dense valued tensors, still differs from how the brainworks based on sparse spike events.

Original: Being another sub-category of brain-inspired learning models,the Spiking Neural Networks (SNNs) utilize 0 / 1 binary spike signals to transmit information betweenunits and spatio-temporal dynamics to mimic brain behaviors.
Revised: Being another sub-category of brain-inspired learning models,the Spiking Neural Networks (SNNs) utilize 0 / 1 binary spike signals to transmit information betweenunits and spatio-temporal dynamics to mimic brain behaviors.

Original: Unlike the ANN, where all neurons inthe same layer need to be evaluated before passing information to the next layer, the SNN processesinformation asynchronously and sparsely based on sparse spike events.
Revised: Unlike the ANN, where all neurons inthe same layer need to be evaluated before passing information to the next layer, the SNN processesinformation asynchronously and sparsely based on sparse spike events.

Original: Benefitting from such aninformation processing paradigm, the SNN can efficiently run on specialized neuromorphic hardware( e.g. , Loihi [4]) and has achieved substantial progress in recent years.
Revised: Benefitting from such aninformation processing paradigm, the SNN can efficiently run on specialized neuromorphic hardware( e.g. , Loihi [4]) and has achieved substantial progress in recent years.

Original: Although the SNN is more energy-efficient compared with the ANN, it still has some limitationsto achieve satisfactory task performance.
Revised: Although the SNN is more energy-efficient compared with the ANN, it still has some limitationsto achieve satisfactory task performance.

Original: Due to the usage of spikes for information receiving andtransmitting, the SNN needs to quantize real analog values into 0 / 1 -formed spikes in implementation.
Revised: Due to the usage of spikes for information receiving andtransmitting, the SNN needs to quantize real analog values into 0 / 1 -formed spikes in implementation.

Original: This quantization causes information loss in forward propagation.
Revised: This quantization causes information loss in forward propagation.

Original: There are some remarkable existingworks that alleviate the problem of information loss of SNNs by adjusting the spike rate by learningmore parameters.
Revised: There are some remarkable existingworks that alleviate the problem of information loss of SNNs by adjusting the spike rate by learningmore parameters.

Original: In [19] and [31], a method that aims to learn the correct membrane leak-and-firingthreshold for each layer of the SNN was proposed.
Revised: In [19] and [31], a method that aims to learn the correct membrane leak-and-firingthreshold for each layer of the SNN was proposed.

Original: Results showed that learning the correct thresholdcan help increase the information capacity of the SNN.
Revised: Results showed that learning the correct thresholdcan help increase the information capacity of the SNN.

Original: A wise choice of the firing threshold must besmall enough that sufficient spikes are fired while also being large enough that avoids all neurons
Revised: A wise choice of the firing threshold must besmall enough that sufficient spikes are fired while also being large enough that avoids all neurons

Original: [Figure]
Revised: [Figure]

Original: Post
Revised: Post

Original: [Figure]
Revised: [Figure]

Original: Leaky Integrate-Fire
Revised: Leaky Integrate-Fire

Original: [Figure]
Revised: [Figure]

Original: fired at the same time.
Revised: fired at the same time.

Original: Similarly, by incorporating the learnable membrane time parameter intothe SNN, the parametric neuron [6; 26] can not only increase the diversity of neurons but also theinformation expressiveness of the network.
Revised: Similarly, by incorporating the learnable membrane time parameter intothe SNN, the parametric neuron [6; 26] can not only increase the diversity of neurons but also theinformation expressiveness of the network.

Original: However, all these methods require more parameters andattempt to increase the information expressiveness of the SNN in an indirect manner.
Revised: However, all these methods require more parameters andattempt to increase the information expressiveness of the SNN in an indirect manner.

Original: Meanwhile,the non-differentiable spike activities also prevent the SNN from producing accurate gradients inbackward propagation, which increase the difficulty of the model optimization.
Revised: Meanwhile,the non-differentiable spike activities also prevent the SNN from producing accurate gradients inbackward propagation, which increase the difficulty of the model optimization.

Original: In this work, we propose information maximization (IM) loss to deal with the information loss in
Revised: In this work, we propose information maximization (IM) loss to deal with the information loss in

Original: SNNs.
Revised: SNNs.

Original: More specifically, we first analyze the information expressiveness ability of the SNN from theview of information entropy of the spikes.
Revised: More specifically, we first analyze the information expressiveness ability of the SNN from theview of information entropy of the spikes.

Original: Then a spike distribution loss is introduced for calculatingthe information loss, i.e. , IM-Loss, of the SNN, thus the information expressiveness of the SNN canbe maximized directly by minimizing this information loss, which benefits the accuracy improvementof the SNN.
Revised: Then a spike distribution loss is introduced for calculatingthe information loss, i.e. , IM-Loss, of the SNN, thus the information expressiveness of the SNN canbe maximized directly by minimizing this information loss, which benefits the accuracy improvementof the SNN.

Original: To overcome the non-differentiable spike activities during the model optimization in
Revised: To overcome the non-differentiable spike activities during the model optimization in

Original: SNNs, we also introduce Evolutionary Surrogate Gradients (ESG) in the paper.
Revised: SNNs, we also introduce Evolutionary Surrogate Gradients (ESG) in the paper.

Original: The ESG can evolveautomatically during training and provides a pathway to bridge the gap between the pseudo derivativeand appropriate derivative.
Revised: The ESG can evolveautomatically during training and provides a pathway to bridge the gap between the pseudo derivativeand appropriate derivative.

Original: With evolving, it not only ensures sufficient updating at the beginningbut also accurate gradients calculation at the end of the training, resulting in networks with betterperformance.
Revised: With evolving, it not only ensures sufficient updating at the beginningbut also accurate gradients calculation at the end of the training, resulting in networks with betterperformance.

Original: The overall workflow of our algorithm can be seen in Fig.1.
Revised: The overall workflow of our algorithm can be seen in Fig.1.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We propose the IM-Loss to preserve the information in SNNs.
Revised: • We propose the IM-Loss to preserve the information in SNNs.

Original: It presents a new perspectiveto understand the ability of the SNN to express information.
Revised: It presents a new perspectiveto understand the ability of the SNN to express information.

Original: IM-Loss also plays a part
Revised: IM-Loss also plays a part

Original: of the role of normalization but without introducing any multiply-and-accumulate (MAC)
Revised: of the role of normalization but without introducing any multiply-and-accumulate (MAC)

Original: operations in the inference phase.
Revised: operations in the inference phase.

Original: With the IM-Loss, we can train a more accurate and
Revised: With the IM-Loss, we can train a more accurate and

Original: deeper SNN.
Revised: deeper SNN.

Original: • We also introduce the ESG to effectively approximate the gradients of the spike activityfunction.
Revised: • We also introduce the ESG to effectively approximate the gradients of the spike activityfunction.

Original: The ESG can ensure both sufficient updating of weights at the beginning andaccurate gradients at the end of the training in an evolutionary way so that the SNN wouldbe more accurate and easier to converge.
Revised: The ESG can ensure both sufficient updating of weights at the beginning andaccurate gradients at the end of the training in an evolutionary way so that the SNN wouldbe more accurate and easier to converge.

Original: • Our method is evaluated on both static and dynamic datasets.
Revised: • Our method is evaluated on both static and dynamic datasets.

Original: Extensive experimental resultsshow that our algorithm performs remarkably well across various network structures.
Revised: Extensive experimental resultsshow that our algorithm performs remarkably well across various network structures.

Original: To our
Revised: To our

Original: best knowledge, this is the first work that can directly train deep SNNs from scratch without
Revised: best knowledge, this is the first work that can directly train deep SNNs from scratch without

Original: normalization or warm-start techniques.
Revised: normalization or warm-start techniques.

Original:  Related Work
Revised:  Related Work

Original: In this section, we first briefly review the learning methods of SNNs to provide readers an overallunderstanding.
Revised: In this section, we first briefly review the learning methods of SNNs to provide readers an overallunderstanding.

Original: Then we introduce the information theory used in DNNs which gives us insight todesign our method for handling information loss in SNNs.
Revised: Then we introduce the information theory used in DNNs which gives us insight todesign our method for handling information loss in SNNs.

Original: 2.1 Learning Methods of Spiking Neural Networks
Revised: 2.1 Learning Methods of Spiking Neural Networks

Original: The learning algorithms for SNNs can be divided into three categories: (i) converting ANN to SNN(ANN2SNN)
Revised: The learning algorithms for SNNs can be divided into three categories: (i) converting ANN to SNN(ANN2SNN)

Original: [21]; (ii) unsupervised learning [17]; and (iii) supervised learning [16].
Revised: [21]; (ii) unsupervised learning [17]; and (iii) supervised learning [16].

Original: ANN2SNNconverts a special trained ANN to an SNN that yields the same input-output mapping for a giventask.
Revised: ANN2SNNconverts a special trained ANN to an SNN that yields the same input-output mapping for a giventask.

Original: Since the research of the ANN has been very successful, ANN2SNN can gain competitiveaccuracy.
Revised: Since the research of the ANN has been very successful, ANN2SNN can gain competitiveaccuracy.

Original: Some recent conversion methods have achieved nearly loss-less accuracy with VGG and
Revised: Some recent conversion methods have achieved nearly loss-less accuracy with VGG and

Original: ResNet as original ANN models [13].
Revised: ResNet as original ANN models [13].

Original: However, this kind of method is restricted to rate-codingand can only handle static datasets.
Revised: However, this kind of method is restricted to rate-codingand can only handle static datasets.

Original: Moreover, ANN2SNN requires long timesteps (usually needshundreds of timesteps) to converge and cannot exploit the spatio-temporal features during training forsolving temporal tasks.
Revised: Moreover, ANN2SNN requires long timesteps (usually needshundreds of timesteps) to converge and cannot exploit the spatio-temporal features during training forsolving temporal tasks.

Original: Unsupervised learning is based on biological plausible local learning rulesand is usually considered more biologically plausible than other categories.
Revised: Unsupervised learning is based on biological plausible local learning rulesand is usually considered more biologically plausible than other categories.

Original: However, unsupervisedlearning can only train shallow SNNs and is hard to achieve desirable performance.
Revised: However, unsupervisedlearning can only train shallow SNNs and is hard to achieve desirable performance.

Original: Supervisedlearning uses the derivable approximation to overcome the non-differentiability of the spike activities.
Revised: Supervisedlearning uses the derivable approximation to overcome the non-differentiability of the spike activities.

Original: Hence, the SNN can be optimized with gradient descent algorithms as the ANN and achieves highperformance.
Revised: Hence, the SNN can be optimized with gradient descent algorithms as the ANN and achieves highperformance.

Original: Moreover, Supervised learning can greatly reduce the number of timesteps and handledynamic datasets.
Revised: Moreover, Supervised learning can greatly reduce the number of timesteps and handledynamic datasets.

Original: Thus, supervised learning has increasingly aroused researchers’ great interest inrecent years.
Revised: Thus, supervised learning has increasingly aroused researchers’ great interest inrecent years.

Original: We focus on providing some insights to improve the performance of the supervisedlearning-based SNNs.
Revised: We focus on providing some insights to improve the performance of the supervisedlearning-based SNNs.

Original: 2.2 Information Theory in Deep Neural Networks
Revised: 2.2 Information Theory in Deep Neural Networks

Original: Information theory is widely used in DNNs, ranging from theoretical investigations ( e.g. , the generalization bound induced by mutual information [23]) to practical applications ( e.g. , the variationalinformation bottleneck in representation learning [2]).
Revised: Information theory is widely used in DNNs, ranging from theoretical investigations ( e.g. , the generalization bound induced by mutual information [23]) to practical applications ( e.g. , the variationalinformation bottleneck in representation learning [2]).

Original: It provides a useful way to understand theunderlying behavior of random variables (or processes) and this is a key factor in developing and analyzing deep models.
Revised: It provides a useful way to understand theunderlying behavior of random variables (or processes) and this is a key factor in developing and analyzing deep models.

Original: There are many popular information-theoretic principles that have demonstratedgreat potential in DNNs [28].
Revised: There are many popular information-theoretic principles that have demonstratedgreat potential in DNNs [28].

Original: In [27], deep energy-based models are trained with the f -divergence(a generalization of KL divergence) minimization.
Revised: In [27], deep energy-based models are trained with the f -divergence(a generalization of KL divergence) minimization.

Original: In reinforcement learning, the maximum entropyregularization is adopted to encourage exploration and avoid getting stuck in a local optima [1].
Revised: In reinforcement learning, the maximum entropyregularization is adopted to encourage exploration and avoid getting stuck in a local optima [1].

Original: Mutual information maximization is used in [18] to improve the expressiveness of a model by shiftingthe distribution of weights to zero-mean in every training iteration.
Revised: Mutual information maximization is used in [18] to improve the expressiveness of a model by shiftingthe distribution of weights to zero-mean in every training iteration.

Original: In [3], training DNNs for classification is investigated via minimizing the information bottleneck (IB) functional.
Revised: In [3], training DNNs for classification is investigated via minimizing the information bottleneck (IB) functional.

Original: Despite the greatsuccess of information theory in DNNs, we find no related works in supervised learning of SNNs.
Revised: Despite the greatsuccess of information theory in DNNs, we find no related works in supervised learning of SNNs.

Original: In the paper, we propose the IM-Loss, which is based on the information maximization principle,to improve SNNs.
Revised: In the paper, we propose the IM-Loss, which is based on the information maximization principle,to improve SNNs.

Original: Furthermore, our implementation and idea of the information maximization arevery different from these existing works, which may also provide a new idea to further improve thesemethods.
Revised: Furthermore, our implementation and idea of the information maximization arevery different from these existing works, which may also provide a new idea to further improve thesemethods.

Original:  Preliminary
Revised:  Preliminary

Original: For better expression, we will first introduce the iterative Leaky Integrate-and-Fire (LIF) model andsurrogate gradients of SNNs in detail, which are also adopted in our method.
Revised: For better expression, we will first introduce the iterative Leaky Integrate-and-Fire (LIF) model andsurrogate gradients of SNNs in detail, which are also adopted in our method.

Original: 3.1 Explicitly Iterative LIF Model
Revised: 3.1 Explicitly Iterative LIF Model

Original: The LIF neuron is a commonly used mathematical model to describe the behavior of neural activities,which includes spike firing and update of membrane potential.
Revised: The LIF neuron is a commonly used mathematical model to describe the behavior of neural activities,which includes spike firing and update of membrane potential.

Original: Formally, a LIF neuron in layer l ofan SNN with index i can be given by
Revised: Formally, a LIF neuron in layer l ofan SNN with index i can be given by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where u li is the membrane potential, u rest is the membrane resting potential, τ is the membrane timeconstant to describe the membrane potential decaying, I li is the pre-synaptic input, and V th is a givenfiring threshold.
Revised: where u li is the membrane potential, u rest is the membrane resting potential, τ is the membrane timeconstant to describe the membrane potential decaying, I li is the pre-synaptic input, and V th is a givenfiring threshold.

Original: From Eq.
Revised: From Eq.

Original: 1, we can see that u li actually acts as a leaky integrator of the pre-synaptic input I li .
Revised: 1, we can see that u li actually acts as a leaky integrator of the pre-synaptic input I li .

Original: And Eq.
Revised: And Eq.

Original: 2 shows that when a neuron membrane potential reaches the firing threshold V th , itemits a spike to transfer its information to other neurons and reset its membrane potential to u rest .
Revised: 2 shows that when a neuron membrane potential reaches the firing threshold V th , itemits a spike to transfer its information to other neurons and reset its membrane potential to u rest .

Original: The above differential expressions described in the continuous domain are difficult to be implementedon mainstream machine learning frameworks ( e.g. , Pytorch, Tensorflow), where the embeddedautomatic differentiation mechanism executes discretely.
Revised: The above differential expressions described in the continuous domain are difficult to be implementedon mainstream machine learning frameworks ( e.g. , Pytorch, Tensorflow), where the embeddedautomatic differentiation mechanism executes discretely.

Original: Alternatively, the iterative LIF model [25]used the Euler method to solve the first-order differential equation of the LIF model.
Revised: Alternatively, the iterative LIF model [25]used the Euler method to solve the first-order differential equation of the LIF model.

Original: The iterativeexpression of the LIF model can be governed by
Revised: The iterativeexpression of the LIF model can be governed by

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where τ decay is a constant to describe membrane potential decaying, u li ( t ) is the membrane potentialin timestep t , and I li ( t ) is the pre-synaptic input.
Revised: where τ decay is a constant to describe membrane potential decaying, u li ( t ) is the membrane potentialin timestep t , and I li ( t ) is the pre-synaptic input.

Original: Since I li ( t ) is the accumulation of spikes from theneurons of layer l − 1 connected to the current neuron, it can be described as I li ( t )
Revised: Since I li ( t ) is the accumulation of spikes from theneurons of layer l − 1 connected to the current neuron, it can be described as I li ( t )

Original: = (cid:80) j w ij o j ( t ) .
Revised: = (cid:80) j w ij o j ( t ) .

Original: Where w ij is the weight of j -th pre-synaptic of the current neuron, o j ( t ) is the output of the neuronfrom the previous layer, which interacts on the current neuron’s j -th pre-synaptic at the moment oft .
Revised: Where w ij is the weight of j -th pre-synaptic of the current neuron, o j ( t ) is the output of the neuronfrom the previous layer, which interacts on the current neuron’s j -th pre-synaptic at the moment oft .

Original: If we set u reset = 0 , the whole iterative LIF model in both spatial and temporal domain can bedetermined by
Revised: If we set u reset = 0 , the whole iterative LIF model in both spatial and temporal domain can bedetermined by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The iterative LIF model is friendly to be implemented on general machine learning programmingframeworks.
Revised: The iterative LIF model is friendly to be implemented on general machine learning programmingframeworks.

Original: 3.2 Surrogate Gradients of SNNs
Revised: 3.2 Surrogate Gradients of SNNs

Original: The spike activity function shown as Eq.6 can be viewed as a variant of the sign function.
Revised: The spike activity function shown as Eq.6 can be viewed as a variant of the sign function.

Original: Its derivativeis 0 everywhere except for the infinity value at V th .
Revised: Its derivativeis 0 everywhere except for the infinity value at V th .

Original: Since the non-differentiability of spike activityfunction, the back-propagation cannot be applied directly for training SNN.
Revised: Since the non-differentiability of spike activityfunction, the back-propagation cannot be applied directly for training SNN.

Original: To demonstrate thischallenge, we show the expression to compute gradients of weights obtained by the spatial-temporalback-propagation (STBP) as follows,
Revised: To demonstrate thischallenge, we show the expression to compute gradients of weights obtained by the spatial-temporalback-propagation (STBP) as follows,

Original: [Equation]
Revised: [Equation]

Original: where y denotes the target output vector, W is the matrix of weights of the SNN, L is the lossfunction, and ∂ y t
Revised: where y denotes the target output vector, W is the matrix of weights of the SNN, L is the lossfunction, and ∂ y t

Original: ∂
Revised: ∂

Original: u t is the gradient of the spike activity function.
Revised: u t is the gradient of the spike activity function.

Original: The weight update is computed as
Revised: The weight update is computed as

Original: [Equation]
Revised: [Equation]

Original: where η is the learning rate.
Revised: where η is the learning rate.

Original: Considering that, ∂ y t ∂ u t is either 0 almost everywhere or infinity at V th ,∆ W is either 0 everywhere or a very large value in a rare case, hence weights would likely not bechanged or updated to a large value some times.
Revised: Considering that, ∂ y t ∂ u t is either 0 almost everywhere or infinity at V th ,∆ W is either 0 everywhere or a very large value in a rare case, hence weights would likely not bechanged or updated to a large value some times.

Original: Most prior works adopted surrogate gradients (SG) to overcome the non-differentiability challenge [30; 11; 24; 15; 22].
Revised: Most prior works adopted surrogate gradients (SG) to overcome the non-differentiability challenge [30; 11; 24; 15; 22].

Original: However, the SGs in existing methods are fixed during training, such astrategy would limit the learning capacity of the network.
Revised: However, the SGs in existing methods are fixed during training, such astrategy would limit the learning capacity of the network.

Original: We notice that the SG methods are alsoadopted in Quantization Neural Networks (QNNs) which also suffer from non-differentiability.
Revised: We notice that the SG methods are alsoadopted in Quantization Neural Networks (QNNs) which also suffer from non-differentiability.

Original: Somechangeable surrogate gradient methods that can achieve better performance by changing SG in thetraining are proposed in QNNs [18; 7].
Revised: Somechangeable surrogate gradient methods that can achieve better performance by changing SG in thetraining are proposed in QNNs [18; 7].

Original: Combining characteristics of SNNs, an evolutionary SG thatcan maintain strong updating ability in the early training phase and approach the accurate gradientsgradually at the end of the training is introduced in the paper.
Revised: Combining characteristics of SNNs, an evolutionary SG thatcan maintain strong updating ability in the early training phase and approach the accurate gradientsgradually at the end of the training is introduced in the paper.

Original:  Methodology
Revised:  Methodology

Original: In this paper, we argue that the challenge of training highly accurate spiking neural networks ismainly limited by the severe information loss caused by the spike activity function.
Revised: In this paper, we argue that the challenge of training highly accurate spiking neural networks ismainly limited by the severe information loss caused by the spike activity function.

Original: To mitigate theproblem, the IM-Loss is proposed.
Revised: To mitigate theproblem, the IM-Loss is proposed.

Original: Furthermore, we also introduce the ESG to obtain the appropriatesurrogate gradient for the non-differentiable spike activity in backward propagation.
Revised: Furthermore, we also introduce the ESG to obtain the appropriatesurrogate gradient for the non-differentiable spike activity in backward propagation.

Original: 4.1 Information Maximization Loss
Revised: 4.1 Information Maximization Loss

Original: In the forward propagation, the membrane potential to 0 / 1 spikes quantization of the spike activityfunction will cause information loss unquestionably.
Revised: In the forward propagation, the membrane potential to 0 / 1 spikes quantization of the spike activityfunction will cause information loss unquestionably.

Original: This is one of the main reasons why an SNNmodel cannot achieve as satisfactory performance as its ANN counterpart.
Revised: This is one of the main reasons why an SNNmodel cannot achieve as satisfactory performance as its ANN counterpart.

Original: Ideally, the spike tensor Oshould reflect the information of the membrane potential tensor U as much as possible.
Revised: Ideally, the spike tensor Oshould reflect the information of the membrane potential tensor U as much as possible.

Original: In the view ofthe information flow, to maximize the information flow from the full-precision tensor U to the binarytensor O , the mutual information I ( U ; O ) of the random variables U
Revised: In the view ofthe information flow, to maximize the information flow from the full-precision tensor U to the binarytensor O , the mutual information I ( U ; O ) of the random variables U

Original: and O should be maximized asfollows,
Revised: and O should be maximized asfollows,

Original: [Equation]
Revised: [Equation]

Original: where H ( O ) is the entropy of O and H ( O | U ) is the conditional entropy of O given U .
Revised: where H ( O ) is the entropy of O and H ( O | U ) is the conditional entropy of O given U .

Original: It can beproved that, maximizing the mutual information
Revised: It can beproved that, maximizing the mutual information

Original: I ( U ; O ) is equivalent to maximize the informationentropy H ( O ) for SNNs (see Appendix A.1 for detailed proofs) as follows,
Revised: I ( U ; O ) is equivalent to maximize the informationentropy H ( O ) for SNNs (see Appendix A.1 for detailed proofs) as follows,

Original: [Equation]
Revised: [Equation]

Original: Moreover, by studying the correlation between the entropy H ( O ) and the random variables U , wenotice that the spike activity function has a fixed firing threshold and decision levels, thus Proposition1 about H ( O ) and the distribution of U can be obtained.
Revised: Moreover, by studying the correlation between the entropy H ( O ) and the random variables U , wenotice that the spike activity function has a fixed firing threshold and decision levels, thus Proposition1 about H ( O ) and the distribution of U can be obtained.

Original: Proposition 1.
Revised: Proposition 1.

Original: When the distribution of U satisfies p (0) = p (1) = 0 .
Revised: When the distribution of U satisfies p (0) = p (1) = 0 .

Original: 5 , the entropy H ( O ) ismaximized.
Revised: 5 , the entropy H ( O ) ismaximized.

Original: The proof of Proposition 1 is shown in Appendix A.2.
Revised: The proof of Proposition 1 is shown in Appendix A.2.

Original: This proposition reveals that to maximize theinformation flow from the full-precision tensor U to the spike tensor O , the spike values should beevenly distributed, i.e. , (cid:80) u<V th p ( u ) =
Revised: This proposition reveals that to maximize theinformation flow from the full-precision tensor U to the spike tensor O , the spike values should beevenly distributed, i.e. , (cid:80) u<V th p ( u ) =

Original: (cid:80) u ≥ V th p ( u ) = 0 .
Revised: (cid:80) u ≥ V th p ( u ) = 0 .

Original: We set U ( q ) to denotes the q -quantileof U ’s elements in ascend order where 0 ≤ q ≤ 1 .
Revised: We set U ( q ) to denotes the q -quantileof U ’s elements in ascend order where 0 ≤ q ≤ 1 .

Original: To maximize the H ( O ) , we should have
Revised: To maximize the H ( O ) , we should have

Original: U (0 . 5 − ) <
Revised: U (0 . 5 − ) <

Original: V th and U (0 .
Revised: V th and U (0 .

Original: 5 + ) ≥ V th .
Revised: 5 + ) ≥ V th .

Original: Since the U follows a Gaussian distribution [30], U (0 . 5) is almostequal to the mean value ¯ U .
Revised: Since the U follows a Gaussian distribution [30], U (0 . 5) is almostequal to the mean value ¯ U .

Original: Then ( ¯ U − V th ) 2 can be regarded as a metric of the information-losscaused by the quantization from the full-precision tensor U to the spike tensor O .
Revised: Then ( ¯ U − V th ) 2 can be regarded as a metric of the information-losscaused by the quantization from the full-precision tensor U to the spike tensor O .

Original: Hereby, we definethe proposed IM-Loss as:
Revised: Hereby, we definethe proposed IM-Loss as:

Original: [Equation]
Revised: [Equation]

Original: where ¯ U l is the average membrane potential at all timesteps of l -th layer, and L is the total number oflayers.
Revised: where ¯ U l is the average membrane potential at all timesteps of l -th layer, and L is the total number oflayers.

Original: Then the overall loss can be calculated as follows,
Revised: Then the overall loss can be calculated as follows,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where L CE is the cross-entropy loss, and λ is a coefficient to balance the classification loss and theinformation loss.
Revised: where L CE is the cross-entropy loss, and λ is a coefficient to balance the classification loss and theinformation loss.

Original: We set λ as 2 in this paper.
Revised: We set λ as 2 in this paper.

Original: 4.2 The Effect of IM-Loss
Revised: 4.2 The Effect of IM-Loss

Original: SNNs utilize 0 / 1 spikes to communicate between neurons.
Revised: SNNs utilize 0 / 1 spikes to communicate between neurons.

Original: Benefitting from this informationprocessing paradigm, SNNs run efficiently.
Revised: Benefitting from this informationprocessing paradigm, SNNs run efficiently.

Original: However, since the spike rate of the network with randomweights decreases sharply through layers, an SNN is difficult to train.
Revised: However, since the spike rate of the network with randomweights decreases sharply through layers, an SNN is difficult to train.

Original: In other words, the spikerate will usually soon become 0 after several layers when training models from scratch, resulting ininformation flow disappearing and difficulty for training large and deep SNNs.
Revised: In other words, the spikerate will usually soon become 0 after several layers when training models from scratch, resulting ininformation flow disappearing and difficulty for training large and deep SNNs.

Original: For better understanding, the layer-wise average spike rates of spiking CIFARNet and ResNet-19initialized with random weights on the CIFAR10 test set are visualized in Fig.
Revised: For better understanding, the layer-wise average spike rates of spiking CIFARNet and ResNet-19initialized with random weights on the CIFAR10 test set are visualized in Fig.

Original: Both networks
Revised: Both networks

Original: [Figure]
Revised: [Figure]

Original: were trained with a timestep of 4 and without normalization.
Revised: were trained with a timestep of 4 and without normalization.

Original: When epoch is 0, i.e. , the untrainednetworks, spike rates are becoming 0 in the deeper layers, resulting in untrainable networks.
Revised: When epoch is 0, i.e. , the untrainednetworks, spike rates are becoming 0 in the deeper layers, resulting in untrainable networks.

Original: Thismeans deep SNNs are not easy to train without any special techniques.
Revised: Thismeans deep SNNs are not easy to train without any special techniques.

Original: Some works attempted toalleviate this phenomenon by using normalization techniques.
Revised: Some works attempted toalleviate this phenomenon by using normalization techniques.

Original: However, normalization requires otheradditional operations.
Revised: However, normalization requires otheradditional operations.

Original: For instance, the bias operation consumes extra computations in the inferencephase when implementing models on some specialized neuromorphic hardware.
Revised: For instance, the bias operation consumes extra computations in the inferencephase when implementing models on some specialized neuromorphic hardware.

Original: While for the caseof epoch=1, where the proposed IM-Loss was applied, the spike rate in each layer can be restricted toan appropriate range ( i.e. , about 50% ), which enables the learning algorithm to train a large and deep
Revised: While for the caseof epoch=1, where the proposed IM-Loss was applied, the spike rate in each layer can be restricted toan appropriate range ( i.e. , about 50% ), which enables the learning algorithm to train a large and deep

Original: SNN.
Revised: SNN.

Original: By maintaining an appropriate range for the spike rate, IM-Loss can effectively update theweights of the two deep SNNs even with random initialization.
Revised: By maintaining an appropriate range for the spike rate, IM-Loss can effectively update theweights of the two deep SNNs even with random initialization.

Original: In this sense, IM-Loss also possessesthe effect of normalization but without other additional operations or well-designed initializationstrategies.
Revised: In this sense, IM-Loss also possessesthe effect of normalization but without other additional operations or well-designed initializationstrategies.

Original: To further demonstrate the information retention ability of IM-Loss, the case of epoch=1000 in Fig.2 shows the spike rate of the SNNs at the end of training.
Revised: To further demonstrate the information retention ability of IM-Loss, the case of epoch=1000 in Fig.2 shows the spike rate of the SNNs at the end of training.

Original: For ResNet-19, IM-Loss did not restrictthe layers inside the residual blocks, since these layers tend to learn small perturbations which areincompatible with the restriction by IM-Loss.
Revised: For ResNet-19, IM-Loss did not restrictthe layers inside the residual blocks, since these layers tend to learn small perturbations which areincompatible with the restriction by IM-Loss.

Original: It can be seen that the spike rate of each layer restrictedby IM-Loss is near 0.5, meaning the information flow can be maximized by IM-Loss as analyzedabove.
Revised: It can be seen that the spike rate of each layer restrictedby IM-Loss is near 0.5, meaning the information flow can be maximized by IM-Loss as analyzedabove.

Original: 4.3 Evolutionary Surrogate Gradients
Revised: 4.3 Evolutionary Surrogate Gradients

Original: The zero-derivative of spike activity function almost everywhere extremely prevents SNNs fromtraining adequately.
Revised: The zero-derivative of spike activity function almost everywhere extremely prevents SNNs fromtraining adequately.

Original: To our best knowledge, nearly all the prior works attempted to alleviate thisproblem by following the principle of replacing the gradient of the spike activity function with afixed SG in SNNs.
Revised: To our best knowledge, nearly all the prior works attempted to alleviate thisproblem by following the principle of replacing the gradient of the spike activity function with afixed SG in SNNs.

Original: As aforementioned, changeable surrogate gradient (CSG) methods in QNNs canachieve better performance.
Revised: As aforementioned, changeable surrogate gradient (CSG) methods in QNNs canachieve better performance.

Original: We guess that designing a suitable CSG can also increase accuracy in
Revised: We guess that designing a suitable CSG can also increase accuracy in

Original: SNNs.
Revised: SNNs.

Original: To this end, we first introduce a differentiable asymptotic function to approximate the spikeactivity function as follows,
Revised: To this end, we first introduce a differentiable asymptotic function to approximate the spikeactivity function as follows,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the coefficient k determines the shape of the asymptotic function.
Revised: where the coefficient k determines the shape of the asymptotic function.

Original: The gradient of theasymptotic function, φ ( · ) is used to replace the gradient of the spike activity function.
Revised: The gradient of theasymptotic function, φ ( · ) is used to replace the gradient of the spike activity function.

Original: For the sake of intuitive explanation, Fig. 3 illustrates the influences of k on the asymptotic function(left) and its corresponding gradient (right).
Revised: For the sake of intuitive explanation, Fig. 3 illustrates the influences of k on the asymptotic function(left) and its corresponding gradient (right).

Original: It can be seen that, when setting k with a large value, amore accurate gradient in backward can be obtained, so as to increase the difficulty of the training.
Revised: It can be seen that, when setting k with a large value, amore accurate gradient in backward can be obtained, so as to increase the difficulty of the training.

Original: Conversely, when k becomes smaller, the gradient of the asymptotic function will tend to be uniformand keep some relatively large value, thus the strong weight updating ability can be obtained.
Revised: Conversely, when k becomes smaller, the gradient of the asymptotic function will tend to be uniformand keep some relatively large value, thus the strong weight updating ability can be obtained.

Original: Then we design the ev olutionary a symptotic f unction (EvAF) as follows,
Revised: Then we design the ev olutionary a symptotic f unction (EvAF) as follows,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1.7 2.2 x 0.
Revised: 1.7 2.2 x 0.

Original: ( x ) k =1 k = k = 1.2 0.
Revised: ( x ) k =1 k = k = 1.2 0.

Original: 1.7 2.2 x     d ( x ) d x k =1 k = k
Revised: 1.7 2.2 x     d ( x ) d x k =1 k = k

Original: = where φ ( x ) is the backward approximation substitute for the spike activity function and K ( i ) is thecoefficient that changes dynamically with the training epoch as follows,
Revised: = where φ ( x ) is the backward approximation substitute for the spike activity function and K ( i ) is thecoefficient that changes dynamically with the training epoch as follows,

Original: [Equation]
Revised: [Equation]

Original: where K min and K max denotes the lower bound and the upper bound of the value range of K ,respectively, and i is the index of epoch starting from 0 to N − 1 .
Revised: where K min and K max denotes the lower bound and the upper bound of the value range of K ,respectively, and i is the index of epoch starting from 0 to N − 1 .

Original: In practice, we set K min = 10 0and K max = 10 1 .
Revised: In practice, we set K min = 10 0and K max = 10 1 .

Original: Without using any additional hyper-parameter, K ( i ) will increase with theaccumulation of training epochs.
Revised: Without using any additional hyper-parameter, K ( i ) will increase with theaccumulation of training epochs.

Original: Driven by K ( i ) , EvAF can gradually evolve to the spike activityfunction, and a more accurate gradient will be obtained, correspondingly.
Revised: Driven by K ( i ) , EvAF can gradually evolve to the spike activityfunction, and a more accurate gradient will be obtained, correspondingly.

Original: Such dynamically changinggradient will be used as the SG in our method, namely Evolutionary Surrogate Gradient (ESG).
Revised: Such dynamically changinggradient will be used as the SG in our method, namely Evolutionary Surrogate Gradient (ESG).

Original: Byintroducing the modified asymptotic function, EvAF into the SNNs, both weight updating ability inthe early stage of training and more accurate backward gradient at the end of training can be fulfilledby the proposed ESG.
Revised: Byintroducing the modified asymptotic function, EvAF into the SNNs, both weight updating ability inthe early stage of training and more accurate backward gradient at the end of training can be fulfilledby the proposed ESG.

Original: The rule for designing K ( i ) and its effect is detailed in Appendix A.3.
Revised: The rule for designing K ( i ) and its effect is detailed in Appendix A.3.

Original: 4.4 The Effect of ESG
Revised: 4.4 The Effect of ESG

Original: To demonstrate the effect of ESG, we compared the accuracy curves of spiking CIFARNet and
Revised: To demonstrate the effect of ESG, we compared the accuracy curves of spiking CIFARNet and

Original: ResNet-19 (with timestep = 4) on CIFAR10 using the gradient of the asymptotic function withk = 10 0 , k = 10 1 and ESG derived from EvAF with K ∈ [10 0 , 10 1 ] in backward.
Revised: ResNet-19 (with timestep = 4) on CIFAR10 using the gradient of the asymptotic function withk = 10 0 , k = 10 1 and ESG derived from EvAF with K ∈ [10 0 , 10 1 ] in backward.

Original: From thevisualized comparison results shown in Fig.
Revised: From thevisualized comparison results shown in Fig.

Original: 4, we can see that the SNNs with a small-valued k havestronger updating ability and are easier to converge compared to the SNNs with a large k .
Revised: 5, we can see that the SNNs with a small-valued k havestronger updating ability and are easier to converge compared to the SNNs with a large k .

Original: The SGs of
Revised: The SGs of

Original: SNNs with a large k are very close to the gradients of the spike activities, in such case, most weightsof these SNNs cannot be updated sufficiently.
Revised: SNNs with a large k are very close to the gradients of the spike activities, in such case, most weightsof these SNNs cannot be updated sufficiently.

Original: Hence these SNNs will suffer slow convergence andface severe performance degradation.
Revised: Hence these SNNs will suffer slow convergence andface severe performance degradation.

Original: Our ESG starts with K min at the early stage of training, so thatthe strong updating ability can make the SNN converge easily; then it ends with K max at the end oftraining thus the backward gradient will be more accurate resulting in a better-performed SNN.
Revised: Our ESG starts with K min at the early stage of training, so thatthe strong updating ability can make the SNN converge easily; then it ends with K max at the end oftraining thus the backward gradient will be more accurate resulting in a better-performed SNN.

Original: Ingeneral, by using a dynamically changing coefficient, our ESG encourages the SNN with both easierconvergence and higher accuracy.
Revised: Ingeneral, by using a dynamically changing coefficient, our ESG encourages the SNN with both easierconvergence and higher accuracy.

Original:  Experiments
Revised:  Experiments

Original: In this section, extensive experiments were conducted to demonstrate the effectiveness of the proposedmethods.
Revised: In this section, extensive experiments were conducted to demonstrate the effectiveness of the proposedmethods.

Original: The algorithm of the training process of our method is presented in Appendix A.4.
Revised: The algorithm of the training process of our method is presented in Appendix A.4.

Original: Itscomplexity analysis is in Appendix A.5.
Revised: Itscomplexity analysis is in Appendix A.5.

Original: The experiments include widely-used network structuresincluding spiking CIFARNet [25], ResNet-19 [30], modified VGG-16 [19], and ResNet-34 [8] forboth popular non-spiking static and neuromorphic datasets: CIFAR10/100, ImageNet (ILSVRC12)and CIFAR10-DVS [12].
Revised: The experiments include widely-used network structuresincluding spiking CIFARNet [25], ResNet-19 [30], modified VGG-16 [19], and ResNet-34 [8] forboth popular non-spiking static and neuromorphic datasets: CIFAR10/100, ImageNet (ILSVRC12)and CIFAR10-DVS [12].

Original: For being friendly with neuromorphic hardware, the max-pooling layerwas replaced with the average-pooling layer in the used network architectures.
Revised: For being friendly with neuromorphic hardware, the max-pooling layerwas replaced with the average-pooling layer in the used network architectures.

Original: The details of datasetdescriptions, data pre-processing, hyper-parameter configurations, and training setups are in Appendix
Revised: The details of datasetdescriptions, data pre-processing, hyper-parameter configurations, and training setups are in Appendix

Original: Appendix A.6.
Revised: Appendix A.6.

Original: We set λ as 2.
Revised: We set λ as 2.

Original: The ablation study for λ is shown in Appendix A.7.
Revised: The ablation study for λ is shown in Appendix A.7.

Original: Besides, we givethe special reason for choosing L 2 -formed loss as information loss in Appendix A.8.
Revised: Besides, we givethe special reason for choosing L 2 -formed loss as information loss in Appendix A.8.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: 5.1 Ablation Study
Revised: 5.1 Ablation Study

Original: To understand how our method works in practice
Revised: To understand how our method works in practice

Original: [Table]
Revised: [Table]

Original: the ablation experiments for IM-Loss and ESGwith spiking CIFARNet and ResNet-19 (timestep = 4) were first conducted on CIFAR10.
Revised: the ablation experiments for IM-Loss and ESGwith spiking CIFARNet and ResNet-19 (timestep = 4) were first conducted on CIFAR10.

Original: To make afair comparison with tdBN, the rectangular surrogate function was used for the model without ESGto compute gradients, as in tdBN.
Revised: To make afair comparison with tdBN, the rectangular surrogate function was used for the model without ESGto compute gradients, as in tdBN.

Original: The performances of the networks with IM-Loss and ESG aloneand their combinations are shown in Table 1.
Revised: The performances of the networks with IM-Loss and ESG aloneand their combinations are shown in Table 1.

Original: For the networks without normalization or IM-Loss,there is a certain probability that they might not be trainable from scratch with inappropriate randominitialization as aforementioned.
Revised: For the networks without normalization or IM-Loss,there is a certain probability that they might not be trainable from scratch with inappropriate randominitialization as aforementioned.

Original: Therefore, in this series of experiments, the models trained by
Revised: Therefore, in this series of experiments, the models trained by

Original: IM-Loss after several iterations (training was stopped when the spike rates of the last layer were not0) as the initialization, which can also be regarded as a warm-start for the networks.
Revised: IM-Loss after several iterations (training was stopped when the spike rates of the last layer were not0) as the initialization, which can also be regarded as a warm-start for the networks.

Original: Results in table 1show that both IM-Loss and ESG can improve the networks’ performances alone.
Revised: Results in table 1show that both IM-Loss and ESG can improve the networks’ performances alone.

Original: And by using thecombination of these two methods, the performances can be further improved.
Revised: And by using thecombination of these two methods, the performances can be further improved.

Original: Moreover, our methodis also compatible with other advanced technologies, such as tdBN [30], hence it can be applied in aplug-in manner to enhance the learning capacity of SNNs together with other techniques.
Revised: Moreover, our methodis also compatible with other advanced technologies, such as tdBN [30], hence it can be applied in aplug-in manner to enhance the learning capacity of SNNs together with other techniques.

Original: 5.2 Comparisons with State-of-the-art Methods
Revised: 5.2 Comparisons with State-of-the-art Methods

Original: As reported in the ablation study, our algorithm combined with tdBN can achieve better results, hencethis training technique was also adopted in our method.
Revised: As reported in the ablation study, our algorithm combined with tdBN can achieve better results, hencethis training technique was also adopted in our method.

Original: The results are listed in table 2.
Revised: The results are listed in table 2.

Original: On CIFAR10, our models achieve better performances than the other state-of-the-art methods (92.20%top-1 accuracy with only 4 timesteps by CIFARNet, 93.85% top-1 accuracy with only 5 timesteps by
Revised: On CIFAR10, our models achieve better performances than the other state-of-the-art methods (92.20%top-1 accuracy with only 4 timesteps by CIFARNet, 93.85% top-1 accuracy with only 5 timesteps by

Original: VGG-16, and 95.49% top-1 accuracy with only 6 timesteps by ResNet-19).
Revised: VGG-16, and 95.49% top-1 accuracy with only 6 timesteps by ResNet-19).

Original: Moreover, with only 2timesteps, our method using ResNet-19 as backbone still outperforms the other ResNet-19-basedmodel, i.e. , STBP-tdBN with 6 timesteps by 0.69% accuracy.
Revised: Moreover, with only 2timesteps, our method using ResNet-19 as backbone still outperforms the other ResNet-19-basedmodel, i.e. , STBP-tdBN with 6 timesteps by 0.69% accuracy.

Original: And our CIFARNet-based model withonly 4 timesteps can also outperform the CIFARNet in [25] with 12 timesteps and the one in [29]
Revised: And our CIFARNet-based model withonly 4 timesteps can also outperform the CIFARNet in [25] with 12 timesteps and the one in [29]

Original: [Table]
Revised: [Table]

Original: -DVS
Revised: -DVS

Original: [Table]
Revised: [Table]

Original: Streaming
Revised: Streaming

Original: [Table]
Revised: [Table]

Original: STBP-
Revised: STBP-

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Our method
Revised: Our method

Original: SNN training
Revised: SNN training

Original: ResNet72.60% ± 

Revised: ResNet72.60% ±

Original: .with 5 timesteps by 1.67% and 0.79% accuracy, respectively.
Revised: .with 5 timesteps by 1.67% and 0.79% accuracy, respectively.

Original: These comparison results clearly showthe effective latency reducing the ability of our method.
Revised: These comparison results clearly showthe effective latency reducing the ability of our method.

Original: And the same situation can also be seen fromthe results on a more complex dataset, CIFAR100.
Revised: And the same situation can also be seen fromthe results on a more complex dataset, CIFAR100.

Original: ResNet-34 and VGG-16 were used as backbones on ImageNet.
Revised: ResNet-34 and VGG-16 were used as backbones on ImageNet.

Original: Our ResNet-34 and VGG-16achieve 67.43% and 70.03%
Revised: Our ResNet-34 and VGG-16achieve 67.43% and 70.03%

Original: top-1 accuracy with only 6 and 5 timesteps, respectively.
Revised: top-1 accuracy with only 6 and 5 timesteps, respectively.

Original: Comparedto STBP-tdBN and Diet-SNN [19], our ResNet-34 and VGG-16 obtain 3.71% and 1.65% accuracyimprovement, respectively.
Revised: Comparedto STBP-tdBN and Diet-SNN [19], our ResNet-34 and VGG-16 obtain 3.71% and 1.65% accuracyimprovement, respectively.

Original: These results show the potential of our method to train more accuratenetworks on a large-scale dataset than the other state-of-the-art methods.
Revised: These results show the potential of our method to train more accuratenetworks on a large-scale dataset than the other state-of-the-art methods.

Original: To further verify the effectiveness of our methods, the neuromorphic dataset, CIFAR10-DVS, wasalso used to evaluate the proposed method.
Revised: To further verify the effectiveness of our methods, the neuromorphic dataset, CIFAR10-DVS, wasalso used to evaluate the proposed method.

Original: With ResNet-19 as the backbone, it achieves the bestperformance with 72.60% accuracy in 10 timesteps, which is 4.80% higher than STBP-tdBN.
Revised: With ResNet-19 as the backbone, it achieves the bestperformance with 72.60% accuracy in 10 timesteps, which is 4.80% higher than STBP-tdBN.

Original:  Conclusion
Revised:  Conclusion

Original: This work aims at addressing two main problems for training accurate and deep SNNs: (i) theinformation flow loss caused by the 0/1 spike quantization; (ii) the training obstacle caused by thenon-differentiable spike activity function.
Revised: This work aims at addressing two main problems for training accurate and deep SNNs: (i) theinformation flow loss caused by the 0/1 spike quantization; (ii) the training obstacle caused by thenon-differentiable spike activity function.

Original: Inspired by the correlation between membrane potentialsand binary spike signals from the view of information entropy, we design a novel loss, IM-Loss,which can directly maximize the information expressiveness of an SNN, so that a well-performed
Revised: Inspired by the correlation between membrane potentialsand binary spike signals from the view of information entropy, we design a novel loss, IM-Loss,which can directly maximize the information expressiveness of an SNN, so that a well-performed

Original: SNN can be obtained.
Revised: SNN can be obtained.

Original: It also plays part of the normalization’s role but does not require additionaloperations or well-designed initialization strategies.
Revised: It also plays part of the normalization’s role but does not require additionaloperations or well-designed initialization strategies.

Original: Then, by incorporating a dynamically changingcoefficient into a differentiable asymptotic function, we introduce the ESG method for training SNNin a more appropriate way.
Revised: Then, by incorporating a dynamically changingcoefficient into a differentiable asymptotic function, we introduce the ESG method for training SNNin a more appropriate way.

Original: ESG encourages the SNN to enjoy both easy convergence and highaccuracy by adjusting the weight updating ability and the accuracy of the gradients.
Revised: ESG encourages the SNN to enjoy both easy convergence and highaccuracy by adjusting the weight updating ability and the accuracy of the gradients.

Original: Combining thesetwo methods, our method outperforms the state-of-the-art methods.
Revised: Combining thesetwo methods, our method outperforms the state-of-the-art methods.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes](b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [No](c) Did you discuss any potential negative societal impacts of your work?
Revised: [No](c) Did you discuss any potential negative societal impacts of your work?

Original: [No]
Revised: [No]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] The proof is available
Revised: [Yes] The proof is available

Original: in appendix
Revised: in appendix

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] Our code willbe available in the supplemental.
Revised: [Yes] Our code willbe available in the supplemental.

Original: (b) Did you specify all the training details (e
Revised: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] All implementations are described in the experiments section and
Revised: [Yes] All implementations are described in the experiments section and

Original: appendix
Revised: appendix

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We report the mean as well as the standard deviationaccuracy.
Revised: [Yes] We report the mean as well as the standard deviationaccuracy.

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g.,type of GPUs, internal cluster, or cloud provider)?
Revised: g.,type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] The computation resources
Revised: [Yes] The computation resources

Original: description is provided in the appendix.
Revised: description is provided in the appendix.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a)
Revised: (a)

Original: If your work uses existing assets, did you cite the creators?
Revised: If your work uses existing assets, did you cite the creators?

Original: [Yes]
Revised: [Yes]

Original: (b) Did
Revised: (b) Did

Original: you mention license of assets?
Revised: you mention license of assets?

Original: No public datasets.
Revised: No public datasets.

Original: ( you include assets material
Revised: ( you include assets material

Original: No]
Revised: No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: [Table]

Original: using/curating?
Revised: using/ curating?

Original: [No]
Revised: No](e)

Original: [Table]
Revised: Did you

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable [Table] information or offensive content?
Revised: discuss whether the data you are using/curating contains personally identifiableinformation or offensive content

Original: [No]
Revised: No]

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects..

Original: [Table]
Revised: .(a)

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to InstitutionalReview
Revised: ](b) Did you describe any potential participant risks, with links to Institutional

Original: [Table]
Revised: Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals , if applicable?

Original: [N/A]
Revised: [N/A

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid toparticipants and the total amount
Revised: ](c) Did you include the estimated hourly wage paid to

Original: [Table]
Revised: participants and the total

Original: spent on participant compensation?
Revised: amountspent on participant compensation?

Original: [N/A]
Revised: [N/A

Revised: A Appendix

Revised: Here, we provide the membrane potential distribution of the first layer of the second block in

Revised: ResNet20 on CIFAR-10 test sets before and after being trained with IM-Loss.

Revised: It can be seen that themembrane potential distribution truely follow the Gaussian-alike distribution with free mean value inour experiments.

Original: [Table]
Revised: [Figure]