Original: IM-Loss: Information Maximization Loss for Spiking Neural Networks
Revised: IM-Loss: Information Maximization Loss for Spiking Neural Networks

Original: Abstract
Revised: Abstract

Original: Spiking Neural Network (SNN), recognized as a type of biologically plausible architecture, has recently drawn much research attention.
Revised: Spiking Neural Network (SNN), recognized as a type of biologically plausible architecture, has recently drawn much research attention.

Original: It transmits information by 0 / 1 spikes.
Revised: It transmits information by 0 / 1 spikes.

Original: This bio-mimetic mechanism of SNN demonstrates extreme energy efficiency since it avoids any multiplications on neuromorphic hardware.
Revised: This bio-mimetic mechanism of SNN demonstrates extreme energy efficiency since it avoids any multiplications on neuromorphic hardware.

Original: However, the forward-passing 0 / 1 spike quantization will cause information loss and accuracy degradation.
Revised: However, the forward-passing 0 / 1 spike quantization will cause information loss and accuracy degradation.

Original: To deal with this problem, the Information maximization loss (IM-Loss) that aims at maximizing the information flow in the SNN is proposed in the paper.
Revised: To deal with this problem, the Information maximization loss (IM-Loss) that aims at maximizing the information flow in the SNN is proposed in the paper.

Original: The IM-Loss not only enhances the information expressiveness of an
Revised: The IM-Loss not only enhances the information expressiveness of an

Original: SNN directly but also plays a part of the role of normalization without introducing any additional operations ( e.g. , bias and scaling) in the inference phase.
Revised: SNN directly but also plays a part of the role of normalization without introducing any additional operations ( e.g. , bias and scaling) in the inference phase.

Original: Additionally, we introduce a novel differentiable spike activity estimation, Evolutionary
Revised: Additionally, we introduce a novel differentiable spike activity estimation, Evolutionary

Original: Surrogate Gradients (ESG) in SNNs.
Revised: Surrogate Gradients (ESG) in SNNs.

Original: By appointing automatic evolvable surrogate gradients for spike activity function, ESG can ensure sufficient model updates at the beginning and accurate gradients at the end of the training, resulting in both easy convergence and high task performance.
Revised: By appointing automatic evolvable surrogate gradients for spike activity function, ESG can ensure sufficient model updates at the beginning and accurate gradients at the end of the training, resulting in both easy convergence and high task performance.

Original: Experimental results on both popular non-spiking static and neuromorphic datasets show that the SNN models trained by our method outperform the current state-of-the-art algorithms.
Revised: Experimental results on both popular non-spiking static and neuromorphic datasets show that the SNN models trained by our method outperform the current state-of-the-art algorithms.

Original:  Introduction
Revised:  Introduction

Original: Recent progress in Artificial Neural Networks (ANNs) has been successfully applied in many machine intelligence fields.
Revised: Recent progress in Artificial Neural Networks (ANNs) has been successfully applied in many machine intelligence fields.

Original: Although ANNs imitate the structure of the brain to a certain extent, their signal processing mechanism using operations on real, dense valued tensors, still differs from how the brain works based on sparse spike events.
Revised: Although ANNs imitate the structure of the brain to a certain extent, their signal processing mechanism using operations on real, dense valued tensors, still differs from how the brain works based on sparse spike events.

Original: Being another sub-category of brain-inspired learning models, the Spiking Neural Networks (SNNs) utilize 0 / 1 binary spike signals to transmit information between units and spatio-temporal dynamics to mimic brain behaviors.
Revised: Being another sub-category of brain-inspired learning models, the Spiking Neural Networks (SNNs) utilize 0 / 1 binary spike signals to transmit information between units and spatio-temporal dynamics to mimic brain behaviors [20; 12; 13].

Original: Unlike the ANN, where all neurons in the same layer need to be evaluated before passing information to the next layer, the SNN processes information asynchronously and sparsely based on sparse spike events.
Revised: Unlike the ANN, where all neurons in the same layer need to be evaluated before passing information to the next layer, the SNN processes information asynchronously and sparsely based on sparse spike events.

Original: Benefitting from such an information processing paradigm, the SNN can efficiently run on specialized neuromorphic hardware ( e.g. , Loihi [4]) and has achieved substantial progress in recent years.
Revised: Benefitting from such an information processing paradigm, the SNN can efficiently run on specialized neuromorphic hardware ( e.g. , Loihi [4]) and has achieved substantial progress in recent years [27].

Original: Although the SNN is more energy-efficient compared with the ANN, it still has some limitations to achieve satisfactory task performance.
Revised: Although the SNN is more energy-efficient compared with the ANN, it still has some limitations to achieve satisfactory task performance.

Original: Due to the usage of spikes for information receiving and transmitting, the SNN needs to quantize real analog values into 0 / 1 -formed spikes in implementation.
Revised: Due to the usage of spikes for information receiving and transmitting, the SNN needs to quantize real analog values into 0 / 1 -formed spikes in implementation.

Original: This quantization causes information loss in forward propagation.
Revised: This quantization causes information loss in forward propagation.

Original: There are some remarkable existing works that alleviate the problem of information loss of SNNs by adjusting the spike rate by learning more parameters.
Revised: There are some remarkable existing works that alleviate the problem of information loss of SNNs by adjusting the spike rate by learning more parameters.

Original: In [22] and [34], a method that aims to learn the correct membrane leak-and-firing threshold for each layer of the SNN was proposed.
Revised: In [25] and [38], a method that aims to learn the correct membrane leak-and-firing threshold for each layer of the SNN was proposed.

Original: Results showed that learning the correct threshold
Revised: Results showed that learning the correct threshold

Original: Pre
Revised: Pre

Original: [Figure]
Revised: [Figure]

Original: Post
Revised: Post

Original: [Figure]
Revised: [Figure]

Original: Leaky Integrate-Fire
Revised: Leaky Integrate-Fire

Original: [Figure]
Revised: [Figure]

Original: can help increase the information capacity of the SNN.
Revised: can help increase the information capacity of the SNN.

Original: A wise choice of the firing threshold must be small enough that sufficient spikes are fired while also being large enough that avoids all neurons fired at the same time.
Revised: A wise choice of the firing threshold must be small enough that sufficient spikes are fired while also being large enough that avoids all neurons fired at the same time.

Original: Similarly, by incorporating the learnable membrane time parameter into the SNN, the parametric neuron [8; 29] can not only increase the diversity of neurons but also the information expressiveness of the network.
Revised: Similarly, by incorporating the learnable membrane time parameter into the SNN, the parametric neuron [8; 33] can not only increase the diversity of neurons but also the information expressiveness of the network.

Original: However, all these methods require more parameters and attempt to increase the information expressiveness of the SNN in an indirect manner.
Revised: However, all these methods require more parameters and attempt to increase the information expressiveness of the SNN in an indirect manner.

Original: Meanwhile, the non-differentiable spike activities also prevent the SNN from producing accurate gradients in backward propagation, which increase the difficulty of the model optimization.
Revised: Meanwhile, the non-differentiable spike activities also prevent the SNN from producing accurate gradients in backward propagation, which increase the difficulty of the model optimization.

Original: In this work, we propose information maximization (IM) loss to deal with the information loss in
Revised: In this work, we propose information maximization (IM) loss to deal with the information loss in

Original: SNNs.
Revised: SNNs.

Original: More specifically, we first analyze the information expressiveness ability of the SNN from the view of information entropy of the spikes.
Revised: More specifically, we first analyze the information expressiveness ability of the SNN from the view of information entropy of the spikes.

Original: Then a spike distribution loss is introduced for calculating the information loss, i.e. , IM-Loss, of the SNN, thus the information expressiveness of the SNN can be maximized directly by minimizing this information loss, which benefits the accuracy improvement of the SNN.
Revised: Then a spike distribution loss is introduced for calculating the information loss, i.e. , IM-Loss, of the SNN, thus the information expressiveness of the SNN can be maximized directly by minimizing this information loss, which benefits the accuracy improvement of the SNN.

Original: To overcome the non-differentiable spike activities during the model optimization in
Revised: To overcome the non-differentiable spike activities during the model optimization in

Original: SNNs, we also introduce Evolutionary Surrogate Gradients (ESG) in the paper.
Revised: SNNs, we also introduce Evolutionary Surrogate Gradients (ESG) in the paper.

Original: The ESG can evolve automatically during training and provides a pathway to bridge the gap between the pseudo derivative and appropriate derivative.
Revised: The ESG can evolve automatically during training and provides a pathway to bridge the gap between the pseudo derivative and appropriate derivative.

Original: With evolving, it not only ensures sufficient updating at the beginning but also accurate gradients calculation at the end of the training, resulting in networks with better performance.
Revised: With evolving, it not only ensures sufficient updating at the beginning but also accurate gradients calculation at the end of the training, resulting in networks with better performance.

Original: The overall workflow of our algorithm can be seen in Fig.1.
Revised: The overall workflow of our algorithm can be seen in Fig.1.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: • We propose the IM-Loss to preserve the information in SNNs.
Revised: • We propose the IM-Loss to preserve the information in SNNs.

Original: It presents a new perspective to understand the ability of the SNN to express information.
Revised: It presents a new perspective to understand the ability of the SNN to express information.

Original: With the IM-Loss, we can train a more accurate and deeper SNN.
Revised: With the IM-Loss, we can train a more accurate and deeper SNN.

Original: • We also introduce the ESG to effectively approximate the gradients of the spike activity function.
Revised: • We also introduce the ESG to effectively approximate the gradients of the spike activity function.

Original: The ESG can ensure both sufficient updating of weights at the beginning and accurate gradients at the end of the training in an evolutionary way so that the SNN would be more accurate and easier to converge.
Revised: The ESG can ensure both sufficient updating of weights at the beginning and accurate gradients at the end of the training in an evolutionary way so that the SNN would be more accurate and easier to converge.

Original: • Our method is evaluated on both static and dynamic datasets.
Revised: • Our method is evaluated on both static and dynamic datasets.

Original: Extensive experimental results show that our algorithm performs remarkably well across various network structures.
Revised: Extensive experimental results show that our algorithm performs remarkably well across various network structures.

Original: To our best knowledge, this is the first work that can directly train deep SNNs from scratch without normalization or warm-start techniques.
Revised: To our best knowledge, this is the first work that can directly train deep SNNs from scratch without normalization or warm-start techniques.

Original:  Related Work
Revised:  Related Work

Original: In this section, we first briefly review the learning methods of SNNs to provide readers an overall understanding.
Revised: In this section, we first briefly review the learning methods of SNNs to provide readers an overall understanding.

Original: Then we introduce the information theory used in DNNs which gives us insight to design our method for handling information loss in SNNs.
Revised: Then we introduce the information theory used in DNNs which gives us insight to design our method for handling information loss in SNNs.

Original: 2.1 Learning Methods of Spiking Neural Networks
Revised: 2.1 Learning Methods of Spiking Neural Networks

Original: The learning algorithms for SNNs can be divided into three categories: (i) converting ANN to SNN (ANN2SNN) [24; 16]; (ii) unsupervised learning [6]; and (iii) supervised learning [20; 22; 28; 17; 10].
Revised: The learning algorithms for SNNs can be divided into three categories: (i) converting ANN to SNN (ANN2SNN) [28; 18]; (ii) unsupervised learning [6]; and (iii) supervised learning [23; 25; 32; 19; 10].

Original: ANN2SNN converts a special trained ANN to an SNN that yields the same input-output mapping for a given task.
Revised: ANN2SNN converts a special trained ANN to an SNN that yields the same input-output mapping for a given task.

Original: Since the research of the ANN has been very successful, ANN2SNN can gain competitive accuracy.
Revised: Since the research of the ANN has been very successful, ANN2SNN can gain competitive accuracy.

Original: Some recent conversion methods have achieved nearly loss-less accuracy with VGG and
Revised: Some recent conversion methods have achieved nearly loss-less accuracy with VGG and

Original: ResNet as original ANN models [16].
Revised: ResNet as original ANN models [18].

Original: However, this kind of method is restricted to rate-coding and can only handle static datasets.
Revised: However, this kind of method is restricted to rate-coding and can only handle static datasets.

Original: Moreover, ANN2SNN requires long timesteps (usually needs hundreds of timesteps) to converge and cannot exploit the spatio-temporal features during training for solving temporal tasks.
Revised: Moreover, ANN2SNN requires long timesteps (usually needs hundreds of timesteps) to converge and cannot exploit the spatio-temporal features during training for solving temporal tasks.

Original: Unsupervised learning is based on biological plausible local learning rules and is usually considered more biologically plausible than other categories.
Revised: Unsupervised learning is based on biological plausible local learning rules and is usually considered more biologically plausible than other categories.

Original: However, unsupervised learning can only train shallow SNNs and is hard to achieve desirable performance.
Revised: However, unsupervised learning can only train shallow SNNs and is hard to achieve desirable performance.

Original: Supervised learning uses the derivable approximation to overcome the non-differentiability of the spike activities.
Revised: Supervised learning uses the derivable approximation to overcome the non-differentiability of the spike activities.

Original: Hence, the SNN can be optimized with gradient descent algorithms as the ANN and achieves high performance.
Revised: Hence, the SNN can be optimized with gradient descent algorithms as the ANN and achieves high performance.

Original: Moreover, Supervised learning can greatly reduce the number of timesteps and handle dynamic datasets.
Revised: Moreover, Supervised learning can greatly reduce the number of timesteps and handle dynamic datasets.

Original: Thus, supervised learning has increasingly aroused researchers’ great interest in recent years.
Revised: Thus, supervised learning has increasingly aroused researchers’ great interest in recent years.

Original: We focus on providing some insights to improve the performance of the supervised learning-based SNNs.
Revised: We focus on providing some insights to improve the performance of the supervised learning-based SNNs.

Original: 2.2 Information Theory in Deep Neural Networks
Revised: 2.2 Information Theory in Deep Neural Networks

Original: Information theory is widely used in DNNs, ranging from theoretical investigations ( e.g. , the generalization bound induced by mutual information [26]) to practical applications ( e.g. , the variational information bottleneck in representation learning [2]).
Revised: Information theory is widely used in DNNs, ranging from theoretical investigations ( e.g. , the generalization bound induced by mutual information [30]) to practical applications ( e.g. , the variational information bottleneck in representation learning [2]).

Original: It provides a useful way to understand the underlying behavior of random variables and this is a key factor in developing and analyzing deep models.
Revised: It provides a useful way to understand the underlying behavior of random variables and this is a key factor in developing and analyzing deep models.

Original: There are many popular information-theoretic principles that have demonstrated great potential in DNNs [31].
Revised: There are many popular information-theoretic principles that have demonstrated great potential in DNNs [35].

Original: In [30], deep energy-based models are trained with the f -divergence (a generalization of KL divergence) minimization.
Revised: In [34], deep energy-based models are trained with the f -divergence (a generalization of KL divergence) minimization.

Original: In reinforcement learning, the maximum entropy regularization is adopted to encourage exploration and avoid getting stuck in a local optima [1].
Revised: In reinforcement learning, the maximum entropy regularization is adopted to encourage exploration and avoid getting stuck in a local optima [1].

Original: Mutual information maximization is used in [21] to improve the expressiveness of a model by shifting the distribution of weights to zero-mean in every training iteration.
Revised: Mutual information maximization is used in [24] to improve the expressiveness of a model by shifting the distribution of weights to zero-mean in every training iteration.

Original: In [3], training DNNs for classification is investigated via minimizing the information bottleneck functional.
Revised: In [3], training DNNs for classification is investigated via minimizing the information bottleneck functional.

Original: Despite the great success of information theory in DNNs, we find no related works in supervised learning of SNNs.
Revised: Despite the great success of information theory in DNNs, we find no related works in supervised learning of SNNs.

Original: In the paper, we propose the IM-Loss, which is based on the information maximization principle, to improve SNNs.
Revised: In the paper, we propose the IM-Loss, which is based on the information maximization principle, to improve SNNs.

Original: Our implementation and idea of the information maximization are very different from these existing works, which may also provide a new idea to further improve these methods.
Revised: Our implementation and idea of the information maximization are very different from these existing works, which may also provide a new idea to further improve these methods.

Original:  Preliminary
Revised:  Preliminary

Original: For better expression, we will first introduce the iterative Leaky Integrate-and-Fire (LIF) model and surrogate gradients of SNNs in detail, which are also adopted in our method.
Revised: For better expression, we will first introduce the iterative Leaky Integrate-and-Fire (LIF) model and surrogate gradients of SNNs in detail, which are also adopted in our method.

Original: 3.1 Explicitly Iterative LIF Model
Revised: 3.1 Explicitly Iterative LIF Model

Original: The LIF neuron is a commonly used mathematical model to describe the behavior of neural activities, which includes spike firing and update of membrane potential.
Revised: The LIF neuron is a commonly used mathematical model to describe the behavior of neural activities, which includes spike firing and update of membrane potential.

Original: Formally, a LIF neuron in layer l of an SNN with index i can be given by
Revised: Formally, a LIF neuron in layer l of an SNN with index i can be given by

Original: [Equation]
Revised: [Equation]

Original: where u li is the membrane potential, u rest is the membrane resting potential, τ is the membrane time constant to describe the membrane potential decaying, I li is the pre-synaptic input, and V th is a given firing threshold.
Revised: where u li is the membrane potential, u rest is the membrane resting potential, τ is the membrane time constant to describe the membrane potential decaying, I li is the pre-synaptic input, and V th is a given firing threshold.

Original: From Eq.
Revised: From Eq.

Original: 1, we can see that u li actually acts as a leaky integrator of the pre-synaptic input I li .
Revised: 1, we can see that u li actually acts as a leaky integrator of the pre-synaptic input I li .

Original: And Eq.
Revised: And Eq.

Original: 2 shows that when a neuron membrane potential reaches the firing threshold V th , it emits a spike to transfer its information to other neurons and reset its membrane potential to u rest .
Revised: 2 shows that when a neuron membrane potential reaches the firing threshold V th , it emits a spike to transfer its information to other neurons and reset its membrane potential to u rest .

Original: The above differential expressions described in the continuous domain are difficult to be implemented on mainstream machine learning frameworks ( e.g. , Pytorch, Tensorflow), where the embedded automatic differentiation mechanism executes discretely.
Revised: The above differential expressions described in the continuous domain are difficult to be implemented on mainstream machine learning frameworks ( e.g. , Pytorch, Tensorflow), where the embedded automatic differentiation mechanism executes discretely.

Original: Alternatively, the iterative LIF model [28] used the Euler method to solve the first-order differential equation of the LIF model.
Revised: Alternatively, the iterative LIF model [32] used the Euler method to solve the first-order differential equation of the LIF model.

Original: The iterative expression of the LIF model can be governed by
Revised: The iterative expression of the LIF model can be governed by

Original: [Equation]
Revised: [Equation]

Original: where τ decay is a constant to describe membrane potential decaying, u li ( t ) is the membrane potential in timestep t , and I li ( t ) is the pre-synaptic input.
Revised: where τ decay is a constant to describe membrane potential decaying, u li ( t ) is the membrane potential in timestep t , and I li ( t ) is the pre-synaptic input.

Original: Since I li ( t ) is the accumulation of spikes from the neurons of layer l − 1 connected to the current neuron, it can be described as I li ( t )
Revised: Since I li ( t ) is the accumulation of spikes from the neurons of layer l − 1 connected to the current neuron, it can be described as I li ( t )

Original: = (cid:80) j w ij o j ( t ) .
Revised: = (cid:80) j w ij o j ( t ) .

Original: Where w ij is the weight of j -th pre-synaptic of the current neuron, o j ( t ) is the output of the neuron from the previous layer, which interacts on the current neuron’s j -th pre-synaptic at the moment of t .
Revised: Where w ij is the weight of j -th pre-synaptic of the current neuron, o j ( t ) is the output of the neuron from the previous layer, which interacts on the current neuron’s j -th pre-synaptic at the moment of t .

Original: If we set u reset = 0 , the whole iterative LIF model in both spatial and temporal domain can be determined by
Revised: If we set u reset = 0 , the whole iterative LIF model in both spatial and temporal domain can be determined by

Original: [Equation]
Revised: [Equation]

Original: The iterative LIF model is friendly to be implemented on general machine learning programming frameworks.
Revised: The iterative LIF model is friendly to be implemented on general machine learning programming frameworks.

Original: 3.2 Surrogate Gradients of SNNs
Revised: 3.2 Surrogate Gradients of SNNs

Original: The spike activity function shown as Eq.6 can be viewed as a variant of the sign function.
Revised: The spike activity function shown as Eq.6 can be viewed as a variant of the sign function.

Original: Its derivative is 0 everywhere except for the infinity value at V th .
Revised: Its derivative is 0 everywhere except for the infinity value at V th .

Original: Since the non-differentiability of spike activity function, the back-propagation cannot be applied directly for training SNN.
Revised: Since the non-differentiability of spike activity function, the back-propagation cannot be applied directly for training SNN.

Original: To demonstrate this challenge, we show the expression to compute gradients of weights obtained by the spatial-temporal back-propagation (STBP) as follows,
Revised: To demonstrate this challenge, we show the expression to compute gradients of weights obtained by the spatial-temporal back-propagation (STBP) as follows,

Original: [Equation]
Revised: [Equation]

Original: where y denotes the target output vector, W is the matrix of weights of the SNN, L is the loss function, and ∂ y t
Revised: where y denotes the target output vector, W is the matrix of weights of the SNN, L is the loss function, and ∂ y t

Original: ∂ u t is the gradient of the spike activity function.
Revised: ∂ u t is the gradient of the spike activity function.

Original: The weight update is computed as
Revised: The weight update is computed as

Original: [Equation]
Revised: [Equation]

Original: where η is the learning rate.
Revised: where η is the learning rate.

Original: Considering that, ∂ y t ∂ u t is either 0 almost everywhere or infinity at V th , ∆ W is either 0 everywhere or a very large value in a rare case, hence weights would likely not be changed or updated to a large value some times.
Revised: Considering that, ∂ y t ∂ u t is either 0 almost everywhere or infinity at V th , ∆ W is either 0 everywhere or a very large value in a rare case, hence weights would likely not be changed or updated to a large value some times.

Original: Most prior works adopted surrogate gradients (SG) to overcome the non-differentiability challenge [33; 14; 27; 19; 25].
Revised: Most prior works adopted surrogate gradients (SG) to overcome the non-differentiability challenge [37; 16; 31; 22; 29].

Original: However, the SGs in existing methods are fixed during training, such a strategy would limit the learning capacity of the network.
Revised: However, the SGs in existing methods are fixed during training, such a strategy would limit the learning capacity of the network.

Original: We notice that the SG methods are also adopted in Quantization Neural Networks (QNNs) which also suffer from non-differentiability.
Revised: We notice that the SG methods are also adopted in Quantization Neural Networks (QNNs) which also suffer from non-differentiability.

Original: Some changeable surrogate gradient methods that can achieve better performance by changing SG in the training are proposed in QNNs [21; 9].
Revised: Some changeable surrogate gradient methods that can achieve better performance by changing SG in the training are proposed in QNNs [24; 9].

Original: Combining characteristics of SNNs, an evolutionary SG that can maintain strong updating ability in the early training phase and approach the accurate gradients gradually at the end of the training is introduced in the paper.
Revised: Combining characteristics of SNNs, an evolutionary SG that can maintain strong updating ability in the early training phase and approach the accurate gradients gradually at the end of the training is introduced in the paper.

Original:  Methodology
Revised:  Methodology

Original: In this paper, we argue that the challenge of training highly accurate spiking neural networks is mainly limited by the severe information loss caused by the spike activity function.
Revised: In this paper, we argue that the challenge of training highly accurate spiking neural networks is mainly limited by the severe information loss caused by the spike activity function.

Original: To mitigate the problem, the IM-Loss is proposed.
Revised: To mitigate the problem, the IM-Loss is proposed.

Original: Furthermore, we also introduce the ESG to obtain the appropriate surrogate gradient for the non-differentiable spike activity in backward propagation.
Revised: Furthermore, we also introduce the ESG to obtain the appropriate surrogate gradient for the non-differentiable spike activity in backward propagation.

Original: 4.1 Information Maximization Loss
Revised: 4.1 Information Maximization Loss

Original: In the forward propagation, the membrane potential to 0 / 1 spikes quantization of the spike activity function will cause information loss unquestionably.
Revised: In the forward propagation, the membrane potential to 0 / 1 spikes quantization of the spike activity function will cause information loss unquestionably.

Original: This is one of the main reasons why an SNN model cannot achieve as satisfactory performance as its ANN counterpart.
Revised: This is one of the main reasons why an SNN model cannot achieve as satisfactory performance as its ANN counterpart.

Original: Ideally, the spike tensor O should reflect the information of the membrane potential tensor U as much as possible.
Revised: Ideally, the spike tensor O should reflect the information of the membrane potential tensor U as much as possible.

Original: In the view of the information flow, to maximize the information flow from the full-precision tensor U to the binary tensor O , the mutual information
Revised: In the view of the information flow, to maximize the information flow from the full-precision tensor U to the binary tensor O , the mutual information

Original: I ( U ; O ) of the random variables U
Revised: I ( U ; O ) of the random variables U

Original: and O should be maximized as follows,
Revised: and O should be maximized as follows,

Original: [Equation]
Revised: [Equation]

Original: where H ( O ) is the entropy of O and H ( O | U ) is the conditional entropy of O given U .
Revised: where H ( O ) is the entropy of O and H ( O | U ) is the conditional entropy of O given U .

Original: It can be proved that, maximizing the mutual information
Revised: It can be proved that, maximizing the mutual information

Original: I ( U ; O ) is equivalent to maximize the information entropy H ( O ) for SNNs (see Appendix A.1 for detailed proofs) as follows,
Revised: I ( U ; O ) is equivalent to maximize the information entropy H ( O ) for SNNs (see Appendix A.1 for detailed proofs) as follows,

Original: [Equation]
Revised: [Equation]

Original: Obviously, when the distribution of O satisfies p (0) = p (1) = 0 .
Revised: Obviously, when the distribution of O satisfies p (0) = p (1) = 0 .

Original: 5 the entropy H ( O ) is maximized, where the p is the marginal probability mass functions of the discrete variables O .
Revised: 5 the entropy H ( O ) is maximized, where the p is the marginal probability mass functions of the discrete variables O .

Original: This means that to maximize the information flow from the full-precision tensor U to the spike tensor O , the spike values should be evenly distributed, i.e. , (cid:80) u<V th p ( u ) =
Revised: This means that to maximize the information flow from the full-precision tensor U to the spike tensor O , the spike values should be evenly distributed, i.e. , (cid:80) u<V th p ( u ) =

Original: (cid:80) u ≥ V th p ( u ) = 0 .
Revised: (cid:80) u ≥ V th p ( u ) = 0 .

Original: We set U ( q ) to denotes the q -quantile of U ’s elements in ascend order where 0 ≤ q ≤ 1 .
Revised: We set U ( q ) to denotes the q -quantile of U ’s elements in ascend order where 0 ≤ q ≤ 1 .

Original: To maximize the H ( O ) , we should have U (0 . 5 − ) <
Revised: To maximize the H ( O ) , we should have U (0 . 5 − ) <

Original: V th and U (0 .
Revised: V th and U (0 .

Original: 5 + ) ≥ V th .
Revised: 5 + ) ≥ V th .

Original: Since the U follows a Gaussian distribution [33; 10],
Revised: Since the U follows a Gaussian distribution [37; 10],

Original: U (0 . 5) is almost equal to the mean value ¯ U .
Revised: U (0 . 5) is almost equal to the mean value ¯ U .

Original: Then ( ¯ U − V th ) 2 can be regarded as a metric of the information-loss caused by the quantization from the full-precision tensor U to the spike tensor O .
Revised: Then ( ¯ U − V th ) 2 can be regarded as a metric of the information-loss caused by the quantization from the full-precision tensor U to the spike tensor O .

Original: Hereby, we define the proposed IM-Loss as:
Revised: Hereby, we define the proposed IM-Loss as:

Original: [Equation]
Revised: [Equation]

Original: where ¯ U l is the average membrane potential at all timesteps of l -th layer, and L is the total number of layers.
Revised: where ¯ U l is the average membrane potential at all timesteps of l -th layer, and L is the total number of layers.

Original: Then the overall loss can be calculated as follows,
Revised: Then the overall loss can be calculated as follows,

Original: [Equation]
Revised: [Equation]

Original: where L CE is the cross-entropy loss, and λ is a coefficient to balance the classification loss and the information loss.
Revised: where L CE is the cross-entropy loss, and λ is a coefficient to balance the classification loss and the information loss.

Original: We set λ as 2 in this paper.
Revised: We set λ as 2 in this paper.

Original: 4.2 The Effect of IM-Loss
Revised: 4.2 The Effect of IM-Loss

Original: SNNs utilize 0 / 1 spikes to communicate between neurons.
Revised: SNNs utilize 0 / 1 spikes to communicate between neurons.

Original: Benefitting from this information processing paradigm, SNNs run efficiently.
Revised: Benefitting from this information processing paradigm, SNNs run efficiently.

Original: However, since the spike rate of the network with random weights decreases sharply through layers, an SNN is difficult to train.
Revised: However, since the spike rate of the network with random weights decreases sharply through layers, an SNN is difficult to train.

Original: In other words, the spike rate will usually soon become 0 after several layers when training models from scratch, resulting in information flow disappearing and difficulty for training large and deep SNNs.
Revised: In other words, the spike rate will usually soon become 0 after several layers when training models from scratch, resulting in information flow disappearing and difficulty for training large and deep SNNs.

Original: For better understanding, the layer-wise average spike rates of spiking CIFARNet and ResNetinitialized with random weights on the CIFAR10 test set are visualized in Fig.
Revised: For better understanding, the layer-wise average spike rates of spiking CIFARNet and ResNetinitialized with random weights on the CIFAR10 test set are visualized in Fig.

Original: Both networks were trained with a timestep of 4 and without normalization.
Revised: Both networks were trained with a timestep of 4 and without normalization.

Original: When epoch is 0, i.e. , the untrained networks, spike rates are becoming 0 in the deeper layers, resulting in untrainable networks.
Revised: When epoch is 0, i.e. , the untrained networks, spike rates are becoming 0 in the deeper layers, resulting in untrainable networks.

Original: This means deep SNNs are not easy to train without any special techniques.
Revised: This means deep SNNs are not easy to train without any special techniques.

Original: Some works attempted to alleviate this phenomenon by using normalization techniques.
Revised: Some works attempted to alleviate this phenomenon by using normalization techniques.

Original: However, normalization requires other additional operations.
Revised: However, normalization requires other additional operations.

Original: For instance, the bias operation consumes extra computations in the inference phase when implementing models on some specialized neuromorphic hardware.
Revised: For instance, the bias operation consumes extra computations in the inference phase when implementing models on some specialized neuromorphic hardware.

Original: While for the case of epoch=1, where the proposed IM-Loss was applied, the spike rate in each layer can be restricted to an appropriate range ( i.e. , about 50% ), which enables the learning algorithm to train a large and deep
Revised: While for the case of epoch=1, where the proposed IM-Loss was applied, the spike rate in each layer can be restricted to an appropriate range ( i.e. , about 50% ), which enables the learning algorithm to train a large and deep

Original: SNN.
Revised: SNN.

Original: By maintaining an appropriate range for the spike rate, IM-Loss can effectively update the weights of the two deep SNNs even with random initialization.
Revised: By maintaining an appropriate range for the spike rate, IM-Loss can effectively update the weights of the two deep SNNs even with random initialization.

Original: In this sense, IM-Loss also possesses the effect of normalization but without other additional operations or well-designed initialization strategies.
Revised: In this sense, IM-Loss also possesses the effect of normalization but without other additional operations or well-designed initialization strategies.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: To further demonstrate the information retention ability of IM-Loss
Revised: To further demonstrate the information retention ability of IM-Loss

Original: [Figure]
Revised: [Figure]

Original: the case of epoch=1000 in Fig.
Revised: the case of epoch=1000 in Fig.

Original: 2 shows the spike rate of the SNNs at the end of training.
Revised: 2 shows the spike rate of the SNNs at the end of training.

Original: For ResNet-19, IM-Loss did not restrict the layers inside the residual blocks, since these layers tend to learn small perturbations which are incompatible with the restriction by IM-Loss.
Revised: For ResNet-19, IM-Loss did not restrict the layers inside the residual blocks, since these layers tend to learn small perturbations which are incompatible with the restriction by IM-Loss.

Original: It can be seen that the spike rate of each layer restricted by IM-Loss is near 0.5, meaning the information flow can be maximized by IM-Loss as analyzed above
Revised: It can be seen that the spike rate of each layer restricted by IM-Loss is near 0.5, meaning the information flow can be maximized by IM-Loss as analyzed above

Original: [Figure]
Revised: [Figure]

Original: 4.3 Evolutionary Surrogate Gradients
Revised: 4.3 Evolutionary Surrogate Gradients

Original: The zero-derivative of spike activity function almost everywhere extremely prevents SNNs from training adequately.
Revised: The zero-derivative of spike activity function almost everywhere extremely prevents SNNs from training adequately.

Original: To our best knowledge, nearly all the prior works attempted to alleviate this problem by following the principle of replacing the gradient of the spike activity function with a fixed SG in SNNs.
Revised: To our best knowledge, nearly all the prior works attempted to alleviate this problem by following the principle of replacing the gradient of the spike activity function with a fixed SG in SNNs.

Original: As aforementioned, changeable surrogate gradient (CSG) methods in QNNs can achieve better performance.
Revised: As aforementioned, changeable surrogate gradient (CSG) methods in QNNs can achieve better performance.

Original: We guess that designing a suitable CSG can also increase accuracy in
Revised: We guess that designing a suitable CSG can also increase accuracy in

Original: SNNs.
Revised: SNNs.

Original: To this end, we first introduce a differentiable asymptotic function to approximate the spike activity function as follows,
Revised: To this end, we first introduce a differentiable asymptotic function to approximate the spike activity function as follows,

Original: [Equation]
Revised: [Equation]

Original: where the coefficient k determines the shape of the asymptotic function.
Revised: where the coefficient k determines the shape of the asymptotic function.

Original: The gradient of the asymptotic function, φ ( · ) is used to replace the gradient of the spike activity function.
Revised: The gradient of the asymptotic function, φ ( · ) is used to replace the gradient of the spike activity function.

Original: For the sake of intuitive explanation, Fig. 3 illustrates the influences of k on the asymptotic function (left) and its corresponding gradient (right).
Revised: For the sake of intuitive explanation, Fig. 3 illustrates the influences of k on the asymptotic function (left) and its corresponding gradient (right).

Original: It can be seen that, when setting k with a large value, a more accurate gradient in backward can be obtained, so as to increase the difficulty of the training.
Revised: It can be seen that, when setting k with a large value, a more accurate gradient in backward can be obtained, so as to increase the difficulty of the training.

Original: Conversely, when k becomes smaller, the gradient of the asymptotic function will tend to be uniform and keep some relatively large value, thus the strong weight updating ability can be obtained.
Revised: Conversely, when k becomes smaller, the gradient of the asymptotic function will tend to be uniform and keep some relatively large value, thus the strong weight updating ability can be obtained.

Original: [Figure]
Revised: [Figure]

Original: Then we design the ev olutionary a symptotic f unction (EvAF) as follows,
Revised: Then we design the ev olutionary a symptotic f unction (EvAF) as follows,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where φ ( x ) is the backward approximation substitute for the spike activity function and K ( i ) is the coefficient that changes dynamically with the training epoch as follows,
Revised: where φ ( x ) is the backward approximation substitute for the spike activity function and K ( i ) is the coefficient that changes dynamically with the training epoch as follows,

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where K min and K max denotes the lower bound and the upper bound of the value range of K , respectively, and i is the index of epoch starting from 0 to N − 1 .
Revised: where K min and K max denotes the lower bound and the upper bound of the value range of K , respectively, and i is the index of epoch starting from 0 to N − 1 .

Original: In practice, we set K min = 10and K max = 10 1 .
Revised: In practice, we set K min = 10and K max = 10 1 .

Original: Without using any additional hyper-parameter, K ( i ) will increase with the accumulation of training epochs.
Revised: Without using any additional hyper-parameter, K ( i ) will increase with the accumulation of training epochs.

Original: Driven by K ( i ) , EvAF can gradually evolve to the spike activity function, and a more accurate gradient will be obtained, correspondingly.
Revised: Driven by K ( i ) , EvAF can gradually evolve to the spike activity function, and a more accurate gradient will be obtained, correspondingly.

Original: Such dynamically changing gradient will be used as the SG in our method, namely Evolutionary Surrogate Gradient (ESG).
Revised: Such dynamically changing gradient will be used as the SG in our method, namely Evolutionary Surrogate Gradient (ESG).

Original: By introducing the modified asymptotic function, EvAF into the SNNs, both weight updating ability in the early stage of training and more accurate backward gradient at the end of training can be fulfilled by the proposed ESG.
Revised: By introducing the modified asymptotic function, EvAF into the SNNs, both weight updating ability in the early stage of training and more accurate backward gradient at the end of training can be fulfilled by the proposed ESG.

Original: 4.4 The Effect of ESG
Revised: 4.4 The Effect of ESG

Original: To demonstrate the effect of ESG, we compared the accuracy curves of spiking CIFARNet and
Revised: To demonstrate the effect of ESG, we compared the accuracy curves of spiking CIFARNet and

Original: ResNet-19 (with timestep = 4) on CIFAR10 using the gradient of the asymptotic function with k = 10 0 , k = 10 1 and ESG derived from EvAF with K ∈ [10 0 , 10 1 ] in backward.
Revised: ResNet-19 (with timestep = 4) on CIFAR10 using the gradient of the asymptotic function with k = 10 0 , k = 10 1 and ESG derived from EvAF with K ∈ [10 0 , 10 1 ] in backward.

Original: From the visualized comparison results shown in Fig.
Revised: From the visualized comparison results shown in Fig.

Original: 4, we can see that the SNNs with a small-valued k have stronger updating ability and are easier to converge compared to the SNNs with a large k .
Revised: 4, we can see that the SNNs with a small-valued k have stronger updating ability and are easier to converge compared to the SNNs with a large k .

Original: The SGs of
Revised: The SGs of

Original: SNNs with a large k are very close to the gradients of the spike activities, in such case, most weights of these SNNs cannot be updated sufficiently.
Revised: SNNs with a large k are very close to the gradients of the spike activities, in such case, most weights of these SNNs cannot be updated sufficiently.

Original: Hence these SNNs will suffer slow convergence and face severe performance degradation.
Revised: Hence these SNNs will suffer slow convergence and face severe performance degradation.

Original: Our ESG starts with K min at the early stage of training, so that the strong updating ability can make the SNN converge easily; then it ends with K max at the end of training thus the backward gradient will be more accurate resulting in a better-performed SNN.
Revised: Our ESG starts with K min at the early stage of training, so that the strong updating ability can make the SNN converge easily; then it ends with K max at the end of training thus the backward gradient will be more accurate resulting in a better-performed SNN.

Original: In general, by using a dynamically changing coefficient, our ESG encourages the SNN with both easier convergence and higher accuracy.
Revised: In general, by using a dynamically changing coefficient, our ESG encourages the SNN with both easier convergence and higher accuracy.

Original:  Experiments
Revised:  Experiments

Original: In this section, extensive experiments were conducted to demonstrate the effectiveness of the proposed methods.
Revised: In this section, extensive experiments were conducted to demonstrate the effectiveness of the proposed methods.

Original: The algorithm of the training process of our method is presented in Appendix
Revised: The algorithm of the training process of our method is presented in Appendix

Original: A.2.
Revised: A.2.

Original: The experiments include widely-used network structures including spiking CIFARNet [28],
Revised: The experiments include widely-used network structures including spiking CIFARNet [32],

Original: ResNet-19 [33], modified VGG-16 [22], and ResNet-34 [11] for both popular non-spiking static and neuromorphic datasets: CIFAR10/100, ImageNet (ILSVRC12) and CIFAR10-DVS [15].
Revised: ResNet-19 [37], modified VGG-16 [25], and ResNet-34 [11] for both popular non-spiking static and neuromorphic datasets: CIFAR10/100, ImageNet (ILSVRC12) and CIFAR10-DVS [17].

Original: For being friendly with neuromorphic hardware, the max-pooling layer was replaced with the average-pooling layer in the used network architectures.
Revised: For being friendly with neuromorphic hardware, the max-pooling layer was replaced with the average-pooling layer in the used network architectures.

Original: [Table]
Revised: [Table]

Original: 5.1 Ablation Study
Revised: 5.1 Ablation Study

Original: To understand how our method works in practice, the ablation experiments for IM-Loss and ESG with spiking CIFARNet and ResNet-19 (timestep = 4) were first conducted on CIFAR10.
Revised: To understand how our method works in practice, the ablation experiments for IM-Loss and ESG with spiking CIFARNet and ResNet-19 (timestep = 4) were first conducted on CIFAR10.

Original: To make a fair comparison with tdBN, the rectangular surrogate function was used for the model without ESG to compute gradients, as in tdBN.
Revised: To make a fair comparison with tdBN, the rectangular surrogate function was used for the model without ESG to compute gradients, as in tdBN.

Original: The performances of the networks with IM-Loss and ESG alone and their combinations are shown in Table 1.
Revised: The performances of the networks with IM-Loss and ESG alone and their combinations are shown in Table 1.

Original: For the networks without normalization or IM-Loss, there is a certain probability that they might not be trainable from scratch with inappropriate random initialization as aforementioned.
Revised: For the networks without normalization or IM-Loss, there is a certain probability that they might not be trainable from scratch with inappropriate random initialization as aforementioned.

Original: Therefore, in this series of experiments, the models trained by
Revised: Therefore, in this series of experiments, the models trained by

Original: IM-Loss after several iterations (training was stopped when the spike rates of the last layer were not 0) as the initialization, which can also be regarded as a warm-start for the networks.
Revised: IM-Loss after several iterations (training was stopped when the spike rates of the last layer were not 0) as the initialization, which can also be regarded as a warm-start for the networks.

Original: Results in tableshow that both IM-Loss and ESG can improve the networks’ performances alone.
Revised: Results in tableshow that both IM-Loss and ESG can improve the networks’ performances alone.

Original: And by using the combination of these two methods, the performances can be further improved.
Revised: And by using the combination of these two methods, the performances can be further improved.

Original: Moreover, our method is also compatible with other advanced technologies, such as tdBN [33], hence it can be applied in a plug-in manner to enhance the learning capacity of SNNs together with other techniques.
Revised: Moreover, our method is also compatible with other advanced technologies, such as tdBN [37], hence it can be applied in a plug-in manner to enhance the learning capacity of SNNs together with other techniques.

Original: 5.2 Comparisons with State-of-the-art Methods
Revised: 5.2 Comparisons with State-of-the-art Methods

Original: As reported in the ablation study, our algorithm combined with tdBN can achieve better results, hence this training technique was also adopted in our method.
Revised: As reported in the ablation study, our algorithm combined with tdBN can achieve better results, hence this training technique was also adopted in our method.

Original: The results are listed in table 2.
Revised: The results are listed in table 2.

Original: On CIFAR10, our models achieve better performances than the other state-of-the-art methods (92.20% top-1 accuracy with only 4 timesteps by CIFARNet, 93.85% top-1 accuracy with only 5 timesteps by VGG-16, and 95.49% top-1 accuracy with only 6 timesteps by ResNet-19).
Revised: On CIFAR10, our models achieve better performances than the other state-of-the-art methods (92.20% top-1 accuracy with only 4 timesteps by CIFARNet, 93.85% top-1 accuracy with only 5 timesteps by VGG-16, and 95.49% top-1 accuracy with only 6 timesteps by ResNet-19).

Original: Moreover, with only 2 timesteps, our method using ResNet-19 as backbone still outperforms the STBP-tdBN with 6 timesteps by 0.69% accuracy.
Revised: Moreover, with only 2 timesteps, our method using ResNet-19 as backbone still outperforms the STBP-tdBN with 6 timesteps by 0.69% accuracy.

Original: And our CIFARNet-based model with only 4 timesteps can also outperform the CIFARNet in [28] with 12 timesteps and the one in [32] with 5 timesteps by 1.67% and 0.79% accuracy, respectively.
Revised: And our CIFARNet-based model with only 4 timesteps can also outperform the CIFARNet in [32] with 12 timesteps and the one in [36] with 5 timesteps by 1.67% and 0.79% accuracy, respectively.

Original: These comparison results clearly show the effective latency reducing the ability of our method.
Revised: These comparison results clearly show the effective latency reducing the ability of our method.

Original: And the same situation can also be seen from the results on a more complex dataset, CIFAR100.
Revised: And the same situation can also be seen from the results on a more complex dataset, CIFAR100.

Original: ResNet-34 and VGG-16 were used as backbones on ImageNet.
Revised: ResNet-34 and VGG-16 were used as backbones on ImageNet.

Original: Our ResNet-34 and VGGachieve 67.43% and 70.03%
Revised: Our ResNet-34 and VGGachieve 67.43% and 70.03%

Original: top-1 accuracy with only 6 and 5 timesteps, respectively.
Revised: top-1 accuracy with only 6 and 5 timesteps, respectively.

Original: Compared to
Revised: Compared to

Original: STBP-tdBN and TET, our ResNet-34 obtains 3.71% and 2.64% accuracy improvement, respectively.
Revised: STBP-tdBN and TET, our ResNet-34 obtains 3.71% and 2.64% accuracy improvement, respectively.

Original: Compared to Diet-SNN [22] , our VGG-16 obtains 1.65% accuracy improvement.
Revised: Compared to Diet-SNN [25] , our VGG-16 obtains 1.65% accuracy improvement.

Original: The Dspike is some better than our method.
Revised: The Dspike is some better than our method.

Original: However, the Dspike is very time-consuming than our method since it is computed by finite difference manner.
Revised: However, the Dspike is very time-consuming than our method since it is computed by finite difference manner.

Original: Evaluating finite difference could be time-consuming, where for each single weight, the model should run two times to evaluate the difference of the loss, and a model can have more than ten million parameters.
Revised: Evaluating finite difference could be time-consuming, where for each single weight, the model should run two times to evaluate the difference of the loss, and a model can have more than ten million parameters.

Original: To further verify the effectiveness of our methods, the neuromorphic dataset, CIFAR10-DVS, was also used to evaluate the proposed method.
Revised: To further verify the effectiveness of our methods, the neuromorphic dataset, CIFAR10-DVS, was also used to evaluate the proposed method.

Original: With ResNet-19 as the backbone, it achieves the best performance with 72.60% accuracy in 10 timesteps, which is 4.80% higher than STBP
Revised: With ResNet-19 as the backbone, it achieves the best performance with 72.60% accuracy in 10 timesteps, which is 4.80% higher than STBP

Original: [Table]
Revised: [Table]

Original:  Conclusion
Revised:  Conclusion

Original: This work aims at addressing two main problems for training accurate and deep SNNs: (i) the information flow loss caused by the 0/1 spike quantization; (ii) the training obstacle caused by the non-differentiable spike activity function.
Revised: This work aims at addressing two main problems for training accurate and deep SNNs: (i) the information flow loss caused by the 0/1 spike quantization; (ii) the training obstacle caused by the non-differentiable spike activity function.

Original: Inspired by the correlation between membrane potentials and binary spike signals from the view of information entropy, we design a novel loss, IM-Loss, which can directly maximize the information expressiveness of an SNN, so that a well-performed
Revised: Inspired by the correlation between membrane potentials and binary spike signals from the view of information entropy, we design a novel loss, IM-Loss, which can directly maximize the information expressiveness of an SNN, so that a well-performed

Original: SNN can be obtained.
Revised: SNN can be obtained.

Original: It also plays part of the normalization’s role but does not require additional operations or well-designed initialization strategies.
Revised: It also plays part of the normalization’s role but does not require additional operations or well-designed initialization strategies.

Original: Then, by incorporating a dynamically changing coefficient into a differentiable asymptotic function, we introduce the ESG method for training SNN in a more appropriate way.
Revised: Then, by incorporating a dynamically changing coefficient into a differentiable asymptotic function, we introduce the ESG method for training SNN in a more appropriate way.

Original: ESG encourages the SNN to enjoy both easy convergence and high accuracy by adjusting the weight updating ability and the accuracy of the gradients.
Revised: ESG encourages the SNN to enjoy both easy convergence and high accuracy by adjusting the weight updating ability and the accuracy of the gradients.

Original: Combining these two methods, our method outperforms the state-of-the-art methods.
Revised: Combining these two methods, our method outperforms the state-of-the-art methods.

Original: Acknowledgment
Revised: Acknowledgment

Original: This work is supported by grants from the National Natural Science Foundation of China under contracts No.12202412 and No.12202413.
Revised: This work is supported by grants from the National Natural Science Foundation of China under contracts No.12202412 and No.12202413.