Paper: Towards Energy Efficient Spiking Neural Networks: An Unstructured Pruning Framework (ICLR-2024)

Reviewer comment (Theory):
The difference between ANNs and SNNs in pruning (especially unstructured neuron pruning) is not fully explained. It is not clear why unstructured neuron pruning is not used in ANNs. Why is it more suitable for SNNs?

Author response:
Thanks for pointing it out! The reason why unstructured neuron pruning is more suitable for SNNs lies in the sparse computing manner of SNNs and neuromorphic hardware. In ANNs, neurons primarily function as stateless activation functions, such as Sigmoid or ReLU. These activation functions typically yield limited output sparsity. Consequently, ANNs heavily rely on dense matrix computations and are commonly implemented on hardware architectures optimized for dense computing, such as GPUs. These architectures are not designed to effectively exploit the unstructured sparsity of neurons. In contrast, SNNs employ sparse event-driven computing as their neurons communicate with each other by firing sparse, discrete spikes. Thus, SNNs are typically implemented on neuromorphic hardware that is highly optimized for sparsity. Therefore, unstructured neuron pruning is more suitable for SNNs.

It is important to note that the above analysis is only for the common case. In fact, some work has successfully utilized sparsity in ANNs by designing novel hardware [1], utilizing AVX instructions on CPUs [2], or designing new algorithms for GPUs [3]. We highly appreciate these efforts. However, it is crucial to note that these methods of exploiting sparsity are not widely adopted yet. Therefore, the analysis presented above is generally reasonable as it applies to most cases.

We have added this detailed analysis in the appendix.

Reviewer comment (Reproducibility):
The authors mainly illustrate how to optimize the penalty, but there was no explanation of how to optimize the first term of the loss function in Eq.11.

Author response:
If we understand correctly, the first term of the loss function here refers to $\sigma(\beta{\boldsymbol \alpha}_w)\odot{\boldsymbol w}$, and you are asking why the element-wise multiplication form in this term isn't improved as the penalty term does. In fact, the form of this term does not require further improvement, since the element-wise multiplication here is not problematic.

This is because the loss function term is different from the penalty term. The original form of the penalty term in Eq. 8 is a simple summation of the products of the three types of elements. For each of these products, if one of the three elements takes 0, the product achieves the minimum value of 0. Therefore, this penalty term tends to prune only one type of element, making it an ill-posed problem that is very sensitive to initial conditions. In contrast, the loss function term measures the loss between the target and the model output. From the perspective of the term $\sigma(\beta{\boldsymbol \alpha}_w)\odot{\boldsymbol w}$, the loss function term is a complex nonlinear term with respect to $\sigma(\beta{\boldsymbol \alpha}_w)\odot{\boldsymbol w}$, thus there is not a trivial solution where $\boldsymbol \alpha_w$ or $\boldsymbol w$ being 0 would make the loss function term achieve the minimum value of 0. Thus using element-wise multiplication here is not problematic.

For how to solve the optimization problem in Eq. 11, please refer to the pseudo-code using the stochastic gradient descent (SGD) method in Section A.1 on page 14.

Reviewer comment (Reproducibility):
How to optimize the first term of the loss function in Eq.11?

Author response:
If we understand correctly, the first term of the loss function here refers to $\sigma(\beta{\boldsymbol \alpha}_w)\odot{\boldsymbol w}$, and you are asking why the element-wise multiplication form in this term isn't improved as the penalty term does. In fact, the form of this term does not require further improvement, since the element-wise multiplication here is not problematic.

This is because the loss function term is different from the penalty term. The original form of the penalty term in Eq. 8 is a simple summation of the products of the three types of elements. For each of these products, if one of the three elements takes 0, the product achieves the minimum value of 0. Therefore, this penalty term tends to prune only one type of element, making it an ill-posed problem that is very sensitive to initial conditions. In contrast, the loss function term measures the loss between the target and the model output. From the perspective of the term $\sigma(\beta{\boldsymbol \alpha}_w)\odot{\boldsymbol w}$, the loss function term is a complex nonlinear term with respect to $\sigma(\beta{\boldsymbol \alpha}_w)\odot{\boldsymbol w}$, thus there is not a trivial solution where $\boldsymbol \alpha_w$ or $\boldsymbol w$ being 0 would make the loss function term achieve the minimum value of 0. Thus using element-wise multiplication here is not problematic.

For how to solve the optimization problem in Eq. 11, please refer to the pseudo-code using the stochastic gradient descent (SGD) method in Section A.1 on page 14.

Reviewer comment (Experiments):
How to choose the parameter $\beta$ for different datasets?

Author response:
We would like to clarify that there is no necessity to choose different values of $\beta$ for different datasets, as the parameter $\beta$ is robust across different datasets. We consistently utilize the settings of $\beta_0=5$ and $\beta_T=1000$ in all of our experiments conducted on different datasets. Experimental results show that our parameter setting achieves good performance across diverse datasets, demonstrating the robustness of parameter $\beta$. Moreover, this robustness facilitates the application of our method to different tasks without the need for additional hyperparameter searches.

Reviewer comment (Evaluation):
The cost function associated with number of synaptic operations ignores the cost associated with managing the sparsity. In CNNs it is well known that it is hard to take advantage of irregular non-structured sparsity (see e.g., https://arxiv.org/pdf/1907.02124.pdf) and this is somewhat well understood in SNNs as well. Many hardware accelerators claim to try to take advantage of sparsity, but the cost associated with managing the sparsity reduces the overall energy benefits. See https://arxiv.org/pdf/2309.03388.pdf for an example of a good paper that identifies a high-level energy model of SNNs which captures this issue by more accurately capturing the associated memory cost of managing an SNN.

Author response:
Thank you for your valuable feedback on our work. Your insights on accelerator architectures are greatly appreciated and will help us to improve our paper. We agree with you that our model may not be suitable for all architectures, since many architectures claim that they take advantage of sparsity but have not fully exploited it, leading to additional costs for managing the sparsity. In the following, we discuss in detail which architectures are suited to use SOPs as an energy consumption metric and which are not.

First, we would like to emphasize the significance of SOPs as an important metric in both the hardware and software domains. Many works on neuromorphic hardware provide the energy consumption per SOP (or equally, energy per synaptic spike op, energy per connection, SOPs per second per Watt (SOPS/W), etc.) as an important metric for assessing hardware energy efficiency. Notable examples include Loihi [1] (23.6pJ per synaptic spike op), TrueNorth [2] (46 GSOPS/W (typical), 400 GSOPS/W (max)), and TianJic [3] (649 GSOPS/W (max)). As stated in [4], "Since synaptic processing dominates the system energy cost, this (denotes energy per (synaptic) connection) is the best guide to the overall energy efficiency of the system." For this reason, many SNN works use SOPs as the metric for estimating their energy consumption estimation, such as ESLSNN [5] compared in our manuscript. Therefore, it is reasonable to use SOPs as a meaningful energy consumption metric.

While SOPs is an important metric, we recognize that it does not cover all aspects of hardware energy consumption like memory access and data movement. Therefore, for architectures where other aspects of energy consumption, such as memory accesses, dominate and are not proportional to SOPs, it may not be accurate to use SOPs as the only metric of energy consumption. We list two types of these architectures. The first is the memory access-dominated architectures, such as SATA [6]. As demonstrated in [7], the energy consumption associated with memory access in SATA is significantly higher than that of computing and significantly depends on the dataflow. The second is the analog-based architectures such as [8]. Architectures with analog synapses typically consume much more energy to generate a spike compared to synaptic computation, thus their energy consumption may not be proportional to SOPs.

However, we believe that there are architectures that match our energy model due to their full exploitation of sparsity. By fully utilizing sparsity, the energy consumption of the whole system may be reduced to a level that is linear to the sparsity of synaptic operations, if the other aspects of energy consumption are also sparsely triggered by the synaptic operations. However, due to the wide variety of hardware architectures, it is almost impossible to distinguish which architecture can fully exploit sparsity by summarizing certain characteristics like many-core, weight reuse, the role of dataflow, etc. Thus, we list two different architectures that we believe may match our energy model and discuss which kind of architectures may match our energy model by analyzing their characteristics.

1. Loihi [1]. Loihi is a highly optimized neuromorphic chip designed for efficient neuromorphic computing. It incorporates two key features to enhance energy efficiency: local memory access and asynchronous spike communication. By colocating memory and computing within the core, Loihi minimizes the need for global memory access and reduces data travel distances. Although the detailed architecture design of Loihi has not been made public, the Loihi team claims that Loihi’s neuromorphic mesh and cores are built with asynchronous circuits and, at the transistor level, communicate event-driven tokens of information between logic stages, which helps utilize the sparsity [9]. While there is no direct evidence specific to SNNs, experimental results on other tasks indicate that the energy consumption of Loihi is roughly proportional to the computing.
2. SpinalFlow [10]: SpinalFlow differs significantly from Loihi. It is based on the DCNN accelerator Eyeriss [11], making its overall architecture more similar to an ANN accelerator rather than neuromorphic hardware. It doesn't have local memory access and asynchronous communication features like Loihi. Instead, it employs a global buffer and is built with the synchronous circuit. SpinalFlow employs an SNN-tailored dataflow and increases the reuse of neuron potential, input spikes, and weights. The architecture is designed to effectively leverage the inherent sparsity in spike-based computations. Experimental results demonstrate that SpinalFlow successfully takes advantage of sparsity and reduces energy consumption to a level proportional to sparsity.

Reviewer comment (Evaluation):
Which accelerator architectures match well to using SOP as a metric and which do not?

Author response:
Thank you for your valuable feedback on our work. Your insights on accelerator architectures are greatly appreciated and will help us to improve our paper. We agree with you that our model may not be suitable for all architectures, since many architectures claim that they take advantage of sparsity but have not fully exploited it, leading to additional costs for managing the sparsity. In the following, we discuss in detail which architectures are suited to use SOPs as an energy consumption metric and which are not.

First, we would like to emphasize the significance of SOPs as an important metric in both the hardware and software domains. Many works on neuromorphic hardware provide the energy consumption per SOP (or equally, energy per synaptic spike op, energy per connection, SOPs per second per Watt (SOPS/W), etc.) as an important metric for assessing hardware energy efficiency. Notable examples include Loihi [1] (23.6pJ per synaptic spike op), TrueNorth [2] (46 GSOPS/W (typical), 400 GSOPS/W (max)), and TianJic [3] (649 GSOPS/W (max)). As stated in [4], "Since synaptic processing dominates the system energy cost, this (denotes energy per (synaptic) connection) is the best guide to the overall energy efficiency of the system." For this reason, many SNN works use SOPs as the metric for estimating their energy consumption estimation, such as ESLSNN [5] compared in our manuscript. Therefore, it is reasonable to use SOPs as a meaningful energy consumption metric.

While SOPs is an important metric, we recognize that it does not cover all aspects of hardware energy consumption like memory access and data movement. Therefore, for architectures where other aspects of energy consumption, such as memory accesses, dominate and are not proportional to SOPs, it may not be accurate to use SOPs as the only metric of energy consumption. We list two types of these architectures. The first is the memory access-dominated architectures, such as SATA [6]. As demonstrated in [7], the energy consumption associated with memory access in SATA is significantly higher than that of computing and significantly depends on the dataflow. The second is the analog-based architectures such as [8]. Architectures with analog synapses typically consume much more energy to generate a spike compared to synaptic computation, thus their energy consumption may not be proportional to SOPs.

However, we believe that there are architectures that match our energy model due to their full exploitation of sparsity. By fully utilizing sparsity, the energy consumption of the whole system may be reduced to a level that is linear to the sparsity of synaptic operations, if the other aspects of energy consumption are also sparsely triggered by the synaptic operations. However, due to the wide variety of hardware architectures, it is almost impossible to distinguish which architecture can fully exploit sparsity by summarizing certain characteristics like many-core, weight reuse, the role of dataflow, etc. Thus, we list two different architectures that we believe may match our energy model and discuss which kind of architectures may match our energy model by analyzing their characteristics.

1. Loihi [1]. Loihi is a highly optimized neuromorphic chip designed for efficient neuromorphic computing. It incorporates two key features to enhance energy efficiency: local memory access and asynchronous spike communication. By colocating memory and computing within the core, Loihi minimizes the need for global memory access and reduces data travel distances. Although the detailed architecture design of Loihi has not been made public, the Loihi team claims that Loihi’s neuromorphic mesh and cores are built with asynchronous circuits and, at the transistor level, communicate event-driven tokens of information between logic stages, which helps utilize the sparsity [9]. While there is no direct evidence specific to SNNs, experimental results on other tasks indicate that the energy consumption of Loihi is roughly proportional to the computing.
2. SpinalFlow [10]: SpinalFlow differs significantly from Loihi. It is based on the DCNN accelerator Eyeriss [11], making its overall architecture more similar to an ANN accelerator rather than neuromorphic hardware. It doesn't have local memory access and asynchronous communication features like Loihi. Instead, it employs a global buffer and is built with the synchronous circuit. SpinalFlow employs an SNN-tailored dataflow and increases the reuse of neuron potential, input spikes, and weights. The architecture is designed to effectively leverage the inherent sparsity in spike-based computations. Experimental results demonstrate that SpinalFlow successfully takes advantage of sparsity and reduces energy consumption to a level proportional to sparsity.

Reviewer comment (Writing):
Some portions of the paper lacks clarity. For example, Figure 1 is not sufficiently elucidated. The authors illustrate in the motivation section that Figure 1 shows how fine-grained pruning significantly reduces synaptic connections, but it is challenging to grasp. Furthermore, some variables are not clearly defined, such as $d_n$ and $d_w$ in Equation (6).

Author response:
> Figure 1 is not sufficiently elucidated.

Thanks for pointing it out. The purpose of Figure 1 is to provide readers with a visual representation of fine-grained pruning. The left figure shows neurons, weights, and synaptic connections in a convolutional layer with $5\times 5$ input and a $3\times 3$ filter.  In order to avoid visual clutter caused by an excessive number of synaptic connections, we have chosen to display only 9 synaptic connections that correspond to one of the postsynaptic neurons. The right figure shows the effect of different pruning techniques on the number of synaptic connections. Each subfigure in the right figure is labeled with the number of synaptic connections that remain after pruning. Similarly, due to the excessive number of synaptic connections, we have not visualized the synaptic connections, which seems to have caused confusion. We have added these notes in the revised version.

> some variables are not clearly defined, such as $d_n$ and $d_w$ in Equation (6).

Thanks for your comment. In Equation (6), $d_n$ and $d_w$ denote the number of neurons and weights in the network $f$, respectively. We have added these notes in the revised version.

Reviewer comment (Writing):
There are some grammar errors that require improvement.  
Page 4: 'better trade-off' should be 'a better trade-off',  
Page 7: 'refers to' should be ' refer to',  
Page 9: 'outperformes' should be 'outperforms'.

Author response:
> There are some grammar errors that require improvement.
> Before Equation (6), it appears that the order may be reversed.

Thank you for pointing out these errors. We have fixed them in the revised version.

Reviewer comment (Theory):
Why there is not a element-wise multiplication for $m_n$ in Equation (6)? Why it is different from the masks of weights $m_w$?

Author response:
> Why there is not an element-wise multiplication for $m_n$ in Equation (6)? Why it is different from the masks of weights $m_w$?

This is because neurons are not considered as learnable parameters within the network, and as a result, they do not appear in Equation 6. Therefore, we only include $\boldsymbol m_n$ to denote the mask of neurons in Equation 6.

Reviewer comment (Novelty):
The central idea doesn't appear to offer a breakthrough. The hyperparameter search may not sufficiently differentiate this work. Although combining energy factors into training is a novel approach for SNNs, it has previously been utilized in other contexts, such as artificial neural networks, for various objectives. Such as in the following paper: Salehinejad, H., & Valaee, S. (2021). Edropout: Energy-based dropout and pruning of deep neural networks. IEEE Transactions on Neural Networks and Learning Systems, 33(10), 5279-5292.

Author response:
We would like to clarify that the concept of "energy" in the energy consumption model proposed in our paper and the "energy" referred to in [1] are distinct and unrelated. In our paper, the energy consumption model is a model for estimating the energy consumption of SNNs, with "energy" specifically referring to the energy consumption of the network itself. On the other hand, the Energy-Based Model (EBM) mentioned in [1] is a type of machine learning model where "energy" represents a mathematical function that measures the compatibility between input data and the model's parameters, without any connection to energy consumption. Therefore, it is important to note that our proposed method and [1] are entirely different approaches with distinct goals. Our method mainly focuses on reducing the energy consumption of SNNs, while the approach in [1] is designed to reduce the model parameters.

Reviewer comment (Experiments):
The comparison with the state-of-the-art, especially concerning the ImageNet dataset, lacks comprehensiveness. Only one previous work is considered.

Author response:
Thanks for pointing it out! Actually, there is limited existing work on SNN pruning, and currently, only STDS [2], the state-of-the-art SNN pruning method, provides experimental results on the ImageNet dataset. As a result, we primarily focus on comparing our method with STDS. It is worth noting that the original paper of STDS mentions the results of other pruning methods (GradR [3] and ADMM [4]) on the ImageNet dataset, which were implemented by the authors themselves. However, the performance of these methods significantly lags behind STDS, and no relevant code implementations are provided. Consequently, we have not included these methods in our comparison.

To provide a more comprehensive evaluation, we are working on reproducing the results of these pruning methods (GadR and ADMM) by our implementation. Unlike the comparison in STDS, our comparison requires additional metrics such as the number of SOPs and connections, in addition to the accuracy and number of parameters already provided. Therefore it is necessary to reproduce these experiments and obtain the trained models to measure these metrics, which is a time-consuming process. We have reproduced the experiments of ADMM and have presented the results in Table R1. The experimental results indicate that compared to both STDS and ADMM, our method achieves comparable performance while significantly reducing energy consumption. 

We have added the comparison in the appendix. The GradR experiments are still in progress and we will report the results as soon as they are available.

**Table R1. Comparison on ImageNet dataset**

| Method   | Arch.        | T    | Top-1 Acc. (%) | Acc. Loss (%) | Avg. SOPs (M) | Ratio     | Conn. (%) | Param (M) |
| -------- | ------------ | ---- | -------------- | ------------- | ------------- | --------- | --------- | --------- |
| ADMM     | SEW ResNet18 | 4    | 59.48          | -3.74         | 402.75        | 3.33      | 29.25     | 1.94      |
| ADMM     | SEW ResNet18 | 4    | 55.85          | -7.37         | 262.39        | 5.11      | 21.04     | 1.24      |
| ADMM     | SEW ResNet18 | 4    | 50.57          | -12.65        | 173.10        | 7.76      | 15.68     | 0.75      |
| STDS     | SEW ResNet18 | 4    | 62.84          | -0.34         | 1082.60       | 1.24      | 77.54     | 7.42      |
| STDS     | SEW ResNet18 | 4    | 61.51          | -1.67         | 512.95        | 2.61      | 34.79     | 2.38      |
| STDS     | SEW ResNet18 | 4    | 61.30          | -1.88         | 424.07        | 3.15      | 29.29     | 1.94      |
| STDS     | SEW ResNet18 | 4    | 59.93          | -3.25         | 273.39        | 4.89      | 21.08     | 1.24      |
| STDS     | SEW ResNet18 | 4    | 58.06          | -5.12         | 182.49        | 7.33      | 15.71     | 0.75      |
| STDS     | SEW ResNet18 | 4    | 56.28          | -6.90         | 134.64        | 9.94      | 12.95     | 0.52      |
| **Ours** | SEW ResNet18 | 4    | **61.89**      | **-1.29**     | **311.70**    | **4.29**  | **28.60** | 3.93      |
| **Ours** | SEW ResNet18 | 4    | **60.00**      | **-3.18**     | **177.99**    | **7.52**  | **19.31** | 3.10      |
| **Ours** | SEW ResNet18 | 4    | **58.99**      | **-4.19**     | **116.88**    | **11.45** | **14.19** | 2.62      |

Reviewer comment (Theory):
In paragraph 3 of Section 5.2, the proposed energy model is claimed to be validated by energy estimates of GradR and STDS. However, as the energy measure used (SOPs) is itself defined by the energy model, this claim seems circular.

Author response:
We apologize for any confusion caused by our previous statements. We would like to clarify our intended meaning. In our analysis, we have found that the number of SOPs is proportional to the number of connections. Our energy consumption model, as described in Equation 1, aims to refine the SOPs into two components: the number of spikes and the number of connections. Assuming that the firing rates of neurons remain constant, we expect the SOPs to be proportional to the number of connections. The results we have obtained (41.89M SOPs/2.15% connections vs. 26.81M SOPs/1.35% connections) confirm our assumption and provide validation for our objective of reducing energy consumption by decreasing the number of connections. We have altered the expression in the revised version.

Reviewer comment (Evaluation):
In Table 2 (comparison with state-of-the-art), the accuracy for GradR-CIFAR10 and STDS-ImageNet are obtained from the original papers, but the percentage of connections reported differs from the original papers. Please clarify these differences (i.e. if there is a difference in definition).

Author response:
Yes, there is a difference in the definition of "connections" in our paper compared to GradR and STDS. In our paper, "connections" specifically refers to synaptic connections, as described in Section 3.1 of the Energy Consumption Model. On the other hand, in GradR and STDS, "connections" actually refers to the weights of the model. The "Connection (%)" data in the original paper of GradR corresponds to the "Param." in Table 2, which is calculated by multiplying the "Connection (%)" obtained from the original paper with the number of parameters of the dense model.

We use a different definition of "connection" in order to distinguish the synaptic connections and their corresponding weights. It is important to note that weights can be reused in multiple connections, whereas synaptic connections represent the actual connections between neurons. For instance, in the convolution layer illustrated in Figure 1 of the main text, there are 169 synaptic connections, while the weights are only 9. 

To clarify this distinction, we have provided an explanation in the footnotes of Table 2 in the revised version of our paper.