Paper: Spatio-Temporal Approximation: A Training-Free SNN Conversion for Transformers (ICLR-2024)

Reviewer comment (Experiments):
The latency/runtime benefit of SNN-based transformer with different timesteps T needs to be compared to ANN accelerators.

Author response:
As this paper primarily focuses on algorithm development on neuromorphic platforms, our knowledge regarding Transformers on ANN accelerators is limited. We plan to promptly engage with our hardware collaborators to conduct a more comprehensive analysis if possible. Besides, we provide a more detailed explanation of the theoretical energy consumption calculation method used in our work in the response to Weakness 5.

Reviewer comment (Evaluation):
The proposed method can have a high fidelity with a small timestep, but there is still 1% gap compared to ANNs, even with a large T. Compared to the training/calibration-based method, the training-free one shows a higher accuracy gap. (Table2, SNM, Calib on resnet20 can fully recover the accuracy). Can the authors comment on that?

Author response:
We acknowledge the phenomenon you have pointed out. This gap primarily stems from the structural differences between the two ANN backbones, ResNet and Transformer. In ResNet, the non-linearities can be represented as piecewise linear functions, resulting in quantization and truncation errors that cause only minor accuracy loss. However, the non-linear operations in Transformer, such as GELU, softmax, and exp, are more complex. Consequently, we employ a group of neurons (UGOs) to fit and approximate these complex functions. The errors introduced during this approximation can never be recovered by increasing timesteps, but can be further optimized during pre-training UGOs.

Reviewer comment (Evaluation):
Is there any randomness in the spikes-based multiplications given the current data encoding? If there is, the output of the computing result is not deterministic. How robust is it to randomness? To have a deterministic output, the effective resolution will be reduced, thus harming accuracy. Can the authors comment on that?

Author response:
From an algorithmic perspective, with fixed input values, the existing encoding methods adhere to deterministic arithmetic rules. Consequently, the spike results of the input and output on each neuron are always deterministic. Our algorithm only focuses on the rate statistics at the end of all time steps, and the temporal resolution is solely used to adjust the quantization accuracy of spike encoding without other significant affect.

Reviewer comment (Novelty):
How does the spiking-based multiplication differentiate from the standard multiplication mechanism in stochastic computing?

Author response:
We appreciate your insightful observation. Indeed, both stochastic computing and SNN multiplications share the computational objective of encoding firing rates with two bit streams and performing multiplications on them. However, in stochastic computing, the result is influenced by the correlation between the occurrence positions of spikes in the two sequences, leading to instability. In SNNs, where spikes are often sparse, this instability is more pronounced and can result in significant errors, particularly in deep networks when timesteps are insufficient. Our proposed algorithm addresses this challenge by achieving stable and accurate computation of the multiplication of the two sequences, albeit at the cost of increased computation compared with stochastic computing.

Reviewer comment (Novelty):
There are other acceleration methods to speed up and reduce energy consumption by a large factor without sacrificing accuracy, e.g., model compression and better architecture design. Moving to a new hardware platform with 30-40% energy reduction seems not very convincing.

Author response:
The energy reductions mentioned (30%~40%) are not based on actual measurements on a hardware platform, but rather serve as reference values at the algorithmic level. These estimates facilitate performance comparisons with existing SNN conversion works. Specifically, we adopt the typical energy values proposed in the 2014 paper [1], such as $E_{MAC}=4.6pJ, E_{AC}=0.9pJ. These values are outdated as hardware has evolved since then. However, for fair comparisons, many mainstream works in the field, such as [2], [3], [4], still employ these values as the standard. Furthermore, the energy reductions achieved by these existing conversions also ranges from 30% to 50%.

Our contribution primarily lies on enabling the direct deployment of ANN Transformers on neuromorphic hardware. In order to directly transfer pre-trained model parameters, we could not redesign the model structure to optimize energy consumption. However, we may explore some model compression methods in the future that do not compromise the structure.

Thank you again for your valuable feedback, which has allowed us to further elaborate on these weaknesses and provide additional context for our research. We appreciate your time and consideration.

[1] Horowitz, Mark. "1.1 computing's energy problem (and what we can do about it)." 2014 IEEE international solid-state circuits conference digest of technical papers (ISSCC). IEEE, 2014.

[2] Li, Yuhang, et al. "A free lunch from ANN: Towards efficient, accurate spiking neural networks calibration." International conference on machine learning. PMLR, 2021. 

[3] Wang, Yuchen, et al. "Signed neuron with memory: Towards simple, accurate and high-efficient ann-snn conversion." International Joint Conference on Artificial Intelligence. 2022.

[4] Li, Yang, and Yi Zeng. "Efficient and accurate conversion of spiking neural network with burst spikes." arXiv preprint arXiv:2204.13271 (2022).

Reviewer comment (Evaluation):
From Tab.1-2, it seems that the author's approximate fitting methods for nonlinear operations such as LayerNorm require relatively long time-steps ($\geq 32$) to be effectively implemented, which will result in more significant time latency and energy consumption. In addition, even under 256 time-steps, the author's approximate fitting and error correction methods cannot completely eliminate the conversion error (there is a ~1% accuracy loss).

Author response:
1.	Time Latency: The latency observed during ANN-SNN conversion is not solely caused by the approximation method proposed in our work. It primarily stems from the inherent requirement for higher precision in large-scale pre-trained models, which is far less sparse than typical SNN models. Additionally, we would like to highlight that in our subsequent experiments, our method also achieve satisfactory outcomes at T=16. Notably, the performance reduction of our approach is even lower that of the ResNets at the same time step.

| Dataset   | Model      | Method              | ANN Acc. | T=16 | T=32  | T=64  | T=128 | T=256 |
| --------- | ---------- | ------------------- | -------- | ----- | ----- | ----- | ----- | --------- |
| CIFAR-10  | ResNet-50  | Calib. (Li et al.)  | 72.35    |59.71  | 64.08 | 68.13 | 71.04 | 71.19 |
| CIFAR-10  | ResNet-50  | SNM (Wang et al.)   | 72.35    |/  | 58.69 | 61.22 | 70.68 | 70.88 |
| *CIFAR-10* | *ResNet-50* | *Combined Conversion* | *72.35*  |/  | *63.22* | *67.05* | *71.15* | *70.92* |
| CIFAR-10  | ResNet-101 | Calib. (Li et al.)  | 79.64    |31.65  | 38.21 | 55.37 | 67.44 | 71.21 |
| CIFAR-10  | ResNet-101 | SNM (Wang et al.)   | 79.64    |/  | 43.25 | 52.68 | 68.42 | 72.96 |
| *CIFAR-10* | *ResNet-101* | *Combined Conversion* | *79.64*  |/  | *55.03* | *64.62* | *68.37* | *73.64* |
| **CIFAR-10** | **ViT-B/32** | **STA (Ours)**  | **89.74** | **83.92** | 87.71 | 88.20 | 88.29 | 88.34 |
| CIFAR-100 | ResNet-50  | Calib. (Li et al.)  | 41.01    |22.53  | 24.67 | 33.41 | 38.20 | 39.01 |
| CIFAR-100 | ResNet-50  | SNM (Wang et al.)   | 41.01    |/  | 35.64 | 34.71 | 39.95 | 41.13 |
| *CIFAR-100* | *ResNet-50* | *Combined Conversion* | *41.01* |/  | *37.41* | *39.26* | *40.31* | *40.78* |
| **CIFAR-100** | **ViT-B/32** | **STA (Ours)** | **64.26** | **54.12** | 62.55 | 62.74 | 62.98 | 63.01 |
| ImageNet-200     | ResNet-50 | Calib. (Li et al.) | 45.63 | /          | 22.50 | 34.51 | 41.82 | 42.03 |
| ImageNet-200     | ResNet-50 | SNM (Wang et al.) | 45.63 | /          | 25.43 | 38.17 | 42.25 | 42.95 |
| **ImageNet-200** | **ViT-B/32** | **STA (Ours)** | **62.25** | **44.72** | 59.79 | 61.24 | 61.53 | 61.66 |
| **ImageNet** | **ViT-B/32** | **STA (Ours)** | **57.93** | **46.62** | 55.50 | 56.39 | / | / |

2.	Conversion Error under large time-steps: The error in our method is caused by the approximation error of nonlinear operators with our Universal Group Operators (UGOs), as in Fig.6. The limited number of neurons are not sufficient to accurately fit complex functions, so such error cannot be eliminated even with infinite timesteps. Further optimization of the pre-training process of UGO can help alleviate this issue.

Reviewer comment (Evaluation):
Regarding the fitting calculation of LayerNorm and Softmax layers, as well as the self-attention layer error correction calculation in Eq.9, it seems that the calculation steps and costs involved are still relatively large. I think this may hinder the algorithm's practical application.

Author response:
Compared to convolutional networks, the ANN-SNN conversion process for Transformers does indeed incur higher computational costs. The main contribution of our work lies in demonstrating the feasibility of this conversion and exploring its computational cost in preliminary terms. In practical applications, there are various energy optimization methods available, such as spatial pruning and model compression, as well as temporal approaches based on the dynamics of different layers in the network. Furthermore, it is important to note that our SNN design is tailored specifically for hardware implementation, as we have removed nonlinear operators that hinder efficient hardware inference. Such operators are implicitly retained in many other SNN works.

Reviewer comment (Novelty):
I noticed that a previous work [1] achieved similar ANN-SNN Conversion performance to this paper when using BatchNorm layers directly (without involving nonlinear operations) and without error correction for attention modules. So I think the value of this approximate fitting and error correction method still needs to be further evaluated.

[1] Ziqing Wang, Yuetong Fang, Jiahang Cao, Qiang Zhang, Zhongrui Wang, Renjing Xu. Masked Spiking Transformer. ICCV 2023.

Author response:
The key distinction between our work and [1] is that [1] modifies and trains the original ANN model before conversion, which is not involved in our approach. Our goal is to rapidly obtain a fully equivalent SNN counterpart for widely used large-scale pre-trained ANN models and deploy them directly in production environments. This necessitates adopting a training-free approach to prevent any degradation in generalization performance caused by fine-tuning. Furthermore, the reason for not using Batch Normalization is that it is not commonly adopted in mainstream pre-trained Transformer models.

Reviewer comment (Evaluation):
The proposed Universal Group Operators use extensive spiking neurons to model fine-grained ANN operations. This high model complexity reduces power efficiency and incurs large memory usage.

Author response:
We acknowledge your observation on computational cost. However, considering that the operations themselves in ViT are more complex compared to basic ReLU neurons, a consequent decrease in efficiency may be inevitable. Besides, we actively utilize larger memory to achieve higher accuracy within limited timesteps. Further optimization can be conducted to control the power efficiency and memory cost through sparsification on the converted model.

Reviewer comment (Experiments):
Would employing model compression strategies, such as pruning, enhance the efficiency and reduce the size of the Universal Group Operators?

Author response:
You are correct in suggesting that Universal Group Operators (UGO) can be further compressed. In fact, in the UGO model utilized in our paper, a significant portion of the IF neurons were rarely activated. This perspective indicates that the non-linear operations in ViT can be compressed and represented by smaller models. In future work, we plan to optimize these operations targeting language Transformers.

Reviewer comment (Experiments):
The conversion implementation integrates multiple existing ANN-SNN conversion algorithms, including SNM and Burst. Using the same combined conversion for ResNet baselines could enable a more fair comparison.

Author response:
We conducted these evaluations on the CIFAR-10 and CIFAR-100 datasets, supplementing the results in Table 1. The additional results are as follows:

| Dataset   | Model      | Method              | ANN Acc. | T=32  | T=64  | T=128 | T=256 |
| --------- | ---------- | ------------------- | -------- | ----- | ----- | ----- | ----- |
| CIFAR-10  | ResNet-50  | Calib. (Li et al.)  | 72.35    | 64.08 | 68.13 | 71.04 | 71.19 |
| CIFAR-10  | ResNet-50  | SNM (Wang et al.)   | 72.35    | 58.69 | 61.22 | 70.68 | 70.88 |
| CIFAR-10  | ResNet-50  | *Combined Conversion* | *72.35*    | *63.22* | *67.05* | *71.15* | *70.92* |
| CIFAR-10  | ResNet-101 | Calib. (Li et al.)  | 79.64    | 38.21 | 55.37 | 67.44 | 71.21 |
| CIFAR-10  | ResNet-101 | SNM (Wang et al.)   | 79.64    | 43.25 | 52.68 | 68.42 | 72.96 |
| CIFAR-10  | ResNet-101  | *Combined Conversion* | *79.64*    | *55.03* | *64.62* | *68.37* | *73.64* |
| CIFAR-10  | ViT-B/32   | **STA (Ours)**          | **89.74**    | **87.71** | **88.20** | **88.29** | **88.34** |
| CIFAR-100 | ResNet-50  | Calib. (Li et al.)  | 41.01    | 24.67 | 33.41 | 38.20 | 39.01 |
| CIFAR-100 | ResNet-50  | SNM (Wang et al.)   | 41.01    | 35.64 | 34.71 | 39.95 | 41.13 |
| CIFAR-100 | ResNet-50 | *Combined Conversion* | *41.01* | *37.41* | *39.26* | *40.31* | *40.78* |
| CIFAR-100 | ViT-B/32   | **STA (Ours)** | **64.26**    | **62.55** | **62.74** | **62.98** | **63.01** |

In comparison to existing baselines, this conversion approach did not yield significant performance improvements on ResNets. Moreover, it is important to note that even if it achieves accurate conversion, the performance still lags far behind ViT due to the limitations of a weak backbone.

Reviewer comment (Experiments):
What hurdles might one encounter when adapting this technique to language Transformers? Would the method necessitate modifications?

Author response:
Please refer to the response provided for Weakness 2, as it addresses the applicability of our method to language models.

Reviewer comment (Experiments):
What challenges might arise when adapting this method to larger-scale transformer models?

Author response:
This question is also related to Weakness 2. The main engineering challenges we encountered include the increased memory requirements resulting from larger input sizes and wider blocks, as well as the error accumulation introduced by deeper network architectures. Despite these challenges, our conversion method remains applicable since the basic modules are consistent with that of ViTs.