Paper: TAB: Temporal Accumulated Batch Normalization in Spiking Neural Networks (ICLR-2024)

Reviewer comment (Evaluation):
The paper lacks a detailed analysis of the computational complexity of TAB, which could be crucial for practical applications. Although it mentions that TAB requires more computations than other normalization techniques, it does not provide a detailed cost analysis or the impact of different hyperparameters on computational complexity, making it difficult to evaluate its practicality in real-world applications.

Author response:
Thanks for the advice. The Table summarizes the Complexity Cost of different BN methods, including computational complexity and Memory Cost. 


| BN methods | Computational Complexity | Memory Cost | 
| -------|------------| -------------| 
| Conventional BN | $\mathcal{O}(BCHW) $  |  $\mathcal{O}(CHW)$ |  
| BNTT [1] | $\mathcal{O}(TBCHW)$ |  $\mathcal{O}(TCHW)$ | 
| tdBN [2] | $\mathcal{O}(TBCHW)$ | $\mathcal{O}(CHW)$ | 
| TEBN [3] | $\mathcal{O}(TBCHW)$  |  $\mathcal{O}(CHW) + \mathcal{O}(T)$ | 
| TAB (ours) | $\mathcal{O}(TBCHW)$ |  $\mathcal{O}(CHW) + \mathcal{O}(T)$ | 


- (1) We have obtained this Complexity analysis based on the statistics and parameters of different BN methods in Table 1 in the main paper. We use here T to refer to the number of time-steps, B to refer to the batch size, C,H,W are the number of channels, height, and width. So the input at time-step t, $X_t$, to the normalization layer in SNNs has a dimension of $\mathbf{R}^{C \times H \times W}$.  

- (2) In the conventional BN method in ANNs, there is no time dimension, therefore, the traditional BN method in ANNs has the computational complexity of $\mathcal{O}(BCHW) $, and a Memory Cost of $\mathcal{O}(CHW)$ to save the mean and variance which have the same dimension as $X$. 


**Memory Cost:**  
- a) As BNTT independently normalizes data at each time-step, it needs $\mathcal{O}(TCHW)$ space to save the mean and variance at each time-step, which is T times more than the traditional BN method, tdBN, and TEBN. 
- b) The tdBN and TEBN jointly normalize data across all time-steps, so they only need $\mathcal{O}(CHW)$ space to save one overall mean and variance which is the same as the traditional BN method for saving mean and variance. 
- c) The total memory cost of tdBN is $\mathcal{O}(CHW)$, as it needs only shared overall batch parameters just as the traditional BN, so there is no additional memory cost for saving the batch parameters. 
- d) The total memory cost of TEBN is $\mathcal{O}(CHW) + \mathcal{O}(T)$, as it needs to save batch parameters just as the traditional BN and additional $T$ values to scale the data at each time-step. But $T$ is usually small compared to $\mathcal{O}(CHW)$, this memory cost can be approximated by $\mathcal{O}(CHW)$, but we still keep the original one in order to show directly the difference of using the additional $T$ memory cost. 
- e) The total memory cost of our proposed TAB method is $\mathcal{O}(CHW) + \mathcal{O}(T)$, as it needs to scale the data as well as TEBN. 


**Computational Complexity:** 
- All the BN methods for SNNs need to deal with all the data spanning the time-dimension, no matter how to get the mean and variance,, all at once (e.g. tdBN and TEBN) or get that in a moving averaging way as in our proposed TAB method, the Computational Complexity is always $\mathcal{O}(TBCHW)$. 


```
[1] Youngeun Kim and Priyadarshini Panda. Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in neuroscience, 2021.
[2] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence, 2021.
[3] Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. In Advances in Neural Information Processing Systems, 2022.
```

Reviewer comment (Evaluation):
What is the impact of TAB on energy efficiency, and how does it compare to other methods for training SNNs? While the authors mention that SNNs are attractive for their energy-efficient computing when implemented on neuromorphic hardware, they do not provide a detailed analysis of the impact of TAB on energy efficiency. This is an important question, as it would help to assess the potential impact of TAB on energy efficiency and identify potential areas for future research.

Author response:
Thank you for your valuable feedback. In response to your suggestion, we have examined the average firing rate, specifically focusing on activity sparsity, to illustrate the network's energy efficiency. Detailed plots illustrating this analysis can be found [here](https://anonymous.4open.science/r/ICLR2024_TAB_Rebuttal-BF2C/spike_density_plot.pdf).

Upon careful examination of the plots, it is evident that the average spiking firing rate hovers around 10%-20%. This observation implies that merely 10%-20% of the neurons are actively firing spikes. In practical terms, this means that only a fraction of the neurons, approximately 10%-20%, are utilized for transmitting information between layers. This level of sparsity underscores the network's remarkable efficiency in transmitting information through the utilization of a minimal subset of neurons. We believe these findings significantly contribute to the energy-efficient nature of our proposed approach.

Reviewer comment (Theory):
Can you provide a more detailed analysis of the computational complexity of TAB and how it compares to other methods?

Author response:
Thanks for the advice. The Table summarizes the Complexity Cost of different BN methods, including computational complexity and Memory Cost. 


| BN methods | Computational Complexity | Memory Cost | 
| -------|------------| -------------| 
| Conventional BN | $\mathcal{O}(BCHW) $  |  $\mathcal{O}(CHW)$ |  
| BNTT [1] | $\mathcal{O}(TBCHW)$ |  $\mathcal{O}(TCHW)$ | 
| tdBN [2] | $\mathcal{O}(TBCHW)$ | $\mathcal{O}(CHW)$ | 
| TEBN [3] | $\mathcal{O}(TBCHW)$  |  $\mathcal{O}(CHW) + \mathcal{O}(T)$ | 
| TAB (ours) | $\mathcal{O}(TBCHW)$ |  $\mathcal{O}(CHW) + \mathcal{O}(T)$ | 


- (1) We have obtained this Complexity analysis based on the statistics and parameters of different BN methods in Table 1 in the main paper. We use here T to refer to the number of time-steps, B to refer to the batch size, C,H,W are the number of channels, height, and width. So the input at time-step t, $X_t$, to the normalization layer in SNNs has a dimension of $\mathbf{R}^{C \times H \times W}$.  

- (2) In the conventional BN method in ANNs, there is no time dimension, therefore, the traditional BN method in ANNs has the computational complexity of $\mathcal{O}(BCHW) $, and a Memory Cost of $\mathcal{O}(CHW)$ to save the mean and variance which have the same dimension as $X$. 


**Memory Cost:**  
- a) As BNTT independently normalizes data at each time-step, it needs $\mathcal{O}(TCHW)$ space to save the mean and variance at each time-step, which is T times more than the traditional BN method, tdBN, and TEBN. 
- b) The tdBN and TEBN jointly normalize data across all time-steps, so they only need $\mathcal{O}(CHW)$ space to save one overall mean and variance which is the same as the traditional BN method for saving mean and variance. 
- c) The total memory cost of tdBN is $\mathcal{O}(CHW)$, as it needs only shared overall batch parameters just as the traditional BN, so there is no additional memory cost for saving the batch parameters. 
- d) The total memory cost of TEBN is $\mathcal{O}(CHW) + \mathcal{O}(T)$, as it needs to save batch parameters just as the traditional BN and additional $T$ values to scale the data at each time-step. But $T$ is usually small compared to $\mathcal{O}(CHW)$, this memory cost can be approximated by $\mathcal{O}(CHW)$, but we still keep the original one in order to show directly the difference of using the additional $T$ memory cost. 
- e) The total memory cost of our proposed TAB method is $\mathcal{O}(CHW) + \mathcal{O}(T)$, as it needs to scale the data as well as TEBN. 


**Computational Complexity:** 
- All the BN methods for SNNs need to deal with all the data spanning the time-dimension, no matter how to get the mean and variance,, all at once (e.g. tdBN and TEBN) or get that in a moving averaging way as in our proposed TAB method, the Computational Complexity is always $\mathcal{O}(TBCHW)$. 


```
[1] Youngeun Kim and Priyadarshini Panda. Revisiting batch normalization for training low-latency deep spiking neural networks from scratch. Frontiers in neuroscience, 2021.
[2] Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu, and Guoqi Li. Going deeper with directly-trained larger spiking neural networks. In Proceedings of the AAAI conference on artificial intelligence, 2021.
[3] Chaoteng Duan, Jianhao Ding, Shiyan Chen, Zhaofei Yu, and Tiejun Huang. Temporal effective batch normalization in spiking neural networks. In Advances in Neural Information Processing Systems, 2022.
```

Reviewer comment (Evaluation):
Can you provide evidence that TAB solves TCS? I think the illustration of distribution is capable of demonstrating the solution.

Author response:
Thank you for your insightful suggestion. 

- (1) In response, we have thoroughly investigated the distribution of presynaptic activities both before and after the application of TAB batch normalization. 

- (2) For our experimentation, we trained a 7-layer CNN model on DVS-CIFAR10 using the TAB method. The temporal distribution of normalization layers was visualized across various time-steps. Notably, all selected time-steps exhibited an almost identical distribution. 

- (3) This observation has shown evidence that TAB solves TCS, leads us to the conclusion that our TAB method effectively contributes to the adjustment of the current distribution in DVS data. These findings provide valuable insights into the adaptive capabilities of TAB in normalizing presynaptic activities in SNNs.

Reviewer comment (Experiments):
The authors mention the phenomenon of covariance concept drift encountered by SNN models, so why don't they verify it on time-domain related datasets?

Author response:
Thanks for your advice. 

- (a) The phenomenon of covariance concept drift encountered by SNN models is raised directly by the LIF spiking neuron model and its neuron dynamics, where each neuron maintains its membrane potential dynamics $u(t)$ over time, "integrates" the received input with a leakage, and fires a spike if the accumulated membrane potential value exceeds a threshold. SNNs use binary spike trains to transmit information between layers. The spike train is a series of 0's and 1's like `spike-train=[0,0,0,1,0,0,1]` with 0 indicating the neuron is silent, and 1 indicating the neuron is active.

- (2) The neuron dynamics refer to the changes in the membrane potential of a neuron over time as it integrates input signals and generates spikes. It can be formulated as a first-order differential equation (ODE), 
$\tau \frac{d u(t)}{d t} = -u(t) + I(t) $. 


- (3) Within SNNs, synaptic currents $I(t)$ are sequentially fed into spiking neurons, with spike-triggered asynchronous currents accumulating in the membrane potential. Whenever this accumulated membrane potential exceeds a threshold, a spike is generated. This temporal dependency on membrane accumulation has the potential to amplify the internal covariate shift across the temporal domain which is termed the Temporal Covariate Shift (TCS) phenomenon. 


- (4) In ANNs, the internal covariate shift (ICS) is incurred due to changes in the distribution of layer inputs caused by updates of preceding layers. In SNNs, this covariate shift does not only come from the layer updating, but also the temporal updating. The Temporal Covariate Shift phenomenon is due to updates of preceding layers and prior time-steps, which specially transpire along the additional temporal dimension compared to ANNs. Therefore, the intertwining of this temporal dependency of membrane potential dynamics $u(t)$ with the TCS phenomenon, presents a significant challenge in direct training of SNNs.

Reviewer comment (Evaluation):
We can see that the improvement on the CIFAR-10 and CIFAR-100 datasets is very small, even just 0.6%. Can the current approach be regarded as a theoretically feasible strategy?

Author response:
- (1) Yes, our proposed TAB approach provides a theoretically feasible strategy of Batch Normalization in training SNNs, as it addresses the temporal covariate shift issue by aligning with neuron dynamics and utilizing temporal accumulated statistics for data normalization. A detailed theoretical analysis has been provided in the main paper.

- (2) Meanwhile, as for the improvement of our proposed TAB method with other SOTA methods, for example on the CIFAR-10 and CIFAR-100 datasets, results of the statistical hypothesis have shown that there is a statistically significant improvement of our TAB method over other SOTA methods. 


- (3) As suggested by *Reviewer ZLsc*, we have conducted a statistical hypothesis testing to check if the improvement of our TAB method is statistically significant over other SOTA methods. We have also done the Wilcoxon signed-rank test. Results show p-value=0.000244140625 which is less than 0.001 (the critical value), indicating that there is a significant difference in the performance between TAB and TEBN, statistically. Please check ***Response to All Reviewers (part 1/3)*** for the details. 


- (4) We also checked the rank of the performance of our TAB method and TEBN, our TAB method always ranks 1 among all the paired TAB-TEBN performance results.

Reviewer comment (Experiments):
In addition, the authors have not looked at the impact of other network skeletons, such as ResNet34.

Author response:
- (1) Thanks for your comments! We have provided the experiment results of ResNet34 on the ImageNet dataset. The following table summarizes the results. 
Please refer to ***Response to All Reviewers (part 3/3)*** for the details.

- (2) Due to the time limit of the rebuttal period, we have enabled training 80 epochs of ResNet-34 on ImageNet with $T=2$, and 10 epochs for $T=4$. 
Although we only train our model for 80 epochs for $T=2$, the TAB method achieves a $1.09\%$ increment on ResNet-34 with smaller latency over TEBN on $T=4$ (i.e. TAB=$ 65.38\% $ with $T=2$  v.s. TEBN=$ 64.29\%$ with $T=4$). Even for only 10 epochs, the accuracy of our TAB method on ImageNet already reaches $ 65.98\%$ for $T=4$, with $1.7\%$ increment compared to TEBN with an accuracy of $64.29\%$ for $T=4$, which shows our TAB method is very promising. 


| Dataset  | Model  |  Methods |  Architecture  |  Time-steps | Accuracy (%) | 
| -------  | -------|------------| -------------| ------------|  ------------| 
| ImageNet | SPIKE-NORM [4] | ANN-to-SNN  |  ResNet-34 |  2500 | 69.96 | 
| ImageNet | RTS [5]  | ANN-to-SNN  |  VGG-16 |  16 | 55.80 |
| ImageNet | QCFS [6] | ANN-to-SNN  | ResNet-34 |  16 | 59.35 |
| ImageNet | SlipReLU [7] | ANN-to-SNN  | ResNet-34 |  32 | 66.61 |
| ImageNet | SNNC-AP [8] | ANN-to-SNN  | ResNet-34 |  32 | 64.54 |
| ImageNet | Hybrid Conversion [9] | Hybrid  | ResNet-34 |  250 | 61.48 |
| ImageNet | TET [10] | Surrogate Gradient | Spiking-ResNet-34 |   6   | 64.79 | 
| ImageNet | tdBN [3] | Surrogate Gradient | ResNet-34 |   6   | 63.72 | 
| ImageNet | TEBN [11] | Surrogate Gradient | ResNet-34 |   4   | 64.29 | 
| ImageNet | TAB (ours) | Surrogate Gradient | ResNet-34  |   4   | 65.98 (10 epochs) |
| ImageNet | TAB (ours) | Surrogate Gradient | ResNet-34  |   2   | 65.38 (80 epochs) |

Reviewer comment (Theory):
Another issue reviewer is concerned about is the relationship between TAB and neuron dynamics. Does the exponential nonlinear operation of the SNN-based model on the time coefficient lead to an error in approximating a first-order linear ODE?

Author response:
- (1) Yes, but there will be an approximation error, but this is due to the discretization process.

- (2) The analytical closed-form solution for the first-order Initial Value Problem of the LIF dynamics ODE is 
    $$
        U(t) =  \exp{(-\frac{t}{\tau}) } \left( \int_0^t \frac{R}{\tau} I(s) \exp{ (\frac{s}{\tau}) } ds + U_0 \right) \ .
    $$
    When the neuron initiates at the value $U_0$ with no further input, i.e., $I(t)=0$, the closed-form solution of the ODE shows that the membrane potential $U(t)$ will start at $U_0$ and exponentially decay with a time constant $\tau$, $U(t) = U_0 \exp{ (-\frac{t}{\tau}) }$. 
    However, for the exponential term, it is absorbed in $\lambda$, as $\lambda = \frac{U(t+\Delta t)}{ U(t)} = \frac{U_0 \exp{ (-\frac{t+\Delta t}{\tau}) }}{ U_0 \exp{ (-\frac{t}{\tau}) }} = \exp{ (-\frac{\Delta t}{\tau}) }$. This relationship enables us to formulate the discretization scheme $U[t+1] = \lambda U[t]$. 


- (3) Through further rigorous derivation, we derive another equivalent form of the closed-form solution for the LIF dynamics, 
    $$
    U(t) = (U_0 -R I_0) \exp{ (-\frac{t}{\tau}) } + R I(t) -\int_0^t R \exp{(\frac{s-t}{\tau}) } dI(s)
    $$
    The commonly used discrete LIF model, as denoted by $U[t] = \lambda U[t-1] + X[t]$, is derived from the first two terms of the discretization version of the closed-form solution.

- (4) Recalling our TAB method normalizes data utilizing temporal accumulated batch statistics ($\mu_{1:t}, \sigma^2_{1:t}$) across an expanding window $[1, t]$, where $\mu_{1:t}$ and $\sigma^2_{1:t}$ represent the temporal accumulated information up to time-step $[t]$. Computation of these temporal accumulated statistics is performed dynamically, employing a moving averaging approach, obviating the need to store batch data from all previous time-steps. 
The utilization of the temporal accumulated batch statistics aligns well with the accumulation mechanism of the membrane potential through the above equation.