Paper: Online Stabilization of Spiking Neural Networks (ICLR-2024)

Reviewer comment (Reproducibility):
If the reviewer understands the OTS method correctly, the threshold $\theta[t]$ is dynamically adjusted for each sample batch during both the training and inference phases. However, the reviewer thinks that $\theta[t]$ should be precalculated and fixed during the inference stage. Otherwise, the batch size will highly influence the performance. Much worsely, the obtained network cannot be implemented on normal neuromorphic chips because the chips do not support calculating the mean and variance. The reviewer suggests the authors to maintain a running $\theta[t]$ used for inference.

Author response:
Thank you for your valuable suggestion. Your understanding is right that the threshold $\theta[t]$ is dynamically adjusted for each sample batch during both the training and inference phases.
We have conducted two extra experiments:
- 1. We have tested the performance of using fixed running $\theta[t]$ on Imagenet, its performance is 64.06\% (original accuracy is 64.14\%) (using the saved model of OSR+OTS). This result shows that your suggestion works well.
- 2. We have tested the performance for $batch size=1$ on Imagenet (using our saved model), and the performance is 62.64\%. Although there is a performance drop, it is still better than the baseline.

In addition, we think that $\theta[t]$ could be dynamical during the inference stage if it is incorporated in the neuron model, since it can be viewed as a neural adaptation mechanism. This is a potential direction for future research.

Reviewer comment (Experiments):
This paper could be regarded as an engineering work. Then the reviewer thinks that the ablation experiments are not abundant.

Author response:
Thank you for mentioning this crucial point. Actually, the experimental baseline (BN(Vanilla) for the Imagenet dataset in Table 1) in the paper is almost the same (with slight difference) as you mentioned in (ii). The details are provided in **Experimental settings** in **Response to All Reviewers**.
- For (i), We have conducted extra experiments on CIFAR10 and CIFAR100 dataset, and the results are shown in the following table:

|                     | CIFAR10 | CIFAR100 |
| ------------------- | ------- | -------- |
| VGG+Vanilla BN      | 92.6    | 75.17    |
| VGG+OSR             | 94.05   | 75.65    |
| VGG+OSR+OTS         | 94.35   | 76.48    |
| Resnet19+Vanilla BN | 92.96   | 73.68    |
| Resnet19+OSR        | 95.14   | 74.03    |
| Resnet19+OSR+OTS    | 95.20   | 77.86    |

Note that Resnet19+OSR on CIFAR100 has a significant performance drop compared with Resnet19+OSR+OTS.
In addition, when conducting these experiments, we find that only adding OSR is sensitive to the $weight$ $decay$ parameter. It often requires lower $weight$ $decay$ to get a better result. The ablation experiments we reported in Table 1 uses the same set of parameters for vanilla BN, BN+OSR, BN+OTS, and BN+OSR+OTS. Maybe turning down the $weight$ $decay$ parameter for BN+OSR will promote its performance, but we do not have enough time to finish tuning this parameter during rebuttal. We are working on this and will provide a better result if we can.

- For (iii), we have tested this ''linear transformation'' on the CIFAR100 dataset using VGGSNN and find it very hard to train. The final result we get is 53.25\%, which is significantly worse than OSR.

Reviewer comment (Experiments):
When OSR is incorporated individually, there appears to be a significant performance degradation. It is important to clarify whether such a phenomenon is typical across various datasets.

Author response:
Thank you for mentioning this crucial point. Actually, the experimental baseline (BN(Vanilla) for the Imagenet dataset in Table 1) in the paper is almost the same (with slight difference) as you mentioned in (ii). The details are provided in **Experimental settings** in **Response to All Reviewers**.
- For (i), We have conducted extra experiments on CIFAR10 and CIFAR100 dataset, and the results are shown in the following table:

|                     | CIFAR10 | CIFAR100 |
| ------------------- | ------- | -------- |
| VGG+Vanilla BN      | 92.6    | 75.17    |
| VGG+OSR             | 94.05   | 75.65    |
| VGG+OSR+OTS         | 94.35   | 76.48    |
| Resnet19+Vanilla BN | 92.96   | 73.68    |
| Resnet19+OSR        | 95.14   | 74.03    |
| Resnet19+OSR+OTS    | 95.20   | 77.86    |

Note that Resnet19+OSR on CIFAR100 has a significant performance drop compared with Resnet19+OSR+OTS.
In addition, when conducting these experiments, we find that only adding OSR is sensitive to the $weight$ $decay$ parameter. It often requires lower $weight$ $decay$ to get a better result. The ablation experiments we reported in Table 1 uses the same set of parameters for vanilla BN, BN+OSR, BN+OTS, and BN+OSR+OTS. Maybe turning down the $weight$ $decay$ parameter for BN+OSR will promote its performance, but we do not have enough time to finish tuning this parameter during rebuttal. We are working on this and will provide a better result if we can.

Reviewer comment (Experiments):
A simpler approach to integrating BN into the online training regime involves calculating the statistics solely based on data from each time step. In this approach, at every time step during training, the normalized I[t] is computed based on $\mu[t]$ and $\sigma[t]$, as illustrated in eq. 8. Additionally, the running mean and variance are also updated at each time step. This vanilla BN method should be considered as the baseline. It would be valuable to assess whether this method outperforms OSR and whether the combination of the baseline with OTS outperforms the combination of OSR with OTS.

Author response:
Thank you for mentioning this crucial point. Actually, the experimental baseline (BN(Vanilla) for the Imagenet dataset in Table 1) in the paper is almost the same (with slight difference) as you mentioned in (ii). The details are provided in **Experimental settings** in **Response to All Reviewers**.
- For (i), We have conducted extra experiments on CIFAR10 and CIFAR100 dataset, and the results are shown in the following table:

|                     | CIFAR10 | CIFAR100 |
| ------------------- | ------- | -------- |
| VGG+Vanilla BN      | 92.6    | 75.17    |
| VGG+OSR             | 94.05   | 75.65    |
| VGG+OSR+OTS         | 94.35   | 76.48    |
| Resnet19+Vanilla BN | 92.96   | 73.68    |
| Resnet19+OSR        | 95.14   | 74.03    |
| Resnet19+OSR+OTS    | 95.20   | 77.86    |

Note that Resnet19+OSR on CIFAR100 has a significant performance drop compared with Resnet19+OSR+OTS.
In addition, when conducting these experiments, we find that only adding OSR is sensitive to the $weight$ $decay$ parameter. It often requires lower $weight$ $decay$ to get a better result. The ablation experiments we reported in Table 1 uses the same set of parameters for vanilla BN, BN+OSR, BN+OTS, and BN+OSR+OTS. Maybe turning down the $weight$ $decay$ parameter for BN+OSR will promote its performance, but we do not have enough time to finish tuning this parameter during rebuttal. We are working on this and will provide a better result if we can.

Reviewer comment (Experiments):
In the backward stage, the authors utilize a "double transformation" trick to facilitate more meaningful backpropagation. What if we directly conduct backprop based on the "linear transformation"? Is OSR better than that?

Author response:
Thank you for mentioning this crucial point. Actually, the experimental baseline (BN(Vanilla) for the Imagenet dataset in Table 1) in the paper is almost the same (with slight difference) as you mentioned in (ii). The details are provided in **Experimental settings** in **Response to All Reviewers**.
- For (i), We have conducted extra experiments on CIFAR10 and CIFAR100 dataset, and the results are shown in the following table:

|                     | CIFAR10 | CIFAR100 |
| ------------------- | ------- | -------- |
| VGG+Vanilla BN      | 92.6    | 75.17    |
| VGG+OSR             | 94.05   | 75.65    |
| VGG+OSR+OTS         | 94.35   | 76.48    |
| Resnet19+Vanilla BN | 92.96   | 73.68    |
| Resnet19+OSR        | 95.14   | 74.03    |
| Resnet19+OSR+OTS    | 95.20   | 77.86    |

Note that Resnet19+OSR on CIFAR100 has a significant performance drop compared with Resnet19+OSR+OTS.
In addition, when conducting these experiments, we find that only adding OSR is sensitive to the $weight$ $decay$ parameter. It often requires lower $weight$ $decay$ to get a better result. The ablation experiments we reported in Table 1 uses the same set of parameters for vanilla BN, BN+OSR, BN+OTS, and BN+OSR+OTS. Maybe turning down the $weight$ $decay$ parameter for BN+OSR will promote its performance, but we do not have enough time to finish tuning this parameter during rebuttal. We are working on this and will provide a better result if we can.

- For (iii), we have tested this ''linear transformation'' on the CIFAR100 dataset using VGGSNN and find it very hard to train. The final result we get is 53.25\%, which is significantly worse than OSR.

Reviewer comment (Writing):
eq.2: $s^{l-1}[t]W^l$ -> $W^ls^{l-1}[t]$, eq.4: $u^{l} [t] (1-s^{l} [t] )$  -> $u^{l}[t] \odot (1-s^{l}[t])$.

Author response:
Thank you for such a careful review. We have revised these equations in the latest version.

Reviewer comment (Writing):
The statement preceding Section 5: “our OTS mechanism helps our OTS mechanism”.

Author response:
Thank you for the careful review. We have changed it to ''our OTS mechanism helps our OSR mechanism'' in the latest version.

Reviewer comment (Experiments):
How to implement the proposed method on NF-Resnet-34 which is a normalizer-free architecture? The reviewer knows that the goal of this experiment is to compare the proposed method and OTTT. But still, the proposed method and normalizer-free nets are totally orthogonal.

Author response:
Thank you for bringing up this point. The NF-Resnet-34 in OTTT uses a normalizer-free network adding membrane potential in the shortcut connection.
Briefly speaking, we keep adding membrane potential in the shortcut connection while replacing the normalizer-free part with our normalization modules.
Specifically, we do not use scaled weight standardization, and drop scaling factors $\alpha, \beta$ along with the corresponding operations to keep variance stable. Instead, we use our normalization modules here.

Reviewer comment (Evaluation):
Although the online training approaches save the memory cost, the proposed method falls short of BPTT in performance.

Author response:
Online training saves memory cost by backpropagating each time-step using information generated at the current time-step or at previous time-steps.
This advantage is the direct origin of its disadvantage that it cannot use the information from future time-steps (compared with BPTT-based training algorithms).
At the same time, it is hard to find a way to completely offset the information from future time-steps in online training, especially when the network has multiple layers (perfect solution such as eligibility traces [1][2] only works for single-layer networks without other mechanisms like BN).
As a result, the performance of online training approaches still fall short of BPTT.

    [1] Bellec, G., Scherr, F., Subramoney, A., Hajek, E., Salaj, D., Legenstein, R., & Maass, W. (2020). A solution to the learning dilemma for recurrent networks of spiking neurons. Nature communications, 11(1), 3625.
    [2] Bohnstingl, T., Woźniak, S., Pantazi, A., & Eleftheriou, E. (2022). Online spatio-temporal learning in deep neural networks. IEEE Transactions on Neural Networks and Learning Systems.

Reviewer comment (Writing):
The online calculation of all-time mean and variance is interesting. However, where is it used? Is it a part of OSR?

Author response:
Yes, it is a part of OSR.
The process of OSR is described as follows:

It is applied on $I[t]$ to get the normalized $\tilde{I}[t]$:.
  During training, first we normalize $I[t]$ to $\hat{I}[t]$ (note $\hat{I}[t]$ passes gradients to $\mu[t]$ and $\sigma^2[t]$):
  $$\hat{I}[t] = \frac{I[t] - \mu[t]}{\sqrt{\sigma^2[t] + \epsilon}}$$
Then we apply linear transformation from $\hat{I}[t]$ to $\tilde{I}[t]$ (note $\tilde{I}[t]$ does not pass gradients to $\mu[t]$ and $\sigma^2[t]$ due to $no\\_grad()$, and $\gamma, \beta$ are learnable):
  $$\tilde{I}[t] = \gamma \cdot \left( \hat{I}[t] \cdot no\\_grad\left( \frac{\sqrt{\sigma^2[t] + \epsilon}}{\sqrt{\hat{\sigma^2} + \epsilon}} \right) + no\\_grad\left(\frac{\mu[t] - \hat{\mu}}{\sqrt{\hat{\sigma^2} + \epsilon}} \right) \right) + \beta$$
  As a result, $\tilde{I}[t] = \gamma \cdot \frac{ I[t] - \hat{\mu}}{\sqrt{\hat{\sigma^2} + \epsilon}} + \beta$ in the forward pass in training. Note that we apply conventional linear transformation $\tilde{I}[t] = \gamma \cdot \frac{ I[t] - \hat{\mu}}{\sqrt{\hat{\sigma^2} + \epsilon}} + \beta$ in the testing phase, so the input goes through the same linear transformations **among all time-steps in both training and inference**.

The calculation of $\mu[t], \sigma^2[t], \hat{\mu}, \hat{\sigma^2}$ during training can be found in **Response to All Reviewers**.

Reviewer comment (Theory):
The authors have made an assumption that the membrane potential follows a normal distribution in OTS. Although experiments have shown the effectiveness of OTS+OSR, I am still curious about the real distribution of the membrane potential.

Author response:
Thanks for your valuable advice. To verify the assumption that the membrane potential follows a normal distribution, we collect the membrane potential of a VGG network trained with OSR and OTS on the CIFAR-10 dataset and visualize the distribution of membrane potentials for each layer and each time step. We display the results in Fig. 3 in the appendix of the revised paper. These distributions displays the shape of bell curves, which similars to normal distributions. Thus, we think it is reasonable to use the assumption of normal distribution in OTS.

Reviewer comment (Experiments):
I suggest to provide the experimental results to verify the Gaussion assumption.

Author response:
Thanks for your advice. In the main content, we make the assumption that the membrane potential have a Gaussian distribution. To verify the assumption, we collect the membrane potential of a VGG network trained with OSR and OTS on the CIFAR-10 dataset and visualize the distribution of membrane potentials for each layer and each time step. We display the results in Fig. 3 in the appendix of the revised paper. These distributions display the shape of bell curves, which similars to Gaussian distributions.

Reviewer comment (Writing):
I did not find the calculation form of $\hat{\mu}$ and $\hat{\sigma}$. I suggest to provide a clear definition of them in the paper.

Author response:
Thank you for mentioning this. We have summarized the calculation of $\mu[t], \sigma^2[t], \hat{\mu}, \hat{\sigma^2}$ during training in **Response to All Reviewers**.

Reviewer comment (Writing):
I did not find the meaning of m in (8).

Author response:
It means the number of elements in a group of normalization.
For an image feature map at time $t$ of size $B \times C \times H \times W$ where $B$ is the batch size, $C$ is the number of channels, $H$ and $W$ are height and width, a group contains all elements in a channel, so $m = B \times H \times W$.

Reviewer comment (Reproducibility):
The presentation of the new idea in the online training setting is not clear at all; it depends on the running mean and running variance, but how to calculate them is not mentioned clearly.

Author response:
Thank you for your constructive feedback. We have clarified the calculation of $\mu[t], \sigma^2[t], \hat{\mu}, \hat{\sigma^2}$ during training in **Response to All Reviewers**.
If you have any further questions, please feel free to ask.

Reviewer comment (Reproducibility):
It is still unclear how to save memory by using the proposed modules.

Author response:
Thank you for your question. Saving memory is a property of online training instead of our proposed modules.
The reason that online training approaches can save memory compared with BPTT approaches is as follows:
- BPTT methods require information of all time-steps in backpropagation since they need to backpropagate gradients from the last time-step to the first time-step. As a result, they need to save gradient information of all time-steps during training, which is $O(T)$ where $T$ denotes the total number of time-steps.
- Online training algorithms backpropagates at each time-step, and they only require information at (or before) the current time-step. If information before the current time-step is required, they can be aggregated to some variables (called eligibility traces in prior works) costing constant memory. Overall, it only require $O(1)$ memory cost with respect to $T$.

Reviewer comment (Theory):
In the theoretical parts, the authors tend to make unrealistic assumptions, such as assuming the spike train is iid and expecting the weights to be iid.

Author response:
It should be mentioned that BN itself relies on the fact that elements to be normalized are i.i.d samples from the training distribution [1][2].
Moreover, some works analyzing BN-like mechanisms make assumptions of independence [3].
As a result, our i.i.d assumptions are not that unrealistic.

    [1] Ioffe, S. (2017). Batch renormalization: Towards reducing minibatch dependence in batch-normalized models. Advances in neural information processing systems, 30.
    [2] Summers, C., & Dinneen, M. J. (2019). Four things everyone should know to improve batch normalization. International Conference on Learning Representations.
    [3] Yan, J., Wan, R., Zhang, X., Zhang, W., Wei, Y., & Sun, J. (2020). Towards stabilizing batch statistics in backward propagation of batch normalization. International Conference on Learning Representations.

Reviewer comment (Theory):
The purpose and usefulness of the conclusions of the theorems are unclear, in particular why we need the expectation of the average of \(u[t]\) and the expectation of the average of \(\sigma[t]\), and how these results help with the main proposed modules.

Author response:
Thank you for your question. The aim of our theoretical analysis is to show that our OTS helps stabilize our OSR mechanism. To achieve this goal, we show that OTS helps reduce the (expectation of) sample variance. 
This variance reduction will lead to more accurate estimation of running mean and running variance (for all time-steps) since it reduces the variance in estimation.
In addition, this variance reduction come from the part of variance of mean among time-steps (lower 1st moment of distribution). This reduces the temporal covariate shift (which indicates the distribution difference) among time-steps.

Reviewer comment (Reproducibility):
How to calculate \(\hat{u}\) and \(\hat{\sigma}\) is not explained.

Author response:
We have summarized the calculation of $\hat{\mu}$ and $\hat{\sigma}$ in the **The calculation of $\mu[t], \sigma^2[t], \hat{\mu}, \hat{\sigma^2}$ during training** part in **Response to All Reviewers**.
In addition, $\gamma$ and $\beta$ are learnable.

Reviewer comment (Reproducibility):
It is unclear whether \(\gamma\) and \(\beta\) are learnable or fixed.

Author response:
We have summarized the calculation of $\hat{\mu}$ and $\hat{\sigma}$ in the **The calculation of $\mu[t], \sigma^2[t], \hat{\mu}, \hat{\sigma^2}$ during training** part in **Response to All Reviewers**.
In addition, $\gamma$ and $\beta$ are learnable.

Reviewer comment (Evaluation):
There seems to be no point to the Online Threshold Stabilizer (OTS) to stabilize the firing rate of each layer.

Author response:
We have plotted the firing rate statistics of different configurations in Figure.2 (b). Results have shown that the firing rate across time-steps is stabilized when OTS is added.

Reviewer comment (Experiments):
There are no experimental results on ResNet on CIFAR datasets.

Author response:
We have conducted experiments of Resnet-19 on CIFAR10 and CIFAR100.
The test accuracy for Resnet-19 is 95.20\% on CIFAR10 and 77.86\% on CIFAR100.