From: ICLR.cc/2023/Conference/Paper2535/Authors
title: Submission Withdrawn by the Authors

From: ICLR.cc/2023/Conference/Paper2535/Reviewer_5NBM
title: Two timesteps
comment: Thanks for your response, and sorry for the delay.

I now understand that there are two sorts of timesteps:
1) The timesteps of the stimulus, which cause one frame per timestep.
2) The shorter timesteps of the SNN: each static frame is processed through a multi-timestep stateful SNN (Eq 1)
And the potentials are reset between the frames.

In my opinion, this is not clearly explained in your manuscript.

To conclude, I maintain my rating of 6.

From: ICLR.cc/2023/Conference/Paper2535/Authors
title: Response to Reviewer Tbtq
comment: We appreciate you further comments and would like to address your questions as follows. 

**Q1: Regarding A1: I do not agree with the authors. Please take a look at a bunch of papers reporting SOTA results on the same datasets. Comparison of the authors data with just two or three outdated results is not fair.**

Thank you for your suggestion. We updated the table have more exiting results as follows for comparison. We also added one more column to show whether the network can have fully asynchronous operation. In contrast, some SNN models fail to meet the criteria for being fully asynchronous due to down-scaling input or analogue activation.

**Cifar10-DVS**

| Models | Methods | F-Async.      | Acc. | Ave. t     |
|    :----:   |          :---: | :---:        |    :----:   |         :---: |
|   Deng et al (2022)       | DT   | x | 83.17%| 1.3s|
|   Meng et al (2022)       | DT   | x | 77.27%| 1.3s|
|   Li et al (2022)  | Graph   | √  | 68.40%| -|
|   Fang et al (2021a)       | DT   | √ | 74.40%| 1.3s|

**N-Caltech101**

| Models | Methods | F-Async.      | Acc. | Ave. t     |
|    :----:   |          :---: | :---:        |    :----:   |         :---: |
|   Schaefer et al (2022)       | Graph   | √ | 66.80%| -|
|   Li et al (2022)  | Graph   | √ | 76.10%| -|

**DVS128 Gesture**
| Models | Methods | F-Async.      | Acc. | Ave. t     |
|    :----:   |          :---: | :---:        |    :----:   |         :---: |
|   Wu et al (2021)       | DT   | x | 97.56%| 1.5s|
|   Fang et al (2021a)  | DT  | √ | 97.92%| 6.0s|

[1] Deng et al. Temporal efficient training of spiking neural network via gradient re-weighting. ICLR, 2022

[2] Meng et al. Training high-performance low-latency spiking neural networks by differentiation on spike representation. CVPR, 2022

[3] Li et al. Graph-based asynchronous event processing for rapid object recognition. ICCV, 2021

[4] Fang et al. Deep residual learning in spiking neural networks. NeurIPS, 2021a

[5]  Schaefer et al. Aegnn: Asynchronous event-based graph neural network. CVPR, 2022

[6]  Wu et al. Liaf-net: Leaky integrate and analog fire network for lightweight and efficient spatiotemporal information processing. TNNLS, 2021.

**Q2: I meant that without knowing for a dataset subject to inference, the proposed method cannot be used. Further, event-data include a significant amount of uncontrollable noises. That is, event-streams from the same object most likely differ for each other, so that each event-stream is likely of different . Given this situation, it feels hard to apply the proposed method.**

We can re-use the training dataset for the estimation of $g(X)$, without requiring a new dataset. So, we do not understand why ``the proposed method cannot be used''. Nevertheless, it is possible that the noise in runtime may cause the estimation of $g(X)$ with training data to be less accurate. In this case, the application of cut-off might lead to the decrease of the accuracy. However, our experimental results show that this is a minor issue, e.g., SNN-ROE remains similar performance on accuracy when applying cutoff, please refer the accuracy for entire spike train in Figure 5 (bracket) and slightly lower accuracy for cutoff in Table 2. Therefore, we would appreciate if you could elicit in more details why you feel the method is hard to apply. 

We hope our new answers have addressed you concerns. Please let us know if you have further questions, the discussion is warmly welcomed that helps improve our work.

From: ICLR.cc/2023/Conference/Paper2535/Authors
title: Response to Reviewer ozZz (2/2) - II
comment: 
**Q5: SNN for robotic control tasks.**

Thanks for the suggested reference. In addition to studying plausibility of SNNs in robotic control, it will be interesting in the future to validate our cutoff in robot control tasks and see if robot control can benefit from adaptive inference time.

**Q6: SNNs themselves, without recurrent, can deal with temporal information, which can be achieved by the membrane potential accumulation and dynamic threshold.**

We do agree with you that the accumulation of membrane potential does affect the temporal prediction. However, we guess (please let us know if we didn't get your point correctly) that "temporal information" you mentioned is about the input dominated by temporal information, such as audio. At this point, we believe that deploying recurrent connection is essential for SNN to achieve high accuracy for such input, which has been introduced in [2]. Note that, the dataset in [2] has small spatial dimension but rich sequential information. In contrast, DVS has much larger spatial resolution, e.g., $128\times128\times2$ for the benchmark in our work. This physical feature in DVS allows us to consider more spatial information during processing.  

[2] Accurate and efficient time-domain classification with adaptive spiking recurrent neural networks, Nature Machine Intelligence, 2021

**Q7: Did the authors process each DVS pixel using a single IF neuron?**

In our work, the input events in one pixel can be regarded as the spikes generated in one neuron and sent to post IF neurons. 

**Q8: In addition, "generating the events when the environment changes" is inaccurate! Camera movement can also trigger events.**

The change of environment is relative to camera. 

We hope our new answers have addressed you concerns. Please let us know if you have further questions, the discussion is warmly welcomed that helps improve our work.

From: ICLR.cc/2023/Conference/Paper2535/Authors
title: Response to Reviewer ozZz (1/2) - II
comment: We appreciate you further comments and would like to address your questions as follows. 

**Q1: ANN-to-SNN itself should not even be a topic of SNN study. SNNs leverage the bioplausible, especially for the temporal domain. ANN cannot really learn the dynamic of the temporal part. As such, ANN-to-SNN does not make sense from a bioplausible perspective. The fact that many accepted papers leverage ANN-to-SNN is true, but it does not mean ANN-to-SNN is meaningful!**

We agree with you that the plausibility is an important feature in SNN, which makes SNN different from ANN. For example, the adaptive threshold benefits SNN in processing temporal information. However, in our work, we are focusing on another significant feature of SNN --- asynchronicity. This feature also differentiates SNN from ANN, and together with our cutoff, it makes SNNs adaptive in inference time and reduces the average latency. 

We respectfully disagree with you that ANN-to-SNN is not meaningful. ANN-to-SNN technique manifest the mature training regime of SNN and bridges the gap between ANN and SNN. To optimise event-driven or asynchronous SNN, as the first attempt, we choose IF neuron as our objective, due to its simplicity, while leveraging the techniques from ANN to improve the theoretical lower bound of SNN toward processing event-based input. The theoretical analysis proposed a new metric - the ratio of threshold to accumulated current - to estimate the performance of SNN. This is not in conflict with extension to other SNN neurons in the future when considering biological plausibility. 

Furthermore, although direct training of SNN considers the biological plausibility during training, it suffers from large memory consuming to train a fast response SNN, which has been argued in our related work. As an evidence, we also have shown that our SNN outperform those LIF-based SNNs using direct training in Table 2, including the results on DVS Gesture. It should be noted that we applied temporal training to ANN-to-SNN to minimise the limitation of ANN-to-SNN in processing temporal input. Figure A-2 in appendix represents the impact of temporal training. 

**Q2: N-imagenet**

We have updated our manuscript to include the reference and will consider it in our future work.

**Q3: The authors also pointed out, "A low power, fully event-based gesture recognition system." The approach requires a sequence for capturing the temporal information, which makes sense to me! However, the other two classification tasks used in the submission do not require temporal information.**

Even for Cifar10-DVS and N-Caltech101 where non-temporal information dominates, they are still characterized by asynchronicity and sparsity. In our work, we are focusing on optimising SNN for event-based input with optimal adaptive inference time.

**Q4: BTW, the energy-efficiency is mainly introduced by TrueNorth hardware.**

We believe that the energy-efficiency in TrueNorth is due to the event-driven manner of SNN. The statement below can found in supplementary of [1].

'The TrueNorth architecture aims to minimize active power using an event-driven design.

Despite using a synchronous flow, the neuron circuit operates in an event-driven manner. That is, the neuron’s clock is generated by the asynchronous Controller block, which issues on-the-fly clock pulses only when necessary for the neuron to execute synaptic integration, leak subtraction, and threshold checking.' 

[1] A million spiking-neuron integrated circuit with a scalable communication network and interface, Science, 2014

From: ICLR.cc/2023/Conference/Paper2535/Reviewer_Tbtq
title: not convincing
comment: Regarding A1: I do not agree with the authors. Please take a look at a bunch of papers reporting SOTA results on the same datasets. Comparison of the authors data with just two or three outdated results is not fair.

Regarding A2: I meant that without knowing $g(x)$ for a dataset subject to inference, the proposed method cannot be used. Further, event-data include a significant amount of uncontrollable noises. That is, event-streams from the same object most likely differ for each other, so that each event-stream is likely of different $g(x)$. Given this situation, it feels hard to apply the proposed method.

From: ICLR.cc/2023/Conference/Paper2535/Reviewer_ozZz
title: Many disagreement!
comment: Thanks for the feedback! But I disagree with many points of the rebuttal!

"Actually, ANN-to-SNN conversion is a popular research direction with many ICLR/ICML papers published in the past years." It does not make sense to me! ANN-to-SNN itself should not even be a topic of SNN study. SNNs leverage the bioplausible, especially for the temporal domain. ANN cannot really learn the dynamic of the temporal part. As such, ANN-to-SNN does not make sense from a bioplausible perspective. The fact that many accepted papers leverage ANN-to-SNN is true, but it does not mean ANN-to-SNN is meaningful! 

Thanks for pointing out the "N-imagenet", I think it is necessary to test the proposed approach on the N-imagenet dataset. The authors also pointed out, "A low power, fully event-based gesture recognition system." The approach requires a sequence
for capturing the temporal information, which makes sense to me! However, the other two classification tasks used in the submission do not require temporal information. BTW, the energy-efficiency is mainly introduced by TrueNorth hardware.    

Recently, many papers have leveraged SNN for robotic control tasks. I would suggest the authors check the following papers. 
1. Reinforcement co-learning of deep and spiking neural networks for energy-efficient mapless navigation with neuromorphic hardware. 
2. Deep reinforcement learning with population-coded spiking neural network for continuous control.
3. Biologically Inspired Dynamic Thresholds for Spiking Neural Networks.

Based on the feedback, I feel the authors do not deeply understand SNNs. SNNs are not a variant of ANN in general, either! SNNs themselves, without recurrent, can deal with temporal information, which can be achieved by the membrane potential accumulation and dynamic threshold.  

My question on directly processing asynchronous output from a DVS camera is about something other than where the asynchronous is from. Did the authors process each DVS pixel using a single IF neuron? In addition, "generating the events when the environment changes" is inaccurate! Camera movement can also trigger events. 

There is no experimental evidence to show the proposed approach is practical with LIF. 

Therefore, I keep my original score and do not suggest accepting submissions.

From: ICLR.cc/2023/Conference/Paper2535/Authors
title: New response to Reviewer 5NBM
comment: We appreciate your increase of rating and the additional comments. We would like to have one more round of discussions with you, as follows. 

**Resetting at every timestep is not necessarily a bad idea, but it has a lot of implications that are not discussed**

Firstly, we need to clarify that $t$ is an analog value, representing time point, which is different from the discrete timestep in prior work. In addition, we use $t$ to multiply with spiking resolution $S_r$ to approximate the timestep that has been introduced in the text below Equation 6. Secondly, frame number $F$ and timestep $tS_r$ are different from each other. Specifically, SNN can have more than one timestep within one frame. We let the duration of one frame be $T$ and the membrane potential could accumulate and reset by subtract though $T$ within each frame. The membrane potential of the hidden layer is reset after each frame, while that of the output layer is reset after the last frame.

We have now updated the statement on "reset to zero" more precise in manuscript.

**Q1: Their SNN does not do any temporal integration, therefore it cannot learn temporal dependencies**

While it is unlikely for SNN/ANN without memory units to deal with temporal dependencies very well, there are two kinds of temporal integration in our SNN as follows 

- Neurons at hidden layer integrate and fire within one frame.

- Neurons at output layer integrate and fire before the last frame. 

**Q2: Their SNN can be seen as a feedforward, stateless, ANN that uses Heaviside activation functions (aka McCulloch \& Pitts neurons aka threshold gates).**

When $F>1$, the membrane potential in output layer can remember the state from previous frames. 

**Q3: It can also be seen as semi-quantized ANN, where activations are quantized on one bit, and weights are not quantized.**

We believe there are major differences with semi-quantized ANN, because in SNN the membrane potential integrates and fires over time.

**Q4-5: Eq 1 is confusing, because V(t-1) is always 0. The soft reset that follows firing is unuseful, because the potential will be zeroed for the next time step in any case.**

V(t-1) and soft reset are needed, as the integrate-and-fire process happens within one frame.

From: ICLR.cc/2023/Conference/Paper2535/Reviewer_5NBM
title: Accuracy is excellent, but the paper is somewhat confusing
comment: Thank you for your answers.
I wasn't aware that you reset all the potentials at each time step!
I see that you now say it in the new version of the manuscript.
Resetting at every timestep is not necessarily a bad idea, but it has a lot of implications that are not discussed:

* Their SNN does not do any temporal integration, therefore it cannot learn temporal dependencies.

* Their SNN can be seen as a feedforward, stateless, ANN that uses Heaviside activation functions (aka McCulloch & Pitts neurons aka threshold gates).

* It can also be seen as semi-quantized ANN, where activations are quantized on one bit, and weights are not quantized.

* Eq 1 is confusing, because V(t-1) is always 0.

* The soft reset that follows firing is unuseful, because the potential will be zeroed for the next time step in any case.

But as I've already said the accuracy they reach is very good!

For all these reasons, my rating is now 6.

From: ICLR.cc/2023/Conference/Paper2535/Authors
title: Response to Reviewer 5NBM (2/2)
comment: **Q4: With F>1 the ANN makes several predictions that are averaged through Eq 4. But the ANN is stateless thus each prediction is independent from the previous ones. This is not the case for the SNN, unless you reset the potentials during the stimulus (you don't, do you?). That causes a discrepancy between the ANN and the SNN which should be discussed.**

In our training for the case of $F>1$, the predictions between frames are independent, and the potential is reset once a frame is processed. We remark that, the reset mechanism is hardware enabled [1-2]. 

**Q5:  Will the authors share their code? Previous work should be included in Table 2?.**

The code is attached as part of the supplementary materials, and is now in a GitHub repository which we will disclose the link once the paper accepted. 

We have updated Table 2 to include the reference you suggested. Thanks!

From: ICLR.cc/2023/Conference/Paper2535/Authors
title: Response to Reviewer ozZz (2/2)
comment: **Q6: The author should provide experimental results to prove Theorem 4.1. I expect the experimental results also show the positive correlation between classification accuracy and $\Big(\sqrt{n^l} \frac{V^l_{thr}}{\Vert \boldsymbol{a}^l \Vert_2} \Big)^{-1}$**

The positive correlation as requested in the comment has been shown in our existing experiments in Appendix. Specifically, 
Figure A-4 shows that the increase of $\alpha$ -- hyper-paramter for ROE -- from 0.0 to 0.003 can increase the $cos(\phi)$ at each layer, while the accuracy loss of SNN at the beginning of inference drops. We also compared the cosine similarity on ROE, QA and COE in Figure A-5, and it is clear that ROE performs better.

From: ICLR.cc/2023/Conference/Paper2535/Authors
title: Response to Reviewer Tbtq
comment: Thank you for your thoughtful comments. In the following, we explain how we have updated our paper by addressing the comments.

**Q1: Benchmark is weak. The authors compared the classification accuracy of ROE with COE and QA which have not been reported in any previous publications. This may undermine the efficacy of ROE. Therefore, the authors should consider systematic and fair comparison of the proposed method with previous methods.**

We respectfully disagree that the benchmark is weak. Actually, we implement the three methods (QA, COE, and ROE) onto the same platform (a state-of-the-art conversion method and several event driven datasets), so that they can compare in a fair way. Specifically, the SNN-QA, introduced at the beginning of the experiment section, replicates an algorithm in [1]. COE removes the floor operation in SNN-QA and does not need to pre-configure the quantization length in training, which eliminates the impact of quantisation on event-based input.

In terms of the comparison results, we also show that ROE can achieve better performance than [2] in Table 2, which applied conversion method on event-based dataset without ROE, QA and COE.

**Q2: Regarding the confidence rate evaluation, I assume that $g(x)$ should be known a priori for all samples to evaluate the confidence rate. That is, this method cannot be used for any test datasets whose $g(x)$ are not available a priori, which significantly undermines the value of the proposed early termination method. Therefore, the proposed early termination method has a certain limitation.**

Theoretically, $g(X)$ is based on the data distribution so we do not know in prior. However, like  most of the machine learning problems, in practice (and in our experiments), we estimate it with a set of data instances sampled from the data distribution.

**Q3: Conversion method is not elaborated. The authors should detail the objective function minimization sequence and all related sub-methods. Particularly, ROE minimization method should be detailed.**

As stated in the paper, for the limitation of spaces we inherit the conversion method and most of the notations from [3] and [1]. Particularly, the relationship between ReLU and IF is from [3] and the relationship between quantised ReLU and IF is from [1]. Moreover, ROE minimisation is based on [3] and operates as a penalty term during ANN training. In the revised manuscript, we have included in the Appendix all the necessary materials. Thank you for pointing this out. 

[1] Bu, Tong et al. Optimal ANNSNN conversion for high-accuracy and ultra-low-latency spiking neural networks. ICLR 2022. 

[2] Kugele, Alexander et al Efficient processing of spatio-temporal data streams with spiking neural networks. Frontiers in Neuroscience, 2020.

[3] Wu, Dengyu  et al. A little energy goes a long way: Build an energyefficient, accurate spiking neural network from convolutional neural network. Frontiers in neuroscience, 2022.

From: ICLR.cc/2023/Conference/Paper2535/Authors
title: Response to Reviewer zEep
comment: **Q1: Overall, the paper applies a lot of prior knowledge and research to the problem of training spiking networks, and goes into explicit detail and proofs for the proposed work. The results show clear improvements over the prior works, although on relatively simple datasets and a simple task (image classification). Also, the proposed method only improves upon prior works when the early cutoff is applied (results without it are mixed). This seems a little surprising that terminating early could have such improvements.**

We appreciate your positive comments, and agree with you that early termination has potential to be applied more widely in the SNN. In the following, we would like to take this opportunity to elicit several settings of our work: 



- Research contributions beyond prior works. Firstly, we design a regulariser to train a low-latency SNN, and support the design with a theoretical analysis. Our Theorem 4.1 suggests that low-latency SNN needs a good distribution of accumulated current, which provides  a new metric to measure the performance of SNN. This is in contrast with previous conversion-based methods [1-3], where the algorithms tend to minimise the Mean Square Error (MSE) between ANN and resulted SNN. Secondly, as a first attempt, the proposed cutoff is hardware-friendly and compatible with different SNN neuron models (e.g. IF and LIF), as long as the prediction is based on the number of output spikes. It also highlights the advantage in SNN, which is more adaptable in inference than traditional ANN. 
    
- Datasets and classification task. Unlike static image dataset, e.g., Cifar10/100, the event-based dataset exhibits two significant features, sparsity and asynchronicity. Together with neuromorphic chip, [4] has shown that these two feastures makes the recognition system consume lower power and have lower latency.  In our work, these unique features are fully utilised by our techniques, i.e.,  regulariser and cutoff. The experimental results show that our approach can improve the performance of SNN for event-based input, in terms of accuracy and latency, which is potential for practical implementation. 

[1] Bu, Tong et al. Optimal ANN-SNN conversion for high-accuracy and ultra-low-latency spiking neural networks. ICLR, 2022.

[2] Deng, Shikuang et al. Optimal conversion of conventional artificial neural networks to spiking neural networks. ICLR, 2021. 

[3] Li, Yuhang et al. A free lunch from ann: Towards efficient, accurate spiking neural networks calibration. ICML, 2021.

[4] Amir, Arnon et al. A low power, fully event-based gesture recognition system. CVPR, 2017.

**Q2: Many symbols are also introduced, and perhaps it would be useful to have a glossary to keep track of them all.**

Thanks. A notation table has been added to the Appendix.

From: ICLR.cc/2023/Conference/Paper2535/Authors
title: Response to Reviewer 5NBM (1/2)
comment: We appreciate the insightful comments from you and would like to address your concerns as follows.

**Q1: The neurons do not leak, thus old inputs cannot be forgotten. Maybe this is not an issue with the datasets they used, because the different patterns to be recognized are in different files, and all the potentials are reset in between (right?). However, it could be an issue if deployed in a "streaming" setting where potentials are never reset. Can the authors elaborate on this? In addition, could their approach work with LIF instead of IF?**

Actually, our method can work with "streaming" input, and in the meantime reset the potentials between frames.  We believe the key technical argument can be on whether or not the reset can be implemented in hardware, i.e., whether or not our method is hardware friendly. For this, we would like to refer to a few recent work on implementing SNN in hardware, including [1] and [2]. 

Both work has a flag signal that indicates the beginning or end of input spike train and can be used to achieve 'Reset in between'. %'Reset in between' can be achieved by using the flat signal. The flag signal in [1] can be found in their open-source design, named 'TIME\_TICK'. Moreover, [2] mentioned the flag signal in page 6 with description, suggesting that 

'This signal can be used to reset membrane potentials to zero if required'

Regarding LIF vs IF, our approach consists of ROE and Cutoff. ROE may limit to IF because it is based on ANN-to-SNN conversion, but cutoff can be easily extended to LIF and direct training, as it only cares about the number of output spikes. Actually, ANN-to-SNN conversion is a popular research direction with many papers published in the past years. More importantly, as we have argued in Related Work, leveraging ANN-to-SNN conversion can result in fast response SNN and improve the latency, both of which are required in practical applications such as edge computing

[1] Frenkel, Charlotte et al. ReckOn: A 28nm Sub-mm2 Task-Agnostic Spiking Recurrent Neural Network Processor Enabling On-Chip Learning over Second-Long Timescales. ISSCC, 2022.

[2] Khodamoradi, Alireza  et al. S2n2: A fpga accelerator for streaming spiking neural networks. ACM/SIGDA,  2021.

**Q2: Conversely, IF is the way to go with static images. Could their method work with static images?**

Our method can work with static images, which is not surprising. However, the paper is  focused on the dedicated features of DVS camera inputs (i.e., asynchronicity and sparsity), which we believe contribute as an important -- and specialised -- application scenario for SNNs. 

**Q3: The ANN that is trained ignores the time, in the sense that shuffling the frames in Eq 4 would have no effect on the loss. This wouldn't work on a truly temporal task in which the order of the sequence matter. The most temporal dataset here is DVS Gesture, but even with this one, most gestures can be categorized based on spatial information only (even the clockwise and counterclockwise hand gesture can usually be distinguished if using the event polarities).**

While considering a fully temporal application is tempting, neither ANN nor SNN is designed for it when there is no recurrent mechanism (where there are memory units to store state information) as we do in this paper. By working with non-recurrent SNN, this paper is dealing with the other two important features of DVS, asynchronicity and sparsity, which lead to the lower latency and lower power in recognition system [3], even if the machine learning tasks based on DVS input are not necessarily fully temporal. As the evidence to demonstrate that these two properties are practically useful, we note that [4] has shown that the object detection with sparse events can be more computational efficiency than the traditional ANN --- dense and synchronous operations. Moreover, these features also encourage the authors in [5] to develop an event-based dataset based on ImageNet.

[3] Amiret, Arnon  et al. A low power, fully event-based gesture recognition system. CVPR,  2017.

[4] Messikommer, Nico et al. Event-based asynchronous sparse convolutional networks. ECCV, 2020.

[5] Kim, Junho et al,  N-imagenet: Towards robust, fine-grained object recognition with event cameras. CVPR, 2021.

From: ICLR.cc/2023/Conference/Paper2535/Authors
title: Response to Reviewer uret
comment: **Q1: it would be interesting to understand why the introduction of such mechanisms improves performance, for example by analyzing the statistics of the learned networks, such as the distribution of spike frequencies**

Thanks for the suggestion. According to your comments, we have updated the Appendix to include an additional experiment on the distribution of activation over different layers in Figure 7-A. Comparing with QA, ROE helps ANN retain a Gaussian-like distribution. On the other hand, ROE brings the $99.99th$ percentile value closer to the mean of the activation than COE.

**Q2: Another contribution would be to understand why spikes are beneficial in this network. From the equations, I understand that the neural model is an integration and firing model, but it is not clear if the precise timing of spikes plays an important role in such a network, especially after learning.**

We guess (please let us know if we didn't get your point correctly) that your question is on whether or not the ordering of frames and spikes are important to the inference. Actually, these have been implicitly stated -- but rigorously presented -- in our theoretical reasoning. For example, from Eq. 4, it is clearly that the ordering of frames does not have an impact on the performance. In addition, Eq. 6 shows that it is the average spiking rate that plays the most important role in classification,  that is, the  ordering between spikes in a frame does  not have a significant impact on the performance.

From: ICLR.cc/2023/Conference/Paper2535/Authors
title: Response to Reviewer ozZz (1/2)
comment: Thank you for your thoughtful comments. We would like to address your concerns and your questions in the following.

__Q1: The proposed approach only limits to the IF model, the simplest SNN neuron model. In addition, I disagree that SNNs are just a variant of ANNs. If so, why not just ANNs directly? Why suffer so much, especially for classification tasks?__

This paper includes two main techniques: ROE and Cutoff. It is true that ROE is for IF because it is based on ANN-to-SNN conversion, but cutoff can be easily extended to LIF and direct training, as it simply deals with the number of output spikes. Actually, ANN-to-SNN conversion is a popular research direction with many ICLR/ICML papers published in the past years. More importantly,  as we have argued in Related Work, leveraging ANN-to-SNN conversion can result in fast response SNN and improve the latency, both of which are required in practical applications such as edge computing.

Regarding the term "variant", we used it in a general way, to refer to the discussion that SNN was regarded as a next generation  ANN. We have now made the statement more precise in the updated version. 

**Q2: Leveraging DVS for classification itself does not make much sense. DVS is sensitive to motion, and it can provide rich temporal information but lacks texture cues. The classification does not require temporal information but requires texture cues.**

We respectfully disagree on the comment that classification task is not for DVS. First of all, this is in contrast with the current development of the field that most SNN tasks are for classifications [1-3]. We agree that DVS is sensitive to motion, but the term "motion" does not ecessarily be captured/formalised as temporal behaviour, which can only be handled well with recurrent ANN/SNN where the memory units are embedded. Instead, this paper focuses on a different way of formalising the motion in DVS, i.e., asynchronicity and sparsity, which can be dealt with non-current ANN/SNN.  

Actually, asynchronicity and sparsity lead to the lower latency and lower power in recognition system [1]. [2] has shown that the object detection with sparse events can be more computational efficiency than the traditional system --- dense and synchronous operations. These features also encourage the authors in [3] to develop an event-based dataset based on ImageNet.

In summary, while the same as other SNNs w/o memory units, our SNN is not designed for the fully temporal DVS inputs, we are exploiting the asynchronicity and sparsity of DVS inputs and achieved a performance that is superior to the state-of-the-art.  

[1] Amiret, Arnon et al. A low power, fully event-based gesture recognition system. CVPR,  2017.

[2] Messikommer, Nico  et al. Event-based asynchronous sparse convolutional networks. ECCV, 2020.

[3] Kim, Junho   et al,  N-imagenet: Towards robust, fine-grained object recognition with event cameras. CVPR, 2021.

**Q3: The author claimed that they directly process the raw events in an asynchronous manner. But, it is not clear to me how did authors achieve it.**

We do not need to take extra action/effort to obtain asynchronicity as it is a natural property of the DVS inputs. The asynchronicity can be seen by forwarding the events from DVS to SNN directly. DVS is event-driven, generating the events when the environment changes. Accordingly, the binary event with address location can be sent to its corresponding spiking neurons. The neurons only operate when they are receiving spikes. That is, the whole system is asynchronous to the environment.

**Q4: In section 4.1, the authors claimed, "When t is large enough to make $\Delta$l(t) negligible," but why?**

Equation 6 shows that $\boldsymbol{\Delta}^l(t) \triangleq  {\boldsymbol{V}^l(t)}/{(tS_r V^l_{thr})}$. According to Equation 1, we can get $\boldsymbol{V}^l(t)/V^l_{thr} \in [0,1)$. Since $S_r$ is a constant,  $\boldsymbol{\Delta}^l(t)$ tends to be zero when $t >> 1$.

**Q5: For ROE,  represents a batch of  during training. Then, it means the batch size could impact the performance of the trained model. However, the authors did not show any experimental results about it.**

The performance of ROE, like all other training schemes utilising mini-batches, is related to  the size of batches. To respond to your comment, we have now updated the Appendix to include an experiment on batch size in Figure A-6.

The result indicates that the change of $\alpha$ and batch size can influence the performance of the resulted SNN. We can see that SNN with ROE can achieve better accuracy at $0.125T_{total}$ with larger batch size. However, too small or too large batch size may significantly degrade the normal ANN training. Thus, we set the batch size wisely that can result in an optimal SNN.

From: ICLR.cc/2023/Conference/Paper2535/Reviewer_ozZz
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors proposed anytime-optimal-inference SNNs (AOI-SNNs) to process raw events obtained from DVS cameras. Essentially, the authors claimed, for any t, the performance of the SNN is of negative correlation with threshold and positive correlation with L2 norm over the output of ReLU activation.

strength_and_weaknesses: Weaknesses:
-1. The proposed approach only limits to the IF model, the simplest SNN neuron model. In addition, I disagree that SNNs are just a variant of ANNs. If so, why not just ANNs directly? Why suffer so much, especially for classification tasks?

-2. Leveraging DVS for classification itself does not make much sense. DVS is sensitive to motion, and it can provide rich temporal information but lacks texture cues. The classification does not require temporal information but requires texture cues. 

-3. The author claimed that they directly process the raw events in an asynchronous manner. But, it is not clear to me how did authors achieve it. 

-4. In section 4.1, the authors claimed, "When t is large enough to make ∆l(t) negligible," but why? 

-5. For ROE, $A^l$ represents a batch of $a^l$ during training. Then, it means the batch size could impact the performance of the trained model. However, the authors did not show any experimental results about it.

-6. The author should provide experimental results to prove Theorem 4.1. I expect the experimental results also show the positive correlation between classification accuracy and $(\sqrt{n_l}\frac{V_{the}^l}{||a^l||_2})^{-1}$.

clarity,_quality,_novelty_and_reproducibility: Clarity: It is not very clear, especially the section 3.3.

Novelty: I did not see much novelty.


summary_of_the_review: The proposed approach is only limited to the IF model and lacks experimental evidence of the main Theorem. The writing also needs to be improved. Therefore, I would not recommend this submission. 
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 1: The contributions are neither significant nor novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 3: reject, not good enough

From: ICLR.cc/2023/Conference/Paper2535/Reviewer_uret
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: Based on the observation that biological neural networks should be able to provide an answer at any time, the authors introduce two mechanisms in a range of existing SNNs - first, they introduce a regularization in the cost function to provide an optimization of the classification accuracy at any time. Second, they provide a confidence calculation method that introduces a cutoff mechanism. The resulting model is analyzed at the theoretical level and numerical simulations show that such a framework provides an effective trade-off between accuracy and processing speed, which could benefit SNNs as implemented in neuromorphic hardware. 
strength_and_weaknesses: The main strength of the paper is that it clearly defines a gap in existing networks, as they often assume that inference is given after a given fixed processing time. By introducing mechanisms to efficiently perform computations at any time, the authors provide an innovative complement to neural networks. The experimental results are robust and illustrate well the efficiency of the algorithm through numerical simulations. 

A weakness of the paper is the lack of analysis of the resulting network. Indeed, it would be interesting to understand why the introduction of such mechanisms improves performance, for example by analyzing the statistics of the learned networks, such as the distribution of spike frequencies. For example, is the activity more sparse? Another contribution would be to understand why spikes are beneficial in this network. From the equations, I understand that the neural model is an integration and firing model, but it is not clear if the precise timing of spikes plays an important role in such a network, especially after learning. 

clarity,_quality,_novelty_and_reproducibility: The paper is very clearly written and the mathematical details are well and progressively detailed. 

summary_of_the_review: In summary, the authors provide two mechanisms for augmenting traditional SNNs with the ability to infer a response efficiently at any time. The numerical simulations are clearly convincing, while further analysis of the network would give additional strength to the paper.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2535/Reviewer_5NBM
confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
summary_of_the_paper: The authors work on ANN2SNN conversion, in the context of DVS processing. They use IF neurons, and introduce a new regularization method called  "regularizer for outlier elimination" (ROE), and a cutoff that allows making decisions before the stimulus is over.


strength_and_weaknesses: STRENGTHS:

The accuracy outperforms the SOTA by a large margin (Table 2)

WEAKNESSES:

The limits are not correctly identified:

* The neurons do not leak, thus old inputs cannot be forgotten. Maybe this is not an issue with the datasets they used, because the different patterns to be recognized are in different files, and all the potentials are reset in between (right?). However, it could be an issue if deployed in a "streaming" setting where potentials are never reset. Can the authors elaborate on this? In addition, could their approach work with LIF instead of IF?

* Conversely, IF is the way to go with static images. Could their method work with static images?

* The ANN that is trained ignores the time, in the sense that shuffling the frames in Eq 4 would have no effect on the loss. This wouldn't work on a truly temporal task in which the order of the sequence matter. The most temporal dataset here is DVS Gesture, but even with this one, most gestures can be categorized based on spatial information only (even the clockwise and counterclockwise hand gesture can usually be distinguished if using the event polarities).

* With F>1 the ANN makes several predictions that are averaged through Eq 4. But the ANN is stateless thus each prediction is independent from the previous ones. This is not the case for the SNN, unless you reset the potentials during the stimulus (you don't, do you?). That causes a discrepancy between the ANN and the SNN which should be discussed. 

clarity,_quality,_novelty_and_reproducibility: * Will the authors share their code?

* https://arxiv.org/abs/2102.04159 should be included in Table 2


summary_of_the_review: Accuracy is excellent, but the limits are not properly identified.


correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2535/Reviewer_zEep
confidence: 2: You are willing to defend your assessment, but it is quite likely that you did not understand the central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
summary_of_the_paper: This paper presents a method to improve spiking neural networks by adding a regularizer to training and defining a metric to determine whether the network is able to return a result during inference. The regularizer aims to reduce the impact of outliers during training, by improving the similarity between the predicted spiking rate and desired spiking rate, based on prior work. The metric, then, optimizes for a point in time before the total time when the most confident prediction is significantly more confident that the second most confident. Experiments are performed on three event camera datasets for classification, where the union of the two proposed contributions outperforms prior state of the art.
strength_and_weaknesses: Overall, the paper applies a lot of prior knowledge and research to the problem of training spiking networks, and goes into explicit detail and proofs for the proposed work. The results show clear improvements over the prior works, although on relatively simple datasets and a simple task (image classification). Also, the proposed method only improves upon prior works when the early cutoff is applied (results without it are mixed). This seems a little surprising that terminating early could have such improvements.
clarity,_quality,_novelty_and_reproducibility: My main issue with the paper is the clarity. Overall, the method is very dense, and assumes a lot of background knowledge on theory. Many symbols are also introduced, and perhaps it would be useful to have a glossary to keep track of them all.
summary_of_the_review: Overall, the proposed method seems promising, and does provide improvements over prior work. The ability to return early during inference is also very interesting, and could unlock additional benefits for SNNs over ANNs. However, the overall text is too dense, and this is a little beyond my field of expertise.
correctness: 3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.
technical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
empirical_novelty_and_significance: 3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
flag_for_ethics_review: NO.
recommendation: 6: marginally above the acceptance threshold

From: ICLR.cc/2023/Conference/Paper2535/Reviewer_Tbtq
confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
summary_of_the_paper: The authors propose regularizer for outlier elimination (ROE) for ANN-to-SNN conversion. ROE aims to suppress activation outliers to enhance conversion accuracy. Additionally, a cutoff mechanism for early inference-termination is proposed to reduce inference latency. The feasibility of the proposed methods was verified on various event datasets, e.g., DVS-CIFAR10, N-Caltech101, DVS128 Gesture.  
strength_and_weaknesses: Strength:

It feels ROE helps conversion reduce inference error compared with COE and QA. Also, the cutoff mechanism was successfully able to indicate the minimally required time for successful early termination.

Weaknesses:

1. Benchmark is weak. The authors compared the classification accuracy of ROE with COE and QA which have not been reported in any previous publications. This may undermine the efficacy of ROE. Therefore, the authors should consider systematic and fair comparison of the proposed method with previous methods.   

2. Regarding the confidence rate evaluation, I assume that $g(\boldsymbol{X})$ should be known a priori for all samples to evaluate the confidence rate. That is, this method cannot be used for any test datasets whose $g(\boldsymbol{X})$ are not available a priori, which significantly undermines the value of the proposed early termination method. Therefore, the proposed early termination method has a certain limitation.

3. Conversion method is not elaborated. The authors should detail the objective function minimization sequence and all related sub-methods. Particularly, ROE minimization method should be detailed.

4. Overall, I do not see any technical and/or theoretical breakthroughs in the authors’ contribution.  
clarity,_quality,_novelty_and_reproducibility: Clarity: the paper barely addresses the detail of the proposed conversion method and minimization of ROE.

Quality, Novelty, Significance: the technical progress is marginal, ROE appears novel to some degree though. Given the limited impact of the proposed method, its significance is also marginal.

Reproducibility: I appreciate authors' providing the code.
summary_of_the_review: Given the weaknesses addressed above, mainly regarding the marginal significance and novelty, the impact of the proposed method is also marginal.
correctness: 2: Several of the paper’s claims are incorrect or not well-supported.
technical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
empirical_novelty_and_significance: 2: The contributions are only marginally significant or novel.
flag_for_ethics_review: NO.
recommendation: 5: marginally below the acceptance threshold