Paper: BEEF: Building a BridgE from Event to Frame (ICLR-2024)

Reviewer comment (Experiments):
The authors have conducted a comparison experiment with a fixed number of times, as shown in Table 3. Nevertheless, it is advisable for the authors to include experiments with a fixed time window. Furthermore, the authors should investigate how various parameters for fixed events or fixed time windows compare to BEEF. Additionally, it would be beneficial for the authors to provide more visual comparison results of event representations.

Author response:
Thank you very much for your valuable suggestions! In the original experiments presented in Table 3, we used the slicing of fixed event count for comparison. We will include more specific details about this in the revised version of our manuscript. To facilitate a more complete comparison between dynamic slicing and traditional fixed slicing methods, we have supplemented our experiments in the object recognition task. These include slicing based on a fixed number of events and a fixed duration of events. We have also compared three different event representation methods, with the results as follows:

| DVSGesture |Event Frame | Event Spike Tensor | Voxel Grid |
|:------|:----------------|:--------|:----|
| Fix Duration | 93.75% | 93.75% | 88.54% |
|Fix Event Count |93.06%|94.79%|88.19%|
|**BEEF(ours)** |**94.79%**|**95.49%**|**89.24%**|


The results show that our dynamic slicing approach, BEEF, outperforms the fixed slicing approach in downstream tasks across different event representations. This highlights the efficacy of BEEF in handling event streams.

Additionally, we plan to include more illustrative figures in the revised version to vividly depict the differences and respective advantages of dynamic and fixed slicing. More visualization results related to the experiments will also be added to the appendix.

Thank you again for your suggestion! Your feedback is instrumental in enhancing the clarity and comprehensiveness of our research.

Reviewer comment (Novelty):
There are articles exploring adaptive event stream splitting strategies. The author should consider citing some relevant references [1, 2] that utilize hyperparameters for implementation.

[1] EDFLOW: Event driven optical flow camera with keypoint detection and adaptive block matching, IEEE TCSVT 2022.  
[2] Asynchronous spatio-temporal memory network for continuous event-based object detection, IEEE TIP 2022.

Author response:
Thank you very much for your suggestion! We acknowledge the importance of incorporating relevant references, particularly those that explore adaptive event stream splitting strategies. We will make sure to include the references you have mentioned [1,2] in the revised version of our manuscript! Thanks!

***Reference:***

[1] Liu M, Delbruck T. EDFLOW: Event driven optical flow camera with keypoint detection and adaptive block matching[J]. IEEE TCSVT 2022.

[2] Li J, Li J, Zhu L, et al. Asynchronous spatio-temporal memory network for continuous event-based object detection[J]. IEEE TIP 2022.

Reviewer comment (Theory):
I am aware that with event and spiking cameras it is quite popular to convert the event/spike streams into a sort of frame based representation. However I have a fundamental objection with this type of an approach (which is shared by quite a few of my colleagues around the world, in private conversations at least) as to why should these fundamentally asynchronous event streams representations should be converted to a rather synchronous representation, simply to be able to map them into algorithms that were originally developed for synchronous frame like data. I think a more thorough discussion on this is needed in the paper to better motivate the work.

Author response:
Thank you for your insightful comment. First, I wholeheartedly agree with your statement regarding the conversion of asynchronous event data into synchronous formats, which indeed can undermine the inherent advantages of the asynchronous nature. This is not an optimal representation, and we acknowledge this limitation. However, the reasons for converting asynchronous event data into synchronous representations in current practices can be summarized as follows:

1. Why choose synchronous over asynchronous processing? Given that existing GPU hardware architectures and programming models are designed for highly synchronous and parallel tasks, the conversion of asynchronous event stream data into a synchronous representation becomes a necessity for algorithm simulations performed on GPUs. In our paper, we aim to slice the event stream into very fine event cells to represent the original event data as closely as possible (Sec.4.1), hoping to minimize the errors introduced by synchronous representation.

2. The synchronous simulations conducted on GPUs in our work do not preclude the possibility of future asynchronous implementations on neuromorphic hardware. Both SNNs and event stream data are inherently asynchronous. However, due to hardware constraints, current software simulations are temporarily unable to achieve true asynchronous processing. If deployed on neuromorphic hardware (e.g., Loihi [1], TrueNorth [2]), the asynchronous processing of event streams by SNNs would be extremely low-energy and low-lantancy[3,4,5]. Therefore, this work lays the theoretical groundwork for future hardware deployment, and efficiently implementing SNN processing of asynchronous event streams in conjunction with ANNs on neuromorphic hardware is one of our future goals.

We hope this explanation addresses your concerns. We are committed to further discussing this topic in our manuscript to provide a comprehensive motivation for our work. 


***Reference:***

[1] Davies M, Srinivasa N, Lin T H, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro 2018.

[2] Akopyan F, Sawada J, Cassidy A, et al. Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip. IEEE TCAD 2015.

[3] Roy A, Nagaraj M, Liyanagedera C M, et al. Live Demonstration: Real-time Event-based Speed Detection using Spiking Neural Networks. In CVPRW 2023.

[4] Viale A, Marchisio A, Martina M, et al. Carsnn: An efficient spiking neural network for event-based autonomous cars on the loihi neuromorphic research processor. In IJCNN 2021.

[5] Yu F, Wu Y, Ma S, et al. Brain-inspired multimodal hybrid neural network for robot place recognition. Science Robotics 2023.

Reviewer comment (Experiments):
Clarify better what are the alternative methods to which this is being compared? What exactly is meant by "fixed slice" approaches to which this is being compared? Many approaches for producing frame like representations (such as getting the max or union of all events in a time window) result in the introduction of significant amounts of noise. In contrast morphological operands like erosion and dilation can introduce much better quality frames. To what extent is the good performance of the algorithm attributable simply to noisy frame generation in competing approaches?

Author response:
*Symbol Description: the total event stream $E$; the resulting sliced sub-event stream list by BEEF: $E_{beef}=[E^b_{1},..E^b_{N_1}]$; the resulting sliced sub-event stream list by fixed slicing method: $E_{fixtime}=[E^f_{1},..E^f_{M_1}]$.*

Thank you for your insightful suggestions! First, let me clarify our dynamic slicing approach, which allows the event stream to be sliced at any timestamp and then converted into frames for downstream tasks. In Table 1, the term "fixed slice" refers to a method that we employ a fixed slicing approach with a constant number of events per sub-event stream to segment the entire event stream into $E_{fixtime}=[E^f_{1},..E^f_{M_1}]$. These sub-event streams are then transformed into different representations using the same event representation method $F$ (we used Event Frame [1]) and fed into downstream tasks (tracking/recognition) for testing. We will revise the terminology to "slicing with fixed event number" in our updated manuscript to make the experimental settings clearer.

We greatly appreciate your mention of the potential use of morphological operands. These methods are more commonly found in event stream denoising algorithms or event representation algorithms [2], and our paper primarily focuses on the event stream slicing process. The techniques you mentioned, like erosion or dilation, could be integrated with our current approach; for example, denoising the original event stream before applying BEEF’s dynamic slicing. We intend to investigate it further in future research. Thank you for this valuable suggestion!

***Reference:***


[1] Maqueda A I, Loquercio A, Gallego G, et al. Event-based vision meets deep learning on steering prediction for self-driving cars. CVPR 2018.

[2] Baldwin R W, Liu R, Almatrafi M, et al. Time-ordered recent event (TORE) volumes for event cameras. TPAMI 2022.

Reviewer comment (Reproducibility):
Unless I missed it, will source code be provided?

Author response:
Absolutely！We will provide the source code upon acceptance.

Reviewer comment (Novelty):
This paper fails to fully review the topic of this work: event representation. As suggested in [1][2], there are several existing event representation strategies including stacking based on time/event counts, voxel grid, histogram of time surfaces, event spike tensor, and a recent work introduces neural representation [3]. However, this paper only mentions two of them. In addition, the motivation to consider temporal information is similar with event counts integration, which is mentioned by the authors.

Author response:
Thank you for your recommendations. I understand your concern regarding why our paper does not address a broader range of event representation methods. The reason is that **our focus is on event slicing rather than event representation.** Event stream data conversion into frames/representations involves two steps: Step 1: slicing the event stream into multiple sub-event streams, and Step 2: converting these sub-streams into frames using different event representation methods. There is a significant body of work dedicated to optimizing event representation (Step 2), including the voxel grid, time surface, and other methods you mentioned. However, these do not address the issues arised with fixed slicing (e.g., resulting non-uniform event in scenarios with changing motion speed). Hence, our paper primarily addresses the first step: event slicing.

After the dynamic slicing with BEEF, the events can indeed be transformed into different representational formats using various event representation methods. Although event slicing and representation are different processing steps, either better slicing or representation method benefits the feature extraction with neural network, thus improving performance. Experiments with different slicing methods and different event representation have been supplemented in **Reply 3**. 

The references you have mentioned will also be included in the updated version of our manuscript. And the confusing description of motivation in the original article that you pointed out will also be revised shortly!

I hope this explanation addresses your query. Thank you once again!

Reviewer comment (Evaluation):
The necessity of a very lightweight SNN is not clear. Since SNN works with ANN cooperatively, SNN has only very limited contribution to the overall computational cost. As implied in Table 2, considering the ANN is the major cost for the process, the contribution and necessity for low energy and fast speed of SNN is reduced.

Author response:
Thank you for your question. Let me elucidate the necessity of using SNN.

1.The reason **why we choose SNN as the event slicing trigger** is twofold:
- Utilizing SNNs on neuromorphic hardware for processing event streams is low-energy and low-latency [1,2].

- Deployed on neuromorphic hardware, SNNs can process event streams asynchronously [3,4,5], conserving energy when there is no data input—a capability that GPUs, operating synchronously, lack.

Due to the aforementioned reasons, there is a considerable amount of research [6,7,8,9,10] employing Spiking Neural Networks (SNNs) for event data. Although these SNNs are simulated on GPU platforms, the models resulting from such simulations could be deployed on neuromorphic hardware [3,4].


2.The rationale behind our aim for **low-energy and fast-speed SNN processing** is:

We design the BEEF as a **plug-and-play** algorithm, intending for the SNN to dynamically slice the event stream without adversely affecting the latency or energy consumption of the main network.

Regarding the comparison of resource consumption between SNNs and ANNs (Table 2), it is crucial to note that our dynamic slicing is performed in real-time, not in an offline manner.
By employing a low-energy SNN model as a dynamic event stream slicer, we ensure that the speed does not impede the downstream processing rate and also enhances the overall performance of downstream tasks. This is one of the core motivations of our paper, and we hope it addresses your concerns.

***Reference:***

[1] Davies M, Srinivasa N, Lin T H, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro 2018.

[2] Akopyan F, Sawada J, Cassidy A, et al. Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip. IEEE TCAD 2015.

[3] Roy A, Nagaraj M, Liyanagedera C M, et al. Live Demonstration: Real-time Event-based Speed Detection using Spiking Neural Networks. In CVPRW 2023.

[4] Viale A, Marchisio A, Martina M, et al. Carsnn: An efficient spiking neural network for event-based autonomous cars on the loihi neuromorphic research processor. In IJCNN 2021.

[5] Yu F, Wu Y, Ma S, et al. Brain-inspired multimodal hybrid neural network for robot place recognition. Science Robotics 2023.

[6] Hagenaars J, Paredes-Vallés F, De Croon G. Self-supervised learning of event-based optical flow with spiking neural networks. NeurlPS 2021.

[7] Yao M, Gao H, Zhao G, et al. Temporal-wise attention spiking neural networks for event streams classification. ICCV 2021.

[8] Zhu L, Wang X, Chang Y, et al. Event-based video reconstruction via potential-assisted spiking neural network. CVPR 2022

[9] Kosta A K, Roy K. Adaptive-spikenet: event-based optical flow estimation using spiking neural networks with learnable neuronal dynamics. ICRA 2023.

[10] Hussaini S, Milford M, Fischer T. Spiking neural networks for visual place recognition via weighted neuronal assignments. RAL 2023.

Reviewer comment (Experiments):
The compared methods in the experiment are not sufficient. More event representation/stacking methods should be considered to compare with the proposed methods, including the methods mentioned in [1-3].

Author response:
Thanks for your suggestion! Event representation refers to the process of event information extraction that is performed after the event stream has been sliced into sub-event stream, and the resulting event representation meets the neural network input requirements. Thus, our dynamic slicing process and event representation can be used at the same time. 

To validate the effectiveness of our slicing approach, we assess the downstream task performance using three distinct event representation methods, namely Event Frame [1], Event Spike Tensor (EST) [2], and Voxel Grid [3], on the DVSGesture dataset. We measure these against both fixed (slice by fixed duration and fixed event count) and dynamic slicing approaches to provide a comprehensive analysis:

| DVSGesture |Event Frame | Event Spike Tensor | Voxel Grid |
|:------|:----------------|:--------|:----|
| Fix Duration | 93.75% | 93.75% | 88.54% |
|Fix Event Count |93.06%|94.79%|88.19%|
|**BEEF(ours)** |**94.79%**|**95.49%**|**89.24%**|


The results show that our dynamic slicing approach, BEEF, outperforms the fixed slicing approach in downstream tasks across different event representations. This highlights the efficacy of BEEF in handling event streams. 

Thanks for your suggestion! We will cite the event representation methods you mentioned in the revised paper and try to integrate them with BEEF in the future.

***Reference:***

[1] Maqueda A I, Loquercio A, Gallego G, et al. Event-based vision meets deep learning on steering prediction for self-driving cars. CVPR 2018.

[2] Gehrig D, Loquercio A, Derpanis K G, et al. End-to-end learning of representations for asynchronous event-based data. ICCV 2019.

[3] Zhu A Z, Yuan L, Chaney K, et al. Unsupervised event-based learning of optical flow, depth, and egomotion. CVPR 2019.

Reviewer comment (Experiments):
More details about the experimental settings are required. The proposed methods use adaptive slicing time, how to create GT accordingly? And how to compare with fixed-sliced methods that have different timestamps for event frames?

Author response:
Thank you for your question. Below are more detailed explanations of our experimental settings and the methodology for generating Ground Truth (GT). We will include these details in the appendix of the revised version:

**Experimental settings：**

1. *Network Structure*:
We used a Spiking Neural Network architecture comprising {16C3-IF-AP2-32C3-IF-AP2-64C3-IF-AP2-LNIF-LN-IF}, where IF denotes the use of integrated-fire neurons. (Appendix E)

2. *Training Setup*:
We adopted the SGD optimizer with an initial learning rate of 1e-4, complemented by a cosine learning rate scheduler. SNN models were trained for 50 epochs with a batch size of 32. (Appendix E)

3. *Data Processing Setup:*
For single object tracking tasks, we utilized the FE108 dataset, while for object recognition, we used the DVS-gesture and N-caltech101 datasets. Each dataset was divided into training, testing, and validation sets. The validation set was used to infer the ANN model to provide feedback for supervising SNN training. All results in the table represent the ANN's performance on the test set. 

4. *GT Setting:*
Since the SNN might choose to spike and segment the event stream at any position, the GT at any timestamp was obtained through linear interpolation from the GT provided by the original dataset. 

**Details of Fixed-sliced method:**

*Symbol Description: the total event stream $E$; the resulting sliced sub-event stream list by BEEF: $E_{beef}=[E^b_{1},..E^b_{N_1}]$; the resulting sliced sub-event stream list by fixed slicing method: $E_{fixtime}=[E^f_{1},..E^f_{M_1}]$.*

Suppose an event stream (duration = $T$) is sliced into $N'$ slices ($E_{beef}=[E^b_{1},..E^b_{N_1}]$) after dynamic slicing. Then for a fair comparison, the number of sub-event stream generated by the fixed slicing method should also be $N'$ (i.e., $len(E_{fixtime})=N'$), where the duration of each sub-event stream in $E_{fixtime}$ is $\frac{T}{N'}$. Since it was mentioned earlier that we know the GT of each timestamp, we can likewise obtain the GT corresponding to each sub-event stream after fixed slicing. Then we can compare the results of the fixed slicing method with our dynamic slicing method.

Reviewer comment (Experiments):
Frame-like inputs to transformers or CNNs where frames have been derived from events may be sensitive to slicing. We need a toy experiment to study this hypothesis with a smaller network and different slicing techniques.

Author response:
Thank you very much for your constructive suggestion! In response, we have conducted total 60 experiments with **different models** to investigate the impact of **different slicing techniques** and **different number of slices** on the performance in downstream tasks, thereby affirming the hypothesis that event streams are sensitive to slicing.

In our experiment, we employed two fixed slicing methods: (1). *Slicing with a fixed number of events* and (2). *Slicing with a fixed duration*. $N$ denotes the number of resulting event slices. Experimental results are detailed as follows: 

| NCaltech101   | $N$  |          2 |          4 |          6 |          8 |         10 |         12 |         14 |         16 |         18 |         20 |         22 |         24 |         26 |         28 |         30 |     Mean|     Var|
|:---------------|:-------------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|:-----------|
| ResNet18       | Fixed Count     |   70.96 | 75.26 | 75.39 | 75.30  | 76.09 | 73.95 | 74.09 | 73.80  | 76.40  | 75.39 | 75.45 | 73.60  | 71.94 | 71.01 | 71.17 | **73.98**|**3.33**|
| ResNet18       | Fixed Time     | 62.90  | 72.64 | 76.38 | 74.48 | 74.91 | 73.70  | 74.30  | 74.69 | 76.95 | 74.75 | 74.46 | 74.42 | 71.61 | 71.52 | 69.69 |**73.16**|**10.80**|
| ResNet34       | Fixed Count| 72.19 | 75.55 | 76.98 | 78.22 | 77.14 | 77.40  | 76.78 | 76.90  | 78.14 | 77.06 | 76.91 | 74.85 | 74.76 | 76.91 | 73.07 |**76.19**|**2.90**|
| ResNet34       | Fixed Time| 65.42 | 75.92 | 78.29 | 78.20  | 78.48 | 76.22 | 77.76 | 76.57 | 75.94 | 76.80  | 76.61 | 75.91 | 75.11 | 74.76 | 74.19 |**75.74**|**9.15**|

The results indicate significant fluctuations (large variance) in downstream performance based on the slicing method and the number of slices used. We believe this addition effectively demonstrates the sensitivity of event streams to fixed slicing techniques, confirming the need for our motivation to propose dynamic slicing of event streams. 
Additionally, the accuracy achieved using the dynamic slicing method (82.54% by ResNet34) surpasses that of any fixed slicing approach (with the highest being 78.48%), further substantiating the efficacy of the dynamic method in our study.

Thanks for your suggestion!

Reviewer comment (Experiments):
The exposition is really hard to follow. As stated directly after eq. 4, the slicing is done by grouping together events whose timestamps are between two output spikes of the SNN. Here, an experiment is needed on the statistics of this slicing and why such an approach makes sense.

Author response:
To demonstrate the effectiveness of our proposed dynamic slicing method, we provide the following statistical results:

### **1. Statistics of dynamic slicing (BEEF) vs. fixed slicing.**
*Symbol Description: the total event stream $E$; the resulting sliced sub-event stream list by BEEF: $E_{beef}=[E^b_{1},..E^b_{N_1}]$; the resulting sliced sub-event stream list by fixed slicing method: $E_{fixtime}=[E^f_{1},..E^f_{M_1}]$.*

In the tracking task, the average duration of each sub-event stream $E^b_{k}(k\in[1,N_1])$ is 65ms (corresponding to 13 event cells, and the duration of the event stream contained in each event cell is 5ms). The maximum duration of each sub-event stream is 100ms, and the minimum duration is 30ms, while for our comparison of the slicing-by-fixed-time approach, the duration of each sub-event stream $E^f_{j}(j\in[1,M_1])$ is fixed at 75ms. The following are specific statistics:

|Method| Avg Cell Num| Var Cell Num | Avg Duration| Min Duration | 25th Duration| 75th Duration| Max Duration|
|:--------|:--------------------------------------|:---------------------------------------|:------------------------------|:--------|:-----|:-----|:----|
| BEEF                                  |  12.99                          | 3.96    | ~65ms                                  |25ms    | 50ms   | 80ms  | 100ms |
| Slice by fixed duration               |  15                             | 0       | 75ms                                  |//   | //   | //  | //  |

We will put the visualization of the statistic results in the appendix of the revised version.

### **2. Statistics of event density.**

We also counted the average density of sub-event streams after fixed slicing and dynamic slicing for comparison.

*Symbol Description: For each sub-event stream $E^b_{k}$ or $E^f_{j}$, it contains several event points $e_i=[x_i,y_i,t_i,p_i]$. We define a matrix $C$ to represent the event count, where $C_{xy}$ represents the number of events at coordinates $(x,y)$. Given a threshold $T$, we define the event density to be: $D = \frac{\sum_{x,y}\mathbb{1}\{C_{x,y}\geq T\}}{\sum_{x,y}\mathbb{1}\{C_{x,y}> 0\}}$. The threshold $T$ is determined based on the percentile of the event count $C_{xy}$.*

**Density Analysis**: We define the density of events as a metric reflecting the amount of event information within sub-event streams. The smaller the fluctuation in the density of each sub-stream (the smaller the variance), the more stable the event information contained. The stability of the event density is crucial for scenarios where the distribution of events is not uniform (e.g., scenarios with changing motion speed).


We chose $T = 30$% and $90$% (lower $T$ to observe overall event activity, including those less frequent events; higher $T$ for focusing on repetitive or frequent events) to validate the effectiveness of BEEF to dynamically slice the event stream. Below are the statistics of event density, where the data (left vs right) on the left are statistics of BEEF and on the right for statistics of fixed slicing. We select 4 classes in the FE108 tracking dataset.

| T=30% | airplane_mul222 | box_hdr | dog | tank_low |
|:------|:----------------|:--------|:----|:---------|
| Mean  | 0.9196 vs 0.9074 | 0.9850 vs 0.9850 | 0.9208 vs 0.8984 | 0.9584 vs 0.9575 |
| Var $\downarrow$| **0.0151** vs 0.0164 | **0.0038** vs 0.0039 | **0.0144** vs 0.0168 | **0.0096** vs 0.0100 |
| Std $\downarrow$ | **0.1231** vs 0.1283 | **0.0622** vs 0.0626 | **0.1200** vs 0.1299 | **0.0984** vs 0.1003 |

| T=90% |airplane_mul222 | box_hdr | dog | tank_low|
|:------|:----------------|:--------|:----|:---------|
| Mean | 0.1256 vs 0.1263 | 0.1338 vs 0.1405 | 0.1120 vs 0.1131 | 0.9584 vs 0.9575 |
| Var $\downarrow$| 0.0004 vs 0.0004 | **0.0007** vs 0.0053 | 0.0001 vs 0.0001 | **0.0096** vs 0.0100 |
| Std $\downarrow$| 0.0215 vs 0.0215 | **0.0267** vs 0.0732 | **0.0073** vs 0.0078 | **0.0984** vs 0.1003 |

The results show that the event stream after BEEF's dynamic slicing has a more stable event density (low variance), which verifies that BEEF has a certain ability to perceive the event information, and ensures that BEEF is robust in different motion scenarios, as shown in Table 3.

Reviewer comment (Theory):
4.3.1 has to be elaborated. While the math derivations are sound, it is not clear to the reader why the starting point of the derivations is the desire for $S_{out}$ to spike at $n^{*}$. I tried to understand it also through the observations in 4.3.2 but could not.

Author response:
Thank you for your query which provides us with an opportunity to clarify our methodology. 

Contrary to fixed slicing methods (such as predetermined time intervals or event counts), our approach dynamically determines event stream slicing based on spike occurrences in the SNN, detailed in Figure 1.

In order to supervise the SNN to slice the event stream at the optimal position, we feed the events sliced at the SNN-determined position as well as the events sliced at its neighboring positions into the downstream model.
The downstream model then returns the loss (e.g., classification loss, tracking loss), where the position corresponding to the minimum loss indicates the optimal slicing point (Eq.10). Thus, we obtain a position label $n^*$, which in turn guides the SNN to make the best slicing strategy through supervised learning. 

In summary, when ANN feedback indicates that slicing at the $n^*$ position can enhance model performance, the SNN discriminator is directed to spike at the $n^*$. Section 4.3 of our paper delves into formulating the SPA-CE loss function, which aids the SNN to spike at this specified $n^*$ position. 

We hope this response clarifies our approach and methodology. We are committed to improving the manuscript in its revised version for better understanding. Should there be any points needing further clarification, please feel free to reach out.

Reviewer comment (Writing):
The beginner's arena was meant to explain the above but is incomprehensible. What does it mean “to slice at a specified time step $T^{*}$”?

Author response:
Following up on the explanation of Reply 3, consider the downstream ANN's feedback indicates that the SNN should spike at a specific moment $n^*$ to slice the event stream. In this case, our goal is to supervise the SNN to ensure that it spikes precisely at $n^*$. The purpose of "The beginner's arena" is to demonstrate that our proposed loss function SPA-CE can effectively guide the SNN to pulse at $n^*$, in contrast to common loss functions like Cross-Entropy (CE) or Mean Squared Error (MSE), which could not achieve this.

The term $T^*$ used in the text may have caused some confusion. In future revisions of our manuscript, we will standardize this notation to $n^*$ to avoid any ambiguity. We hope this clarification helps and apologize for any confusion caused. Thank you for the reminder.

Reviewer comment (Evaluation):
It is not clear what purpose the energy computations of the SNN serve when the task will be solved with ultra consuming GPUs.

Author response:
Thank you for your comment. Indeed, you are correct that the implementation of Spiking Neural Networks (SNNs) on GPUs does not currently confer a significant energy advantage. This is largely due to the limitations inherent in current SNN simulation platforms (such as spikingjelly[9] and tonic[10]), which are primarily GPU-based.

However, it is important to note that SNNs demonstrate significant energy efficiency when operated on neuromorphic hardware, such as Loihi [7] and TrueNorth [8]. This advantage is well-established and extensively discussed in recent literature [1,2]. The energy calculations in our paper [3] are intended to provide an estimation of the potential energy efficiency of SNNs with future advancements in hardware and algorithms. In addition, comparing the theoretical energy consumption of SNNs with that of traditional Artificial Neural Networks (ANNs) is commonly-adopted in this field [4,5,6], aiding in the understanding of the energy dynamics of these systems.

***Reference:***

[1] Yin B, Corradi F, Bohté S M. Accurate online training of dynamical spiking neural networks through Forward Propagation Through Time. Nature Machine Intelligence 2023.

[2] Schuman C D, Kulkarni S R, Parsa M, et al. Opportunities for neuromorphic computing algorithms and applications. Nature Computational Science 2022.

[3] Yao M, Zhao G, Zhang H, et al. Attention spiking neural networks. TPAMI 2023.

[4] Kim S, Park S, Na B, et al. Spiking-yolo: spiking neural network for energy-efficient object detection. AAAI 2020. 

[5] Zhou Z, Zhu Y, He C, et al. Spikformer: When spiking neural network meets transformer. ICLR 2023.

[6] Wang Z, Fang Y, Cao J, et al. Masked Spiking Transformer. ICCV 2023.

[7] Davies M, Srinivasa N, Lin T H, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro 2018.

[8] Akopyan F, Sawada J, Cassidy A, et al. Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip. IEEE TCAD 2015.

[9] Fang W, Chen Y, Ding J, et al. SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence. Science Advances 2023.

[10] Eshraghian J K, Ward M, Neftci E O, et al. Training spiking neural networks using lessons from deep learning. Proceedings of the IEEE 2023.

Reviewer comment (Experiments):
The experimental comparison should be with approaches that are asynchronous end to end like HOTS or HATS or Cannici'19, Perot'20 etc. or approaches like the Event Transformer.

Author response:
Thank you for your suggestion! The above methods you mentioned are all about event representation methods. It is worth noting that **our work focuses on the slicing of the event stream rather than focusing on event representation.** Event representation refers to the process of event information extraction that is performed after the event stream has been sliced into sub-event stream, and the resulting event representation meets the neural network input requirements. Thus, our dynamic slicing process and event representation can be used at the same time, either better slicing or representation method benefits the feature extraction with neural network, thus improving performance.

To validate the effectiveness of our slicing approach, we supplement the event-based recognition task below. We compare the downstream performance of three different event representation methods (including Event Frame [1], Event Spike Tensor (EST [2]) and Voxel Grid [3]) on the DVSGesture dataset under fixed slicing and dynamic slicing: 


| DVSGesture |Event Frame | Event Spike Tensor | Voxel Grid |
|:------|:----------------|:--------|:----|
| Fix Duration | 93.75% | 93.75% | 88.54% |
|Fix Event Count |93.06%|94.79%|88.19%|
|**BEEF(ours)** |**94.79%**|**95.49%**|**89.24%**|


The results demonstrate that across different event representation methods, our dynamic slicing approach, BEEF, outperforms fixed slicing methods in downstream tasks. This underscores the efficacy of BEEF in handling event streams. 

Thanks for your suggestion! We will cite the event representation methods you mentioned in the revised paper and try to integrate them with BEEF in the future.

***Reference:***

[1] Maqueda A I, Loquercio A, Gallego G, et al. Event-based vision meets deep learning on steering prediction for self-driving cars. CVPR 2018.

[2] Gehrig D, Loquercio A, Derpanis K G, et al. End-to-end learning of representations for asynchronous event-based data. ICCV 2019.

[3] Zhu A Z, Yuan L, Chaney K, et al. Unsupervised event-based learning of optical flow, depth, and egomotion. CVPR 2019.

Reviewer comment (Writing):
It is unclear whether events are treated differently according to their polarity.

Author response:
We apologize for not talking about the handling of event polarity in the main text. Our approach retains the polarity information of the events. The information for each polarity is accumulated separately. Additionally, we have compared our method with other event representation techniques, which are presented in Reply 6. We will ensure to clearly articulate these experimental details in the subsequent version of our manuscript.

Reviewer comment (Writing):
There is some problem with the definition of ${\cal D}$ because $n_q$ is not defined anywhere but mentioned “where $n_q$ denotes the time of the last spike”.

Author response:
My apologies for the confusion caused by the writing error in Section 4.2. The phrase "where $n_q$ denotes the time of the last spike" should indeed be removed as it was not previously defined. Thank you for bringing this to our attention. We will correct this error shortly!

Reviewer comment (Evaluation):
It would be worth listing the latency from event to GPU output for the particular architectures on tracking and recognition. This is much more critical here than the power consumption of the CNN.

Author response:
I agree. It's of great significance to consider the latency of in the processing of events, especially in practical applications. As documented in Table 2 of our manuscript, the Frames Per Second (FPS) rate achieved by utilizing the SNN to slice the event stream is 111Hz, which corresponds to processing once every 0.009 seconds. This rate is significantly lower than the latency experienced by the downstream ANN models during tracking tasks, which have an FPS rate of 39. Consequently, our BEEF model can facilitate real-time dynamic event slicing. 


|  |Latency | FPS |
|:------|:----------------|:--------|
| SNN | 0.009s per img | 111 Hz| 
|ANN |0.025s per img|39 Hz|

Reviewer comment (Experiments):
The authors need to explain the slicing method more clearly (possible misreadings are listed above). My main concern is the lack of any experimental analysis or motivation for the particular quite elaborate slicing method.

Author response:
To demonstrate the effectiveness of our proposed dynamic slicing method, we provide the following statistical results:

### **1. Statistics of dynamic slicing (BEEF) vs. fixed slicing.**
*Symbol Description: the total event stream $E$; the resulting sliced sub-event stream list by BEEF: $E_{beef}=[E^b_{1},..E^b_{N_1}]$; the resulting sliced sub-event stream list by fixed slicing method: $E_{fixtime}=[E^f_{1},..E^f_{M_1}]$.*

In the tracking task, the average duration of each sub-event stream $E^b_{k}(k\in[1,N_1])$ is 65ms (corresponding to 13 event cells, and the duration of the event stream contained in each event cell is 5ms). The maximum duration of each sub-event stream is 100ms, and the minimum duration is 30ms, while for our comparison of the slicing-by-fixed-time approach, the duration of each sub-event stream $E^f_{j}(j\in[1,M_1])$ is fixed at 75ms. The following are specific statistics:

|Method| Avg Cell Num| Var Cell Num | Avg Duration| Min Duration | 25th Duration| 75th Duration| Max Duration|
|:--------|:--------------------------------------|:---------------------------------------|:------------------------------|:--------|:-----|:-----|:----|
| BEEF                                  |  12.99                          | 3.96    | ~65ms                                  |25ms    | 50ms   | 80ms  | 100ms |
| Slice by fixed duration               |  15                             | 0       | 75ms                                  |//   | //   | //  | //  |

We will put the visualization of the statistic results in the appendix of the revised version.

### **2. Statistics of event density.**

We also counted the average density of sub-event streams after fixed slicing and dynamic slicing for comparison.

*Symbol Description: For each sub-event stream $E^b_{k}$ or $E^f_{j}$, it contains several event points $e_i=[x_i,y_i,t_i,p_i]$. We define a matrix $C$ to represent the event count, where $C_{xy}$ represents the number of events at coordinates $(x,y)$. Given a threshold $T$, we define the event density to be: $D = \frac{\sum_{x,y}\mathbb{1}\{C_{x,y}\geq T\}}{\sum_{x,y}\mathbb{1}\{C_{x,y}> 0\}}$. The threshold $T$ is determined based on the percentile of the event count $C_{xy}$.*

**Density Analysis**: We define the density of events as a metric reflecting the amount of event information within sub-event streams. The smaller the fluctuation in the density of each sub-stream (the smaller the variance), the more stable the event information contained. The stability of the event density is crucial for scenarios where the distribution of events is not uniform (e.g., scenarios with changing motion speed).


We chose $T = 30$% and $90$% (lower $T$ to observe overall event activity, including those less frequent events; higher $T$ for focusing on repetitive or frequent events) to validate the effectiveness of BEEF to dynamically slice the event stream. Below are the statistics of event density, where the data (left vs right) on the left are statistics of BEEF and on the right for statistics of fixed slicing. We select 4 classes in the FE108 tracking dataset.

| T=30% | airplane_mul222 | box_hdr | dog | tank_low |
|:------|:----------------|:--------|:----|:---------|
| Mean  | 0.9196 vs 0.9074 | 0.9850 vs 0.9850 | 0.9208 vs 0.8984 | 0.9584 vs 0.9575 |
| Var $\downarrow$| **0.0151** vs 0.0164 | **0.0038** vs 0.0039 | **0.0144** vs 0.0168 | **0.0096** vs 0.0100 |
| Std $\downarrow$ | **0.1231** vs 0.1283 | **0.0622** vs 0.0626 | **0.1200** vs 0.1299 | **0.0984** vs 0.1003 |

| T=90% |airplane_mul222 | box_hdr | dog | tank_low|
|:------|:----------------|:--------|:----|:---------|
| Mean | 0.1256 vs 0.1263 | 0.1338 vs 0.1405 | 0.1120 vs 0.1131 | 0.9584 vs 0.9575 |
| Var $\downarrow$| 0.0004 vs 0.0004 | **0.0007** vs 0.0053 | 0.0001 vs 0.0001 | **0.0096** vs 0.0100 |
| Std $\downarrow$| 0.0215 vs 0.0215 | **0.0267** vs 0.0732 | **0.0073** vs 0.0078 | **0.0984** vs 0.1003 |

The results show that the event stream after BEEF's dynamic slicing has a more stable event density (low variance), which verifies that BEEF has a certain ability to perceive the event information, and ensures that BEEF is robust in different motion scenarios, as shown in Table 3.

Reviewer comment (Novelty):
There is no motivation to use an SNN since the slicing is only a minimal energy and latency fraction of a pipeline that uses transformers or regression.

Author response:
Thanks again for all your suggestions and questions! Your valuable suggestions greatly help us to improve the content and quality of our articles! Please allow me to revisit and clarify the motivation behind our dynamic event stream slicing algorithm and how we've verified its effectiveness.

### **1. Motivation for Proposing a Dynamic Event Stream Slicing Algorithm**

Let's start by clarifying the process of event-to-frame conversion, which is mainly divided into two steps: **Step 1. Slice the raw event stream into multiple sub-event stream** and **Step 2. convert these sub-event streams into frames using various event representation methods.** While many works has focused on optimizing event representation (Step 2) to extract better event information, including time surface and EST, they do not address the issues arised with fixed slicing (e.g., resulting non-uniform event in scenarios with changing motion speed). Despite event slicing being a small part of the overall pipeline, it is a critical point. This is because the event stream is very sensitive to slicing, and the model performance fluctuates very much for different slicing methods, as proved by extensive experiments in Reply 1.

To better address this issue, we introduced the dynamic slicing framework BEEF. Meanwhile, BEEF is guided by downstream task feedback to ensure that the new sub-streams could enhance downstream task performance.




### **2. Motivation for Using SNN as a Slicing Trigger**

The reason why we choose SNN as the event slicing trigger is twofold:

- Utilizing SNNs on neuromorphic hardware for processing event streams is low-energy and low-latency [1,2].

- Deployed on neuromorphic hardware, SNNs can process event streams asynchronously [3,4,5], conserving energy when there is no data input—a capability that GPUs, operating synchronously, lack.

Due to the aforementioned reasons, there is a considerable amount of research [6,7,8,9,10] employing Spiking Neural Networks (SNNs) for event data. Although these SNNs are simulated on GPU platforms, the models resulting from such simulations could be deployed on neuromorphic hardware [3,4].

### **3. Contribution for Using SNN as a Slicing Trigger**
We propose a new cooperative paradigm where SNN acts as an efficient, low-energy data processor to assist the ANN in improving downstream
performance. This is a brand-new SNN-ANN cooperation way, paving the way for future event-related implementation on neuromorphic chips.

### **4. Comparison with Other Event Representations**
Although our article focuses on dynamic event stream slicing, not event representation, we appreciate your suggestion that verifying the effectiveness of dynamic slicing BEEF through different event representations is necessary. As I mentioned earlier, event slicing and event representation are not in conflict, and a fusion of the two may have potentially better enhancements. Experiments with different slicing methods and different event representation have been supplemented in Reply 6.

Reviewer comment (Experiments):
There is no comparison with architectures that use other event representations like time surfaces.

Author response:
Thanks again for all your suggestions and questions! Your valuable suggestions greatly help us to improve the content and quality of our articles! Please allow me to revisit and clarify the motivation behind our dynamic event stream slicing algorithm and how we've verified its effectiveness.

### **1. Motivation for Proposing a Dynamic Event Stream Slicing Algorithm**

Let's start by clarifying the process of event-to-frame conversion, which is mainly divided into two steps: **Step 1. Slice the raw event stream into multiple sub-event stream** and **Step 2. convert these sub-event streams into frames using various event representation methods.** While many works has focused on optimizing event representation (Step 2) to extract better event information, including time surface and EST, they do not address the issues arised with fixed slicing (e.g., resulting non-uniform event in scenarios with changing motion speed). Despite event slicing being a small part of the overall pipeline, it is a critical point. This is because the event stream is very sensitive to slicing, and the model performance fluctuates very much for different slicing methods, as proved by extensive experiments in Reply 1.

To better address this issue, we introduced the dynamic slicing framework BEEF. Meanwhile, BEEF is guided by downstream task feedback to ensure that the new sub-streams could enhance downstream task performance.




### **2. Motivation for Using SNN as a Slicing Trigger**

The reason why we choose SNN as the event slicing trigger is twofold:

- Utilizing SNNs on neuromorphic hardware for processing event streams is low-energy and low-latency [1,2].

- Deployed on neuromorphic hardware, SNNs can process event streams asynchronously [3,4,5], conserving energy when there is no data input—a capability that GPUs, operating synchronously, lack.

Due to the aforementioned reasons, there is a considerable amount of research [6,7,8,9,10] employing Spiking Neural Networks (SNNs) for event data. Although these SNNs are simulated on GPU platforms, the models resulting from such simulations could be deployed on neuromorphic hardware [3,4].

### **3. Contribution for Using SNN as a Slicing Trigger**
We propose a new cooperative paradigm where SNN acts as an efficient, low-energy data processor to assist the ANN in improving downstream
performance. This is a brand-new SNN-ANN cooperation way, paving the way for future event-related implementation on neuromorphic chips.

### **4. Comparison with Other Event Representations**
Although our article focuses on dynamic event stream slicing, not event representation, we appreciate your suggestion that verifying the effectiveness of dynamic slicing BEEF through different event representations is necessary. As I mentioned earlier, event slicing and event representation are not in conflict, and a fusion of the two may have potentially better enhancements. Experiments with different slicing methods and different event representation have been supplemented in Reply 6.

Reviewer comment (Theory):
The paper claims a fixed event split method fails to generalize. However, event cell $C[N]$ is a discrete 2D representation generated from a fixed event split, and is used as the input for SNN.

Author response:
Thank you for your comment! You are correct in noting that our event cells adopt a 2D format. Ideally, our goal is for the Spiking Neural Network (SNN) to process raw event inputs asynchronously. However, due to the constraints that CNN-based deep learning frameworks can only accept 2D inputs, we also adopted 2D format, which is a commonly-used method for processing event [1,2,3].

In our work, we have endeavored to slice the entire event stream into very fine event cells along the timeline as much as possible, since we want to find the optimal and accurate event slices. 
These event cells contain very little information, and our SNN is able to process the event cell and supervised to slice the event stream adaptively.

In the future, we expect to further our research by deploying SNNs on neuromorphic hardware. This would enable the asynchronous processing of event streams using BEEF, potentially overcoming current constraints and enhancing the efficiency of our approach.

***Reference:***

[1] Zhang J, Yang X, Fu Y, et al. Object tracking by jointly exploiting frame and event domain. ICCV 2021.

[2] Baldwin R W, Liu R, Almatrafi M, et al. Time-ordered recent event (TORE) volumes for event cameras. TPAMI 2022.

[3] Maqueda A I, Loquercio A, Gallego G, et al. Event-based vision meets deep learning on steering prediction for self-driving cars. CVPR 2018.

Reviewer comment (Theory):
BEEF can be used in ANN-based 3D CNN/Transformer seamlessly. Event cameras and SNN are all bio-inspired but do not necessarily imply that SNN is a good fit to event data.

Author response:
Thank you for your comment!
In the original article, we mentioned that ``both SNN and event stream are brain-like inspired, which motivates us to use SNN to process events'', and there is indeed a problem with the logic of this statement. I appreciate you reminding us of this point. Therefore, here we have reorganized the motivation for using SNN to process events:

- Utilizing SNNs on neuromorphic hardware for processing event streams is low-energy and low-latency [1,2].

- Deployed on neuromorphic hardware, SNNs can process event streams asynchronously [3,4,5], conserving energy when there is no data input—a capability that GPUs, operating synchronously, lack.

Due to the aforementioned reasons, there is a considerable amount of research [6,7,8,9,10] employing Spiking Neural Networks (SNNs) for event data. Although these SNNs are simulated on GPU platforms, the models resulting from such simulations could be deployed on neuromorphic hardware [3,4].

We hope this explanation satisfactorily addresses your concerns and illustrates the rationale behind our methodology.

***Reference:***

[1] Davies M, Srinivasa N, Lin T H, et al. Loihi: A neuromorphic manycore processor with on-chip learning. IEEE Micro 2018.

[2] Akopyan F, Sawada J, Cassidy A, et al. Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip. IEEE TCAD 2015.

[3] Roy A, Nagaraj M, Liyanagedera C M, et al. Live Demonstration: Real-time Event-based Speed Detection using Spiking Neural Networks. In CVPRW 2023.

[4] Viale A, Marchisio A, Martina M, et al. Carsnn: An efficient spiking neural network for event-based autonomous cars on the loihi neuromorphic research processor. In IJCNN 2021.

[5] Yu F, Wu Y, Ma S, et al. Brain-inspired multimodal hybrid neural network for robot place recognition. Science Robotics 2023.

[6] Hagenaars J, Paredes-Vallés F, De Croon G. Self-supervised learning of event-based optical flow with spiking neural networks. NeurlPS 2021.

[7] Yao M, Gao H, Zhao G, et al. Temporal-wise attention spiking neural networks for event streams classification. ICCV 2021.

[8] Zhu L, Wang X, Chang Y, et al. Event-based video reconstruction via potential-assisted spiking neural network. CVPR 2022

[9] Kosta A K, Roy K. Adaptive-spikenet: event-based optical flow estimation using spiking neural networks with learnable neuronal dynamics. ICRA 2023.

[10] Hussaini S, Milford M, Fischer T. Spiking neural networks for visual place recognition via weighted neuronal assignments. RAL 2023.

Reviewer comment (Experiments):
Why not experiment with the latest event recognition/single object tracking framework? The latest methods in Tab. 1 and Tab. 3 were published in 2021?

Author response:
To enhance the credibility and robustness of our results, we have incorporated state-of-the-art models: Swin Transformer (SwinT) and Vision Transformer (ViT), to further validate the efficacy of our BEEF algorithm in event-based recognition tasks:


| **Method** | Random Slice | Fixed Slice | **Ours** |
|:-------|:-------|:-------|:------|
| SwinT | 88.19 | 89.93 | **91.67(+1.74%)**|
| ViT | 87.50 | 85.07 |**88.54(+3.47%)**|

*Experiment Settings: We choose SwinT-small and ViT-small for comparisons on DVSGesture dataset. Other settings are consistent with the main experiments.*

BEEF is designed as a **plug-and-play** algorithm for dynamic event stream slicing. It is benchmarked against baselines that employ fixed event stream slicing methods. Our approach is versatile and can be applied in any event recognition or single object tracking framework (including both classical and latest). 

We appreciate your suggestion and hope that this response adequately addresses your query.