Paper: FARSE-CNN: Fully Asynchronous, Recurrent and Sparse Event-Based CNN (ICLR-2024)

Reviewer comment (Evaluation):
I would have liked to see a more thorough discussion on the number of events produced internally by this architecture. To run something like this on neuromorphic hardware efficiently, you need to ensure that a sparse number of events are created internally. A discussion on this in the paper, would improve it

Author response:
We thank the reviewer for pointing out the necessity of an analysis of the number of internal events. This will be included in the updated manuscript.

Reviewer comment (Experiments):
While the paper mentions both FARSE-CNN and SUB-FARSE-CNN, with the latter being an optimized improvement of the former, there is no experimental data to prove the effectiveness of this optimization. For example, there is no performance or computational complexity comparison.

Author response:
Regarding the effectiveness of our optimizations, Temporal Dropout and Submanifold FARSE-Convolutions. In Table 1 we have shown that, by changing the window size parameter of Temporal Dropout from 2 to 4, we can reduce complexity by 4 times at the cost of 1% accuracy loss on the NCars dataset. In Figure 4 we also compare the complexity of our network with one that does not use Temporal Dropout, and one that uses normal FARSE-Convolutions instead of Submanifold ones. The complexity growth in those cases (_e.g._, about 100x when using normal FARSE-Convolutions) makes it impractical to train a network due to time and memory constraints since this complexity is directly related to the number of internal events that are generated, hence why we don’t show performance comparisons.

Reviewer comment (Experiments):
Are the datasets shown here sufficient? I am aware of a few other event vision work that looks at some other event data-streams. Can the authors do more SOTA comparisons?

Author response:
In our evaluation, we chose datasets that have been used by previous asynchronous methods to allow for a direct comparison. We believe that additional SOTA comparisons are unfortunately not feasible at this time due to the lack of public code for the baselines. To the best of our knowledge, no public implementation is available for the strongest competitor, EventFormer, as the official project repository is currently empty (https://github.com/udaykamal20/EventFormer), while only the test/inference code is available for AEGNN, and not the training code.

Reviewer comment (Novelty):
Temporal dropout while interesting seems to be an already existing technique? [1] uses some dynamic temporal exit. Further, there are some temporal coding works [2] that use some interesting forms of temporal representation. Can the authors comment on how dropout is different from these?

Author response:
The reviewer noted that there are connections between our Temporal Dropout and previously existing techniques for SNNs. Indeed, the purpose of Temporal Dropout is to introduce a mechanism resemblant of spiking networks in our model, and guarantee that intermediate activations of the network are sparse temporally, as well as spatially. Without this mechanism, our architecture would propagate all events to the output.  As mentioned in our Conclusions, we believe that more sophisticated techniques to exploit temporal structure could be investigated, including Reinforcement Learning approaches as done in SEENN (Li et al., 2023).

Reviewer comment (Writing):
Abstract: It would be better to clarify or paraphrase, as these two sentences seem to contradict each other: "most successful algorithms for event cameras convert batches of events into dense image-like representations" and "achieves similar or better performance than state-of-the-art asynchronous methods". What was the goal of the paper - to beat the best methods or to develop a sota asynchronous pipeline? I assume the latter, but this needs to be stated more clearly in the abstract.

Author response:
About the confusing sentence in the abstract, the goal of the paper is indeed to develop a SOTA asynchronous pipeline. Therefore, the meaning we intended for our claim is "achieves similar or better performance than the state-of-the-art *among* asynchronous methods" *only*.  We will clarify this.

Reviewer comment (Evaluation):
From table 1, it seems that the accuracy is not the best (or significantly better compared to competition). The compute cost seems not the lowest as well. I believe a better explanation should be provided to explain the results.

Author response:
Regarding the performance of our method compared to competitors. It is true that, although our method achieves among the best trade-offs between accuracy and complexity, it is surpassed by EventFormer on some benchmarks. We already acknowledged this in Section 4.4, where we also provide a brief discussion of these results. EventFormer uses a very light, non-hierarchical architecture augmented with an associative memory that is able to achieve good performance on object recognition, particularly so in the semi-realistic scenario of N-Caltech101 that does not present challenging motion-rich information. On the other hand, we emphasize that our FARSE-CNN is easily adapted to the more complex task of object detection on real data, where it achieves the best results in both accuracy and complexity (especially given the revised complexity numbers of AEGNN). Finally, we believe that different methods could prove more effective in different tasks due to their inherent features, and so investigating diverse approaches is beneficial for a field that is still at its early stages, like event vision.

Reviewer comment (Novelty):
The authors mention, in the introduction, 3D convolutional networks. What is the main difference / advantage of the presented asynchronous scheme compared to 3D cnns, given that both leverage temporal information and, in theory, could be ran event-by-event? An example paper that explores this: https://openaccess.thecvf.com/content_CVPR_2020/papers/Mitrokhin_Learning_Visual_Motion_Segmentation_Using_Event_Surfaces_CVPR_2020_paper.pdf - it would be beneficial to add it to the review section as well.

Author response:
On the differences with 3D convolutional networks. While 3D convolutions can indeed leverage temporal information, standard 3D CNNs require dense inputs, which introduce a lot of redundant computation to the sparse activations of event cameras. For this reason, a line of work focuses on 3D convolutions on sparse graphs that can be updated in event-by-event fashion with less computational burden. The work by Mitrokhin et al. (2020), referenced in the review, also belongs to this family, together with the Graph Neural Network baselines that we considered (NVS-S, EVS-S, AEGNN). We thank the reviewer for pointing us to it, and we will include a reference in the updated manuscript. Differently from these methods, our approach uses a naturally recurrent architecture that can memorize past events in its state, instead of using fixed, explicit time windows. A similar observation, on the difference with LSTM-like approaches, is also made in Mitrokhin et al. (2020) in Section 6.