Paper: Spiking CenterNet: A Distillation-boosted Spiking Neural Network for Object Detection (ICLR-2024)

Reviewer comment (Novelty):
Please highlight more clearly the differences between Cordone et al. (2022) and the proposed method (other than applying knowledge distillation).

Author response:
Thank you for your review. Our main difference to the work of Cordone et al. besides the use of KD is our CenterNet-based, anchor-less object detection via a heatmap which is particularly suited to spikes (and also enables an easy application of KD). It is also composed of very simple building blocks (see more below). We will try to highlight this stronger in our work.

Reviewer comment (Novelty):
Please discuss what are the challenges of applying knowledge distillation for SNN object detection, compared to existing knowledge distillation methods between other types of networks.

Author response:
Thank you for pointing out that the challenges of KD in SNN-based object detection are not clear enough in our work; there are a variety of them, and we will try to showcase them better. We should point out, however, that all previous works of KD with SNNs focused on classification as the topic of SNN-based object detection even without KD is not yet very well researched.

Reviewer comment (Reproducibility):
Please provide more details (setup and tool flow) for the implementation of the proposed method on the neuromorphic hardware.

Author response:
We would like to integrate our model into suitable spiking neuromorphic hardware. That is our main motivation behind choosing relatively simple building blocks of convolution layers immediately followed by spiking neurons to ensure we do not propagate float values over long distances such as the works of Su et al. (2023). However, the only commercially available SNN chip, the Akida Brainchip, is not only aimed at a different neural network framework, but also focuses on running converted, rather than trained SNNs and does not offer many details about the underlying spiking dynamics. Nevertheless, we are in close contact with an institution that is developing their own SNN chip and future works of ours aim at running our models on their hardware."

Reviewer comment (Evaluation):
The innovative KD in this paper has little effect on the accuracy, and only improves by 0.006. Related knowledge is not explained clearly, such as event-based object detection methods. The advantages of using CenterNet, such as the need for an NMS, are not explained.

Author response:
Thank you for your review. We have seen that our Knowledge Distillation(KD) approach does indeed improve performance significantly by a mean difference of 1.8 mAP. A single outlier of our 5  training runs without KD with a performance close to the KD-based trainings may mask that effect, but highlights why the maximum value is not very reliable. The only reason we include it is to keep some comparability to previous works which only report the maximum performance. We do acknowledge that we cut back on explaining some other advantages of CenterNet, which are not related to its suitability for SNNs, due to the page limitation. We will try to remedy this.

Reviewer comment (Experiments):
The authors should explore deeper backbone for the proposed framework, such as ResNet50, ResNet101. Whether the effect is better than shallow network?

Author response:
Exploring the effects on deeper networks is of interest to us but they introduce significant challenges. First, due to the additional time dimension the hardware requirements rise significantly, making training iterations more difficult. Second, the error due to the approximate nature of surrogate gradients occurs in each layer and thus accumulates with increased depth of the network. Additionally, the spike-based nature makes sustaining gradients over deep networks difficult. This means deeper SNNs are generally harder to train by sheer computational limits, even without factoring in the inference of a teacher model. However, it is true that KD might be help with some of these issues.

Reviewer comment (Reproducibility):
The details of the proposed method are not clear. For example, how is the identity mapping between encoder and decoder implemented, and how is it different from the implementation in ANN?

Author response:
Our chosen identity mapping between encoder and decoder is the same for both the ANN and SNN as the skip signal is concatenated to the decoder signal. It thus does not change the binary nature of the SNN skip signal. In contrast, residual identity connections are usually added to a module's output to let it learn the residuals, and thus introduces non-binary results after that addition which violate the binary nature of our spikes. Boolean alternatives which would preserve the binary nature severely limit the advantage of these connections and were not helpful in our initial trials.

Reviewer comment (Evaluation):
The relevant work is not fully introduced, and KD has little effect on performance improvement, which brings large energy consumption.

Author response:
We observed that while our KD did increase the energy consumption, the resulting network is still vastly more energy-efficient than the ANN alternative based on the number of MAC and AC operations.

Reviewer comment (Novelty):
In principle, there’s no new architecture built for object detection.

Author response:
Thank you for your review. We are proposing a new architecture as we not only created a spiking version of CenterNet, but also fundamentally changed its decoder part with structures inspired by M2U-Net. We focused on keeping the basic structures simple to maximize the likelihood that it will be supported by future neuromorphic SNN hardware.

Reviewer comment (Presentation):
The sparsity listed in table 2 is not the sparsity but the density (higher sparsity means sparser).

Author response:
Regarding the 'sparsity', we have used the interpretation of this term established in previous work (Cordone et al, 2022) to avoid confusion when comparing the works. We acknowledge, however, that we find this interpretation also rather unintuitive and will change it accordingly.

Reviewer comment (Evaluation):
Why emphases on KD a lot, given the benefits of KD are quite limited.

Author response:
SNN training due to the on-off behaviour of the spiking neurons is notoriously unstable and difficult. We have seen that while a single training run without Knowledge Distillation (KD) came close to our KD-based training, in the vast majority of cases our KD-based training was significantly better. This can be seen in the more reliable mean performance over our 5 trainings runs, rather than the maximum prone to outliers. We also see that while KD with an ANN teacher decreases the energy efficiency of the SNN, the SNN is still vastly more efficient than the non-spiking ANN as the sparse nature of the SNN has not changed much.