Paper: Can we get the best of both Binary Neural Networks and Spiking Neural Networks for Efficient Computer Vision? (ICLR-2024)

Reviewer comment (Experiments):
Most of the binary neural network research binarizes the weights as well. However, this paper does not report the weight precision and compares the precision scheme with BNN in a comprehensive manner.

Author response:
**BNN weight precision and comprehensive comparison**

The accuracy numbers of our BANNs reported in Table 1 and 2 are with 32-bit floating point weights. We have also shown the BANN accuracies with 1-bit weights in Table 5 for direct comparison with BNNs. As we can see, our BANNs with 1-bit weights yield superior test accuracy compared to existing uni-polar BNNs. However, they do not surpass the test accuracies of state-of-the-art (SOTA) bi-polar BNNs due to the expressivity bottleneck explained in Appendix A.11. We have also shown the accuracies obtained by BANNs for different weight precisions from binary to floating point in Table 4. As we can see, the accuracy decreases progressively with the reduction in bit-precision which is expected.

Reviewer comment (Experiments):
The performance comparison against SNN is not comprehensive enough. E.g., The recent TET [R1], DSpike [R2], and DSR [R3] are excluded from Table 2.

Author response:
**Comprehensive performance comparison against SOTA SNNs**

We have now added the comparison of our proposed method with [1-3] as you suggested in Table 2. We have also compared with a few other recently proposed training methods [4-6] in Table 2. Our approach yields superior trade-off between accuracy and number of time steps for most of these works. Our approach at one time step surpasses the reported test accuracy of some of these techniques at more than one time step; other techniques, that are open-sourced, when extended to one time step, yield accuracies that are significantly lower compared to us on CIFAR10 as shown below. (This is based on the replicated implementation in our setup since the result for one time step was not reported).

| **Work** | **Architecture** | **Accuracy** | **Time steps** |
|------------------|----------------|--------------|------------------|
| SurrModu  [6]      | ResNet18               |87.02       | 1           |
| TET [1]           | ResNet19              | 87.84     | 1         |
| Our work          | ResNet18             | 92.50     | 1        |

Reviewer comment (Evaluation):
The proposed method shows worse performance on ResNet-50 than VGG-16, which is controversial with the full-precision model performance, why is that?

Author response:
**Lower performance of spiking ResNet50 over spiking VGG16**

This phenomenon of ResNets yielding lower performance compared to VGGs is primarily due to the issue of vanishing/exploding gradients in deep ResNets, which was somewhat addressed in the spike-element-wise (SEW) ResNet work [7]. We have now adopted the SEW ResNet architecture to train our SNNs, which yields higher test accuracy in ResNet50 compared to VGG16 on CIFAR10, as shown in the updated Table 1 in the revision. On ImageNet, although we can improve the accuracy of spiking ResNet50 with the SEW architecture, it is still ~1% below compared to VGG16 for one time step. This is consistent with recently published works [3, 8], that observe >3% drop in test accuracy with ResNet34 compared to VGG16 on ImageNet. Note that ResNet50 typically provides <1% accuracy increase from ResNet34 for low-time-step SNNs as shown in [7] and also observed in our experiments. Hence, ResNet50 is expected to yield lower accuracy than VGG16 for the current SNNs. Improving the accuracy of directly trained spiking Resnet models for ultra-low time steps is thus an important research direction.

Reviewer comment (Evaluation):
The proposed method incorporates the learnable potential threshold, I think it is important to report the evolvement of the threshold value during the training process, especially the training stability of the threshold value.

Author response:
**Evolvement of threshold during training**

The threshold value stabilizes fairly well during the later parts of the training. As mentioned in the last paragraph of Section 3.1, the threshold, i.e., the Hoyer extremum in each layer changes only slightly during the later stages of training, which indicates that it is most likely an inherent attribute of the dataset and model architecture. We have also added Appendix A.12, along with Fig. 5 to report this phenomenon.

Reviewer comment (Experiments):
Although the static image-based classification datasets (e.g., ImageNet) are good indicators of model performance, it is also important to evaluate the performance of SNN with DVS datasets due to the property of spatial-temporal events (e.g., DVS-CIFAR10, N-CalTech101, IBM-Gesture).

Author response:
**SNN Performance on DVS datasets**

We had included the performance of our SNNs on DVS-CIFAR10 in Appendix A.6 (now moved to Appendix A.7) of the original manuscript. In the revision, we have also added the performance comparison on another DVS dataset, N-Caltech 101, as you suggested. As shown in Table 10 and below, our BANNs, when extended to multiple time steps for both these tasks, yield $0.9$% higher test accuracy compared to the SOTA TET [1] training approach. This highlights the portability of our approach to DVS tasks. * denotes results based on our replicated implementation.


| **Dataset** | **Ref.** | **Architecture** |**Accuracy** | **Time steps** |
|------------------|----------------|--------------|------------------|------------------|
| DVS-CIFAR10   | TET [1]              |VGGSNN       | 83.17           | 10      |
| DVS-CIFAR10   | TET [1]              |VGGSNN       | 75.20*           | 4*    |
| DVS-CIFAR10   | tdBN-NDA [9]              | VGG11      | 81.7         | 10      |
| DVS-CIFAR10   | Our work          |VGGSNN       | 83.68          | 10      |
| DVS-CIFAR10   | Our work            |VGGSNN       | 76.17           | 4   |
| N-CalTech101  | tdBN-NDA [9]            | VGG11      |78.6        | 10      |
| N-CalTech101  | Our work          |VGG11       | 79.92           | 10    |

Reviewer comment (Novelty):
The proposed method is not closely related to SNN, and it mainly focuses on the BANN. This method can be used in SNNs for better model accuracy, but it can not bridge the gap between BNN and SNN as the title said. It would be preferable to discuss more about the relationship between BNN, SNN, and the proposed method to illustrate how this method bridges BNN and SNN.

Author response:
**Relationship between BNN and SNN**

Our BANNs are identical to one-time-step SNNs, and can be readily extended to traditional multi-time-step SNNs with the incorporation of the membrane potential (state variable). The threshold value of our BANNs remain fixed during inference, as explained in the last paragraph of Section 3.1, and can be used for all time steps in SNNs. This has been illustrated in the newly created Section 5, and added below. 

To extend BANNs to SNNs, we **use the traditional LIF model** for the neurons where **the membrane potential integrates the weight modulated input spikes and leaks over time**.

We use the **soft reset mechanism that reduces the membrane potential by the threshold value** when an output spike is generated. It has been shown that soft reset minimizes the information loss by allowing the spiking neuron to carry forward the surplus potential above the firing threshold to the subsequent time step [1,2]. 

**We use our proposed combination of Hoyer regularized training and Hoyer spike layer to train the per layer threshold, while we train the weights and leak term using BPTT**.

The equations governing our LIF model for our multi-time-step SNNs on static and DVS datasets are shown below.

$U_i^{temp}(t)=\lambda U_i(t-1)+\sum_j W_{ij}{S_j(t)}$  
$S_i(t) = 1 \ \text{if } U_i^{temp}(t)>V^{th} \ \text{else} \ 0$

$U_i(t) = U_i^{temp}(t)-S_i(t)V^{th}$

Here $U_i(t)$ denotes the membrane potential of the $i^{th}$ neuron at time step $t$, $S_i(t)$ denotes the binary output of the $i^{th}$ neuron at time step $t$, $V^{th}$ denotes the threshold, $\lambda$ denotes the leak, and $W_{ij}$ denotes the weight connecting the pre-synaptic neuron $j$ and the neuron $i$. Our BANNs, when extended to SNNs with multiple time steps, can lead to a small but significant increase in test accuracy as shown in Table 6, thereby demonstrating the efficacy of our approach. From another perspective, it can be argued removing the temporal dimension from the LIF model above leads to a BANN.

Our BANNs are also identical to previously proposed uni-polar sparse BNNs [3, 4] when the weight precision is quantized down to 1-bit. Moreover, we show that our training method results in our 1-bit-weighted BANNs having higher test accuracy compared to existing uni-polar BNNs in Table 5.
	
**Bridging the gap between BNN and SNN**

The sparse activation in BANNs bears resemblance to SNNs while the difference is that there is no temporal dimension. On the other hand, the 1-bit sparse activation in BANNs bears resemblance to BNNs, in particular uni-polar BNNs, while the difference is that the weight precision is multi-bit. Thus, it can be argued that BANNs bridge the gap between BNNs and SNNs by bringing the activation sparsity of SNNs to BNNs, and removing the temporal dimension of SNNs for equivalence with BNNs. This has been highlighted in the newly added Section 5 of the revision.

That said, we agree that our current title might not reflect our contributions very accurately. We have changed the title to ‘Can we get the best of both SNNs and BNNs for efficient computer vision?’. Please let us know if this title looks good to you.

Reviewer comment (Evaluation):
The energy evaluation method in the experiments is inaccurate. Usually, memory accesses take a large proportion of energy consumption. However, in the experiments, it only estimates the computed energy. It is better to consider the energy of memory access as well.

Author response:
**Incorporation of memory accesses in energy evaluation**

Our BANNs significantly reduce the memory energy compared to existing multi-time-step SNNs since the latter requires the membrane potentials and weights to be fetched from and read to the on-/off-chip memory for each time step. Our BANNs can avoid these repetitive read/write operations as it does involve any state and lead to a $T\times$ reduction in the number of these accesses compared to a $T$-time-step SNN model.

Compared to traditional bi-polar BNNs, our BANNs can also reduce the number of memory accesses with the support of zero gating logic leveraging the high activation sparsity. This can be achieved by skipping the reading of the spike activation when it is zero.  However, the exact savings in memory energy will depend on the data reuse scheme and the underlying hardware. 

We have now incorporated the memory accesses of the weights (and membrane potentials for SNNs) in our energy evaluation framework assuming a simple weight stationary dataflow, as shown in Appendix A.9 in the revision. We perform FPGA simulations to estimate the memory energy incurred in reading an 8-bit weight or membrane potential from the on-chip memory (BRAM and URAM) to the combinational logic. Based on the suggestions of reviewer X4qq, we have also incorporated the overhead of sparsity in our evaluation. We observe that our BANNs yield $70.28\times$ and $5.15\times$ lower energy compared to a traditional DNN and a 5-time-step SNN respectively with our improved energy (compute+memory) evaluation framework. We have updated our energy numbers in Fig. 2, Table 4, and Table 5 obtained from this framework.

Reviewer comment (Theory):
In section 2.2, “Note that this model is similar to leaky-integrate-and-fire (LIF) model”. This is not very direct. Why they are similar? In this model, there is no membrane potential, which is important in LIF.

Author response:
**Inappropriate use of the term ‘LIF model’ and wrong table reference**

We apologize for the confusion. In the original version, we stated that our proposed model is similar to LIF without the temporal dimension, where there is no concept of membrane potential. However, we agree this may confuse the readers, and we have now removed this sentence. Lastly, thanks for catching the incorrect table reference. We have now corrected the table reference in the revision.

Reviewer comment (Writing):
Typo: in section 4, Accuracy Comparison with SNNs, “Table 7” should be “Table 2”.

Author response:
**Inappropriate use of the term ‘LIF model’ and wrong table reference**

We apologize for the confusion. In the original version, we stated that our proposed model is similar to LIF without the temporal dimension, where there is no concept of membrane potential. However, we agree this may confuse the readers, and we have now removed this sentence. Lastly, thanks for catching the incorrect table reference. We have now corrected the table reference in the revision.