Paper: Are Spiking Neural Networks more expressive than Artificial Neural Networks? (ICLR-2024)

Reviewer comment (Theory):
The theoretical conclusions are great but it seems that they are highly dependent on the realization of SNN. In Theorem 1, it is assumed that the sum of synaptic weights of each neuron is positive. Later in Remark 3, the authors claim that when the assumption is not met, the realization of the network is not well-defined. This seems to be a strong assumption to me and I am wondering how can we guarantee the positive sum of synaptic weights. In practice, is this assumption met in experiments? What will happen in practice if this assumption is not satisfied?

Author response:
* We acknowledge the reviewer’s assessment that the results are dependent on the realization of SNN. If the sum of weights of a neuron is negative, then a neuron remains inactive and fails to encode any information. The assumption that the sum of synaptic weights of each neuron is positive is a sufficient but not necessary condition. In particular, it guarantees the well-definedness of the input-output function on the whole input domain, i.e., for each input the output neuron will fire (which is not necessarily required in practical applications). However, in each layer, there should be at least one neuron which satisfies the positivity condition to make sure that the information is propagated in the forward layers. In practical scenarios, the issue can be mitigated, e.g., by introducing some weight bumping mechanism as in [1]. They introduce a hyperparameter (a threshold value) which controls the number of neurons without any output spike. If the number of non-spiking neurons is above a certain threshold, then the input weights of the silent neurons are increased, thereby boosting the layer with insufficient spikes. The threshold in the output layer can be set such that all the neurons in the output layer always emit a spike. A similar mechanism can also be applied in our case. We are actively working on the practical implementation and it is currently in progress. 

[1] Göltz, J., Kriener, L., Baumbach, A. et al. Fast and energy-efficient neuromorphic deep learning with first-spike times. Nat Mach Intell 3, 823–835 (2021).

Reviewer comment (Writing):
The paper is self-consistent and rigorous but might be hard to follow for common readers. In Section 2, there are many formal definitions, which are general but may lose readability. For example, I think it would be better to provide a concrete formulation (or example) to introduce SNN and leave the formal definition in the appendix. The formal definition using a graph is general but might be hard to understand for most readers.

Author response:
* We appreciate the reviewer’s concern regarding the potential difficulty for common readers to follow the formal definitions presented in Section 2. While we acknowledge the suggestion to place these details in the appendix for improved readability, we believe that it is important to keep these definitions in the main text for a comprehensive understanding of the technical details throughout the paper. We want to point out that the understanding of the computations involving firing time and terminology associated with spiking neurons is essential for fully grasping the significance of our results.

Reviewer comment (Presentation):
To my understanding, Theorem 2 can serve as a comparison between SNN and ANN and implies that ANN can express the ReLU function more efficiently. Then I think it would be better to put Theorem 2 in section 3.2, where Theorem 2 and Example 1 study the advantages of ANN and SNN, respectively.

Author response:
* In response to the reviewer’s suggestion, we have incorporated Theorem 2 into Section 3.2.

Reviewer comment (Theory):
The title directly indicates the central concern of this paper but the answer seems to be incomplete. To my understanding, Theorem 3 implies that SNN is at least as expressive as ANN from the perspective of equivalence of approximation. Then a natural question is can SNN be more expressive than ANN, i.e., how many parameters are needed for an ANN to express an SNN? If we can get a similar result as Theorem 3, then we can conclude that ANN and SNN have similar expressive power, otherwise, a separation result confirms that SNN can be more expressive than ANN. Although this paper provides several results (Example 1 and Theorem 4) to demonstrate the advantage of SNN, Example 1 is not a separation result, and Theorem 4 is not demonstrated through the number of parameters. It seems that Theorem 4 can be translated into a separation result to make Theorem 3 more complete. For more literature about separation results, it might be helpful to read [1,2,3].

Author response:
* The question raised by the reviewer, “Can SNN be more expressive than ANN, i.e. how many parameters are needed for an ANN to express an SNN'' is indeed an important one. Theorems 1 and 3 establish that SNNs are at least as expressive as ReLU ANNs, both generating CPWL mappings. While ReLU ANNs can reproduce the output of any SNN, the bound on the number of parameters, neurons or layers required for this equivalence is an open question. The absence of a separation result in Example1 is acknowledged and the reviewer’s observation about Theorem 4 being translated into a separation result is what we plan to do. Understanding the bound on the number of linear regions in more complex settings (eg. multi-layer case) is pivotal, forming the groundwork for subsequent investigations. We also thank the reviewer for suggesting the literature on separation results. As for the title, we acknowledge that the answer is incomplete. It is essential to emphasize that our central focus is on the fundamental question of equivalence and demonstrating the differences in the structure of computations between SNNs and ANNs. While the prospect of SNNs being advantageous over ANNs is intriguing, our current findings represent an important initial step in that direction. It also provides a foundation for future explorations into more complex settings, for instance, incorporating multi-spikes and the coupled effect of response and threshold functions on the approximation results.

Reviewer comment (Writing):
The abbreviation CPWL occurs on page 3 (the 4th line of the 2nd contribution) for the first time, but its full name is introduced at the beginning of section 3.

Author response:
* We thank the reviewer for pointing this out. We have revised this in our work.

Reviewer comment (Novelty):
About the novelty. The current version of the paper has not sufficiently clarified its novelty. While the study of SNNs for universal approximation of continuous functions is mentioned, the function classes and settings explored in this paper, while different, may still be considered as an incremental progress in this field.

Author response:
* We believe that the novelty is outlined in the Contributions paragraph within the Introduction Section. Firstly, we prove that the SNNs have the capability to generate continuous piecewise linear (CPWL) mapping. Moreover, Theorem 3 shows that SNNs can reproduce the output of any ReLU network. Our expressivity result in Theorem 3 implies that SNNs can essentially approximate any function with the same accuracy and complexity bounds as (deep) ANNs employing a piecewise linear activation function, given the response function satisfies some basic assumptions, which has not been demonstrated before.

Reviewer comment (Novelty):
About the quality. The title may be somewhat overstated. This paper demonstrates that SNNs have the capability to approximate ANNs, rather than fundamentally surpassing them. As a result, the potential advantages of constructing non-continuous functions and utilizing sparse computation are not fully explored. Essentially, this work places a strong emphasis on establishing a bound that demonstrates SNNs can approximate ANNs, rather than thoroughly investigating whether and to what extent SNNs outperform ANNs. Therefore, the overall significance of this work may be somewhat limited.

Author response:
* It is important to highlight that our approach doesn't approximate ANNs. Rather, Theorem 3 shows that SNNs can reproduce the output of any deep ReLU network. Theorem 4 and Example 1 shows the superior expressiveness of SNNs over ANNs. In fact, it highlights a characteristic of spiking neurons that distinguishes it significantly from a ReLU neuron, thereby illustrating differences in the structure of computations between SNNs and ANNs. But we acknowledge that this work doesn’t explore the benefits of SNNs for constructing non-continuous functions and it is planned as future work.

Reviewer comment (Theory):
Is there any unique difficulties for spiking neurons but not ReLU neurons?

Author response:
* The behaviour of spiking neurons is influenced by various parameters, and the analysis becomes intricate when considering the entangled effect of all these parameters on the output produced by the spiking network. Additionally, the results depend on the encoding scheme that one adopts. The introduced encoding scheme translates analog information into input firing times in a continuous manner. For tasks that involve modeling complex time patterns or a sequence of events with multiple spikes, sophisticated encoding schemes may be required, adding another level of difficulty to the analysis. However, our focus in this work lies on exploring the intrinsic capabilities of SNNs, rather than the specifics of the encoding scheme.

Reviewer comment (Novelty):
The proving of Theorem 1, and 2 are similar to those in [Yar17] and [SCC18]. It seems that the only difference is using different neurons.

Author response:
* The references [Yar17] and [SCC18] that the reviewer mentions have not been cited in the manuscript. Can you please provide more details on this?

Reviewer comment (Novelty):
The universal approximation theorem of SNNs has been proved in [Maass, 1996c]. What is the difference between this paper and [Maass, 1996c]?

Author response:
* This reference [Maass, 1996c] has not been cited in the paper.  But assuming it corresponds to [1], the differences between our work and [1] have been highlighted in the Related Work Section of the main paper. In comparison to [1], we show that SNNs have the capability to realize the output of arbitrary ANNs with continuous piecewise linear (CPWL) activation functions and further specify the size of the network needed to achieve the associated realization, which has not been previously demonstrated. Moreover, we also study the expressivity of SNNs in terms of the number of linear regions and provide new insights on realizations generated by SNNs, that were not explored in [1].

Reviewer comment (Experiments):
In fact, the SRM model is not the most popular spiking neuron model in deep SNNs. Why do the authors use the SRM model, rather than the LIF neuron in [Wu et al., 2018]?

Author response:
* In line with our initial approach, we adopted the model choice based on [1] and we intend to expand our investigation to encompass multi-spike responses and refractoriness effects, thus, the choice of this model is appropriate and comprehensive. As mentioned in Section 2, page 4  "when the parameter $\delta$ is large, the simplified Spike Response Model, stemming from the linear response function and the constraint of single-spike dynamics, exhibits similarities to the integrate and fire model. Conversely, if $\delta$ is small, it resembles the leaky integrate and fire model.” As mentioned in the limitations section in the main paper, "our choice of model resides on theoretical considerations and not on practical considerations regarding implementation.” We acknowledge that there might be other models that are more apt for implementation purposes. Moreover, there are other works which in fact use the SRM model, for instance [2,3].