Paper: Soon Filter: Advancing Feed-Forward Neural Architectures for Inference at the Edge (ICLR-2024)

Reviewer comment (Evaluation):
There is a need for a more nuanced discussion on the hardware implications of the method.

Author response:
Thank you for reviewing our paper, providing valuable feedback and helping improve our paper. 

**Weakness 1 and Question 1:**

> The main objective of the paper is to reduce the complexity of implementation using hashing and filtering. However, very little is done to measure this complexity. Only the model size is reported, as proxy to complexity. But what about the overhead of implementing hashing and filtering?

We appreciate your feedback and have revised our paper accordingly to clarify its main objective. Contrary to what may have been implied, our goal is not to reduce implementation complexity through hashing and filtering. This aspect was already addressed by the BloomWiSARD paper and further advanced in ULEEN, as detailed in the Weightless Neural Network paragraph of the Background section. ULEEN extensively investigates the overhead of hashing and filtering, demonstrating its superiority over BNN in terms of latency, memory consumption, and energy efficiency. We direct readers to the ULEEN paper for more detailed insights into these aspects.

The primary focus of our paper is to address a gradient bottleneck in the BloomFilter's continuous relaxation. Our proposed solution, the SoonFilter, is detailed in the Methodology Section. It specifically targets this bottleneck, contributing a significant advancement in the field.

> How does that compare to regular binarization where values are simply clipped to +/-1?

Regular binarization, where values are simply clipped to +/- 1, is known as Binary Neural Networks (BNN). We compare against this model thoroughly in the experiments section. Please take a look at Section 4.1 of our revised paper.

> How does that compare to low bit width implementation (e.g., 4-bit or 8-bit)?

The comparison of multiplication-free models with traditional models, as well as with quantized (4-bit, 8-bit, etc) and sparse models, is already well covered in the literature. Therefore, we chose to focus our comparison on the state-of-the-art multiplication-free methods in our experiments. This approach allows us to provide a more direct and relevant assessment of the advancements our Soon Filter brings to this specific category of models.

> I found the discussion on hardware benefits and limitations of the method very weak.

For deployment on edge devices, our work builds upon the ULEEN framework with minimal modifications to its model, as detailed in the methodology section. This strategic choice ensures that our model inherits ULEEN's hardware performance characteristics, which were thoroughly explored in the ULEEN paper. Our model achieves significantly smaller sizes at the same accuracy levels as ULEEN, necessitating fewer filters for comparable accuracy. This efficiency allows us to set ULEEN's hardware performance as an upper bound for our method. Given our inheritance of ULEEN’s hardware characteristics, we chose to focus more on the theoretical development of our proposed method in our paper, as the hardware aspects were already well covered in the ULEEN work.

Reviewer comment (Experiments):
There is a need for more relevant empirical results on typical baselines employed in the ML community in 2023.

Author response:
**Weakness 2 and Question 3:**

In response to your feedback, we have updated our paper to better articulate our benchmark selection. The benchmarks we employed are standard in the edge inference community. This includes MLPerf Tiny, a subset of the MLPerf benchmark suite specifically designed for low-power, edge devices. It focuses on tasks like image classification, keyword spotting, and anomaly detection, crucial for models in resource-constrained environments. MLPerf Tiny offers a standardized framework for consistent and comparable benchmarking across different models and platforms, demonstrating the practical applicability of our approach in edge computing scenarios.

Additionally, we included a collection of UCI datasets, targeting edge applications that use structured data, and included MNIST as it is consistently used in all other papers we compare against, ensuring a relevant and direct comparison. The results for comparison models on MNIST are sourced directly from their respective papers.

For a detailed explanation of our benchmark choices, please refer to the Subsection 4.1 of our revised paper.

We hope that these updates and clarifications adequately address your concerns. If you have any further issues or need additional clarification, we are more than happy to assist or make necessary revisions as needed.

Reviewer comment (Novelty):
The proposed modification is very small compared to ULEEN.

Author response:
**Weakness:** Concerning the perceived smallness of our contribution, we understand that replacing the AND operation with a SUM in Bloom Filters might seem minor. However, we argue that this is a non-trivial change. Such a substitution is counterintuitive in Bloom Filters as it increases the false positive rate, which is typically undesirable. This alteration is grounded in a robust theoretical framework of gradient maximization, as elaborated in our methodology section. Our work, drawing inspiration from the developments in resnets, mirrors a similar pattern. At first glance, adding a skip connection in resnets might also appear a small contribution. Yet, their work became a cornerstone for many modern deep neural network (DNN) models. We contend that our contribution is significant and impactful, as evidenced by the results we achieved. Notably, our Soon Filter consistently halves the model size of ULEEN, marking a substantial advancement in multiplication-free models. For comprehensive evidence, please refer to the **MLPerf Tiny Results** and **MNIST Results** paragraphs in the experiments section in the revised paper. This size reduction facilitates the deployment of much more efficient models at the edge.

Reviewer comment (Experiments):
The application is quite niche and not well demonstrated.

Author response:
**Questions 1 and 2:** Weightless neural networks (WNNs) hold significant potential for operating with extremely low computational resources. However, they encounter a major challenge that must be addressed to make them applicable in more complex scenarios: their inability to achieve greater depth due to the discrete nature of the RAM indexing, which lacks a derivative, akin to embeddings. As a result, constructing deep models and convolutional models with WNNs is currently unfeasible. This limitation presents a substantial opportunity for future research. Overcoming this challenge could lead to groundbreaking advancements in deep learning, paving the way for the development of even more efficient and effective models. Consequently, future research efforts should concentrate on devising algorithms capable of facilitating back-propagation with discrete operations such as RAM indexing.

Reviewer comment (Evaluation):
The improvements are very small.

Author response:
**Weakness:** Concerning the perceived smallness of our contribution, we understand that replacing the AND operation with a SUM in Bloom Filters might seem minor. However, we argue that this is a non-trivial change. Such a substitution is counterintuitive in Bloom Filters as it increases the false positive rate, which is typically undesirable. This alteration is grounded in a robust theoretical framework of gradient maximization, as elaborated in our methodology section. Our work, drawing inspiration from the developments in resnets, mirrors a similar pattern. At first glance, adding a skip connection in resnets might also appear a small contribution. Yet, their work became a cornerstone for many modern deep neural network (DNN) models. We contend that our contribution is significant and impactful, as evidenced by the results we achieved. Notably, our Soon Filter consistently halves the model size of ULEEN, marking a substantial advancement in multiplication-free models. For comprehensive evidence, please refer to the **MLPerf Tiny Results** and **MNIST Results** paragraphs in the experiments section in the revised paper. This size reduction facilitates the deployment of much more efficient models at the edge.

Reviewer comment (Evaluation):
The evaluation is inadequate.

Author response:
**Question 3:** For deployment on edge devices, our work builds upon the ULEEN framework with minimal modifications to its model, as detailed in the methodology section. This strategic choice ensures that our model inherits ULEEN's hardware performance characteristics, which were thoroughly explored in the ULEEN paper. Our model achieves significantly smaller sizes at the same accuracy levels as ULEEN, necessitating fewer filters for comparable accuracy. This efficiency allows us to set ULEEN's hardware performance as an upper bound for our method. Given our inheritance of ULEEN’s hardware characteristics, we chose to focus more on the theoretical development of our proposed method in our paper, as the hardware aspects were already well covered in the ULEEN work.

Reviewer comment (Novelty):
While the central contribution of the paper revolves around the introduction of the "Soon Filter," it is essential to consider augmenting this with supplementary contributions or by providing more extensive exploration and insights to enrich the overall content and scholarly impact.

Author response:
**Weakness 2:** Our primary focus is indeed on the introduction of the Soon Filter. We argue that this represents a high-impact contribution, as demonstrated by the results we have obtained. Notably, our Soon Filter consistently reduces the model size of ULEEN by half, a significant advancement. For detailed evidence of this, please refer to the **MLPerf Tiny Results** and **MNIST Results** paragraphs in the experiments section in the revised paper. This reduction enables the deployment of far more efficient models at the edge, a contribution we believe to be of considerable scholarly significance.

Reviewer comment (Novelty):
Based on Figure 1 and the main content, is the primary distinction between these methods solely attributable to the replacement of the OR operation with the SUM operation?

Author response:
**Question 1:** Yes, the distinction between the Bloom Filter and the Soon Filter is solely attributable to the substitution of the AND operation with the SUM operation. This key difference is thoroughly detailed in our methodology section. Furthermore, in our experiments, we have ensured that the training regime for both ULEEN and our proposed SULEEN remains the same, with the only difference being the use of the Bloom Filter and the Soon Filter, respectively. Please refer to the  **Hyperparameter Tuning**, **Preprocessing**, and **Training** paragraphs in the experiments section in the revised version of our paper for further details.

Reviewer comment (Experiments):
Regarding the VWW dataset, have you evaluated the performance of these methods on smaller models like MCUNetV1 [1] and MCUNetV2 [2]? Such an analysis could provide insights into the accuracy reduction when transitioning from multiplication-based models and offer valuable insights.

Author response:
**Question 2:** The comparison of multiplication-free models with traditional models, as well as with quantized and sparse models, is already well covered in the literature. Therefore, we chose to focus our comparison on the state-of-the-art multiplication-free methods in our experiments. This approach allows us to provide a more direct and relevant assessment of the advancements our Soon Filter brings to this specific category of models.