Paper: Instant Complexity Reduction in CNNs using Locality-Sensitive Hashing (ICLR-2024)

Reviewer comment (Theory):
The motivation is clearly written but there's a gap between that and the method. In the introduction, with the paragraph starting from "However, the application of existing work is restricted..." clearly motivates post-training methods that are data-free. That makes sense, and your method falls in that category. However, what's less clear is why hashing in particular improves over other methods in this category of data-free, post-training methods. Why is hashing the magic solution that defeats all other post-training methods?

Author response:
**Why is hashing chosen over other methods for pruning in a data-free and post-training setting?** Our proposed approach utilizes locality-sensitive hashing (LSH) primarily due to its ability to run dynamically during model inference. This distinguishes it from existing approaches for data-free and post-training pruning, which are static and perform pre-processing on the trained model's weights to generate a fixed pruned model. To the best of our knowledge, this makes the HASTE module the first method that reduces model complexity, in particular, floating-point operations (FLOPs)
- in a data-free manner
- without requiring any pre-processing of the model's weights
- dynamically (dependent on the module's input) during model inference.

and as such, acts as a plug-and-play replacement for convolutional modules. To cluster redundant information in latent feature maps dynamically, we require an approximate nearest neighbor search that is fast to run online, has limited FLOPs overhead and is able to adapt to the current input feature map. LSH fulfills these strict requirements and has shown its potential in previous work, as stated in Section 2 of our paper.

Reviewer comment (Experiments):
For example, another way to leverage the linearity of the convolution (per first few sentences in sec 3.3) is to simply use kmeans. What does hashing buy us that kmeans could not? In fact, maybe kmeans is a reasonable baseline to include?

Author response:
**Could $k$-means be an alternative to hashing?** Yes, it is possible to replace hashing by $k$-means clustering. However, in our scenario, it is not applicable. The reason for this is two-fold:
- Firstly, with $k$-means, we would need to fix the number of clusters $k$, enforcing a predetermined compression ratio onto each latent input. This can introduce large errors when compressing non-redundant feature channels when $k$ was set too high. Similarly, we might miss potential for pruning when there are more than $k$ clusters in the features.
- Secondly, the computational overhead of $k$-means is infeasible when considering dynamic pruning at runtime. Our proposed locality-sensitive hashing (LSH) approach has time complexity $O(L \cdot n \cdot d \cdot (1-s))$, where $L$ is the number of hyperplanes, $s\in(0,1)$ their degree of sparsity, $n$ is the number of vectors to be hashed and $d$ is their dimensionality. However, $k$-means clustering has time complexity $O(I \cdot n \cdot d \cdot k)$, where $k$ is the number of cluster centers and $I$ is the number of required iterations. We would require multiple iterations $I$ and set $k$ to the number of distinct channels we want to maintain after pruning. This quickly exceeds the cost of hashing (where $14 \leq L \leq 36$) and would require extensive hyperparameter testing to set custom values for $k$ at each layer. This is not required in our LSH approach, where layer-wise dynamic pruning ratios are possible due to variable numbers of hash collisions using the same fixed $L$ throughout all layers.

Reviewer comment (Presentation):
nit: Table 3's organization of the right 3 columns are a bit weird. Maybe have a column for FLOPS reduction and then separately have a "type" column (tuning, training, or data-free). Or, if you really want to use more space, have three columns with checkmarks. It's hard as-is to compare results/numbers. (Hrm, but this is already in Table 1. Why repeat this in Table 3?)

Author response:
**Organization of Table 3 makes it difficult to compare results. Maybe change to one "Type" column or use checkmarks?** We found that the current table format offered the clearest distinction of the three compared pruning approaches. We deemed it necessary to clearly distinguish our results from related work at a glance. As our method is fundamentally different by offering dynamic pruning without training or fine-tuning, the obtained results are not directly comparable.  To further improve readability, we might explore other formats and update Table 3 in the revised version of the paper.

Reviewer comment (Novelty):
In terms of novelty, the idea of merging the inputs based on redundancies obtained by hashing and only computing relevant operations is not new [1]. In particular, the differences between HASTE and [1] are marginal, and the empirical evidence suggests that [1] outperforms HASTE.

Author response:
**The method used by the authors is not new and was already used in [1], and the difference between both papers is marginal. Results suggest that [1] outperforms the proposed HASTE approach.** We disagree with this statement. While both papers share the aspect of using hashing to detect and merge redundant information in CNNs, a detailed analysis reveals fundamental differences.
- Firstly, Red++ [1] offers static pruning, where the model is compressed once and fixed in a pruned state for inference. In contrast, our HASTE module offers dynamic pruning, which adapts the compression to the input tensor during inference and requires no pre-processing of the model. This allows us to change the model's total FLOPs requirement instantly at runtime. We can therefore react to varying availability of hardware resources. The pruning steps in Red++ "require a few minutes in the worst case scenario, never exceeding half an hour" [1, Appendix C], which is infeasible for online inference. When the hashed model weights are not stored throughout inference, an even more expensive re-hashing of the base model is necessary, which can take hours for large models trained on ImageNet [1, Appendix C]. In contrast, the HASTE module does not require adjustments when the underlying model's weights are changed, as the compression is performed online on latent feature maps. This allows for applications in a federated learning or online learning setting with continuous weight updates. We acknowledge the need to clarify this distinction in our paper and will add a corresponding segment in the revised version.
- Secondly, the mechanisms used to compress information differ substantially in both works. In [1], the authors quantize ("hash") the individual weights of the baseline model to introduce redundancies in the weight matrices. We do not alter the model's weights and instead perform hashing using signed random projections, which map entire feature channels to a lower-dimensional space, where we find similarities between them. Red++ then structurally removes redundant components of the convolutional filters based on the quantized weights and generates a static pruned network. In comparison, we keep all filters and their weights throughout the network and compress them on-the-fly, based on redundancies in the current input. This allows us to maintain the model's expressivity. We achieve compression by directly comparing hash codes of input feature map channels and averaging their information content. 
- While the results presented in [1] are impressive, we argue that in the instant pruning setting considered in our paper does not allow for a direct comparison of both approaches. We are limited by the compute budget available at runtime. The authors of [1] are not subject to such limitations and make use of extensive pre-processing of the model's weights, which can take multiple hours and is computationally expensive. Nonetheless, we hypothesize that both approaches can be combined to improve the results. Unfortunately, the authors of [1] do not provide publicly available code.

Reviewer comment (Reproducibility):
The authors claim that the proposed HASTE method is parameter-free (first bullet point in the introduction) but then proceed to define two hyperparameters: L and s. Furthermore, these hyperparameters are dataset-specific, and there is no automatic way to find them.

Author response:
**The authors claim that their approach is parameter-free, but introduce two hyperparameters. Can this claim be removed?** This is correct, and we understand how this statement is misleading. To clarify this and avoid any confusion, we will remove this claim in the revised version of our paper.

Reviewer comment (Experiments):
The benchmarks are lacking. While it is understandable that not all researchers have access to the same compute resources, this research is performed in a data-free context, meaning that the authors should be able to benchmark on more recent and larger scale models. From the reviewer's perspective, this is not sufficient for a venue such as ICLR.

Author response:
**The provided benchmarks are lacking.** We provide a wide range of results for different models and additionally show scaling behavior within different model architectures. Nonetheless, our future work includes extensive testing of our proposed method on a variety of architectures and tasks. We hope to include more experiments in the revised paper.

Reviewer comment (Evaluation):
The authors claim that the method induces a limited overhead, but it seems that this overhead is only benchmarked in terms of FLOPs. While FLOPs as a metric has benefits (simple to benchmark and hardware-agnostic), it should be made explicit that latency and memory footprint are not considered, which limits the interest and impact of the method. On most hardware, having multiple sequential operations at a low level added to a network will significantly slow it down. For instance, it is well known that BN layers have very little overhead in terms of FLOPs, yet removing them can reduce latency by half on CPUs and GPUs.

Author response:
**The proposed approach is only benchmarked with regard to floating-point operations. Can the authors comment on latency and memory requirements?** We are aware of this concern and aim to benchmark our model in terms of memory requirement and latency to provide a more complete analysis in the revised version. Figure 2 shows the potential for memory reduction.

Reviewer comment (Theory):
The LSH used for the HASTE module is a variant of SimHash, but it lacks rigorous justification/theory. Different from SimHash using dense Gaussian projections, the paper used very sparse random projection (VSRP) to reduce the complexity. However, SimHash has a strict mapping from the collision probability to the cosine similarity, but signed VSRP does not (asymptotic analysis under some data assumptions might be doable, but no formal result in the literature as far as I know). More details about the hashing methods should be included to help the readers better understand the similarity preserving property of LSH and why the method works. More references on LSH methods and applications can also be added to help the readers appreciate the approach.

Author response:
**The locality-sensitive hashing (LSH) variant used in the paper lacks rigorous theory.** We agree that the very sparse random projections used in our proposed HASTE module do not offer the same theoretical foundations as dense Gaussian projections. We intend to follow this avenue of research in future work.
The choice of sparse projections is mainly motivated by compute budget limitations during runtime. Sparse projections require less overall floating-point operations and allow us to trade multiplications for cheap additions during the hashing process. 
To the best of our knowledge, there are no theoretical results that explicitly draw a link between the collision probability of two vectors and their cosine similarity in the sparse projection case. However, asymptotically, as the dimensionality of vectors grows to infinity, the collision probability converges to the case of dense Gaussian random projections [1, Lemma 2].

Reviewer comment (Presentation):
Section 3.2 and 3.3 have something in common and can be combined and compressed. This could free up space for more introduction/related context on the theoretical foundation of LSH/SimHash and VSRP.

Author response:
**Sections 3.2. and 3.3. can be compressed to give more space to context on theoretical foundations of LSH.** That is a good proposal, and we agree that a more in-depth background on hashing will be beneficial to the reader. We will rework these sections accordingly.

Reviewer comment (Experiments):
There might be other LSH methods used in HASTE, but they are not tested in the paper. The impact of $s$ (which corresponds to different types of projection) is not explored empirically. In the experiments, $s$ was fixed. An ablation study on $s$ could be useful and make the paper more self-contained. Evaluation of cases such as $s=0$, or using Gaussian projection instead of $\pm 1$ when $s=0$, would be informative.

Author response:
**Can you include an ablation study on $s$, the hyperplane support vector sparsity?** Yes, we believe the addition of such an ablation study will give us insights on the hyperparameter sensitivity and enable broader applicability of our method. While the choice of $s = 2/3$ is also motivated by [1], we acknowledge the need for a more detailed ablation and will add in in the revised paper.

**What happens when setting $s=0$?** In this case, the hyperplane normal vectors are densely filled with entries from $\{-1, 0, 1\}$, which differs from dense Gaussian projections with standard normally distributed vector entries. We will include this special case in our ablation on the hyperparameter $s$.

**What are the results when using dense Gaussian projections**? In this most basic case of LSH with random projections, we use normal vectors with entries drawn from a standard normal distribution. The computational cost increase compared to sparse projections is two-fold: Firstly, the amount of FLOPs required for hashing is increased by using densely filled vectors instead of sparse ones. Secondly, besides additions, we also need to perform multiplications in this case, which are more expensive to compute on a hardware level. We agree that this is an interesting baseline for the reader and will include it in our ablations.

Reviewer comment (Experiments):
For cosine similarity, another possible approach might be the count-sketch (CS) type algorithm and some later developments, including methods such as OPORP and SignOPORP. These algorithms might be even more efficient than VSRP, with a complexity benefit in certain dimensionality settings. As a comparison, for sparse RP used in HASTE, the complexity is $dL(1-s)$, whereas CS can achieve complexity $d$ in the most efficient case. Variations, such as setting $M = 4$ and repeating CS for 2 times when $L=8$, are possible. These alternatives could potentially be incorporated into HASTE.

Author response:
**Potential alternative methods of LSH are not tested in the paper. Algorithms like count-sketches could be even more efficient. A more detailed exploration or discussion of related methods would benefit the paper.** It is correct that we have not explored the usage of count-sketches to replace the hashing component in the HASTE module. We took a general look at count sketches and tensor sketches, but had concerns regarding online runtime. We will provide further motivation for our specific LSH choice and a discussion of related methods such as count-sketches in the revised paper.

Reviewer comment (Experiments):
The paper simply used one LSH method without mentioning other possible approaches. It would be better to more fully explore possible options empirically or, at least, mention more related methods and approaches to give readers a more complete picture and possible alternatives.

Author response:
**Potential alternative methods of LSH are not tested in the paper. Algorithms like count-sketches could be even more efficient. A more detailed exploration or discussion of related methods would benefit the paper.** It is correct that we have not explored the usage of count-sketches to replace the hashing component in the HASTE module. We took a general look at count sketches and tensor sketches, but had concerns regarding online runtime. We will provide further motivation for our specific LSH choice and a discussion of related methods such as count-sketches in the revised paper.

Reviewer comment (Evaluation):
Can you provide the distribution of the average similarity between the input vectors $x$ (flattened channels)? This could be a useful and interesting statistic to better motivate the proposed method, and also useful for algorithm design. Perhaps consider adding it to the paper.

Author response:
**Can you provide the distribution of average similarity between input vectors / input channels?** Yes, this is an interesting idea. Such a visualization could provide insights into which channels are found to be most similar across varying input data, hinting to little class-specific information. We will add the corresponding visualizations to our paper.

Reviewer comment (Evaluation):
The paper would greatly benefit from visualizations of feature maps after the hashing process. These visualizations would help readers understand which channels are considered redundant in the context of image classification. In other words, by visualizing the feature maps, it would become apparent whether the pruned model genuinely focuses on channels crucial for semantic concept recognition.

Author response:
**The paper would benefit from visualizations of the pruned feature maps**. This is a good point, and such visualizations would greatly enhance the understanding of our method. We believe that this will be beneficial to the understanding of our paper and will consequently add visualizations in the revised version of our paper.

Reviewer comment (Experiments):
The paper's experimental focus is primarily on image classification tasks. However, since the core objective of model pruning is to identify redundant feature channels and use the saved channels for accurate recognition of image concepts, it would be valuable to assess whether HASTE performs effectively in other image-related tasks, such as object detection. I strongly recommend that the authors extend their testing to these areas.

Author response:
**Extend testing to other tasks.** We also believe that additional experiments for the downstream usage of our proposed method could provide valuable insights. However, for our research, we focused on image classification, as is customary in the field. Nonetheless, future work includes the extension of our module to said tasks to fortify our claim.

Reviewer comment (Presentation):
There appears to be some missing information in Figure 3(b). It is unclear how the hyperparameter L impacts the method's performance. The values of L should be provided to clarify this aspect.

Author response:
**Is there information missing in Figure 3(b)?** We agree that this figure does not ideally convey the influence of hyperparameter $L$, as stated in its caption. Each data point represents an accuracy / FLOPs trade-off for a certain model, where, from left to right, the hyperparameter $L$ is increased. The corresponding data points are taken from Table 4 in Appendix A.3. To avoid a cluttered figure, we decided to not label the exact choice of $L$ for every point and instead provide a textual description in the fourth paragraph on page 7. However, as stated by the reviewer, this does not convey the full information and should be changed. We will improve upon this figure for a revised version in the updated paper to clarify this.

Reviewer comment (Novelty):
The paper introduces a method for compressing CNNs, which is a well-explored area of research. To better understand the novelty of the proposed HASTE module, the authors should provide a clearer differentiation between their method and existing approaches. A more detailed comparison with state-of-the-art methods, possibly in a tabular form, could help highlight the unique contributions. References to prior works like [Howard et al., 2017] on MobileNets and [Zhang et al., 2018] on Shufflenet could be used to benchmark and contrast the proposed method.

Author response:
**A tabular overview of existing approaches would provide more clarity.** We also believe that a simple overview helps the reader to clearly distinguish our contribution from existing work. For this reason, we included a variety of related approaches, their unique requirements and capabilities for dynamic pruning in Table 1. 

**Contrast your proposed method to prior works like MobileNets or ShuffleNets.** While these prior works consider small and efficient convolutional network architectures, our work focusses on the compression of large architectures that typically exhibit many redundant features. This is motivated in the first paragraph of our introduction: Highly scalable architectures like ResNets are particularly desirable for their performance and ability to generalize, but are not directly employable on embedded hardware. Models like MobileNets or ShuffleNets introduce larger inductive biases as a trade-off. Testing on these models is not the scope of this paper, but a promising direction for future work.

Reviewer comment (Experiments):
The paper presents results primarily on standard architectures like ResNet and VGG. It would be beneficial to see the impact of the HASTE module on a wider range of architectures, including more recent ones like EfficientNet [Tan and Le, 2019] or Vision Transformers [Dosovitskiy et al., 2021]. This would help validate the generalizability of the method.

Author response:
**An analysis on a broader range of architectures would be beneficial.** We agree that the extension of our HASTE module to other CNN architectures is an interesting direction. As stated above, we will explore this in future work.

Reviewer comment (Evaluation):
While the paper discusses FLOPs reduction, a more comprehensive analysis of the actual computational savings in real-world scenarios would be valuable. This includes memory footprint reduction, inference time measurements on hardware where such models are likely to be deployed, and energy efficiency evaluations.

Author response:
**A more detailed analysis regarding memory footprint and latency would be valuable.** This is correct, and we want to include memory benchmarks in the revised version of our paper. Figure 2 hints at the potential for memory reduction that is possible with our method. Further measurements are difficult, as our method is currently not natively supported in modern deep learning frameworks. However, saving memory accesses is crucial for employment on embedded hardware.

Reviewer comment (Evaluation):
The paper could benefit from a robustness analysis, showing how the pruned models perform under different conditions, such as adversarial attacks or out-of-distribution data. An error analysis could also be insightful, particularly if there are specific classes or types of data where the pruned models underperform.

Author response:
**The paper could benefit from an analysis of the method's robustness and potential errors, as well as limitations and failure modes.** Thank you for your proposal. An in-depth analysis of adversarial and out-of-distribution performance would be interesting, especially in comparison to related approaches. We will explore this in future work.

Reviewer comment (Experiments):
The method's performance appears to be sensitive to the choice of hyperparameters, such as the number of hyperplanes L. A more detailed exploration of this sensitivity, along with guidelines for hyperparameter selection in different scenarios, would be useful for practitioners.

Author response:
**Explore hyperparameter sensitivities and give guidelines for hyperparameter selection.** We provide an overview of the influence of hyperparameter $L$ in Figure 3(b). Additionally, we give results for various hyperparameter settings in Table 4 and 5 and therefore provide an outline of reasonable hyperparameter choices. We discuss the theoretically motivated setting for hyperparameter $s$ in Section 4.1. An additional ablation for $s$ is planned for the revised paper.

Reviewer comment (Evaluation):
A discussion on the limitations and potential failure modes of the proposed method would provide a more balanced view. For instance, under what conditions does the HASTE module fail to compress effectively, and what are the trade-offs involved?

Author response:
**The paper could benefit from an analysis of the method's robustness and potential errors, as well as limitations and failure modes.** Thank you for your proposal. An in-depth analysis of adversarial and out-of-distribution performance would be interesting, especially in comparison to related approaches. We will explore this in future work.

Reviewer comment (Reproducibility):
While the authors have stated their intention to release their code upon publication, ensuring that the code is well-documented and easy to use will be crucial for reproducibility. Additionally, providing pre-trained models and a user-friendly interface could facilitate broader adoption and experimentation by the community.

Author response:
**The availability of pre-trained models and a user-friendly interface would facilitate reproducibility.** We fully agree. All pre-trained models used in the paper are already publicly available and easily accessible through the PyTorch framework. As our module is a plug-and-play replacement for regular convolutional modules, the adaptation of our method is very simple.

Reviewer comment (Novelty):
Could you elaborate on the specific aspects of the HASTE module that distinguish it from prior work in CNN compression? Are there particular mechanisms or theoretical underpinnings that are unique to your approach?

Author response:
**What distinguishes HASTE from prior work in CNN compression?**
Our approach distinguishes itself from existing work for CNN compression by offering instant data-free and dynamic pruning. To the best of our knowledge, the proposed HASTE module is the first method that instantly reduces model complexity, in particular, floating-point operations (FLOPs)
- in a data-free manner (no training or fine-tuning)
- without requiring any pre-processing of the model's weights
- dynamically (dependent on the module's input) during model inference,

which allows us to adapt the model's FLOPs in real time to the availability of hardware resources.

**Are there unique mechanisms / theoretical underpinnings used?** We uniquely provide an extension of approximate nearest neighbor search using locality-sensitive hashing to compress latent feature maps in CNN architectures.

Reviewer comment (Experiments):
Have you tested the HASTE module on a broader range of neural network architectures, particularly more recent ones? If not, could you hypothesize on its performance or provide a rationale for the selection of the tested architectures?

Author response:
**An analysis on a broader range of architectures would be beneficial.** We agree that the extension of our HASTE module to other CNN architectures is an interesting direction. As stated above, we will explore this in future work.

Reviewer comment (Evaluation):
Can you provide a more detailed analysis of the computational savings in terms of memory, power, and inference time, particularly on edge devices where such compression techniques are most valuable?

Author response:
**A more detailed analysis regarding memory footprint and latency would be valuable.** This is correct, and we want to include memory benchmarks in the revised version of our paper. Figure 2 hints at the potential for memory reduction that is possible with our method. Further measurements are difficult, as our method is currently not natively supported in modern deep learning frameworks. However, saving memory accesses is crucial for employment on embedded hardware.

Reviewer comment (Evaluation):
Would you be able to include a robustness analysis against adversarial examples or performance on out-of-distribution data? An error analysis on the types of errors introduced post-compression would also be insightful.

Author response:
**The paper could benefit from an analysis of the method's robustness and potential errors, as well as limitations and failure modes.** Thank you for your proposal. An in-depth analysis of adversarial and out-of-distribution performance would be interesting, especially in comparison to related approaches. We will explore this in future work.

Reviewer comment (Evaluation):
Could you discuss any identified limitations or failure modes of the HASTE module? Under what conditions does it fail to deliver the expected compression or performance?

Author response:
**The paper could benefit from an analysis of the method's robustness and potential errors, as well as limitations and failure modes.** Thank you for your proposal. An in-depth analysis of adversarial and out-of-distribution performance would be interesting, especially in comparison to related approaches. We will explore this in future work.

Reviewer comment (Evaluation):
Have you considered the impact of dataset bias on the compression method, and how might this affect the generalizability of the pruned models?

Author response:
**Does your method exhibit dataset bias?** No. As our method is data-free and adapts to input feature maps on-the-fly without any previous learned adaptation to the data, we do not incorporate biases into our pruning approach.

Reviewer comment (Experiments):
How does the compression affect performance on different tasks beyond image classification, such as object detection or segmentation?

Author response:
**How does the compression affect performance of tasks beyond image classification?** This is an interesting question, and we aim to explore image-related tasks in future work. As semantic segmentation and object detection models often use pre-trained classification backbones, we hypothesize that these methods profit in a similar way.