Paper: Faster and Accurate Neural Networks with Semantic Inference (ICLR-2024)

Reviewer comment (Theory):
In Eq 1, if L_eval is a loss (where lower is better), it seems trivial to find subnetworks that perform less well than the original DNN. So I imagine L_eval is not a loss (as the L would suggest), but some performance metric where higher is better. Even in that case the problem is not fully defined, since F_yi = F would satisfy Eq 1 while not being an useful solution.

Author response:
We thank the reviewer for this valuable comment. Yes, the evaluation metric shown in Semantic DNN Subgraph Problem (SDSP) does not correspond to the loss function. We define it as the evaluation criterion used for scoring the network and its subgraphs on their respective datasets. In our work, we have chosen the evaluation metric as accuracy. We understand the confusion regarding the notation which would not arise if the property of the evaluation metric (higher value means better performance) was mentioned. We have updated the manuscript accordingly. We have changed the notation of the evaluation criterion to $\mathcal{B}_{eval}$ and added 

*A higher value of $\mathcal{B}_{eval}$ is assumed to correspond to better performance*. 

As mentioned in the comment, $\mathcal{F}_{\gamma_i} = \mathcal{F}$ would satisfy Equation 1. In our case, the problem definition would be more precise if we used the word *proper subgraph*. This would exclude the trivial solution and would solve the confusion. We have modified the problem statement to incorporate this discussion.

Reviewer comment (Writing):
I find the writing of the paper very hard to follow. How are the semantic clusters built? I’m not sure I could follow the explanation in section 4 and algo 1, which is a bit too cluttered. Something as important for the method as the Discriminative Capability Score is not actually explained anywhere. Same issue with the training of the route predictor.

Author response:
We are sorry that Section 4 and Algorithm 1 have been hard to follow. We have updated the manuscript and tried to make the steps easier to follow. We hope that it will clarify the questions that the reviewer might have regarding the procedure.
Section 4 describes the procedure to obtain the discriminative capability score. The section builds up the intuition behind the discriminative capability score and describes it right before the equation 3. We have updated the manuscript to make it easy to follow the procedure and find the description.
We have added more details to the *Semantic Route Predictor* subsection of Section 5 to illustrate the training of the semantic route predictor. Specifically, we add that 

*To train the auxiliary classifier $\boldsymbol{\chi}$, the section of the base model up to the $M-1$-th layer of $\mathcal{F}$ is frozen and the classifier is trained in supervised fashion using {$\mathbf{A}_{M-1}^j, \gamma_m^j$} , where  j=1...|D| as the dataset.*

*Here, $\mathbf{A}_{M-1}^j$, and $\gamma_m^j$ are respectively the activation of the $M-1$-th layer of the base model and the ground truth semantic cluster for the j-th sample. As we are considering a pre-trained base DNN, we train the auxiliary classifier separately from the base network using the activations obtained from the $M-1$-th layer.*

Reviewer comment (Evaluation):
The authors claim to obtain SOTA results while using relatively old datasets and architectures. I don’t think such a choice invalidates the contribution, but I would weigh the claims accordingly.

Author response:
We thank you for your comment. The setting we are proposing is new and it requires a score-based approach to select the filters in each layer to form the subgraph. As a result, we have chosen the [1], [2] for comparison as recent methods. Apart from that, we compare with the recent [3] which is a published work in ICLR 2023 to prove the efficacy of DCS as a pruning method. We are currently running SINF on ImageNet-1k and will report the results as soon as we obtain them.

[1] Sui, Y., Yin, M., Xie, Y., Phan, H., Aliari Zonouz, S. and Yuan, B., 2021. Chip: Channel independence-based pruning for compact neural networks. Advances in Neural Information Processing Systems, 34, pp.24604-24616.
[2] Lin, M., Ji, R., Wang, Y., Zhang, Y., Zhang, B., Tian, Y. and Shao, L., 2020. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1529-1538).
[3] Murti, C., Narshana, T. and Bhattacharyya, C., 2022, September. TVSPrune-Pruning Non-discriminative filters via Total Variation separability of intermediate representations without fine tuning. In The Eleventh International Conference on Learning Representations.

Reviewer comment (Presentation):
What is Fig 1top supposed to convey? It doesn’t seem to show anything. If the idea is to show the difference in sparsity, a simple plot with the sparsity level of each layer would work better.

Author response:
Figure 1 (top) shows the filter activations for two layers of ResNet50. The figure aims at clearly conveying the sparsity level of each layer. We also focus attention on the similarity of the two classes (otter and seal) in the earlier layer and the distinctness in the later layer. We also quantify this based on the L1 distance. In the earlier layer, the $L_1$ distance is 0.028 while the same for the later layer is 0.111. This shows the specialization of filter activation in deeper layers. From this, we conclude that the deeper layers are more sparse and more specialized for individual classes.

Reviewer comment (Experiments):
Lack the comparison with previous quantization and pruning approaches. There are many quantization and pruning approaches that can speed up the inference without fine-tuning or retraining. Can do more comparison experiments with SINF and show the comparable results of the whole pipeline.

Author response:
We thank you for your comment. To the best of our knowledge, the previous work that does not require fine tuning is very limited -- we were only able to find [1] as a recent paper that does not require fine-tuning. We would be happy to compare against other non fine-tuning techniques if the reviewer shared these papers with us. We would like to clarify that *we are not considering pruning at train-time or retraining approaches*. We are assuming that we start with a pre-trained DNN and extract the sub-graphs corresponding to the semantic clusters defined by the task at hand.

For the extraction of the sub-graph part, we have compared against metrics which are usable in our scenario. For the pruning part, we are considering the most recent work closest to our setting for comparison. Notwithstanding, we point out that designing a pruning algorithm is out of the scope of this paper. Our goal is to design a dynamic DNN which can *immediately* adapt to changing requirements. For this reason, we cannot fine-tune the DNN at run time.

Regarding quantization, as pointed out in the background section (Section II), we believe our proposed work is orthogonal to that approach and can be used on top of quantization and coding approaches to further improve the performance.

[1] Murti, C., Narshana, T. and Bhattacharyya, C., 2022, September. TVSPrune-Pruning Non-discriminative filters via Total Variation separability of intermediate representations without fine tuning. In The Eleventh International Conference on Learning Representations.

Reviewer comment (Experiments):
Table 1 lack the results of ResNet50 on CIFAR100.

Author response:
Thank you for pointing this out. We are comparing our approach to the work which has been published in the last ICLR 2023 [1] . To make a fair comparison, we are using the same DNN model as in [1]. Since the work does not provide any performance result for ResNet50 on CIFAR100, we are not comparing for the same.

Reviewer comment (Evaluation):
Lack the analysis of the poor improvement in Table 1, where there is almost no improvement for VGG19 on CIFAR100 and only a slight improvement on CIFAR10.

Author response:
We thank you for the opportunity to clarify this point. A possible explanation is that since both our technique and the state of the art IterTVSPrune have pruned a significant amount of weights – respectively about 50% and 60% for CIFAR10 and CIFAR 100% – the DNN has reached a lower bound on its predictive capability. In other words, this means that the DNN cannot be pruned more without compromising the accuracy. In the other DNNs (VGG16 and ResNet50) the amounts of weights pruned is less aggressive (up to 40% for VGG16 and up to 35% for ResNet50) so eventually there could be room for improvement. This aspect is nevertheless intriguing and we plan to delve deeper into this aspect in future work. We will include these discussions in the final manuscript.

Reviewer comment (Evaluation):
Is there any proof or metric to evaluate the intrinsic redundancy in latent representations? The idea of intrinsic redundancy is interesting but somehow blurred, it will be great if there is any explicit metric can be proposed or discussed.

Author response:
In Section 3, we have used $L_1$ distance to quantify the similarity of the filter response for inputs of two different classes (otter and seal). We have also utilized the overlapping between filter activation for two classes to quantify this similarity. Although these metrics do not exactly quantify the intrinsic redundancy, it is part of our current work to characterize the redundancy in a more formal manner.

Reviewer comment (Experiments):
There are a lot of existing pruning methods, e.g., DepGraph: Towards Any Structural Pruning, but this paper does not provide any comparison to existing works.

Author response:
We thank the reviewer for this comment. Although the end goal of pruning and our proposed approach is to decrease the computational burden of mobile devices related to executing deep neural networks (DNNs), the primary objective of our work is to do so *without the need to fine-tune the DNN*, as in real-world mobile systems the DNN has to be adapted immediately to changing requirements.

The first key reason behind this choice is that fine-tuning DNNs takes a significant amount of time. For example, fine-tuning one of our sub-graphs even for 20 epochs takes on average a minute, which is just for CIFAR-100. For 20 sub-graphs corresponding to 20 semantic clusters, it takes about 20 minutes. In real-world dynamic mobile scenarios where priorities might change, it may be infeasible and energy-expensive to continuously fine-tune the model. For example, if we are deploying a drone for surveillance in a mountain area, that would encounter certain classes (e.g., animals). However, if the UAV operates in an urban area, then it would require a different set of classes (e.g., cars). Given the dynamic nature of the system, allowing fast switching becomes a compelling necessity.

Another reason is that fine-tuning the DNNs for one set of classes can degrade its performance in other tasks due to catastrophic forgetting [1,2]. Conversely, our approach is to *pre-compute and then select at runtime* the sub-graph with the capability of achieving sufficient accuracy on the given set of semantic classes. 

DepGraph and many other pruning approaches rely on fine-tuning to improve performance. For this reason, we compare to the most recent work for pruning which adopts same scenario as ours [3] in **Table 1**. We also compare the proposed discriminative capability score (DCS) with other score-based metrics usually used for pruning in **Figure 4**.

[1] Pomponi, J., Scardapane, S. and Uncini, A., 2022. Centroids Matching: an efficient Continual Learning approach operating in the embedding space. Transactions on Machine Learning Research.
[2] Davari, M., Asadi, N., Mudur, S., Aljundi, R. and Belilovsky, E., 2022. Probing representation forgetting in supervised and unsupervised continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16712-16721).
[3] Murti, C., Narshana, T. and Bhattacharyya, C., 2022, September. TVSPrune-Pruning Non-discriminative filters via Total Variation separability of intermediate representations without fine tuning. In The Eleventh International Conference on Learning Representations.

Reviewer comment (Experiments):
The acceleration on CIFAR10/CIFAR100 is not satisfied. For example, DepGraph achieves an 8.92× acceleration with a 3.11 loss in accuracy on CIFAR100 with VGG19. However, the results provided by the author (~1.72x and 2.83 loss on acc) are worse than previous work.

Author response:
Although the review is correct, we point out that conversely from DepGraph and other pruning approaches in literature, our results are obtained *without fine-tuning*. When considering the DCS score for pruning, and enabling fine-tuning, our proposed DCS strategy achieves 9.08x speed up for only 3.27% loss in accuracy which is comparable to DepGraph.

Reviewer comment (Writing):
The writing is confusing, it is hard to find the main results.

Author response:
We are sorry to know that the reviewer has found the writing to be confusing and main results hard to find. We have updated the manuscript and tried to highlight the main results. We hope that it will make it the results easier to find.

Reviewer comment (Evaluation):
This method applies an additional predictor to discriminate which hyper-class the input should be, which introduces extra inference cost and training effort. Does the provided inference time include the inference time on the additional predictor? Another concern is that the extra network will hurt model generalization ability.

Author response:
Thank you for giving us the opportunity to clarify this point. The inference time shown in the paper also includes the inference time on the additional predictor. The additional predictor introduces small computational overhead (4.8%) compared to the backbone DNN. This way, it has negligible impact on the inference time. We will clarify this point in the final version of the paper.

We agree that the extra network could hurt model generalization. This is why our proposed approach uses a confidence threshold to avoid this issue. In other words,  if the semantic route predictor is not confident enough, we can bypass the extracted sub-graph and use the base DNN. This decision is made by the feature router. The feature router is presented with the prediction of the semantic route predictor as well as its confidence in its prediction. Provided that the confidence is smaller than the predefined threshold, the feature router activates the full DNN. This way, we can balance the accuracy and latency trade-off. We show in Section 6 (Experimental Evaluation) that with the additional predictor, we can keep the accuracy loss below 0.17%, 3.75%, and 6.75% for VGG19, VGG16, and ResNet50 respectively while reducing inference time by 35%, 29%, and 15% respectively.