Paper: COMPRESSION AND ACCELERATION OF DEEP NEURAL NETWORKS: A VECTOR QUANTIZATION APPROACH (ICLR-2024)

Reviewer comment (Evaluation):
Regarding the compression of operation counts, the method utilizing codebooks and indices primarily offers theoretical compression, and practical acceleration is challenging to achieve. This issue should be argued in the paper.

Author response:
Regarding the practical speed-up, we would like to draw your attention to our response to question 1 from reviewer 1, where we delve into the specifics of achieving practical acceleration with our proposed method. We acknowledge the importance of discussing the practical implications.

3: In addressing the question regarding FLOPs and Params values in our experimental results, it's crucial to note that these values, as well as those in the cited papers, are primarily theoretical. Achieving real acceleration demands either hardware implementation or alternative methods such as implementing CUDA modules. We provide a more comprehensive response to this concern in our reply to the first reviewer's initial questions. This discussion will be clarified in the final paper if accepted.

Reviewer comment (Reproducibility):
How are activation values quantized?

Author response:
1: In the context of this paper, it's important to note that activation values remain in their full precision without quantization. The quantization specifically applies to the weight parameters of the model, which undergo vector quantization. This means that these weights are constrained to discrete values defined by the codebook. To enhance clarity, if our paper is accepted, we intend to provide a more detailed clarification regarding the differentiation between weight parameters and activations in terms of quantization.

Reviewer comment (Evaluation):
In the experimental results, the reported FLOPS and Params values are purely theoretical, correct? Can you discuss the practical acceleration or its applicability? Additionally, how was the base accuracy determined? Is the primary metric accuracy or accuracy reduction? Why is accuracy reduction considered more effective?

Author response:
3: In addressing the question regarding FLOPs and Params values in our experimental results, it's crucial to note that these values, as well as those in the cited papers, are primarily theoretical. Achieving real acceleration demands either hardware implementation or alternative methods such as implementing CUDA modules. We provide a more comprehensive response to this concern in our reply to the first reviewer's initial questions. This discussion will be clarified in the final paper if accepted.

The determination of base accuracy varies depending on the method employed. For post-training approaches applied to pre-trained models, the base accuracy represents the model's accuracy before pruning. As we lack control over external methods, we report the specifics of their procedures. For instance, the base ResNet50/ImageNet model in PScratch (Wang et al., 2020b) exhibited an accuracy of 77.20%, while SG-CNN (Guo et al., 2020) had 76.13%. In the case of methods like our VQ method, which involves training a model from scratch, we ensure consistency by training two models—one non-VQ as a baseline and one VQ model—using identical hyperparameters (epochs, batch size, data augmentations, etc.).

The emphasis on accuracy reduction over absolute accuracy stems from the inherent variability in base accuracies across methods. Given constraints such as training ResNet50/Imagenet for 90 epochs with a batch size of 256 (refer to Appendix A.1 for details), our baseline accuracy is naturally lower than approaches utilizing well-trained public models. Consequently, we believe assessing the degree of accuracy degradation provides a fairer comparison, considering the varied starting points of different methods, rather than fixating on absolute accuracy figures.

Reviewer comment (Presentation):
Experimental results are listed without any highlight, which is very unclear.

Author response:
Thank you for bringing this to our attention. We have addressed the clarity concern by incorporating highlights in the tables in our revised draft. This adjustment aims to improve the visual distinction of experimental results, making it easier for readers to interpret and understand the presented data.

Reviewer comment (Novelty):
The idea of adopting VQ in compressing DNNs is not new, so more effort could be made including giving the guidelines for designing the hyperparameters such as the group numbers.

Author response:
Our sincere thanks for your comment. While it's acknowledged that clustering weights for DNN compression isn't a novel idea, our paper uniquely contributes by being the first, to the best of our knowledge, to directly apply Vector Quantization (VQ) to neural network weights. This distinction is crucial as it pioneers a new application of VQ in the context of weight compression. Our approach introduces two key contributions: firstly, demonstrating the effectiveness of a single compact codebook in encoding layers of large architectures, exemplified by ResNet-50; and secondly, emphasizing the efficiency gains in computation through the use of a small, single codebook. This innovation allows for streamlined operations, multiplying the entire codebook by inputs in each layer, thereby circumventing redundant calculations. We acknowledge the existing work in weight clustering and stress that our unique application of VQ to network weights sets our approach apart, offering a fresh perspective and valuable contributions to the field.

Reviewer comment (Evaluation):
The conclusion regarding the speed-up performance is not convincing. There is no practical speed-up results presented in the work and the reported FLOPs reduction is only calculated theoretically. Moreover, traditional dense matrix multiplication has been highly optimized in recent years on modern GPUs, e.g., cuDNN. Whether or not a network with vector quantization infers faster than its dense counterparts on modern GPUs remains uncertain. Thus, the lack of inferring time and practical speedup results does not adequately justify their work.

Author response:
We appreciate the reviewer's feedback on the lack of practical validation in the paper. We want to clarify that our decision to focus on theoretical aspects is intentional and aligns with the specific scope defined for this study. While we acknowledge the importance of empirical evidence, we deliberately leave the exploration of practical implementation and performance to future research. The theoretical analysis presented here serves as a foundation for potential follow-up studies aimed at empirically validating and optimizing our approach.

We recognize the optimization of traditional dense matrix multiplication on modern GPUs, such as cuDNN. However, we believe that our emphasis on theoretical foundations is a roadmap for future optimizations.

It's important to note that our approach is not unique in this aspect; many SOTA papers in the literature report FLOPs without immediate practical validation (Li et al., 2016; He et al., 2019; Zhang et al., 2022; Lin et al., 2020a; Park et al., 2023; Liu et al., 2018; Wang et al., 2020b; Guo et al., 2020; Zhao & Luk, 2019; Tang et al., 2020; Zhu et al., 2021; Yu et al., 2022; Chen et al., 2021).

Implementing custom Cuda modules may be possible, but a fair comparison would require similar efforts for all previous works, surpassing our current resources. To the best of our knowledge, no previous papers in this domain have implemented Cuda modules or hardware versions of their methods, and this challenge has been acknowledged by reviewers in the field, as exemplified in the discussion regarding "TMI-GKP: Revisit Kernel Pruning with Lottery Regulated Grouped Convolutions" (available at https://github.com/henryzhongsc/lottery_regulated_grouped_kernel_pruning#discussion-regarding-inference-speedup), which was accepted at ICLR 2022 without further implementation.

Reviewer comment (Reproducibility):
Some aspects of the VQ-DNN illustration require further clarification. For instance, the specific process of updating W_jb and e in equation (1) needs to be clarified. Are they updated alternately? Additionally, the method of mapping weights to the codebook is not clear. Does the mapping change during training? It is necessary to provide more detailed explanations for these concerns.

Author response:
While we commit to providing more detailed explanations in a revised version if the paper is accepted, we offer the following insights for the current discussion:

In eq(1), the update operations for W_jb and e are indeed distinct and are performed simultaneously at the end of each batch. This simultaneous update is facilitated by the use of stop-gradient operators, as described in the original paper by Van Den Oord et al. (2017).

Concerning the mapping of weights to the codebook, it is essential to note that this mapping evolves during training. Specifically, the weight stored in the original model (W_jb) undergoes a comparison with the entire codebook and is subsequently replaced with the nearest codeword. This process essentially treats the original weight as an index. As training progresses, these weights undergo updates, causing the mapping to change dynamically.

Reviewer comment (Experiments):
The experimental section is confusing probably due to the unclear selection of compared methods. The related work section does not introduce the compared methods in Table 3, 5, and 6, making it difficult to understand the state of the arts. Furthermore, the authors introduce four categories of model compression in the related work section, such as pruning, and quantization, but they do not present the performance comparison between them and the proposed methods.

Author response:
In our related work section, we aimed for comprehensiveness by covering all categories of model compression found in the literature. Certain methods, such as knowledge distillation or matrix decomposition approaches, have not been included as their popularity is declining due to the superior performance of pruning methods.

Although our method bears the name "vector quantization," it is distinct from the more prevalent category of low-precision parameter methods or traditional quantization. The terminology may introduce some confusion, but we chose to adhere to the established concept of "vector quantization" in the data compression and deep learning literature. Traditional quantization approaches often measure metrics, such as energy consumption (Hubara et al., 2016), or actual latency (Jacob et al., 2018; Lee & Nirjon, 2019), which may not be directly comparable to our method or other pruning methods. Notably, SOTA pruning methods typically compare their performance exclusively with other pruning methods in the literature, given the intrinsic differences in evaluation metrics.

In the event of acceptance, we are committed to refining the related work section to explicitly cover the papers with which we compare our approach, providing a more comprehensive and coherent understanding of the state-of-the-art landscape in model compression.

Reviewer comment (Presentation):
The quality of figures and tables should be improved. For example, the quality of Figure 2 is poor, which makes it difficult to capture useful information. Similarly, the proposed method's key results in the tables should be highlighted for better readability.

Author response:
We will improve the quality of tables and figures.