Paper: LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models (ICLR-2024)

Reviewer comment (Reproducibility):
One big issue in writing that prevents one from reimplementing the proposed method is how to construct the binary matrices and scaling factor matrices from the pre-trained weights. The attached code assumes randomized matrices so that part cannot provide any useful information regarding this.

Author response:
According to this comment, we have included an example implementation in the supplementary material demonstrating the construction of binary matrices and scaling factor matrices from pre-trained weights. Please refer to the examples directory provided in the supplementary material.

In the case of uniform quantization, our process involves initially obtaining scaling factor matrices and integer weight matrices through a uniform quantization method (we utilized the simple RTN method in our example). Subsequently, we proceed to convert the integer weight matrix into binary weight matrices using bitwise operations, as explained in detail in Appendix C of the revised manuscript. Additionally, to form a key for LUT-GEMM, we pack eight consecutive binary weights in a row into a uint8 data type. All these steps are implemented in the example directory.

In the revised manuscript, we’ve clearly highlighted how to construct the binary matrices and scaling factor matrices from the pre-trained weights.
Thank you for your feedback to improve the quality of our manuscript significantly.

Reviewer comment (Theory):
Regarding the scaling factor part, how to determine the group size? In my understanding, if the group size is small, there are no benefits as you have as many matrices as used bits to represent weights. At what group size do the benefits compensate the cost of the scaling factor matrices? Is there any formula to calculate this?

Author response:
Thank you for your insightful comment. We recognize the impact of group size on memory footprint, particularly how scale factors contribute to increased memory usage when the group size decreases. We visualize the formula for calculating the overall memory footprint as a function of both quantization bit and group size in Figure 6.

The determination of the group size typically involves heuristic approaches to identify an optimal point that meets system requirements effectively. This choice significantly influences various aspects such as model size, end-to-end latency, and accuracy. It's worth noting that a commonly adopted group size value is g = 128, as reported in [1-3], owing to its favorable balance across these critical system parameters.

We appreciate your attention to this aspect and hope that our visual representation in Figure 6 provides clarity on the relationship between quantization bit, group size, and memory footprint in our proposed methodology.

Moreover, in Table 5 of our manuscript, we aimed to present the latency and accuracy metrics for the quantized OPT-175B model, specifically compressed for deployment on a single GPU. An interesting observation emerges when comparing the configurations (q, g)=(2, 32) and (q, g)=(3, -). Despite the smaller model size associated with (q, g)=(2, 32), there is an evident increase in end-to-end latency compared to (q, g)=(3, -). This anomaly is primarily attributed to the overhead latency incurred in smaller group sizes, as illustrated in Figure 4(a).

This discrepancy highlights the influence of group size on latency, where a smaller group size can introduce additional overhead despite reducing the model size. We appreciate your understanding and hope that this clarification resolves the complexities associated with latency variations across various configurations of quantization bits and group sizes.

[1] Lin, Ji, et al. "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration." arXiv preprint arXiv:2306.00978 (2023). \
[2] Frantar, Elias, et al. "OPTQ: Accurate quantization for generative pre-trained transformers." The Eleventh International Conference on Learning Representations. 2022. \
[3] Kim, Sehoon, et al. "SqueezeLLM: Dense-and-Sparse Quantization." arXiv preprint arXiv:2306.07629 (2023).

Reviewer comment (Experiments):
Regarding the experiments, why only consider the OPT model? How about the speedups on other candidates like LLaMa at different scales? In that way, we can also understand the scalability of the used kernel better.

Author response:
To take this comment into account, we have expanded our evaluations to include latency assessments across the LLaMA model family (Table 7 in the revised manuscript).

|           | Kernel-q-g     | Quant. Method | \*PPL | 1-GPU   | 2-GPU   | 4-GPU   |
| --------- | -------------- | ------------- | ----- | ------- | ------- | ------- |
| LLaMA-7B  | cuBLAS-16-N/A  | FP16          | 5.68  | 9.99956 | 7.64547 | 5.58484 |
| LLaMA-7B  | LUT-GEMM-4-128 | RTN           | 5.96  | 6.10075 | 5.41022 | 5.70228 |
| LLaMA-7B  | LUT-GEMM-3-128 | RTN           | 7.01  | 5.48141 | 5.10903 | 5.27691 |
| LLaMA-13B | cuBLAS-16-N/A  | FP16          | 5.09  | 18.2208 | 11.969  | 8.57203 |
| LLaMA-13B | LUT-GEMM-4-128 | RTN           | 5.25  | 10.3895 | 8.02672 | 6.90163 |
| LLaMA-13B | LUT-GEMM-3-128 | RTN           | 5.88  | 9.28256 | 7.14944 | 6.35469 |
| LLaMA-30B | cuBLAS-16-N/A  | FP16          | 4.1   | 43.5962 | 25.0504 | 16.9111 |
| LLaMA-30B | LUT-GEMM-4-128 | RTN           | 4.23  | 20.6537 | 15.5217 | 12.1708 |
| LLaMA-30B | LUT-GEMM-3-128 | RTN           | 4.88  | 18.1275 | 13.6595 | 11.2222 |
| LLaMA-65B | cuBLAS-16-N/A  | FP16          | 3.53  | OOM     | 46.8719 | 29.1132 |
| LLaMA-65B | LUT-GEMM-4-128 | RTN           | 3.67  | 35.6644 | 25.4327 | 19.852  |
| LLaMA-65B | LUT-GEMM-3-128 | RTN           | 4.24  | 31.2953 | 22.9804 | 18.342  |

\* PPL from AWQ reference [1] \
[1] Lin, Ji, et al. "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration." arXiv preprint arXiv:2306.00978 (2023).

Reviewer comment (Reproducibility):
The authors did not clearly describe how you derive the scaling factors matrices and binary matrices from the pre-trained weights. It would be better if there is any theoretical analysis of the quantization error or variance.

Author response:
According to this comment, we have included an example implementation in the supplementary material demonstrating the construction of binary matrices and scaling factor matrices from pre-trained weights. Please refer to the examples directory provided in the supplementary material.

In the case of uniform quantization, our process involves initially obtaining scaling factor matrices and integer weight matrices through a uniform quantization method (we utilized the simple RTN method in our example). Subsequently, we proceed to convert the integer weight matrix into binary weight matrices using bitwise operations, as explained in detail in Appendix C of the revised manuscript. Additionally, to form a key for LUT-GEMM, we pack eight consecutive binary weights in a row into a uint8 data type. All these steps are implemented in the example directory.

In the revised manuscript, we’ve clearly highlighted how to construct the binary matrices and scaling factor matrices from the pre-trained weights.
Thank you for your feedback to improve the quality of our manuscript significantly.

Reviewer comment (Experiments):
The author mentioned that the optimization here currently only focuses on the inference of a single batch, which may limit its use.

Author response:
In Section 6, we acknowledge that LUT-GEMM exhibits diminishing performance gains with increasing batch size. This phenomenon arises due to the effective utilization of a potent built-in GEMM accelerator Tensor core, which becomes operational when the batch size exceeds 4. Consequently, the design choice of LUT-GEMM, employing CUDA core for single batch operations, is intentionally tailored for scenarios assumed to process single requests at a time in personal computers, prioritizing privacy considerations.

Reviewer comment (Evaluation):
The LUT-Based method requires a significant amount of memory capacity in exchange for efficiency. In Table 2, the 4-bit quantified LUT-GEMM storage footprint exceeds the 16 bit model of the cuBLAS baseline. In fact, storage resources are also the main focus of quantization in large language models, not just performance. This paper seems to focus mainly on computational efficiency, but lacks a comparison between memory resource usage.

Author response:
We appreciate your feedback, and we acknowledge that there were some confusing descriptions in the original manuscript regarding memory utilization in Table 2. To clarify, the presented memory utilization in Table 2 represents the percentage of time over the past sample period during which global (device) memory was being read or written [4]. In essence, the memory utilization metric is closely linked to memory movement rather than the amount of memory used in a GPU.

In reality, the memory footprint of LUT-GEMM is significantly smaller than that of cuBLAS. This substantial reduction in memory usage allows us to successfully execute a 175B model on a single GPU, as demonstrated in Table 4. In contrast, attempting to run the 16-bit model of the cuBLAS baseline results in an out-of-memory (OOM) occurrence. We have taken your feedback into consideration and, in the revised manuscript, have made efforts to clearly present the definition of memory utilization in Table 2 to avoid any confusion.

[4] https://nvidia.custhelp.com/app/answers/detail/a_id/3751/~/useful-nvidia-smi-queries

Reviewer comment (Experiments):
The experimental data in this paper is not enough, and the main baseline for comparison is 16 bit cuBLAS, lacking a comprehensive comparison of other quantization methods. The experiment only tested the quantization accuracy of PPL indicators using the OPT model on the LAMBADA dataset, which cannot fully demonstrate the effectiveness and universality of the method.

Author response:
We are sorry that there were some confusing descriptions in the original manuscript. LUT-GEMM is designed to be agnostic to the specific quantization method employed. It can seamlessly support any quantization method that can be represented in the form outlined in Equation 3. To address this concern, we have conducted additional evaluations of latency on the LLaMA model family, employing the baseline quantization method, RTN (Table 7 in the revised manuscript). This extended evaluation provides a more comprehensive understanding of the versatility and compatibility of LUT-GEMM across various quantization approaches.

Reviewer comment (Evaluation):
The speed of inference is important, but it is only one part. I hope the author can compare the advantages and disadvantages of other quantization acceleration methods, including compression ratio of GPU memory requirements, the need for post training calibration, etc.

Author response:
Thank you for your feedback. It's important to highlight that the memory footprint of LUT-GEMM exhibits a significant reduction compared to cuBLAS, enabling the successful execution of a 175B model on a single GPU. Similarly, other quantization acceleration such as  AWQ [1] and OPTQ [2] methods successfully execute larger models on a single GPU.

One of the notable advantages of LUT-GEMM lies in its quantization bit agnosticism. Specifically, the OPTQ kernel, optimized for 3-bit quantized weights, utilizes three 32-bit data types to efficiently store the quantized weights. Similarly, the AWQ kernel, optimized for 4-bit quantized weights, employs a single 32-bit data type for efficient weight storage. Unlike previous dequantization-based kernels that require distinct kernels for different quantization bit configurations, LUT-GEMM operates on each binary weight matrix for every bit position. Consequently, LUT-GEMM possesses the capability to support multiple quantization configurations within a single kernel, enhancing its versatility and reducing the need for multiple specialized kernels.

[1] Lin, Ji, et al. "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration." arXiv preprint arXiv:2306.00978 (2023). \
[2] Frantar, Elias, et al. "OPTQ: Accurate quantization for generative pre-trained transformers." The Eleventh International Conference on Learning Representations. 2022.

Reviewer comment (Experiments):
The author claims that the BCQ quantization used is suitable for both uniform and non uniform quantization, but the experiment only considered uniform quantization. Theoretically, non-uniform quantization has better representation ability to achieve higher accuracy. Although testing the performance of non-uniform quantization may be difficult on current hardware, it is still possible to compare the accuracy differences between the two methods.

Author response:
As you pointed out with this comment, uniform quantization methods often exhibit a drawback by allocating excessive precision to the edges of the distribution, making them generally show worse accuracy compared to non-uniform quantization methods. Despite this disadvantage, the active exploration of uniform quantization methods persists, primarily driven by their ease of deployment without requiring specialized kernels. Our contribution lies in the development of an open-source library for LUT-GEMM, which supports both uniform and non-uniform quantization methods. By providing this flexibility, we aim to encourage a shift towards highlighting non-uniform quantization methods which have not been actively explored in the research community due to the lack of practical supporting kernels. We anticipate that this emphasis will contribute to the evolution of more accurate quantization techniques in the future, leveraging the versatility of our library to facilitate advancements in the field.