Paper: PB-LLM: Partially Binarized Large Language Models (ICLR-2024)

Reviewer comment (Writing):
The proposed approach section is comprehensive, but its complexity makes it challenging to navigate and comprehend throughout the entire section.

Author response:
Thank you for your valuable feedback.

To enhance clarity, we have succinctly summarized our PB-LLM approach and included a conceptual illustration in Fig. 1 (a). PB-LLM begins with a pre-trained, full-precision LLM and partially binarizes the weight matrix while preserving salient weights in higher bits. The process is designed to recuperate the quantized LLM's reasoning capacity, employing both post-training quantization (PTQ) and quantization-aware training (QAT) frameworks. For PTQ recovery (GPTQ-PB), we reconstruct the partially-binarized matrix, guided by Hessian metrics. For QAT recovery (PB-LLM), salient weights are frozen during binarization to streamline the training process.

Additionally, we have incorporated your suggestion by adding a detailed flowchart in the Appendix. This flowchart delineates the stages of transforming a PB-LLM model under the frameworks of PTQ and QAT, intending to improve readability and comprehension.   The revisions are highlighted in red for easy identification and reference. We integrate this diagram into the main body of the paper for the camera-ready version.

Reviewer comment (Presentation):
The readability of the paper can be improved by including a flowchart or block diagram of the proposed method by illustrating different stages of transforming a PB-LLM model.

Author response:
Thank you for your valuable feedback.

To enhance clarity, we have succinctly summarized our PB-LLM approach and included a conceptual illustration in Fig. 1 (a). PB-LLM begins with a pre-trained, full-precision LLM and partially binarizes the weight matrix while preserving salient weights in higher bits. The process is designed to recuperate the quantized LLM's reasoning capacity, employing both post-training quantization (PTQ) and quantization-aware training (QAT) frameworks. For PTQ recovery (GPTQ-PB), we reconstruct the partially-binarized matrix, guided by Hessian metrics. For QAT recovery (PB-LLM), salient weights are frozen during binarization to streamline the training process.

Additionally, we have incorporated your suggestion by adding a detailed flowchart in the Appendix. This flowchart delineates the stages of transforming a PB-LLM model under the frameworks of PTQ and QAT, intending to improve readability and comprehension.   The revisions are highlighted in red for easy identification and reference. We integrate this diagram into the main body of the paper for the camera-ready version.

Reviewer comment (Evaluation):
The paper reports primarily the accuracy to evaluate the proposed approach. The addition of the final model size after partial binarization can further solidify the claim.

Author response:
We appreciate your suggestion to detail the model size savings achieved through our method. Our PB-LLM approach, which involves selective binarization of the LLMs' weight matrix, indeed leads to significant storage savings. This is achieved by retaining a portion of the weights in high bits to bolster linguistic capacity, without substantially increasing the overall model size.

The overall bit number, $N_{bit}$, of our model is calculated using the formula: $N_{bit} \leq 1 \times r_{binary} + N_{salient-bit} \times (1 - r_{binary}) + 1$. In this formula, $r_{binary}$ represents the ratio of weights that are binarized, and $N_{salient-bit}$ signifies the bit count allocated for salient weights (e.g., 8 bits), with an additional bit dedicated to a bitmap index mechanism. As an illustrative example, if 10% of the weights are designated as salient and stored in 8 bits, the overall bit requirement would be at most 1$\times$ 90% + 8$\times$ 10% + 1= 2.7 bits. This implies that the storage requirement for our quantized model is just 16.8% of what would be needed for a full 16-bit model (2.7/16 = 16.8%).

Reviewer comment (Experiments):
Only pre-trained base models are experimented with; those models, however, are usually not deployed directly as applications.

Author response:
Thanks for your suggestion. We are testing the performance of PB-LLM on Vicuna-7B (we use this checkpoint from huggingface: https://huggingface.co/AlekseyKorshuk/vicuna-7b). The results will come out in 8 hours.
***
**Nov/19/2023 Results Update:**

We acknowledge the importance of validating our method on models that extend beyond pre-trained base models, as these are more representative of real-world applications. To address this, we have conducted additional experiments using Vicuna-7B. The results of these experiments are as follows:

| Method    | Humanities | STEM | Social Sciences | Other | Average |
|-----------|----------|-----------|----------------|---------|---------|
| Full-precision Vicuna-7B   | 47.6 | 40.4 | 55.9 | 54.5 | 49.6 |
| LLM-QAT (QAT)  | 40.1 | 31.2 | 46.2 | 47.8 | 41.4|
| PB-LLM 30% Salient Weight  | 41.5 | 34.1 | 50.7 | 51.1 | 45.6|

Based on our **codebase provided in the supplementary materials**, we applied PB-LLM to quantize Vicuna-7B. The results, showcased above, are from the MMLU benchmarks. For comparison, we also quantized Vicuna-7B using LLM-QAT, limiting the training to 10K iterations, identical to the training duration for our PB-LLM method, due to resource constraints.

These results demonstrate that even when applied to fine-tuned models like Vicuna, our PB-LLM method maintains its efficacy, showcasing superior performance compared to LLM-QAT under similar training conditions. This illustrates the versatility and robustness of our approach, extending its applicability to a broader range of LLM deployment scenarios.

Reviewer comment (Experiments):
The LLM quantization scheme is motivated using an angle of GPU memory efficiency. However, the actual GPU memory usage before and after binarization/quantization is not studied in this paper.

Author response:
We acknowledge the importance of understanding the actual GPU memory usage before and after binarization/quantization. Unfortunately, our team lacks dedicated GPU implementation experts, which hindered us from providing a comprehensive analysis in the main body of the paper. Realizing real GPU implementation for algorithms often involves a large team. 
To address your concern, we conducted additional experiments focusing on GPU memory savings for PB-LLM. We utilized the CSR (Compressed Sparse Row) sparse format for 8-bit quantized salient weights.The results are summarized below, comparing the memory occupancy of salient weights after various levels of quantization:

| Configuration | PB-LLM 10% | PB-LLM 20%| PB-LLM 30% | 2bit Quant| 4bit Quant|    
|-----------|----------|-----------|----------------|---------|---------|
| Memory Occupancy Comparing with FP16   | 16.25% | 26.25% |  36.25% | 12.50% | 25.00% |


Notably, PB-LLM with 10% salient weights exhibits memory overhead similar to 2-bit quantization, while PB-LLM with 20% salient weights shows memory overhead comparable to 4-bit quantization. Note that some advanced salient location-saving techniques, such as lead zero, can further decrease the ratio. We plan to incorporate these results into the camera-ready version of the paper. Additionally, we recognize the potential for further optimization in sparse storage. Future work may explore enhancements such as leveraging col index and row index stored in the CSR format of salient weights for binarized quantization or employing semi-structured sparse optimization techniques.

Reviewer comment (Experiments):
How does PB-LLM's performance vary among various sizes of Llama models, e.g., 7B-65B?

Author response:
Thank you for acknowledging the various aspects of our work and for your valuable recommendation. In response, we have extended our experimentation to include larger LLaMA models, specifically LLaMA-13B and LLaMA-30B, to gauge the scalability of our PB-LLM method. Unfortunately, due to computational resource constraints, we were unable to conduct experiments on the LLaMA-65B model, as our available resources – an 8 A6000s server – represent the upper limit for our academic lab's resources.

Below are the results showcasing PB-LLM's performance across these additional LLaMA model sizes:
| Method    | BoolQ | PIQA |  HellaSwag | WinoGrande | ARC-E | ARC-C | OBQA | Avg |
|-----|------|----------|-----|------|--------|--------|---------|---------|
| LLaMA-13B Full-precision | 78.1 | 80.0 | 79.2 | 73.6 | 74.5 | 52.6 | 55.0 | 70.5| 
| RTN     | 76.8 | 79.1 | 76.3 | 70.5 | 72.6 | 49.8 | 56.6 | 68.8|
| LLM-QAT | 77.7 | 79.4 | 77.7 | 71.5 | 72.8 | 52.0 | 53.8 | 69.2|
| 30% PB-LLM | 78.9 | 79.9 | 78.5 | 72.0 | 72.9 | 51.3 | 54.1| 69.7|

| Method    | BoolQ | PIQA |  HellaSwag | WinoGrande | ARC-E | ARC-C | OBQA | Avg |
|-----|------|----------|-----|------|--------|--------|---------|---------|
| LLaMA-30B Full-precision | 83.2 | 82.1 |82.9 | 75.6 | 80 | 58 | 59.3 | 74.4| 
| RTN     | 80.8 | 80.1 | 81.6 | 75.8 | 79.3 | 55.8 | 57.2 | 72.9|
| LLM-QAT | 81.8 | 81.0 |81.8 | 75.1 | 79.4 | 56.8 | 54.9 | 72.9|
| 30% PB-LLM | 82.3 | 79.9 | 81.8 | 76.5 | 79.8 | 56.4 | 56.0 | 73.3|

It is important to note that exhaustive hyper-parameter tuning has not been conducted in this expanded setting. However, these results, even without finely tuned hyper-parameters, indicate the generalization capability of our method across various model scales. In the meantime, our method is more training efficient (refer to Responses to Reviewer 92rD).

Reviewer comment (Novelty):
Lack of novelty: The authors propose the optimal scaling factor as their primary contribution, but the core idea itself seems to have already been proposed in the previous work. For example, [R1] proposed an optimal ternary function (eq. (3) of [R1]), but it can be trivially reduced to the binary function when the threshold is zero; then the equation seems to be identical to the equation (8) and (9) proposed in this paper.

Author response:
We appreciate the reviewer’s insight and the opportunity to clarify the novelty and significance of our approach within the context of LLM compression.
Firstly, we wish to emphasize that our paper does not claim undue novelty in the concept of optimal scaling factors, a concept initially popularized by XNOR-Net, which itself has a substantial citation record (a paper in 2016, over 5000 citation). Indeed, as indicated in Section 3.4.2 of our paper, we acknowledge this foundational work. Even Ternary Weight Networks, which you referenced, credits XNOR-Net for its parameter optimization strategy. Our citation of XNOR-Net is, therefore, appropriate and recognizes the original contribution of this seminal work. We believe that the application of the optimal scaling concept in the context of LLMs, as we have done, represents a novel and significant contribution, similar to how Ternary Weight Networks adapted these ideas to their framework.

Secondly, to clarify, our novel contributions are as follows:

-  Pioneering Network Binarization in LLM Quantization: We are the first to introduce and conduct a comprehensive exploration of network binarization specifically for LLM quantization, demonstrating its capability to achieve lower-bit compression.
- Development of PB-LLM: We propose the Partially-Binarized LLM (PB-LLM), which incorporates a unique salient weights freezing mechanism and applies optimal scaling strategies to preserve the functionality of quantized models.
- Exploring PTQ and QAT for Performance Recovery: Our work is the first application of post-training binarization in the context of PTQ and leverages QAT to efficiently train PB-LLM, showcasing both as practical and effective strategies for LLM compression.

We trust these points clarify the originality and value of our research, affirming its unique contribution to the field of LLM compression.

Reviewer comment (Experiments):
Inconsistent Salient Weight Methodology between PTQ and QAT: The absence of a consistent methodology for salient weight protection between PTQ and QAT is concerning. While the effectiveness of using Hessian criteria for identifying salient weights in PTQ is demonstrated through performance comparisons, the rationale for using magnitude criteria to identify salient weights in QAT seems to be missing. Understanding the disparity in the approach to salient weight protection across PTQ and QAT is crucial for a holistic appreciation of the proposed method.

Author response:
We appreciate the opportunity to address the concerns raised regarding the inconsistent methodology for salient weight protection between PTQ and QAT.
In our experimental analysis, we employed the OPT-1.3b network and conducted PB-LLM (QAT) quantization under both Hessian and Magnitude criteria, targeting a 10% salient weight. The performance results, measured in perplexity (PPL), were as follows:

Hessian Criteria:
wikitext2: 37.07
ptb: 47.07
c4: 38.76

Magnitude Criteria:
wikitext2: 37.50
ptb: 44.42
c4: 38.64


We observed that the PPL values were comparable between the two criteria. However, it's worth noting that the Hessian criteria involve an additional calibration step (collecting calibration data to compute the Hessian), adding complexity to the process. In contrast, the Magnitude criteria offer a simpler approach, eliminating the need for this extra calibration step.
Given the similar PPL results and the desire to streamline the quantization process in QAT, we opted for the Magnitude Metric to identify salient weights during QAT. This decision was motivated by the goal of simplifying the process while maintaining comparable performance.

Reviewer comment (Experiments):
Insufficient evidence on PB-LLM efficiency: To claim that PB-LLM is more efficient in terms of training iteration number compared to LLM-QAT, a more thorough comparison seems necessary. Specifically, it needs to be clear whether the LLM-QAT, being compared with PB-LLM, has been fine-tuned on the same dataset as PB-LLM. Detailed experimental setup information regarding the LLM-QAT is required. Moreover, verification is needed on whether the results through PB-LLM QAT have fully enhanced the reasoning capacity of the pre-trained model. Essentially, it appears that the reasoning accuracy of the target model (OPT-1.3B) obtained through FP fine-tuning should be presented as the upper bound in Figure 7. Additionally, there seems to be a lack of information in Table 2 regarding whether FP LLaMA-7B performance is pre-trained or fine-tuned.

Author response:
Actually, our used training dataset is much smaller than the data in LLM-QAT. It even is a subset of the datasets in LLM-QAT  
The full title of LLM-QAT is:  LLM-QAT: Data-Free Quantization Aware Training for Large Language Models, in which the concept of Data-Free is one of their selling points. Specifically, they use a full-precision LLaMA to generate 100K sentence to train the quantized LLaMA. In the Sec.3.3.1 in LLM-QAT, they discussed their data choice. They observe that WikiText (Merity et al., 2016), which is constructed using text extracted from Wikipedia, does not encompass all the information utilized during pre-training. Consequently, a model fine-tuned solely on WikiText tends to overfit on this specific dataset and struggles to generalize well to other datasets. On the other hand, the Crawled Corpus (C4) dataset (Raffel et al., 2020) comprises hundreds of gigabytes of clean English text collected from the web. Using those large datasets performs worse than using their generated dataset.

On the contrary, we use the RedPajama-simple-1B dataset, as the dataset for LLaMA training is not openly accessible. This dataset, RedPajama-1T, is structured to closely resemble the LLaMa paper and serves as a transparent, open-source alternative to LLM training dataset. 
It amalgamates data from diverse sources including Commoncrawl, C4. RedPajama-simple-1B, representing a 0.1\% subset of RedPajama-1T, is substantially smaller than the typical datasets used for training other LLMs, making it a convenient choice for our experiments. As RedPajama-simple-1B is a tiny subset of Commoncrawl and C4, our training dataset can been seen as a subset of training datasets of LLM-QAT. 

We appreciate the opportunity to clarify the nature of our methodology and its positioning within the Large Language Model (LLM) training paradigm.

Our used model optimization paradigm is fundamentally distinct from fine-tuning methods typically employed in LLMs. Generally, LLM development involves two primary phases: (1) training the pre-trained models (such as LLaMA and OPT) and (2) fine-tuning these pre-trained models for specific applications (as seen in models like Alpaca and Vicuna). The data used and the training paradigms for these two phases are markedly different.
Our work specifically targets the pre-trained models. All our experiments and methodologies are therefore aligned with the LLM pre-training framework, rather than the fine-tuning process. This distinction is crucial as it aligns our approach with the initial training phase of LLMs, which is focused on establishing an efficient foundational model.

To provide practical evidence of our approach, we have made our code available in the supplementary materials. In this code, we utilize the HuggingFace Trainer designed for pre-training LLMs, not the SFTTrainer which is typically used for fine-tuning LLMs. This further reinforces that our method is focused on enhancing the pre-trained models' reasoning capacities within their original training framework.

## Minor Concerns:

Thanks for pointing out the typos, we have corrected them and revised the manuscript. The revised version has been uploaded, and all the revised content is highlighted in red.

Reviewer comment (Writing):
Typo: Sec 3.3 bianrize -> binarize.

Author response:
## Minor Concerns:

Thanks for pointing out the typos, we have corrected them and revised the manuscript. The revised version has been uploaded, and all the revised content is highlighted in red.

Reviewer comment (Writing):
Consistent notation should be used in Sec 4.1 -> LLaMA, LLaMa -> LLaMA.

Author response:
## Minor Concerns:

Thanks for pointing out the typos, we have corrected them and revised the manuscript. The revised version has been uploaded, and all the revised content is highlighted in red.

Reviewer comment (Presentation):
There may be an incorrect reference link in Sec 4.1, "showing its fast convergence property (refer to 3.2)" Should it possibly be corrected to "refer to 3.4?".

Author response:
## Minor Concerns:

Thanks for pointing out the typos, we have corrected them and revised the manuscript. The revised version has been uploaded, and all the revised content is highlighted in red.

Reviewer comment (Writing):
There are spacing issues in the Figure 7 caption, "LLMin" should be "LLM in" and "PM-LLMtriumphs" should be "PB-LLM triumphs".

Author response:
## Minor Concerns:

Thanks for pointing out the typos, we have corrected them and revised the manuscript. The revised version has been uploaded, and all the revised content is highlighted in red.

Reviewer comment (Theory):
What is the rationale behind utilizing a Hessian-based approach to identify salient weights in PTQ, while employing a magnitude-based approach to identify salient weights in QAT?

Author response:
We appreciate the opportunity to address the concerns raised regarding the inconsistent methodology for salient weight protection between PTQ and QAT.
In our experimental analysis, we employed the OPT-1.3b network and conducted PB-LLM (QAT) quantization under both Hessian and Magnitude criteria, targeting a 10% salient weight. The performance results, measured in perplexity (PPL), were as follows:

Hessian Criteria:
wikitext2: 37.07
ptb: 47.07
c4: 38.76

Magnitude Criteria:
wikitext2: 37.50
ptb: 44.42
c4: 38.64


We observed that the PPL values were comparable between the two criteria. However, it's worth noting that the Hessian criteria involve an additional calibration step (collecting calibration data to compute the Hessian), adding complexity to the process. In contrast, the Magnitude criteria offer a simpler approach, eliminating the need for this extra calibration step.
Given the similar PPL results and the desire to streamline the quantization process in QAT, we opted for the Magnitude Metric to identify salient weights during QAT. This decision was motivated by the goal of simplifying the process while maintaining comparable performance.

Reviewer comment (Experiments):
Is there a specific reason why only the PB methodology and GPTQ were applied in PTQ? I am curious about the performance of AWQ-PB in comparison.

Author response:
Thank you for raising this insightful question. Indeed, we did explore the integration of the AWQ methodology into our Partially-Binarized (PB) PTQ setting. However, our experiments revealed that incorporating AWQ in this context was not successful. The AWQ approach, when applied to our PB PTQ framework, failed to yield reasonable results. This outcome led us to omit the mention of AWQ in our paper, as we focused on methodologies that demonstrated effective performance in our experiments.

Reviewer comment (Experiments):
Is there a plan to compare the task-specific fine-tuning in QAT with the 2-bit QAT-KD methodology (QuantGPT [R2], TSLD [R3])?

Author response:
Thank you for suggesting a comparison with the 2-bit QAT-KD methodologies like QuantGPT [R2] and TSLD [R3]. We recognize the value of such a comparison in providing a more comprehensive understanding of our method's effectiveness.

However, due to time constraints and computational limitations during the rebuttal period, it is not feasible for us to implement these methods immediately. Specifically, the QuantGPT [R2] methodology lacks an open-source codebase, complicating our efforts to conduct a direct comparison. Additionally, while TSLD [R3] has recently released their codebase (on November 17, 2023), this was already seven days after the start of the ICLR rebuttal period, providing insufficient time for a thorough implementation and comparison.

We have already made our codebase available in the supplementary materials and plan to open-source it soon. In the camera-ready version of our paper, we aim to include these comparisons. Alternatively, we encourage researchers and practitioners in the field to use our open-source resources to perform this comparison.

Reviewer comment (Evaluation):
The optimal scaling factor and salient weight freezing seem to primarily aim at reducing harsh quantization errors before QAT. Is there insight into how these methods improve the QAT training process as depicted in Figure 5?

Author response:
**High-level Insight on Training Efficiency**

As detailed in Sections 3.4 and 4.1 of our paper, our QAT process for PB-LLMs begins with a pre-trained model, significantly reducing the additional training overhead. 
In Section 3.4.1, we discuss training efficiency in depth. Our proposed salient weights frozen mechanism optimally leverages pre-trained weights. As demonstrated in Fig. 1, we initially filter out a specific number of weights—2% by magnitude—at the onset of quantization-aware training. These weights remain frozen throughout the training process, as substantiated by the training efficiency analysis in Fig. 5. The results clearly indicate that these salient weights are instrumental in preserving LLM capacity. By maintaining their high-bit representation, we facilitate the training of quantized LLMs while simultaneously diminishing the complexity of optimization.

Reviewer comment (Experiments):
While the incorporation of Quantization-Aware Training (QAT) in LLM compression is an interesting proposal, its practicality is uncertain given the substantial costs associated with training LLMs. Could the authors elaborate on the overhead implications of QAT for LLMs?

Author response:
Thank you for recognizing the innovative aspect of our approach. We appreciate the opportunity to elaborate on the practicality and overhead implications of QAT for LLMs.

As detailed in Sections 3.4 and 4.1 of our paper, our QAT process for PB-LLMs begins with a pre-trained model, significantly reducing the additional training overhead. Notably, we fine-tune our binarized PB-LLMs for only 10K iterations, a duration sufficient to surpass the performance of LLM-QAT, which requires a considerably longer training period of 100K iterations. Practically, this fine-tuning necessitates just 2 NV-link connected A6000s over approximately 2 days to achieve performance recovery for the binarized LLaMA-7B. In contrast, LLM-QAT demands around 7-10 days with 8 A6000s, underscoring the efficiency of our approach.

Further, in Section 3.4.1, we discuss training efficiency in depth. Our proposed salient weights frozen mechanism optimally leverages pre-trained weights. As demonstrated in Fig. 1, we initially filter out a specific number of weights—2% by magnitude—at the onset of quantization-aware training. These weights remain frozen throughout the training process, as substantiated by the training efficiency analysis in Fig. 5. The results clearly indicate that these salient weights are instrumental in preserving LLM capacity. By maintaining their high-bit representation, we facilitate the training of quantized LLMs while simultaneously diminishing the complexity of optimization, thus contributing to the overall practicality and feasibility of our method.

Reviewer comment (Presentation):
In regards to Table 2, it is unclear whether GPTQ-PB represents the method proposed by the authors. Could you clarify the distinction between GPTQ-PB and PB-LLM within the context of your study?

Author response:
We apologize for any confusion caused by the terminology in Table 2 and appreciate the opportunity to clarify. In Table 2, 'GPTQ-PB' refers to our method based on post-training quantization (PTQ), whereas 'PB-LLM' designates the approach based on quantization-aware training (QAT). Specifically, considering the partially-binarized weight matrix as a novel weight format, we explored two distinct methodologies to enhance the performance of quantized LLMs using this format. 'GPTQ-PB' represents our method following the PTQ paradigm, while 'PB-LLM' aligns with the QAT paradigm.

To improve clarity and unify the terminology in our paper, we have revised this section. Specifically, 'GPTQ-PB' has been renamed to 'PB-GPTQ' in the updated version of the manuscript (aligning with ‘PB-LLM’), which has been uploaded. The revisions are highlighted in red for easy identification and reference.

Reviewer comment (Experiments):
The application of optimal scaling techniques appears to be confined to the specific case presented. Could these techniques be generalized to other bit configurations, and if so, how might this affect the compression performance?

Author response:
Our PB-LLM technique is adaptable to various bit configurations. PB-LLM employs selective binarization of the LLMs' weight matrix, retaining a proportion of weights in high bits to enhance linguistic capacity. The formula governing the overall bit number, $N_{bit}$, is $N_{bit} \leq 1 \times r_{binary} + N_{salient-bit} \times (1 - r_{binary}) + 1$. Here, $r_{binary}$ is the ratio of binarized weights, and $N_{salient-bit}$ is the number of bits reserved for salient weights (e.g., 8 bits), with an additional bit used for a bitmap index mechanism. By varying $r_{binary}$, we can achieve different levels of bit quantization.

The flexibility of our method across various bit settings is demonstrated in Tables 2 and 3 (binary ratio varies from 10% to 50%). These results illustrate the generalization capability of PB-LLM, confirming its effectiveness not just in the specific case presented but also in a range of bit configurations.

Apart from the generalization ability in different bits, we have supplemented some experiments to investigate the generalization ability on larger LLMs during the rebuttal period. This is referred to the responses to Reviewer 9nFP.