Paper: LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (ICLR-2024)

Reviewer comment (Experiments):
The 4-6-16 setting of experiments is a little bit strange and unfriendly to hardware. I think the paper should move 4-4-4 experiments that take 4-bit weight and activations from the appendix to the main body.

Author response:
Thanks for this suggestion. We will move the 4-4-4 experiments to the main table. it's noteworthy that W4A6 quantization can effectively reduce GPU run-time memory usage compared to W4A8. Additionally, it operates at a speed comparable to W4A8 when using dequantization.

Reviewer comment (Presentation):
I find the fifth row in Table 1 looks strange. It seems that the performance of GPTQ is really slow, even slower than RTN with 4-8-16 bits.

Author response:
Regarding our previous experiments with GPTQ for LLaMA-7B, we found that the GPTQ is sensitive to calibration data choice. Initially, our use of C4 to calibrate all tasks wasn't optimal. Adjusting the calibration dataset to match the evaluation dataset yielded improved results for W4A16 GPTQ, as shown in the following table and will be reflected in the updated Table 1.

| boolq | piqa | siqa | hellaswag | winogrande | arc_easy | arc_challenge | obqa | avg. |
|-|-|-|-|-|-|-|-|-|
|68.7 | 78.2 | 49.0 | 74.0 | 68.1 | 69.9 | 45.8 | 57.0 | 63.8 |

Reviewer comment (Experiments):
Experiments can be more convincing if the paper also conducts on other structures like LLaMA-2, and others take the vanilla Transformer structure as LLaMA adopts a different one.

Author response:
Following your suggestion, we conducted experiments on the OPT-125M model, revealing that the W4A16KV16 LLM-QAT model achieves performance on par with full-precision models. Detailed results for other bit settings will be incorporated into the manuscript. 
| #bits | boolq | piqa | siqa | hellaswag | winogrande | arc_easy | arc_challenge | obqa | avg. |
|-|-|-|-|-|-|-|-|-|-|
| full-precision | 40.8 | 24.8 | 60.8 | 62.5 | 41.5 | 33.0 | 32.7 | 49.7 | 43.2 |
| 4-16-16 | 40.6 | 25.1 | 58.3 | 61.1 | 40.7 | 32.8 | 32.4 | 53.8 | 43.1 |

Reviewer comment (Experiments):
Experiments can be better if the paper could compare with some recent papers [3] and [4].

Author response:
Thanks for suggesting additional papers for comparison and ablation experiments. It's crucial to note that [1], [3], and [4] represent concurrent works yet to be published by the ICLR submission deadline. As per the ICLR reviewer guidelines, not comparing to these works shouldn't be grounds for rejection. The rapid evolution within the Language Model Models (LLMs) community makes it challenging to track every new development. Regarding [2], it focuses on using generated data for data augmentation in cross-lingual semantic parsing tasks. In contrast, our emphasis lies in showcasing that exclusive utilization of generated data can effectively fine-tune precise quantized LLMs. The primary focuses between the two approaches are distinct. Nonetheless, these papers are good work and we'll incorporate discussions about them in the related works section.

Reviewer comment (Experiments):
Based on my practical experience and discussion with some people in this field, we find quantizing KV cache with per-token quantization on LLaMA models will not incur much accuracy decline, even on a 4-bit case. However, results in the main table seem to give an opposite conclusion.

Author response:
Regarding your experiment showing a loss-less accuracy in KV-cache quantization, we're curious if you tested the model on generation tasks that construct sequences token by token. Typically, common evaluation frameworks assess perplexity for datasets like wiki and C4, by predicting the next token based on the ground truth context, instead of the generated tokens, which is similar to the training phase. In that case, the KV cache is not used. Additionally, we confirmed that the gate function output distribution isn't symmetric-like. While symmetric quantization works well for LLaMA models, we'll modify the claim to prevent potential misinterpretation.

Reviewer comment (Evaluation):
The paper claims that distribution for LLaMA models is usually symmetric. However, we find products of the output of gate and up functions in LLaMA are not so.

Author response:
Regarding your experiment showing a loss-less accuracy in KV-cache quantization, we're curious if you tested the model on generation tasks that construct sequences token by token. Typically, common evaluation frameworks assess perplexity for datasets like wiki and C4, by predicting the next token based on the ground truth context, instead of the generated tokens, which is similar to the training phase. In that case, the KV cache is not used. Additionally, we confirmed that the gate function output distribution isn't symmetric-like. While symmetric quantization works well for LLaMA models, we'll modify the claim to prevent potential misinterpretation.

Reviewer comment (Experiments):
Compare LLM-QAT only to training-free methods such as round-to-nearest (RTN) and SmoothQuant [4]. Although AdaRound [1], AdaQuant [2], and FlexRound [3] are post-training quantization (PTQ) methods, they still utilize a training set (small calibration set) in a layer-wise or block-wise manner. Given that these PTQ methods adopt a similar knowledge distillation scheme (even though the loss term is very different), it would be beneficial to report results comparing LLM-QAT with these methods (at least one of them), which can be viewed as non-training-free quantization methods.

Author response:
**A1: The comparison to the SmoothQuant paper is extensively discussed in our main Tables (Table 1 and 2). We also compared with another SoTA PTQ method for LLMs, GPTQ.**  These two PTQ methods are the state-of-the-art PTQ methods for LLMs. In the GPTQ paper, thorough comparisons were made between their method and various Post-Training Quantization (PTQ) approaches on different language models, showcasing GPTQ's superior performance. Hence, we select the most effective PTQ method for LLMs as a benchmark for comparison with our approach. Reproducing other PTQ methods on Language Model Models (LLMs) within a short timeframe poses challenges, and we avoid presenting rushed and potentially suboptimal results for other methods.

Reviewer comment (Experiments):
The arguments would be more convincing with additional references and explanations for why LLM-QAT primarily focuses on the W4A8 quantization format. For instance, regarding the W8A8 integer quantization, SmoothQuant [4] showcases actual latency acceleration results.

Author response:
**A2: It's worth noting that we did NOT only do experiments on W4A8; rather, we experimented with various bit settings, including W4A6, W4A16, W8A8, and W8A16, etc, detailed in in our main Tables (Table 1 and 2).** Therefore, we don’t quite agree with reviewer's comment that 'LLM-QAT primarily focuses on the W4A8 quantization format'.

Reviewer comment (Experiments):
Are there experimental results available for other models?

Author response:
**A3**: Following your suggestion, we conducted experiments on the OPT-125M model, revealing that the W4A16KV16 LLM-QAT model achieves performance on par with full-precision models. Detailed results for other bit settings will be incorporated into the manuscript. 
| #bits | boolq | piqa | siqa | hellaswag | winogrande | arc_easy | arc_challenge | obqa | avg. |
|-|-|-|-|-|-|-|-|-|-|
| full-precision | 40.8 | 24.8 | 60.8 | 62.5 | 41.5 | 33.0 | 32.7 | 49.7 | 43.2 |
| 4-16-16 | 40.6 | 25.1 | 58.3 | 61.1 | 40.7 | 32.8 | 32.4 | 53.8 | 43.1 |

Reviewer comment (Novelty):
In Section 4, "Related Works", I kindly suggest modifying the last sentence: "A single random initial token allows LLMs to autonomously generate data suitable for QAT fine-tuning. To the best of our knowledge, this has not been studied in existing literature." LLM-QAT employs a knowledge distillation (KD) scheme for quantization-aware training (QAT), and a previous method named ZeroQuant [1] also uses the KD scheme. Even though these methods primarily focus on post-training quantization, they also utilize a training dataset for quantization. Specifically, ZeroQuant presents data-free quantization that employs random data for generative language models.

Author response:
**A4**: We'll include a discussion on ZeroQuant in related works. We would like to emphasize two key distinctions: (1) the random tokens in ZeroQuant is not generated from the first token using LLMs, rather, they employ entirely random tokens for entire sentences, differing from our methodology. (2) the quality requirements for calibration data in PTQ are significantly less demanding compared to the quantity needed for training data in QAT.

Reviewer comment (Novelty):
The techniques utilized in the paper (QAT using StatsQ and LSQ, MinMax quantization, knowledge distillation) have been shown to work in the quantization setting before, so the contributions are not particularly novel. Effectiveness of MinMax for language models due to outliers has also been demonstrated earlier.

Author response:
The primary focus of our paper lies not in the design of the quantization function but rather in presenting an innovative data generation pipeline facilitating Quantization Aware Training (QAT) for Language Model Models (LLMs) within feasible computational constraints, as detailed in Lines 32-42 of our manuscript. The previous works Vicuna [1] and Alpaca [2] stand as the most relevant works in this domain, while a distinct divergence exists between their methodologies and ours. Vicuna [1] relies on user-uploaded ShareGPT data for GPT instruction fine-tuning, whereas Alpaca [2] necessitates predefined human prompts for data generation, meticulously curating categories for diversity. In contrast, our approach omits human prompts or user data, initiating QAT fine-tuning data generation with just a single random initial token. To our knowledge, this is yet unexplored in current literature. Our quantization method utilizes standard MinMax quantization (Eq.3).  
[1] Chiang W L, Li Z, Lin Z, et al. Vicuna. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.  
[2] Taori R, Gulrajani I, Zhang T, et al. Alpaca: A strong, replicable instruction-following model[J]. Stanford Center for Research on Foundation Models.

Reviewer comment (Experiments):
While the results are generally better than PTQ techniques, especially as precision is lowered (this is usually the case for low precision scenarios - QAT performs better than PTQ), the gaps are still quite large compared to full precision to be practically useful, especially for the 4-bit case, and the reviewer is not entirely convinced from the limited ablation studies that fine-tuning on a subset of the original data would not yield comparable results.

Author response:
Thanks for the suggestion. While the full-training data of the LLaMA model remains unavailable publicly, we utilized the Red Pajama dataset (a substantial LLM pre-training dataset containing 1 trillion tokens) to address concerns. We sampled 100k sentences from this dataset for a fair comparison to our method. In the following table, we obtained an average score of 61.60 on zero-shot reasoning tasks, slightly surpassing fine-tuning on the C4 dataset but still falling short of the results achieved through finetuning on the generated dataset. This result underscores that sampling a small subset from an extensive pretraining dataset struggles to replicate the dataset's distribution adequately. Conversely, the diversity inherent in data extracted from pretrained LLMs via diverse initial tokens proves more suitable for QAT model finetuning. Nonetheless, this comparison holds value and will be included in our manuscript.

| data | #bits | boolq | piqa | siqa | hellaswag | winogrande | arc_easy | arc_challenge | obqa | avg. |
|-|-|-|-|-|-|-|-|-|-|-|
| C4 | 4-6-16 |  61.7 | 77.7 | 48.8 | 73.2 | 67.2 | 67.8 | 43.6 | 52.2 | 61.5 |
| Red Pajama subset | 4-6-16 | 70.0 | 74.9 | 47.1 | 71.2 | 66.1 | 66.8 | 43.4 | 53.0 | 61.6 | 
| generated da​​ta | 4-6-16 | 72.9 | 76.8 | 47.9 | 72.4 | 68.3 | 68.8 | 44.2 | 53.2 | 63.1 |

Reviewer comment (Experiments):
For the dataset ablation study, have the authors considered using a subset of each dataset and comparing that to generated data, instead of exclusively fine-tuning using a single dataset?

Author response:
Thanks for the suggestion. While the full-training data of the LLaMA model remains unavailable publicly, we utilized the Red Pajama dataset (a substantial LLM pre-training dataset containing 1 trillion tokens) to address concerns. We sampled 100k sentences from this dataset for a fair comparison to our method. In the following table, we obtained an average score of 61.60 on zero-shot reasoning tasks, slightly surpassing fine-tuning on the C4 dataset but still falling short of the results achieved through finetuning on the generated dataset. This result underscores that sampling a small subset from an extensive pretraining dataset struggles to replicate the dataset's distribution adequately. Conversely, the diversity inherent in data extracted from pretrained LLMs via diverse initial tokens proves more suitable for QAT model finetuning. Nonetheless, this comparison holds value and will be included in our manuscript.

| data | #bits | boolq | piqa | siqa | hellaswag | winogrande | arc_easy | arc_challenge | obqa | avg. |
|-|-|-|-|-|-|-|-|-|-|-|
| C4 | 4-6-16 |  61.7 | 77.7 | 48.8 | 73.2 | 67.2 | 67.8 | 43.6 | 52.2 | 61.5 |
| Red Pajama subset | 4-6-16 | 70.0 | 74.9 | 47.1 | 71.2 | 66.1 | 66.8 | 43.4 | 53.0 | 61.6 | 
| generated da​​ta | 4-6-16 | 72.9 | 76.8 | 47.9 | 72.4 | 68.3 | 68.8 | 44.2 | 53.2 | 63.1 |

Reviewer comment (Experiments):
Have the authors considered using a larger teacher model for the distillation data generation?

Author response:
Regarding leveraging logits from a superior pre-trained model, we conducted experiments by distilling a LLaMA-7B model using the LLaMA-13B model, shown in the following table. Interestingly, employing the 13B model as the teacher resulted in decreased accuracy compared to using the 7B model. We consider it crucial to employ the same architecture for both teacher and student models. This shared architecture ensures initial weight alignment and promotes the closest distribution match. As a result, the teacher model can effectively guide the student model, aiding in error compensation introduced by quantization.

| #bits | boolq | piqa | siqa | hellaswag | winogrande | arc_easy | arc_challenge | obqa | avg. |  **prev. avg. ** |
|-|-|-|-|-|-|-|-|-|-|-|
| 4-8-4 | 69.3 | 74.7 | 46.4 | 67.6 | 63.1 | 63.9 | 42.5 | 49.2 | 59.6 | **60.7** |
| 4-8-8 | 73.5 | 77.5 | 47.7 | 71.6 | 67.5 | 69.0 | 46.3 | 52.8 | 63.2 | **64.2** |
| 4-6-16 | 71.2 | 77.1 | 46.5 | 70.6 | 65.4 | 68.1 | 44.7 | 50.6 | 61.8 | **63.1** |
| 4-8-16 | 73.9 | 77.0 | 48.0 | 71.6 | 67.8 | 69.1 | 46.0 | 54.2 | 63.5 | **64.4** |
| 4-16-16 | 74.6 | 77.3 | 48.0 | 71.9 | 67.6 | 69.4 | 47.8 | 54.6 | 63.9 | **64.4** |
| 8-8-4 | 71.1 | 76.4 | 47.1 | 70.1 | 64.9 | 68.2 | 44.0 | 49.2 | 61.4 | **61.6** |
| 8-8-8 | 74.8 | 79.0 | 47.6 | 73.7 | 70.1 | 71.3 | 47.5 | 56.6 | 65.1 | **66.0** |
| 8-8-16 | 73.9 | 78.9 | 47.8 | 73.8 | 69.3 | 71.1 | 48.2 | 55.0 | 64.8 | **65.7** |

Reviewer comment (Evaluation):
Even though the quantized models still maintain the quality closely, there exist non-trivial gaps especially when the weights are quantized to 4-bits.

Author response:
Compared to small models, Language Model Models (LLMs) exhibit distinctive characteristics, where outliers play a significant role. The reviewer's assertion that "additional training (QAT) should yield a quality very close to that of the floating point unquantized model" seems grounded in observations from smaller models, such as BERT or BART. Considering the fact that even for 8-bit quantization, LLMs necessitate special approaches like SmoothQuant [1] or outlier suppression [2], where naive Post-Training Quantization (PTQ) typically suffices for smaller models, it is not surprising that quantizing the LLMs to W4A8 remains a challenging task and still has some gap.  
[1] Xiao, Guangxuan, et al. "Smoothquant: Accurate and efficient post-training quantization for large language models." International Conference on Machine Learning. PMLR, 2023.  
[2] Dettmers, Tim, et al. "LLM.int8(): 8-bit matrix multiplication for transformers at scale." arXiv preprint arXiv:2208.07339 (2022).

Reviewer comment (Experiments):
With additional training (QAT), it is expected the quality is very close to the floating point unquantized model.

Author response:
Compared to small models, Language Model Models (LLMs) exhibit distinctive characteristics, where outliers play a significant role. The reviewer's assertion that "additional training (QAT) should yield a quality very close to that of the floating point unquantized model" seems grounded in observations from smaller models, such as BERT or BART. Considering the fact that even for 8-bit quantization, LLMs necessitate special approaches like SmoothQuant [1] or outlier suppression [2], where naive Post-Training Quantization (PTQ) typically suffices for smaller models, it is not surprising that quantizing the LLMs to W4A8 remains a challenging task and still has some gap.  
[1] Xiao, Guangxuan, et al. "Smoothquant: Accurate and efficient post-training quantization for large language models." International Conference on Machine Learning. PMLR, 2023.  
[2] Dettmers, Tim, et al. "LLM.int8(): 8-bit matrix multiplication for transformers at scale." arXiv preprint arXiv:2208.07339 (2022).

Reviewer comment (Experiments):
Some PTQ methods such as (Li, Qingyuan, et al. "FPTQ: Fine-grained Post-Training Quantization for Large Language Models." arXiv preprint arXiv:2308.15987 (2023).) claim better quality.

Author response:
Regarding the FPTQ paper, it represents concurrent work that is not yet to be formally accepted. Per the ICLR reviewer guidelines, this should not be grounds for rejection. Notably, this paper also shows comparisons with our methodology, as FPTQ adopts a higher granularity, this leads to the higher accuracy.

Reviewer comment (Reproducibility):
Why 3-5 tokens? What exact number is used?

Author response:
We appreciate your feedback on the typos, all of which will be rectified in our manuscript. To provide specifics, our methodology used {3, 4, 5} deterministic tokens in each round for the first token in 32k vocabulary across three rounds, resulting in a total of 3 x 32k = 96k sentences. Additionally, 4k sentences were generated using random numbers within the range of 3-5. These generated sentences composed our 100k-sample finetuning dataset.