Paper: LLM-Codebook for Extreme Compression of Large Language Models (ICLR-2024)

Reviewer comment (Reproducibility):
It is unclear how can this method be combined with downstream LLM finetuning or if it is only effective for post-finetuning compression. After the compression, each linear layer consists of a codebook and an index map, how can the model be further finetuned under this structure?

Author response:
In the codebook-based LoRA during the recovery stage, the original weights of each linear layer are frozen, and an adaptor is assigned to each layer. During the fine-tuning process of all adaptors, a linear layer is randomly selected, and its original weight is merged with the adaptor for compression, resulting in a codebook and index map. Subsequently, the compressed weight is reconstructed based on these components. The compressed weight is no longer updated, meaning that the codebook and index map of that specific linear layer are also not updated. To compensate for the compression error of the layers that have already been compressed, the other linear layers that have not yet been compressed continue to update their adaptors. The above process is repeated until all the linear layer are completely compressed.

Reviewer comment (Reproducibility):
Since the method requires full model finetuning using Lora for performance recovery, would the memory cost be huge when meeting with a larger base model like llama-70B?

Author response:
The memory cost is primarily divided into two components: the base model and the adaptor. Typically, the adaptor accounts for about 1 to 3% of the base model's memory requirement, depending on the chosen value of the rank parameter. The memory consumption of LLaMA-70B exceeds 130GB, and regrettably, we lack sufficient computational resources to support the compression of this model.

Reviewer comment (Experiments):
It is still being determined why the layer is randomly selected for compression during the Lora finetuning procedure.

Author response:
Intel's Incremental Network Quantization[3] adopts a progressive quantization strategy, which divides weights into two groups. One group is quantized and then frozen, while the other is fine-tuned to compensate for the quantization loss of the former. This process is repeated until all weights are fully quantized. Inspired by the aforementioned strategy, the codebook-based LoRA in the recovery stage also represents a form of progressive compression, with specific procedures as outlined in answer1.

Reviewer comment (Experiments):
Eq(5) implies that the estimation of model weight salience depends on the selected dataset and the authors used only 15 randomly selected samples from the Bookcorpus dataset. There is a question about how sensitive this estimation is in terms of the data source (e.g., how about 15 samples from Wiki or a similar corpus) and the number of samples (e.g., is 15 samples enough for a good estimation of salience), as well as the possible influence of the salience estimation error on the later recovery stage.

Author response:
A1: In Table 1 of Official Comment 2, we conducted comparative analyses focusing on the sources of data and the number of samples used. We found that data derived from the Bookcorpus dataset yielded more accurate salience calculations compared to those obtained from Wiki and the C4 corpus. Additionally, increasing the number of samples further enhanced the precision of salience estimation. As shown in Table 2 of Official Comment 2, we performed ablation experiments by omitting the use of salience. These experiments demonstrated that preserving weights with higher salience significantly improves compression performance in the recovery stage.

Reviewer comment (Experiments):
In Table 3, the Lora-based tuning appears more critical to prevent the performance from unacceptable degradation. There is curiosity about whether, with only vanilla K-means, it is possible to match the best performance if more effort is put into tuning the recovery stage, and what the must-be reason is to use Hessian-aware K-means instead of vanilla K-means given its extra computation cost of estimation and secondary role in recovering model performance.

Author response:
A2: In Table 3 of Official Comment 2, we conducted comparative experiments with different LoRA ranks, attempting to achieve optimal performance using only the conventional K-means algorithm while minimally increasing the number of trainable parameters. The results indicate that merely employing the standard K-means algorithm does not suffice to attain optimal performance.  
Contrary to playing a secondary role, Hessian-aware K-means is significantly influential in model compression. In fact, its impact on compression performance exceeds that of the recovery stage at 4/3-bit compression. Conversely, the recovery stage has a more pronounced effect in 2-bit compression scenarios.

Reviewer comment (Experiments):
Following from the previous question, there is a request for clarification on what baselines in Table 1 support Lora-based tuning, and whether, if an extra recovery stage based on Lora tuning was carefully added, those baseline performances would catch up with the proposed method.

Author response:
A3: The QLoRA and LLM-Pruner are based on LoRA tuning, while GPTQ, SparseGPT, AWQ, SqueezeLLM, and Omniquant[3] represent Post-Training Quantization (PTQ) methods or pruning methods not based on LoRA. In Table 3 of Official Comment 2, we have provided data that excludes the use of a recovery stage, and the results surpass those of the aforementioned methods. However, incorporating LoRA into PTQ methods involves substantial engineering efforts, and the adaptor weights fine-tuned are in fp16 format, which does not qualify as strict low-bit quantization. It indicates that our codebook-based LoRA approach in the recovery stage marks a significant innovation. This method deviates from earlier practices in CNN-related research, where updating the codebook was done using gradients of compressed weights through backpropagation – a method not feasible for large language models (LLMs). Instead, we have adopted a strategy inspired by Intel's Incremental Network Quantization, which employs a progressive quantization strategy[4]. We merge linear layer weights with adaptors, compress them, and then freeze them. The compression loss is compensated by adaptors in other uncompressed layers. This effectively encodes LoRA's information into the codebook without necessitating additional memory.

Reviewer comment (Evaluation):
In Table 3 last row, the salience-adopted clustering stage leads to more performance degradation compared to the vanilla baseline (-3.5 vs -6.3). There is a question asking the author to explain the reason for this observation.

Author response:
A4: At this juncture, with a compression ratio of 95%, our method collapses due to substantial compression errors, resulting in the compressed model yielding garbled inference outcomes, which fail to reflect the impact of salience on model compression.

Reviewer comment (Novelty):
This work is essentially an application of production quantization on LLMs. Although the final performance surpasses the baselines, the method itself does not present much novelty. The idea of Hessian-aware k-means has also been utilized in previous works [1][2]

Author response:
A1: We sincerely apologize for our insufficient research, as we failed to recognize that the concept of Hessian-aware k-means has already been utilized in previous works [3][4]. Our implementation of Hessian-aware k-means differs in several key aspects. First, by performing a Taylor expansion on the loss function, we obtain three terms: the first related to the gradient, the second to the Hessian matrix, and the third being a redundant term. Previous works often omit the first term, assuming gradients are close to zero. However, in our case, since the dataset used for computing gradients and Hessian matrices is not extracted from the original training set, gradients are not negligible and should be considered. We regret that our naming convention may have led to the misconception that the significance of weights is solely determined by Hessian calculations. As shown in Table1 of Official Comment 2, we conducted experiments showing that considering both the first and second terms is more effective than considering them separately. Additionally, in our salience-based centroid weighting update, we noticed that many weights have salience values close to zero, leading to significant compression errors. To mitigate this, we added salience.max() to all salience values for smoothing. 

Furthermore, our codebook-based LoRA approach in the recovery stage marks a significant innovation. This method deviates from earlier practices in CNN-related research, where updating the codebook was done using gradients of compressed weights through backpropagation – a method not feasible for large language models (LLMs). Instead, we have adopted a strategy inspired by Intel's Incremental Network Quantization, which employs a progressive quantization strategy[5]. We merge linear layer weights with adaptors, compress them, and then freeze them. The compression loss is compensated by adaptors in other uncompressed layers. This effectively encodes LoRA's information into the codebook without necessitating additional memory.

Reviewer comment (Experiments):
From the perspective of LLM compression ratio, this work does not compare with the state-of-the-art quantization works [2][3][4]

Author response:
A2&A3: 
In Table 2 of Official Comment 2, we have supplemented the low-bit compression comparison with Omniquant[6], SqueezeLLM, and AWQ, demonstrating that our work achieves optimal performance in 2/3-bit compression of LLaMA-7b and LLaMA-13B. Although QuIP represents an excellent piece of work, the absence of open-source code of LLaMA precludes comparison. As shown in Table 3 of Official Comment 2, in exploring the potential of combining low-bit compression with structural clustering for further compression, we conducted experiments with Codebook PTQ, indicating that the results for W8A16 and W4A16 are very close to those of W16A16.

Reviewer comment (Experiments):
The current evaluation is focused solely on the 7b model. Could the author also provide evaluations on larger models, such as Llama-13b?

Author response:
A2&A3: 
In Table 2 of Official Comment 2, we have supplemented the low-bit compression comparison with Omniquant[6], SqueezeLLM, and AWQ, demonstrating that our work achieves optimal performance in 2/3-bit compression of LLaMA-7b and LLaMA-13B. Although QuIP represents an excellent piece of work, the absence of open-source code of LLaMA precludes comparison. As shown in Table 3 of Official Comment 2, in exploring the potential of combining low-bit compression with structural clustering for further compression, we conducted experiments with Codebook PTQ, indicating that the results for W8A16 and W4A16 are very close to those of W16A16.