VPTQ : Extreme Low-bit Vector Post-Training Quantization for Large Language Models (2024)

Abstract: Scaling model size significantly challenges the deployment and inference of Large Language Models (LLMs). Due to the redundancy in LLM weights, recent research has focused on pushing weight-only quantization to extremely low-bit (even down to 2 bits). It reduces memory requirements, optimizes storage costs, and decreases memory bandwidth needs during inference. However, due to numerical representation limitations, traditional scalar-based weight quantization struggles to achieve such extreme low-bit.Recent research on Vector Quantization (VQ) for LLMs has demonstrated the potential for extremely low-bit model quantization by compressing vectors into indices using lookup tables. In this paper, we introduce **Vector Post-Training Quantization (VPTQ)** for extremely low-bit quantization of LLMs. We use Second-Order Optimization to formulate the LLM VQ problem and guide our quantization algorithm design by solving the optimization.We further refine the weights using Channel-Independent Second-Order Optimization for a granular VQ.In addition, by decomposing the optimization problem, we propose a brief and effective codebook initialization algorithm. We also extend VPTQ to support residual and outlier quantization, which enhances model accuracy and further compresses the model.Our experimental results show that VPTQ reduces model quantization perplexity by 0.01-0.34 on LLaMA-2, 0.38-0.68 on Mistral-7B, 4.41-7.34 on LLaMA-3 over SOTA at 2-bit, with an average accuracy improvement of 0.79-1.5% on LLaMA-2, 1% on Mistral-7B, 11-22% on LLaMA-3 on QA tasks on average. We only utilize 10.4-18.6% of the quantization algorithm execution time, resulting in a 1.6- 1.8\times increase in inference throughput compared to SOTA.

## end to end quantization algorithm in this section, we will detail the end-to-end model quantization algorithm (algorithm 2). the algorithm takes the original model, vector length v, centroid number k, and hessian matrices h as inputs. it starts by iterating over each layer l of the model. as each layer's quantization only relates to the current layer and the hessian matrix, we can fully parallelize the quantization of each layer on gpus. in each layer, we first quantize the weight of each linear operator (matrix multiplication of input and weight). if we enable the outlier option, the algorithm first selects outlier columns following section 3.2 and initializes the outlier centroids c outlier . then, vptq is applied to the outlier weights w outlier using the outlier centroids, generating the quantized weights w ′ outlier . next, the algorithm initializes the centroids c for the remaining columns and applies vptq to the weights w using these centroids to produce the quantized weights w ′ . lastly, if residual quantization is enabled, the algorithm initializes the residual centroids c res . it applies vptq to the residual error between the original weights and the quantized weights (w -w ′ ), using the residual centroids. the quantized weight is updated as w ′′ . after processing all the operators, the algorithm will fine-tune the layer l if we enable layer finetuning. the loss function is the mean squared error (mse) between the original and quantized computations. in layer-wise fine-tuning, we only update the normalization operator (e.g. rmsnorm) and centroid. these parameters only comprise a small fraction of the entire layer, and we can complete the fine-tuning quickly with limited memory. after each layer completes quantization and finetuning, we can further fine-tune the entire model as other ptq methods used @xcite @xcite . once the algorithm processes all layers, it outputs the quantized model. the end-to-end vptq algorithm quantizes all the weights in every linear operator in the model into an index and a codebook @xcite . during model inference, we only need to dequantize the weight matrix by reading centroids from the codebook according to the index before executing the operator. ## conclusion in this paper, we propose vector post-training quantization (vptq), a novel approach to achieving extremely low-bit quantization of llms by vector quantization. through the application of second-order optimization, we have formulated the llm vector quantization problem and directed the design of our quantization algorithm. by further refining the weights via channel-independent second-order optimization, we have enabled a more granular vq. vptq also includes a brief and effective codebook initialization algorithm, which is achieved by decomposing the optimization problem. we have extended vptq to support residual and outlier quantization, which not only improves model accuracy but also further compresses the model size. our experimental results demonstrate the effectiveness and efficiency of vptq. the perplexity of quantized model is reduced by 0.01-0.34 on llama-2, 0.38-0.68 on mistral-7b, 4.41-7.34 on llama-3 over sota at 2-bit, with an average accuracy improvement of 0.79-1.5% on llama-2, 1% on mistral-7b, 11-22% on llama-3 on qa tasks. furthermore, we achieved these results only using 10.4-18.6% of the execution time of the quantization algorithm, leading to a 1.6-1.8× increase in inference throughput compared to sota. these results underscore the potential of vptq as an efficient and powerful solution for the deployment and inference of llms, particularly in resourceconstrained settings.