Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLM s (2024)

Abstract: Large Language Models (LLMs) have demonstrated exceptional proficiency in language-related tasks, but their deployment poses significant challenges due to substantial memory and storage requirements. Weight-only quantization has emerged as a promising solution to address these challenges. Previous research suggests that fine-tuning through up and down rounding can enhance performance. In this study, we introduce SignRound, a method that utilizes signed gradient descent (SignSGD) to optimize rounding values and weight clipping within just 200 steps. SignRound integrates the advantages of Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ), achieving exceptional results across 2 to 4 bits while maintaining low tuning costs and avoiding additional inference overhead. For example, SignRound achieves absolute average accuracy improvements ranging from 6.91% to 33.22% at 2 bits, as measured by the average zero-shot accuracy across 11 tasks. It also demonstrates strong generalization to recent models, achieving near-lossless 4-bit quantization in most scenarios. The source code will be made publicly available.

## introduction in recent years, there has been a significant surge in the adoption of large language models (llms), leading to their widespread deployment demand even on devices with constrained resources. however, deploying llms on these devices poses significant challenges due to their extensive memory and storage requirements. additionally, the computational demands of these models create obstacles for real-time applications. therefore, studying techniques such as quantization is crucial for enabling the efficient deployment of llms. quantization techniques can be broadly categorized into two main types: quantization-aware training (qat) * correspondence:wenhua.cheng@intel.com @xcite @xcite and post-training quantization (ptq) @xcite @xcite . qat involves training the model with quantization in mind, using simulated lower-precision representations to allow the model to learn and adapt to the effects of quantization. this approach often results in better accuracy compared to ptq. however, qat has drawbacks, including increased training complexity, longer training times, and the need to tune hyperparameters. the application of qat to llms can be particularly resource-intensive, despite recent efforts @xcite to improve the efficiency of fine-tuning llms. on the other hand, ptq directly quantizes the model without any simulated training or finetuning. while ptq is a more straightforward approach, it is susceptible to significant accuracy drops. this underscores the importance of further advancements in ptq methods to enhance their accuracy preservation capabilities. quantization commonly applies to two types of tensors: activations and weights. quantizing activations for llms can be challenging @xcite @xcite , making weight-only quantization a more practical option. moreover, the main bottleneck in generating new tokens for llms often arises from memory bandwidth limitations @xcite , emphasizing the advantage of weight-only quantization. this study focuses on weight-only quantization. in quantizing weights, a critical step involves rounding, primarily achieved through rounding-tonearest (rtn). rtn quantizes each weight independently by rounding it to the nearest integer, but it overlooks the relationships between weights and between weights and activations. adaptive rounding @xcite explored the potential for an enhanced rounding strategy to improve accuracy. they approached the rounding task by formulating it as a quadratic unconstrained binary optimization problem and approximating the loss using a taylor series expansion. however, relying solely on the second-order term may not yield accurate results, as rounding can significantly modify weights, making other order terms non-negligible. we select signsgd @xcite @xcite as our optimization method to approach the optimal rounding solution within a limited number of steps. the motivation behind this choice, which is elaborated in section 3, stems from the well-defined boundaries of the solution space and the inherent simplicity of the method that necessitates only minimal hyperparameter tuning. figure 1 provides an overview of our method. our contributions primarily lie in three aspects: post-training quantization (ptq). ptq methods simplify the quantization process without the need for additional training. @xcite @xcite @xcite . given its low resource requirement, ptq is particularly suitable for the quantization of large language models. large language models quantization. significant strides have been made in addressing the pressing need for quantizing large language models (llms). gpt3.int8() @xcite introduces a mixed-precision approach to preserve crucial channels in high precision. aqlm @xcite builds upon additive quantization, a classic algorithm from the multi-codebook quantization family, adapting it to llm quantization. zeroquantv2 @xcite employs low-rank matrices to enhance model quality recovery. rptq @xcite addresses range differences between channels by rearranging and quantizing them in clusters. llm-qat @xcite ) employs qat to enhance performance. some other methods, such as spiq @xcite , smoothquant @xcite , and outlier suppression+ @xcite , utilize handcrafted equivalent transformations to mitigate quantization errors. these methods rely on the model architecture to fuse the equivalent transformation operations. lrq @xcite only needs to learn significantly fewer parameters while enabling the individual scaling of weights, thus boosting the generalization capability of quantized llms. ## methodology we begin with an overview of quantization before delving into the specifics of our approach. the following operations can be utilized to quantize and dequantize the weights w: in order to improve the efficacy of the rounding quantization operation, we build upon prior research @xcite ) by introducing a trainable parameter v to adjust the rounding values. these modifications enable a more adaptable quantization process. we utilize block-wise output reconstruction to train these parameters via optimizer, thus framing the optimization as follows. our method distinguishes itself primarily by leveraging signsgd, which optimizes parameters based on the sign of the gradients as follows: figure 1 provides an illustration of our approach. and the pseudocode 1 presents more details of signround. ## experiments this section presents a comprehensive evaluation of signround from multiple perspectives. we begin with a brief overview of the llm architectures and tasks included in our assessment. next, we provide a detailed comparison between our method and several existing approaches, emphasizing the unique advantages of signround. furthermore, we conduct ablation studies to reinforce the efficacy of our choices and investigate the sensitivity of hyperparameters. lastly, we evaluate the generation ability of our method across various recent models. the tuning cost comparisons are provided in appendix a. ## conclusions in this paper, we introduce signround, an efficient and concise approach for optimizing weight rounding in the quantization of large language models. signround employs signed gradient descent for tuning rounding value and weight clipping in 200 steps, completing the quantization of llama-v2-70b in approximately 2.5 hours. our extensive experiments show that signround outperforms other quantization methods across various models and weight bits in the majority of scenarios. additionally, signround shows promising generation capabilities in recent models and achieves enhanced performance through model-specific hyperparameter tuning. despite the advantages, we observed a noticeable gap in accuracy performance for ultra-low bit quantization, particularly with 2-bit quantization, compared to the original model. this challenge could potentially be addressed by exploring nonuniform quantization and mixed-precision quantization, which we leave for future work.