Compensate Quantization Errors: Make Weights Hierarchical to Compensate Each Other (2024)

Abstract: Emergent Large Language Models (LLMs) use their extraordinary performance and powerful deduction capacity to discern from traditional language models. However, the expenses of computational resources and storage for these LLMs are stunning, quantization then arises as a trending conversation. To address accuracy decay caused by quantization, two streams of works in post-training quantization methods stand out. One uses other weights to compensate existing quantization error, while the other transfers the quantization difficulty to other parts in the model. Combining both merits, we introduce Learnable Singular value Increment (LSI) as an advanced solution. LSI uses Singular Value Decomposition to extract singular values of the weights and make them learnable to help weights compensate each other conditioned on activation. Incorporating LSI with existing techniques, we achieve state-of-the-art performance in diverse quantization settings, no matter in weight-only, weight-activation or extremely low bit scenarios. By unleashing the potential of LSI, efficient finetuning on quantized model is no longer a prohibitive problem.

## introduction large language models (llms) have garnered significant attention for their remarkable performance across a wide range of downstream tasks and their ability to exhibit emergent behavior @xcite . furthermore, their prowess in understanding natural language and deductive reasoning can be extended to multimodal domains through alignment training @xcite @xcite . nevertheless, the training and upkeep of such llms are highly resource-intensive, with many gpus being able to support only a single model or parts of one. quantization, as a central paradigm in this field, * corresponding author. quantization methods are typically divided into two categories based on the quantization period. quantization-aware training (qat) @xcite involves tuning the model during training to optimize its compatibility with quantization. although qat can yield superior results compared to post-training quantization (ptq), the significant computational costs associated with the training process are a notable challenge. consequently, ptq methods have gained widespread acceptance and become increasingly popular in recent times. within the ptq field, there are various methodologies to explore. for instance, the gptq series @xcite @xcite employ unquantized weights to gradually offset the quantization errors introduced by previously quantized weights. in this paper, for the sake of convenience in reading, all references to "weight" pertain to the weight matrices in the linear layers of the model. they achieve this by solving a lagrange equation to obtain a new hessian matrix to update. on the other hand, methods like smoothquant @xcite and omni-quant @xcite focus on altering the distribution of weights and activations to mitigate the challenges of quantization. meanwhile, through extensive research and experiments, it has been revealed that a significant portion of errors proposed during the quantization process are caused by a small number of outliers with distinctive weight values. hence, several studies @xcite @xcite concentrate on reducing or mitigating these outliers to minimize the disruption caused by their presence. after thorough research, we have summarized the reasons for the success of previous methods. for the gptq series, in most cases, the weights obtained through manual calculation are not dif-ferent from those obtained by the simple uniform quantization method, detailed in sec 2.2. however, in some later stages of the quantization phase, gptq can alter the inherent hierarchy of the original weights, approximating the weights to other quantization intervals, to achieve a globally optimized solution. for methods like smoothquant, the concept of transferring the difficulty of weight quantization is relatively easy to understand. for example, if the equation 1.3 × 15.4 ≈ 20 is going to be quantized through rounding, quantizing 1.3 to 1 would require changing 15.4 to 20 to maintain the original value. however, if the two multipliers in the equation are scaled to 4.3 × 4.7 ≈ 20 and then quantized, it would only be necessary to quantize it to 4 × 5 to keep the original value unchanged. in this way, the difficulty of quantization is greatly simplified. both types of methods have their own advantages, but they cannot both enjoy the benefits of the other. additionally, approaches like the gptq series have drawbacks such as long quantization times. in this paper, we demonstrate that a good quantization method performs: (1) transformation of the quantization difficulty of weights and activations; after conducting thorough experiments, we have achieved state-of-the-art results across a wide range of quantization settings, while marking a significant breakthrough in the field of quantized model fine-tuning. our contributions encompass the following key points: ## conclusion we introduce lsi to adjust model weights to conform to quantization settings. through integration with established techniques, our approach attains state-of-the-art performance across diverse quantization settings. lsi imparts a hierarchical structure to model weights, enhancing adaptability to quantization parameters without compromising training efficiency. leveraging attributes of lsi allows effective finetuning of a quantized model on various datasets. notably, during inference, lsi introduces no additional parameters and preserves the hardware efficiency inherited from omniquant.