E asy Q uant: An Efficient Data-free Quantization Algorithm for LLM s (2023)

Abstract: Large language models (LLMs) have proven to be very superior to conventional methods in various tasks. However, their expensive computations and high memory requirements are prohibitive for deployment. Model quantization is an effective method for reducing this overhead. The problem is that in most previous works, the quantized model was calibrated using few samples from the training data, which might affect the generalization of the quantized LLMs to unknown cases and tasks. Hence in this work, we explore an important question: Can we design a data-independent quantization method for LLMs to guarantee its generalization performance? In this work, we propose EasyQuant, a training-free and data-independent weight-only quantization algorithm for LLMs. Our observation indicates that two factors: outliers in the weight and quantization ranges, are essential for reducing the quantization error. Therefore, in EasyQuant, we leave the outliers (less than 1%) unchanged and optimize the quantization range to reduce the reconstruction error. With these methods, we surprisingly find that EasyQuant achieves comparable performance to the original model. Since EasyQuant does not depend on any training data, the generalization performance of quantized LLMs is safely guaranteed. Moreover, EasyQuant can be implemented in parallel so that the quantized model could be attained in a few minutes even for LLMs over 100B. To our best knowledge, we are the first work that achieves almost lossless quantization performance for LLMs under a data-independent setting and our algorithm runs over 10 times faster than the data-dependent methods.

## introduction recent work has already proved the superior performance of transformer @xcite based llms @xcite @xcite @xcite @xcite on various tasks over traditional methods, and has attracted massive interest in how to improve and utilize those llms. however, the model size also grows dramatically along with improved performance. hence the memory footprint and computational cost become the bottleneck for deploying those models. one promising solution to alleviate this overhead is model quantization @xcite , where we quantize weight only or weight and activation both i order to reduce memory consumption and computational cost. although model quantization is a well-studied area for normal-sized models, such as bert @xcite and gpt-2 @xcite , it is still a quite challenging task for llms. one major reason is that previous lossless model quantization algorithms require retraining for the quantized model, which is too expensive for models over billions of parameters. beyond this, previous models are usually designed for specific domain tasks, which means the training data are sampled from limited task domains. however, recent llms are usually trained on various domains of data corpus, and they have shown to be quite effective for multi-domain zero-shot tasks. in this case, if we only retrain the quantized llms using partial domain corpus, the generalization ability of llms might get worse. therefore both efficiency and generalization guarantees are very important for designing llms quantization algorithms. to date, for low-bits weight-only quantization, several posttraining algorithms have been proposed @xcite . however, those methods also require a small calibration set sampled from training data, which still takes at least several hours. moreover, the use of those calibration data also brings the risk of making the model overfit to the calibration set. we first find all the outliers in weight and keep them in full precision (fp32/fp16/bf16). afterward, we optimize the quantization range (denoted as q range ) in order to approximate the normal values more precisely. in the end, the normal values are quantized into lower bits (denoted as q[•]) with optimized quantization ranges and we set the outliers unchanged in weight. ## background and motivation the most widely used quantization method, namely rounding to nearest-number (rtn), quantizes a tensor x into k-bits representation according to here s is the quantization scale, l min and l max are the lower and upper bound for clipping, and ⌊•⌉ is the rounding operator. usually we set l @xmath0. there are two major directions for finding the best configuration in weight-only llm quantization. the first is to minimize the reconstruction error of the weight parameter (denoted as w ), which is defined as r(w ) := ∥q[w ] -w ∥ 2 . notice that in this case we only need to have access to the weight itself, therefore it is data-free. beyond this, recent studies @xcite propose to use the output error, defined as data-dependent calibration might weaken the generalization ability of llms however, the performance gain from using calibration data might jeopardize the generalization of the quantized model, because it brings the risk of making the model overfit to the calibration set. for example, the quantization ranges will clip most of the outliers to be very small, hence the perplexity increases severely since those outliers are critical for preserving the model's performance. however, when keeping those outliers unquantized, the quantized model achieves a better performance as the reconstruction error decreases continuously. this result clearly suggests that the outliers are more important than the normal values in weight, and optimizing the quantization ranges using gradient defined in (2) can significantly increase the accuracy of quantized models. more details about the experiment can be found in section 5. data-free quantization is challenging, but very important although it's more challenging to use the reconstruction error as a regulation because it can only optimize the quantized model indirectly, still it is a very important direction for researching because the generalization ability of the model is inherently guaranteed when using data-free quantization since it uses no training data. therefore in this paper, we aim to answer the following question: how can we efficiently recover the performance of the quantized model without using any input data? in this work we propose easyquant, a data-free fast algorithm that could significantly improve the performance of quantized llms in a data-free setting, and more importantly, even outperforms the results from data-dependent quantization algorithms. our experiments reveal that the performance gap of the lower bits (e.g. 4-bits) quantized llms origins from two factors: in easyquant, we use quantization range minimization and outlier isolation to address these two challenges, and our results prove that easyquant achieves a significant improvement over rtn. ## insight behind easyquant as mentioned above, the weight's outliers and quantization ranges are essential to the quantized model's performance. below we present the supporting experiments in detail. ## experiment baselines: we compare easyquant with several baselines in the int4 quantization setting below: experiment setup. for all models, we set the outlier threshold @xmath1.5, 3] in order to ensure that the outliers account less than 1% of all numbers. for bloom and llama, we use @xmath2. when optimizing the quantization ranges, we use adam as the optimizer and set the learning rate 1e -3 for bloom and 1e -4 for llama. we choose the quantization ranges from step 100 for bloom and 500 for llama. we use symmetric quantization since the normal values are symmetrically distributed with the outliers being excluded. for a fair comparison, we use per-channel quantization for weight in all algorithms (which means each column shares one common quantization range). evaluation tasks. as for the evaluation tasks, we mainly focus on perplexity-based tasks, as they are known to be particularly sensitive to model quantization @xcite . the perplexity tasks we include are wikitext2 @xcite , penn treebank @xcite and c4 @xcite . the zero-shot tasks' results are also provided, such as piqa @xcite , arc @xcite and storycloze @xcite . implementation. since each weight can be quantized in parallel, therefore we use 8 * a100 for running easyquant, and we finish the quantization in 1 ∼ 10 mins for all models. we store the index and value for all outliers together with the quantized normal values. our dequantization kernel is built using cuda. ## related work model quantization traditional model quantization algorithms mainly focus on the cases where both parameters and activations of the model are quantized @xcite @xcite . however, directly quantizing the model will greatly decrease the accuracy of the models, and one important technique to improve the performance is quantization aware training (qat) @xcite , where it simulates the quantization procedure in training to improve the accuracy of the quantized model further. for transformer based models, the boundary of the compression level has been continuously advanced. for example, 8-bits quantized transformers as in fullyqt @xcite and q8bert @xcite , 4-bits quantized bert in @xcite and tenary case as in ternarybert @xcite . model quantization for llms. for quantizing llms, due to their prohibitive training expense, we can only use a few training data for calibration. there are two major directions: 1) weight-only quantization, where the weights are quantized into lower bits. in @xcite ; @xcite , authors optimize the output error on the calibration set using obs and gradient descent. 2) activation and weight quantization, where both activations and weights are quantized into lower bits. in this case, the major obstacle is the outliers in activations. llm.int8() @xcite addresses this problem by isolating those outliers in fp16/bf16. however, such implementation leads to large latency overhead and is even slower than fp16 inference. recent studies @xcite found that the outliers only exist in certain channels, and use the layernorm weights @xcite and calibrated scales @xcite to smooth those channels. @xcite has already proved that we can achieve almost lossless w8a8 quantized llms using a few calibration data, without manipulating the original model weights. ## conclusion and limitations in this paper, we propose a data-free fast weightonly quantization algorithm, namely easyquant, for llms, that potentially improves the quantized model's performance without using any training data. our analysis reveals the intrinsic origins of the performance loss when quantizing the model weights into lower bits. we show that by isolating the outliers from quantization, the accuracy of the quantized llm increases accordingly with decreased reconstruction error. our experiment proved that easyquant significantly outperforms rtn in a data-free setting, and also behaves better than data-dependent algorithms. easyquant can finish the quantization for a 176b-sized model within 10 minutes and the overhead of dequantization in easyquant is negligible. however, we also point out some limitations of our work: the outlier recovery functionality in easyquant requires extra cuda kernels for implementation. moreover, weight-only quantization can only reduce the memory footprint without any computation cost reduction, hence the latency of our model cannot be minimized. in addition, this outlier isolation will make the weight/activation quantization more challenging because the weight includes numbers under different precision. we have also noticed that easyquantcannot outperform the data-dependent methods in all tasks, this motivates us to investigate more effective algorithms in future studies.