ATQ : Activation Transformation for W eight-Activation Quantization of Large Language Models (2024)

Abstract: There are many emerging quantization methods to resolve the problem that the huge demand on computational and storage costs hinders the deployment of Large language models (LLMs). However, their accuracy performance still can not satisfy the entire academic and industry community. In this work, we propose ATQ, an INT8 weight-activation quantization of LLMs, that can achieve almost lossless accuracy. We employ a mathematically equivalent transformation and a triangle inequality to constrain weight-activation quantization error to the sum of a weight quantization error and an activation quantization error. For the weight part, transformed weights are quantized along the |in-feature| dimension and the quantization error is compensated by optimizing following in-features. For the activation part, transformed activations are in the normal range and can be quantized easily. We provide comparison experiments to demonstrate that our ATQ method can achieve almost lossless in accuracy on OPT and LLaMA families in W8A8 quantization settings. The increase of perplexity is within 1 and the accuracy degradation is within 0.5 percent even in the worst case.

## introduction large language models (llms) exhibit remarkable performance across various tasks. many different llms were proposed, such as gpt @xcite , opt @xcite , llama @xcite , bloom @xcite and so on. however, the large model size and the huge computation cost prevents their deployment in production. quantization is considered as a promising technique for model compression and inference acceleration, and it can be categorized into two main approaches: quantization-aware training(qat) and post-training quantization(ptq). qat @xcite can achieve comparable performance with the original model. however, qat is not practical due to the huge training cost and the unavailability of training data. researchers prefer to use ptq to quantize llms at the cost of some accuracy degradation. in recent few years, a great number of ptq methods @xcite @xcite @xcite for llms spring up. they are categorized into two classes: weight-only quantization and weightactivation quantization. actually, the inference process of llms composes of two stages: prefilling and decoding. during the prefilling stage, the huge cost is caused by high-precision matrix-matrix multiplication, which can be alleviated by weight-activation quantization. during the decoding stage, it generates only one token using general matrix-vector multiplication. the decoding stage is memory-bound, which means the decoding latency is constrained by the movement of weights between memories. therefore, weight-only quantization methods can reduce the weight movement cost. the number of bits in quantization of llms tends to be lower and lower. for examples, llm.int8() @xcite quantizes the non-outlier columns with 8-bit, and smoothquant @xcite quantizes weight and activation to int8 datatype. gptq quantizes weight to 3 or 4 bits, and omniquant can quantize weight down to 2 bits or quantize weight and activation to 4 bits. quik @xcite and atom @xcite adopt hybrid-precision quantization, where most activations and weights are quantied into int4 keeping only a small part of activations and weights in high precision. llm-fp4 @xcite quantizes both weights and activations down to fp4. decoupleq @xcite achieves 2-bit uniform ptq, and onebit @xcite introduces a 1-bit quantization framework. although these methods can achieve an efficient post-training quantization solution for llms, their accuracy degradation can not satisfy the practical applications. in this paper, we propose atq, a novel weightactivation post-training quantization pipeline. the main contribution can be summarized as follows. ## related work and motivation in this section, we review some related works and present our research motivation. ## atq method traditional ptq methods with gradient optimization is hard to be applied into modern llms due to the huge solution space. in our atq method, we consider the block-wise quantization error minimization problem for each linear layer. the weightactivation quantization problem can be formulated as follows, the first term on the right side of eq.( 2 ), ∥xw -xq w (w )∥, is a weight-only quantization problem and can be resolved by gptq. the second term represents an activation quantization problem for given quantized weights. this error is very large due to activation outliers observed in smoothquant and outlier suppression+. to reduce the activation quantization error, we perform a mathematically equivalent transformation on the linear layer, which can be written as the first term on the right side is still a weightonly quantization problem and can be resolved by the gptq method. the only difference is that the activation used for calculating the hessian matrix h is the transformed x rather than the original x. in the second term, for the given weight q w ( w ), if we transformed activation outliers to be in the normal range of [-1, 1], the quantization error of transformed activations x should be small enough. the problem becomes how to find appropriate shifting and scaling factors, δ and s, to elimate activation outliers. in llms, activation outliers are asymmetric and have large magnitude on certain channels. we first find the center of activations perchannel on a small calibration dataset, which is the shifting factor δ, and move it to 0. mathematically, we have now, the range of outliers in the j th channel is (max(x :,j )min(x :,j ))/2. to minimize activation quantization error, we scale all activations into the range of [-1, 1] unless they are already in this range, which means s @xmath0.0, max(x :,j )min(x :,j ) 2 ). (6) note that, we do not incorporate weight into shifting and scaling factors, which means we do not need to consider the tradeoff between activation and weight. all of quantization difficulties are migrated from activation to weight, and weight quantization errors can be compensated by optimizing following weight in-features, which is exactly our essential difference from other weight-activation quantization methods. ## experiments in the experiments, we compare our atq method with fp16 baselines, gptq @xcite , smoothquant @xcite , outlier suppres-sion+ @xcite and omniquant @xcite . ## conclusion and future work we propose an advanced and accurate post-training quantization scheme called atq. in the scenario of w8a8 quantization, atq exceeds other existing weight-activation quantization methods on accuracy degradation in most cases. considering the limitation discussed in the next section, we plan to incorporate sparsification into this scheme for higher compression ratio, and implement the cuda kernel and operator for computational efficiency and memory saving.