AFPQ : Asymmetric Floating Point Quantization for LLM s (2024)

Abstract: Large language models (LLMs) show great performance in various tasks, but face deployment challenges from limited memory capacity and bandwidth.Low-bit weight quantization can save memory and accelerate inference.Although floating-point (FP) formats show good performance in LLM quantization, they tend to perform poorly with small group sizes or sub-4 bits.We find the reason is that the absence of asymmetry in previous FP quantization makes it unsuitable for handling asymmetric value distribution of LLM weight tensors.In this work, we propose asymmetric FP quantization (AFPQ), which sets separate scales for positive and negative values.Our method leads to large accuracy improvements and can be easily plugged into other quantization methods, including GPTQ and AWQ, for better performance.Besides, no additional storage is needed compared with asymmetric integer (INT) quantization.The code is available at https://github.com/zhangsichengsjtu/AFPQ.

## introduction llms have significantly advanced language understanding, generation, and reasoning @xcite @xcite . however, the increasing size of llms poses great pressure on memory capacity and bandwidth during deployment. low-bit quantization is a widely used solution to decrease both memory capacity and bandwidth requirements. to effectively accommodate llms, new quantization methods have been proposed, such as gptq @xcite and awq @xcite . these methods quantize llms with low-bit int formats. recent studies suggest utilizing low-bit fp formats, such as fp4 and nf4 @xcite , @xcite , our asymmetric fp quantization reduces the wikitext-2 perplexity (the lower the better) in both 3-bit and 4bit fp quantization (nf, short for normalfloat, is an advanced type of fp formats). we use group-size '-1' to represent per-channel quantization. in place of int can lead to improved quantization accuracy of llms @xcite @xcite . this improvement is attributed to the non-uniform distribution of low-bit fp formats, which more effectively align with llm weights, characterized by mostly smaller values and a long tail of larger, significant ones. although generally superior, fp formats tend to be worse than int when quantization with small group sizes or sub-4 bits. we identify this is caused by the absence of asymmetry in fp quantization. given that most weight tensors naturally exhibit asymmetric distributions, it is not suitable to quantize them with standard low-bit fp values, which have a symmetric distribution. furthermore, we find the conventional methods used in asymmetric int quantization, such as scale and zero-point adjustments, do not perform well in the context of fp quantization. in this work, we propose asymmetric floating point quantization (afpq), a simple yet effective approach to fit the weight asymmetry in llms. unlike previous symmetric fp quantization, which uses a uniform scale for positive and negative values within a weight group, afpq sets seperate scales for positive and negative values. afpq ensures that the rescaled fp values can better match the original weight values, thereby enhancing quantization accuracy in llms. in figure 1 , our afpq with fp and np formats show better results in both 3-bit and 4-bit round-to-neare (rtn) quantization. moreover, afpq requires no additional storage compared with asymmetric int quantization. our contributions can be summarized as follows: ## background and motivation post training quantization (ptq) for llms. there are two ptq methods for llms: 1) quantizing both weights (w) and activations (a), for example, w8a8 quantization @xcite ; 2) w-only quantization, for example, w4a16 one @xcite . this article focuses on the w-only method. the naive w-only method is rtn. the advanced methods include gptq @xcite and awq @xcite . gptq uses second-order information to compensate for quantization error, while awq scales salient weights before quantization. both methods use int for quantization. ## asymmetric floating point quantization quantization methods. to make fp quantization applicable to the asymmetric distribution of llm weights, an intuitive approach is to apply the method with one scale and zero-point used in asymmetric int quantization to fp quantization. however, this approach would shift the dense number area of fp from zero to the left of zero, eliminating the advantages of using fp formats, shown in the purple block of figure 4 . to preserve the advantage of fp formats, we propose asymmetric fp quantization with two separate scales, one for positive numbers and another for negative numbers in each weight group. in algorithm 1, we show our fp quantization process of a weight group. each fp16 weight will be divided by scale_pos or scale_neg depending on its sign, then it can be rounded to the nearest fp4 values. in this way, the rescaled fp-asym values can better fit the distribution of original weights (fig- algorithm 1 quantization methods from fp16 weight groups to fp4-asym ones with two scales input :wgrp_16bit output :wgrp_4bit constant :range # the range of fp4 formats # get the positive scale and negative one for this weight group @xmath0, 'fp4') end return wgrp_4bit ure 4 green block). besides, no storage overhead is incurred compared with asymmetric int quantization (both need two parameters for one group). application to sota methods. as afpq operates on each individual sub-tensor or group, it can work as a plugin to other high-level quantization algorithms such gptq @xcite and awq @xcite . to demonstrate the applicability, we integrate afpq with sota methods gptq and awq for better quantization accuracy. ## conclusion in this study, we identify that the lack of asymmetry in previous fp quantization can lead to poor quantization for llm weight tensors with asymmetric distribution. to solve the problem, we propose asymmetric fp quantization which sets separate scales for positive and negative values. besides, our method can be easily plugged into other effective methods, including gptq and awq, for performance improvements. in the future, we plan to push towards other promising formats such as posit (gustafson and yonemoto, 2017) for better llm quantization results.