A Frustratingly Easy Post-Training Quantization Scheme for LLM s (2023)

Abstract: Efficient inference has become crucial for hyper-scale AI models, including large language models, as their parameter count continues to increase for enhanced performance. This necessity holds true regardless of the computing environment, whether it be mobile devices or cloud servers. Quantization emerges as a solution to alleviate the computational burden during inference. By representing models with a reduced bit-width, quantization minimizes the frequency of DRAM access while fully exploiting the parallelism of operations through a dense matrix format. Consequently, quantized models achieve low end-to-end latency and optimize resource utilization by addressing both memory and computing bottlenecks. In this paper, we propose a straightforward post-training quantization scheme, called Z-Fold, that fully utilizes the feature of the Transformer structure widely employed in large language models.

## introduction the transformer @xcite models have revolutionized machine learning across various domains by leveraging attention mechanisms @xcite . neural language processing (nlp) tasks, including large language models (llms) and neural machine translation (nmt), heavily rely on the transformers, extending their impact to vision tasks as well @xcite . in essence, hyperscale ai models heavily depend on the transformer architecture. to enhance the performance of general language tasks, large language models progressively increase the number of parameters, which are achieved with various techniques such as preventing performance saturation or enabling hyperscale model to be trained; layer dropping @xcite figure 1 : the transformer architecture (pre-layernorm). we fully exploit the feature of the architecture to prevent accuracy drop from lowered bit-precision of weights. and he, 2020), pipeline parallelism @xcite , model parallelism @xcite , pre-normalization @xcite , and adamw @xcite are employed and resulting in models with up to 100 trillion parameters. nonetheless, these hyperscale models encounter limitations that hinder their applicability. among the key challenges are the memory and computational costs associated with processing an extensive number of parameters; these lead to increased end-to-end latency and power consumption, regardless of the available computing resources. specifically, a considerable number of parameters need to be transferred between memory and processors, and computed with long input sequences. thus, hyperscale models impose a burden on hardware during inference, despite the performance benefits of parameter scaling. consequently, efficient inference techniques have gained significant attention as much as for scaling out the models, whether in resource-constrained (e.g., edge devices) or resource-abundant (e.g., cloud server) environments. quantization emerges as a promising and attractive solution among various techniques aimed at improving model efficiency (e.g., pruning @xcite , low-rank approximation @xcite and knowledge distillation @xcite ). by representing models with a low bit-width, quantization reduces the frequency of dram access, while fully leveraging the parallelism of operations through a dense matrix format. as a result, quantized models alleviate the bottleneck caused by both memory and computational overheads. in this paper, we propose a post-training quantization method for llms, called z-fold, which exploits the feature of the transformer architecture (pre-layernorm) to enhance the quantized models without introducing additional quantization parameters or computational costs. in a nutshell, we quantize weights into low bit-width (down to 2-bit) using more parameters (ζ) than existing approaches. this contributes to improving quantized networks by further minimizing the loss perturbation as a result of quantization. however, we fold or fuse these additional parameters (ζ) into other existing parameters (α or γ; see figure 1 ) in advance of inference, to avoid imposing further hardware costs. as a result, our approach achieves state-of-the-art performance among quantized llms without any additional parameters or hardware costs. ## experiments 4.1 experimental setup to evaluate the performance of the proposed method, we quantize publicly available llms (e.g., opt @xcite , llama @xcite ) using z-fold. as a calibration dataset to estimate the hessian, we use 128 random 2048 token segments from the c4 dataset @xcite as in optq @xcite . we conduct all experiments using a single nvidia a100 gpu (80 gb) and implement z-fold using pytorch. in our experiments, we quantize only weights and retain activations as full-precision (fp16) since activations do not pose a significant bottleneck during the inference of llms @xcite . we break the alternating updates of α and ζ in z-fold (see lines 4-7 in algorithm 1) when the loss perturbation ∆l (see equation 1) increases after the update or the total number of updates exceeds 30 iterations. after the quantization, we evaluate the performance of the quantized llms using three bench-mark datasets (wikitext-2 @xcite , c4 @xcite , and ptb @xcite ) and several zero-shot tasks. ## conclusion in this paper, we have proposed a post-training quantization scheme called z-fold. by capturing the unique features of the transformer and fully leveraging its architecture (pre-layernorm and successivity of linear layers), we developed a simple yet powerful approach that enhances the perfor-mance of quantized models. our scheme stands out by quantizing models using a greater number of parameters than existing approaches, resulting in enhanced sophistication. importantly, these additional parameters are intelligently fused into existing ones, eliminating the need for extra hardware costs. our findings demonstrate that the z-fold scheme achieves state-of-the-art performance in quantized llms.