Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization (2023)

Abstract: Large Language Models (LLMs) are proficient in natural language processing tasks, but their deployment is often restricted by extensive parameter sizes and computational demands. This paper focuses on post-training quantization (PTQ) in LLMs, specifically 4-bit weight and 8-bit activation (W4A8) quantization, to enhance computational efficiency—a topic less explored compared to weight-only quantization. We present two innovative techniques: activation-quantization-aware scaling (AQAS) and sequence-length-aware calibration (SLAC) to enhance PTQ by considering the combined effects on weights and activations and aligning calibration sequence lengths to target tasks. Moreover, we introduce dINT, a hybrid data format combining integer and denormal representations, to address the underflow issue in W4A8 quantization, where small values are rounded to zero. Through rigorous evaluations of LLMs, including OPT and LLaMA, we demonstrate that our techniques significantly boost task accuracies to levels comparable with full-precision models. By developing arithmetic units compatible with dINT, we further confirm that our methods yield a 2 \times hardware efficiency improvement compared to 8-bit integer MAC unit.

## introduction large language models (llms) have achieved breakthroughs in many natural language processing tasks such as translation, summarization, reasoning, and conversation, often matching or exceeding human performance @xcite @xcite @xcite . however, the extensive parameters of llms present deployment challenges due to the high memory bandwidth needed for high throughput inference. post-training quantization (ptq) addresses this by "compressing" weight pa- * equal contribution † corresponding author rameters, significantly reducing memory requirements and enhancing gpu performance by alleviating memory bandwidth bottlenecks @xcite @xcite . nevertheless, llms' computational complexity remains a concern. for example, gpt-3 @xcite requires at least 350 gflops of computation for a single token, but ptq methods often revert compressed weights to higher precisions like 16-bit floating-point (fp16) for computation, which is inefficient given the resource demands of multiply-accumulate (mac) operations. with computing platforms evolving through high-bandwidth memory @xcite and processingin-memory @xcite to resolve the memory bandwidth bottleneck, addressing llms' computational needs becomes more imperative. a ptq strategy that effectively quantizes both weights and activations is thus appealing as it reduces the hardware complexity of mac units, enhancing computational throughput @xcite @xcite . ptq research specific to llm's computation efficiency is growing, focusing on utilizing int8-int8 mac units, common in gpus @xcite . llm.int8 @xcite , for instance, used int8 quantization for weights and activations, but directed activation outliers through an fp16 datapath, isolating them. smoothquant @xcite extended this by employing activation channel scaling to target outliers and adjusting corresponding weights for balanced quantization. however, these studies do not address challenges faced when weights are reduced to 4 bits, revealing an unexplored area for combined effects on weight and activation quantization. this paper delves into the challenges of posttraining quantization (ptq) for both weights and activations in large language models (llms). we pinpoint two primary hurdles in achieving efficient 4-bit weight and 8-bit activation (w4a8) quantization. first, llms like opt @xcite and llama @xcite have distinct weight and activation range characteristics, making existing ptq methods unsuitable for universal use. for example, awq's @xcite activation-aware scaling makes activations prone to quantization errors, while optq's @xcite weight calibration struggles with varying activation ranges. we propose two novel solutions for this first hurdle: activation-quantization-aware scaling (aqas) and sequence-length-aware calibration (slac). aqas optimizes quantization scales by jointly considering weights and activations, yielding balanced quantization. slac aligns the sequence length of the application task with that of the ptq calibration dataset, mitigating the impact of variations in activation diversity, which significantly affects the ptq calibration process. second, we observe that underflow, where smallmagnitude values round to zero, severely impacts w4a8 quantization in llms because the quantization error associated with values rounding to zero constitutes a significant portion of the output error. while underflow is a well-known issue in reduced-precision formats for deep neural networks (dnns) @xcite @xcite , previous ptq research in llms mainly focuses on outliers, neglecting underflow. we discover that standard int4 representation discards crucial small-magnitude weights when multiplied with activations. as existing data formats like integer, floating-point, or logarithmic formats are inadequate for this underflow issue, we introduce dint, a new integer format with denormal representation. dint merges the uniform coverage of integers with the denormal of floatingpoints, effectively mitigating underflow and improving accuracy. we also propose a mac unit supporting dint to ensure hardware efficiency. we evaluate aqas, slac, and dint on opt and llama, focusing on language modeling, zeroshot reasoning, and 5-shot in-context learning. the results show that integrating these methods for w4a8 ptq significantly improves task accuracies for both opt and llama across a diverse set of benchmarks (wikitext, common sense question answering (csqa), and massive multitask language understanding (mmlu)) and the model sizes ranging from 125m to 65b parameters. ## improving ptq for weight and activation quantization we aim to advance llm quantization beyond the realms of 4-bit weight-only ptq or w8a8 ptq by investigating the combined effects of weight and activation quantization. when quantizing both weight and activation, it is important to note that llms display distinct weight and activation characteristics. for example, opt has been found to have 0.1% activation outliers by @xcite , whereas glm-130b @xcite reported 30% of outliers in its model. in the context of weight, due to varied weight distributions across models, opt-66b experiences a substantial perplexity increase in the wikitext benchmark with int4 weights, soaring from 9.34 to 110 @xcite , whereas glm-130b shows no performance degradation on the mmlu benchmark when int4 weights are applied @xcite . we posit that these discrepancies arise from variances in pre-training configurations such as datasets, learning rates, layer structures, and self-attention directionality, as well as options designed for efficient inference, such as operation fusion techniques like layernorm fusion. significantly, existing ptq research has overlooked these unique traits intrinsic to each model that are pivotal for the combined optimization of activation and weight quantization. therefore, we delve into the weight and activation distributions of widely-used opt and llama models during quantization to understand ptq limitations and develop novel methods to address them. ## overcoming ptq underflow for llms by employing aqas to address activation quantization errors in weight scaling, and utilizing slac to align the sequence length of the calibration dataset with that of the target inference, we achieve a substantial improvement in the performance of our w4a8 models. however, we encounter persistent performance degradation issues. in this section, we unveil "underflow" issues as a previously overlooked cause of accuracy degradation in ptq applied to llms and propose a new numerical format to mitigate this problem. ## conclusion we address post-training quantization (ptq) in large language models (llms), specifically targeting 4-bit weight and 8-bit activation (w4a8) quantization to boost computational efficiency. we present activation-quantization-aware scaling (aqas) and sequence-length-aware calibration (slac), refining ptq by taking into account weights and activations, and aligning sequence lengths. to combat the underflow issue in w4a8 quantization, where small magnitudes are rounded down to zero, we introduce dint, a hybrid format blending integer and denormal representations. through extensive evaluations on llms such as opt and llama, we demonstrate marked improvements in task accuracy and adaptability. additionally, with the development of mac units compatible with dint, we achieve a twofold increase in hardware efficiency.