LLM - FP 4: 4-Bit Floating-Point Quantized Transformers (2023)

Abstract: We propose LLM-FP4 for quantizing both weights and activations in large language models (LLMs) down to 4-bit floating-point values, in a post-training manner. Existing post-training quantization (PTQ) solutions are primarily integer-based and struggle with bit widths below 8 bits. Compared to integer quantization, floating-point (FP) quantization is more flexible and can better handle long-tail or bell-shaped distributions, and it has emerged as a default choice in many hardware platforms. One characteristic of FP quantization is that its performance largely depends on the choice of exponent bits and clipping range. In this regard, we construct a strong FP-PTQ baseline by searching for the optimal quantization parameters. Furthermore, we observe a high inter-channel variance and low intra-channel variance pattern in activation distributions, which adds activation quantization difficulty. We recognize this pattern to be consistent across a spectrum of transformer models designed for diverse tasks such as LLMs, BERT, and Vision Transformer models. To tackle this, we propose per-channel activation quantization and show that these additional scaling factors can be reparameterized as exponential biases of weights, incurring a negligible cost. Our method, for the first time, can quantize both weights and activations in the LLaMA-13B to only 4-bit and achieves an average score of 63.1 on the common sense zero-shot reasoning tasks, which is only 5.8 lower than the full-precision model, significantly outperforming the previous state-of-the-art by 12.7 points. Code is available at: https://github.com/nbasyl/LLM-FP4.

## introduction since the introduction of transformer architecture @xcite , transformers have superseded recursive neural networks, emerging as the dominant architecture in numerous natural language processing (nlp) tasks (kenton and * these authors contributed equally to this work toutanova, 2019; @xcite . the transformative impact of the transformer has been further propelled by the emergence of models like gpt @xcite , catapulting the popularity of this architecture to new heights. meanwhile, the versatility of transformers extends beyond nlp, encompassing diverse domains such as vision @xcite , audio @xcite , etc. this trend towards a unified architecture for different modalities represents a groundbreaking development within the realm of deep learning. however, the advancements in transformer performance are accompanied by a corresponding increase in model size and computational costs @xcite . this poses significant challenges when attempting to leverage the full potential of transformer models in use cases where memory or computational resources are limited. despite the extensive research and widespread adoption of transformers, the field of transformer compression remains relatively underexplored. to address this gap, our study focuses on the compression of transformers, especially through floating-point post-training quantization techniques. post-training quantization (ptq) offers the advantages of simple to use with minimal fine-tuning requirements @xcite . existing ptq solutions for transformers primarily focus on integer (int) quantization @xcite , which can be effective in certain scenarios but often break down when bit widths are below 8 bit. on the other hand, floatingpoint (fp) quantization has gained significant traction as a more flexible alternative, capable of better accommodating various activation and weight distributions. in fact, fp8 has emerged as the default choice in various hardware platforms, including the nvidia h100. different from integer (int) quantization, a particular challenge in floating-point (fp) quantiza-tion is how to select appropriate exponent bits and scale parameters. improper parameter choices can lead to subpar or divergent quantization results. to tackle this challenge, we introduce a robust recipe for fp quantization, which leverage layer-wise reconstruction to jointly search for optimal exponent bits and maximum values. compared to previous approaches that utilize gradient updates for exponent bits @xcite , our search-based method proves to be more stable and consistently delivers desirable quantization results, which establishes a strong baseline for fp-ptq. furthermore, our investigation uncovers an intriguing pattern of activation distributions in transformers, characterized by high inter-channel variance and low intra-channel variance. similar patterns are also observed in previous works @xcite , while we argue that this pattern is inherent to transformer architectures and not limited to specific tasks, as we have observed consistent patterns not only in large language models but also in bert model and even vision transformers. motivated by these findings, we introduce a novel pre-shifted exponent bias for fp quantization of transformers. concretely, we leverage the per-channel activation variance computed from calibration data and reparameterize these scales as the exponential bias of the corresponding fp quantized weight vectors. this approach effectively addresses the challenge posed by high inter-channel variance while incurring negligible computational cost. in summary, we study floating-point posttraining quantization (ptq) for transformer architectures, and the contribution of this paper includes: ## method in this section, we begin by introducing our joint format and max value search, which establishes our strong baseline and already achieves state-ofthe-art results at 8-bit and 6-bit quantization. then we present an efficient pre-shifted exponent bias to tackle the catastrophic high inter-channel activation variance in transformer models and push the quantization limit to 4-bit. ## experiments to validate the effectiveness of the proposed method, we conduct experiments on llama @xcite and bert @xcite ) models in 5.2.1 and sections 5.2.2. further, in section 5.2.3 we show that our method also generalizes well to vision transformer architectures. we present ablation studies on the calibration size and search range in section 5.3, and analyze the hardware costs of implementing fp operators in section 5.4. ## conclusion this paper presents the first successful demonstration of 4-bit floating-point post-training quantization for weights, activations, and embeddings in natural language transformer architectures, including both large language models and bert model. we also extend our method to vision transformers and observe its robust generalization ability. our approach involves a practical search-based technique which establishes a strong baseline and achieves state-of-the-art results for 6-bit and 8-bit quantization. furthermore, we address the challenge of high inter-channel variance in transformers by proposing pre-shifted exponent bias, which proves highly effective in achieving accurate 4-bit quantization.