Outlier Suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling (2023)

Abstract: Post-training quantization (PTQ) of transformer language models faces significant challenges due to the existence of detrimental outliers in activations. We observe that these outliers are concentrated in specific channels and are asymmetric across channels. To address this issue, we propose the Outlier Suppression+ (OS+) framework, which contains the channel-wise shifting for asymmetry and channel-wise scaling for concentration. We show that these operations can be seamlessly migrated into subsequent modules while maintaining equivalence. Second, we propose a fast and stable scheme to calculate effective shifting and scaling values. The channel-wise shifting aligns the center of each channel for removal of outlier asymmetry. The channel-wise scaling quantitatively evaluates changes brought by migration and quantization for better quantization burden balance. We validate our OS+ under both standard and fine-grained quantization settings with models including BERT, OPT, BLOOM, BLOOMZ, and LLaMA. Comprehensive results across various tasks demonstrate the superiority of our approach. Especially, with standard quantization, OS+ can achieve near-floating-point performance on both small models and large language models on 8-bit and 6-bit. Besides, we establish a new state-of-the-art for 4-bit BERT with 15.5% improvement. Our code is available at https://github.com/ModelTC/Outlier_Suppression_Plus .

## introduction transformer language models (e.g., bert, llms) have garnered significant attention due to their remarkable performance and scalable model size. these models have evolved from hundreds of millions of parameters @xcite @xcite to hundreds of billions of parameters @xcite @xcite . this necessitates the employment of compression techniques @xcite @xcite for practical deployment. among these techniques, quantization @xcite has emerged as a general and primary paradigm for reducing both memory footprint and computation overhead. however, quantization, particularly post-training quantization @xcite @xcite under the setting of limited data and gpu resources, has become increasingly challenging on these models (e.g., a 12% accuracy drop in bert @xcite and catastrophic degradation in opt-175b @xcite ). this is caused by the presence of detrimental outliers in activation (e.g., the range of distribution can be 80 in bert and even 140 in opts), which prevents discrete numbers from accurately representing continuous ones. to combat the bottleneck, researchers make indepth investigations and find that outliers mainly concentrate on certain channels. some works @xcite suggest fine-grained quantization schemes and offer extra bit levels for outlier channels. others @xcite take the activation scaling to scale outliers and migrate scaling values to subsequent weights for fp equivalence. however, the former might hurt the quantization acceleration effect while the latter determines scaling values without the consideration of minimizing the change introduced by migration and quantization, which we find is sub-optimal. meanwhile, we also identify a new outlier characteristic that previous works overlooked but is also responsible for the large tensor range. in this paper, we propose the outlier suppres-sion+ framework composed of channel-wise shift-figure 1 : distribution of opt-66b. fig. 1a shows the original distribution with asymmetric outliers consistently occurs at certain channels, owning considerable range @xcite . fig. 1b depicts the channel-wise shifting operation to decrease the tensor range by eliminating the asymmetry. fig. 1c further scales down the outliers to threshold 5 and finally results in a distribution ranging from -5 to 5. our algorithm can be carried out efficiently and enjoy affordability on real hardware, producing more quantization-friendly models in minutes and requiring no extra inference burden on llms. to this end, our main contributions can be summarized into three aspects: ## related work due to the space limit, we give the most relevant papers here and put a complete related work in the appendix a. in the realm of ptq, researchers have discovered that the poor performance of transformer language models should be attributed to extreme outliers in activations, which exhibit special characteristics from both channel and token aspects. thus, we will introduce related works from the two aspects. channel aspect. outliers consistently emerge in certain channels over different inputs. bondarenko et al. (2021) employs a per-embeddinggroup quantization scheme that uses different quantization parameters for distinct channel groups, while dettmers et al. (2022) suggests utilizing fp16 representations for problematic channels holding signals over 6. wei et al. (2022b) introduces an outlier suppression (os) framework with one of components called gamma migration. observing that outliers accumulate in certain channels, it adopts a scaling vector to scale outliers and migrates it to subsequent modules. xiao et al. (2022) further proposes calculating scaling values by equalizing ranges between activations and weights and evaluates on large language models. guo et al. (2023) discards normal values adjacent to outliers, making room for outliers with customized gpu support. to consider the standard quantization, we find that wei et al. (2022b) and xiao et al. (2022) still waste a large portion of quantization levels on the extreme outlier asymmetry across channels. meanwhile, wei et al. (2022b) simply views the scaling parameter in layernorm (ln) as the scaling vector for outliers, which might not always be consistent with the outlier distribution. xiao et al. (2022) that adopts the heuristic way and obtains equalized ranges between activation and weights lacks quantitative evaluation of their output change induced by migration and quantization. token aspect. different tokens exhibit varying degrees of outliers. dettmers et al. (2022); yao et al. (2022) introduce a novel scheme called per-token quantization that dynamically computes quantization parameters for each token. wei et al. (2022b) investigates the clipping impact of outliers and recommends finding an appropriate clipping range in a token-wise manner. in this paper, we focus on the channel aspect and might combine these techniques when necessary. ## method we first present our equivalent shifting and scaling operations, then introduce ways to determine effective values for them. ## conclusion in llms and other transformers. our framework is simple to use, consisting of both scaling and shifting operations, which can be efficiently and effectively implemented. experiments demonstrate the efficacy of our methods for suppressing outliers.