Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models
Abstract
Transformer architecture has become the fundamental element of the widespread natural language processing (NLP) models.
With the trends of large NLP models, the increasing memory and computation costs hinder their efficient deployment on resource-limited devices.
Therefore, transformer quantization attracts wide research interest.
Recent work recognizes that structured outliers are the critical bottleneck for quantization performance.
However, their proposed methods increase the computation overhead and still leave the outliers there.
To fundamentally address this problem, this paper delves into the inherent inducement and importance of the outliers.
We discover that γ in LayerNorm (LN) acts as a sinful amplifier for the outliers, and the importance of outliers varies greatly where some outliers provided by a few tokens cover a large area but can be clipped sharply without negative impacts.
Motivated by these findings, we propose an outlier suppression framework including two components: Gamma Migration and Token-Wise Clipping.
The Gamma Migration migrates the outlier amplifier to subsequent modules in an equivalent transformation, contributing to a more quantization-friendly model without any extra burden.
The Token-Wise Clipping takes advantage of the large variance of token range and designs a token-wise coarse-to-fine pipeline, obtaining a clipping range with minimal final quantization loss in an efficient way.
This framework effectively suppresses the outliers and can be used in a plug-and-play mode.
Extensive experiments prove that our framework surpasses the existing works and, for the first time, pushes the 6-bit posttraining BERT quantization to the full-precision (FP) level.
Our code is available at https://github.com/wimh966/outlier_suppression .
Introduction
Transformer [1] has been one of the most common architectures in natural language processing along with lots of popular self-supervised models, such as BERT [2], RoBERTa [3], XLNet [4] and BART [5].
While these pre-trained models have demonstrated a significant superiority in performance, the memory and computation overheads have been a popular concern, particularly in the real development.
Therefore, model compression [6, 7, 8, 9] has attracted much attention from both academia and industry.
Among them, quantization [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20], working in the low-precision arithmetic fashion, is one of the key approaches to compress large models and fit them into the lightweight devices.
These days, researchers focus more on quantization of Transformer-based models.
[21] proposes an 8-bit quantization scheme for BERT-like models.
[22] advises a group-wise quantization technique and analyzes mixed-precision using second-order Hessian information.
[23, 24] combine distillation with quantization.
[25] approximates nonlinear operations to implement integer-only quantization.
Nonetheless, few studies investigate the inherent bottleneck of quantizing Transformer-based models.
Recently, some papers [26, 27] indicate that the Transformer-based models hold significantly large outliers (even close to 100) and these extreme outliers behave in structured patterns (mainly gather at a few embedding dimensions and even become larger on unique tokens).
These special outliers can bring devastating damage to the quantization performance (e.g., a 12% drop even for the 8-bit [26]).
To combat this challenge, existing method [26] chooses bypassing solutions such as a finer quantization granularity.
However, this scheme causes an increased computation cost and unavoidably hinders the acceleration effect.
In this paper, to suppress the outliers rather than walk around them, we make an in-depth analysis to investigate the inducement of the outliers and the impact of clipping the outliers.
For the inducement, we find that the scaling parameter γ in the LayerNorm structure works as an outlier amplifier, which amplifies the outliers in the output.
By extracting it, the activation becomes more robust for quantization.
Then by further studying the clipping impact, we discover that the influence of final performance when clipping the outliers varies greatly, where some more aggressive outliers covering a large area can be clipped safely without accuracy degradation, but the accuracy can drop suddenly when the important outliers are clipped.
More interestingly, though those less important outliers might present in a long tail form, they are only provided by a few tokens.
Motivated by the analysis, we propose an outlier suppression framework to push the limit of low-bit
Transformer language models.
Such framework contains two key components: Gamma Migration and Token-Wise Clipping, corresponding to the above two findings.
The Gamma Migration produces a more quantization-friendly model by migrating the outlier amplifier γ into subsequent modules in an equivalent transformation and bringing more robust activation for quantization without extra computation burden.
The Token-Wise Clipping further efficiently finds a suitable clipping range with minimal final quantization loss in a coarse-to-fine procedure.
The coarse-grained stage, which leverages the fact that those less important outliers only belong to a few tokens, can obtain a preliminary clipping range quickly in a token-wise manner.
The fine-grained stage then optimizes it.
Our proposed framework can be applied to different models and tasks, and coupled with existing methods.
More essentially, the thought of outlier suppression shall shed new light on the study of
NLP quantization.
To summarize, our contributions are as follows:
We delve into the inducement and clipping impact of outliers in the NLP models and draw two critical findings that help handle the bottleneck of transformer quantization.
Based on the findings, an outlier suppression framework containing Gamma Migration and TokenWise Clipping is proposed.
This framework is efficient, easy to implement, and plug-and-play.
The Gamma Migration suppresses the outliers from the inducement aspect and produces a more quantization-friendly model without any extra inference time.
It transfers the outlier amplifier in LayerNorm to the subsequent modules in an equivalent transformation and contributes to activation with less quantization error.
The Token-Wise Clipping scheme suppresses the outliers from the aspect of importance and produces a superior clipping range efficiently.
It can skip over those unimportant outliers quickly leveraging the large variance of token range and then focus on the influential area.
Extensive experiments on various NLP models (BERT, RoBERTa, BART) and tasks (text classification, question answering
and summarization) prove that our outlier suppression framework sets up a new state of the art for transformer quantization, and for the first time, pushes the 6-bit post-training quantization (PTQ) and 4-bit quantization-aware training (QAT) accuracy of BERT to the full-precision level.
Preliminaries Basic Notations
We mark matrices as X and vectors as x .
Operator · denotes scalar multiplication, and ⊙ is adopted for element-wise multiplication on matrices or vectors.
Also, we use W x as
[Figure] [Figure]
γ 0(
[Figure]
) γ -  -  -  - 0 -  (c) X ′ matrix-vector multiplication.
Specifically, considering the tokens in NLP tasks, X t,j stands for the element at token t and embedding j , and x t represents the embedding of token t .
Quantizer.
Quantization usually includes two operations.
[Equation]
where s (step size), z (zero point) are quantization parameters, b is the bit setting.
The first operation called "Quant" maps continuous numbers ( x ) to discrete points ( ¯ x ) for integer-arithmetic-only matrix computation.
The second operation called "DeQuant" recovers it to ˆ x after multiplication.
Outlier analysis
For Transformer-based models, standard 6/8-bit PTQ or 4-bit QAT would cause severe accuracy degradation.
Investigating each quantizer, we recognize that the output of LayerNorm structures and
GELU functions hold some sharp outliers, which should be responsible for the large quantization error.
Evidence and experimental results in Sec.
B.2.
To deeply investigate the relationship between the harmful outliers and quantization performance, we explore the underlying inducement and impact of clipping the outliers.
Before that, some brief descriptions (see Sec.
C.1 for detailed ones) about the outliers are given first to help understand the following two parts.
The outliers show structured characteristics that they mainly gather at some certain embedding dimensions, and upon these dimensions, the outliers provided by unique tokens like the separate toke and comma even hold more aggressive values.
3.1 Inducement of outliers
For the inducement of outliers, we find that the scaling parameter in LayerNorm amplifies the outliers from embedding dimensions.
And the phenomenon that some tokens have sharper outliers might be caused by the uneven token frequency in the pre-training phase (see Sec. C.2).
In this part, we mainly explain the first inducement to solve these outliers from the origin.
For another one, due to the high cost of adjusting the pre-training, we discuss the clipping impact in the next part to suppress these outliers from the clipping perspective.
Considering the challenges of quantizing the LayerNorm, the natural action is to dive into its internal structure.
For token t at j th embedding dimension, it first normalizes the input using mean ( u t ) and variance ( σ 2 t ) each forward pass, then scales and shifts the value with parameter γ j and β j .
LayerNorm
[Equation]
Then, by observing the parameter distribution of LayerNorm, we surprisingly find that the multiplier γ (Fig. 1b) and the output (cid:102) X (Fig.
1a) hold outliers at the same embedding dimensions.
Besides, the adder β denotes a smaller range (e.g., (0,3)) compared to the output range (e.g., (-60, 0)), so we ignore it for identifying the key point.
That is to say, γ plays a crucial part for the outliers in Fig.
1a, especially can amplify the outliers across tokens by serving as a shared parameter.
This observation enlightens us to remove the amplification effect by extracting γ from Eq.
(2) and use the Non-scaling LayerNorm Eq.
[Equation]
Fig.
1c and Fig.
1a show that the output of the Non-scaling LayerNorm denotes a milder distribution with weaker outliers than the normal one.
It not only coincides with that γ does strengthen the outliers but also reveals that X ′ behaves more friendly than (cid:102) X for quantization.
To quantitatively validate the more quantization-friendly distribution X ′ holds, we adopt the cosine similarity metric to evaluate the quantization loss.
From Table 1, the second row with higher similarity, namely less quantization error, explains that the quantization performance can be improved using
Non-scaling LayerNorm.
[Table]
3.2 Impact of outlier clipping
In this part, we explore the impact of clipping the outliers to design a method that can find an appropriate clipping range for quantization.
The experiments are designed for the clipping impact on the accuracy and token of FP models.
Impact on accuracy.
When clipping the outliers and evaluating the final performance, we find that the importance of outliers is highly varied.
Take the outliers after GELU as an example here (others in Sec.
D.2), Fig.
2 shows that clipping the more aggressive outliers sharply (clipping signals in 10-100 to 10) even does not hurt the full-precision performance with accuracy still at 91.02, while the accuracy drops suddenly to 85.93 with too many outliers cut.
Impact on token.
Another key point is the unimportant outliers which can be clipped without even any accuracy drop in FP models only correspond to a few tokens.
Motivated by [26], they refer that the separator token [SEP] attends to larger values.
We are also aware of the different ranges provided by different tokens.
From the red points in Fig. 2, which represents the proportion of clipped tokens, it can be clearly seen that the more aggressive outliers though occupy a large range from 10 toonly matches with 3% tokens.
Destroying those sharper outliers belonging to a few tokens will not affect the performance.
[Figure]
The former investigation of accuracy impact suggests us taking the final performance into account to find a superior clipping range, where some local optimization methods like [28] are not suitable here.
The latter finding in token impact encourages us to leverage the token’s indication to quickly skip over the unimportant area, especially when it presents in a long tail form where some methods like [29] suffer low efficiency.
Based on these, we will introduce our method in Sec. 4.2.
Method
In this section, we propose our outlier suppression framework based on the above analysis.
Firstly, the
Gamma Migration technique is adopted to obtain a more quantization-friendly model by migrating the gamma into subsequent modules.
Secondly, the Token-Wise Clipping further finds a suitable clipping range efficiently by leveraging the large variance of the token range.
4.1 Gamma Migration
As pointed out in Sec. 3.1, activation without going through the scaling parameter provides less quantization error.
In this way, we split the LayerNorm function, migrate γ into follow-up structures and quantize the output of the Non-scaling LayerNorm.
The transformation is equivalent for the FP model and brings more robust activation for the low-bit one.
The overall flow is illustrated in Fig.
Migration equivalence on FP model.
Naturally, as referred in Eq.
(3), we extract the parameter γ and transform the LayerNorm into Non-scaling one, thus seperate X ′ t,j from (cid:102)
X t,j .
[Equation]
Since the residual connection is frequently adopted after LayerNorm ([30, 31, 32]), it is necessary to illustrate the way to migrate parameter γ into two branches.
To be specific, considering the
LayerNorm after Multi-Head Attention (Fig. 3), γ will be excluded from the LayerNorm and moved to the shortcut branch and weight of the next layer.
Then the LayerNorm becomes the Non-scaling one, the shortcut branch establishes a new parameter γ , and the weight of the next layer can absorb the γ .
Now, we show how the weight absorbs γ .
For linear layers, we have the following equation:
[Equation]
where x serves as a column vector and γ ∈ R n .
The proof is available in Appendix A. Because γ is a shared parameter, each token’s embedding satisfies Eq.
(5), which promises success of transferring the γ into the next layer’s weight.
[Figure]
Quantization after migration.
Deriving from the above equivalent transformation, we outline the quantization pattern after the migration process.
From Fig.
3, the "Quant" process is employed at X ′ , then the quantized output engages in the matrix multiplication on one branch, multiplies parameter γ and experiences the "DeQuant" process on another branch.
In fact, this means delaying the γ calculation from LayerNorm to the shortcut branch.
Hence, this new design will not increase the computation overhead.
Effect of migration.
We then analyze the effect of Gamma Migration on weight and activation, respectively, to reveal that the activation quantization burden has been greatly alleviated with relatively a slight influence on weight.
To begin with, suppose that the absolute max range of output in the original LayerNorm is | max ( X ′ ) | ∗ | max ( γ ) | for the reason that outliers emerge at the same embedding dimensions among γ , activation before X ′ and after (cid:102) X scaling function.
For activation, extracting the γ will reduce the activation range by | max ( γ ) | times.
And the results in Table 1 have already validated the profit the transformation brings to activation.
For weight, the weight matrix does not have the same embedding outlier phenomenon as the activation.
Therefore, the weight range will not be amplified | max ( γ ) | times after the migration.
Experimentally, we also calculate the cosine similarity for the changed weight and observe that γ has little impact on weight (Table 2).
[Table]
4.2 Token-Wise Clipping
Based on the analysis, we propose the Token-Wise Clipping method which considers the final loss when finding a clipping range and takes a coarse-to-fine paradigm to minimize it efficiently in a token-wise manner.
Regarding the very different accuracy impact of clipping the outliers, we search the clipping range, equivalently the step size s , which has the minimal distance between the final quantized output ˆ f ( s ) and the real one f defined as Eq.
To implement the process efficiently, especially when the unimportant outliers cover a wide area, a coarse-to-fine paradigm is designed below.
[Equation]
Coarse-grained Stage.
At this stage, our aim is to quickly skip over the area where clipping causes little accuracy influence.
According to Sec. 3.2, the long tail area only matches with a few tokens.
Therefore, we suggest using the max value of the embedding at token t to be its representatives (min value as representatives for negative outliers).
A new tensor with T elements can be constructed by taking out the maximum signal for each token:
[Equation]
where o u is marked as the collection of upper bounds, o l as the collection of lower bounds.
Then for a clipping ratio α on o u , calculate the corresponding clipping value c u and use it to cut the tensor.
[Equation]
where the quantile function computes the α - th quantiles of o u .
[Figure]
Through grid search of token-wise clipping ratio, step size s = c u − c l 2 b − 1 ( b is the bit-width) with minimal quantization loss Eq.
(6) is obtained.
We mark it as s 0 for later optimization.
Fine-grained Stage.
At this stage, our aim is to make some fine-grained adjustments in the critical area to further provide a guarantee for the final effect.
In detail, with the initialization s 0 , a learning procedure based on gradient descent is used to update parameter s towards loss L ( s ) with learning rate η , as described in Eq.
[Equation]
Benefits.
We mainly explain the benefits of the coarse-grained stage here from efficiency and quantization performance, where the experimental comparisons with other existing approaches are put in Sec.
D.3.
For efficiency, because the wide range of outliers only corresponds to a few tokens, passing through the unimportant area from the token perspective needs much fewer iterations than from the value perspective.
Moreover, the representative collection reduces the size of the tensor ( o u distilled from X ), so the method can run very fast each iteration.
For quantization performance, the first coarse step has already produced a suitable clipping range (Sec. 5.2), which offers a good initialization point for upcoming tuning.
Experiments
In this section, we conduct two sets of experiments to verify the effectiveness of our outlier suppression framework.
Sec.
5.2 shows the effect of each component.
Sec.
5.3 lists the results compared with other existing approaches across text classification, question answering, and summarization tasks.
On the whole, we evaluate GLUE benchmark [33], SQuAD [34, 35], and XSum [36] and
CNN/DailyMail [37] across BERT, RoBERTa, and BART models.
Here, 4-4-4 presents 4-bit weight, embedding, and activation.
And the model size under a certain bit is put in Table 17.
5.1 Experimental setup
Implementation details.
To begin with, we identify the quantization nodes and take a reasonable scheme like the one in FasterTransformer [38] (Details see Sec.
B.1).
For PTQ, equipping our framework, we use 256 samples to calibrate the model.
For QAT, our methods work on the calibration phase and later are combined with LSQ+ [12], a strong baseline for the training phase.
For training, hyper-parameters like learning rate are searched both for our methods and baseline techniques for fair comparisons.
Details see Appendix F.
Baseline.
For PTQ, we compare with the prevalent calibration mechanisms including MinMax [39],
OMSE [28], Percentile [29], EasyQuant [40] and PEG [26].
For QAT, we present the results of
Q-BERT [22], Q8BERT [21] and PEG [26].
Also, because our framework applying in QAT is coupled with LSQ+ [12], we show the results of the pure LSQ+, and another canonical quantization approach PACT [41].
Last but not least, the results combined with knowledge distillation (KD) proposed in TernaryBERT [23] are included as well.
5.2 Ablation Study
In this subsection, we ablate the design elements in the proposed framework (Table 3).
As a general plug-in module, Gamma Migration helps both the MinMax and Token-Wise Clipping.
And the Token-Wise Clipping also surpasses the baseline by a large margin: 17.53% on QNLI, 13.22% on MRPC (comparisons with other calibration algorithms see
Sec. D.3).
About the phenomenon that the fine-grained stage sometimes does not improve much upon the coarse-grained one, we think it’s due to the already good enough results produced by the coarse step.
Besides, Fig.
5 conveys that with a good initialization point provided by our framework, the training of QAT becomes much faster and easier.
[Table]
5.3 Main Results 5.3.1 Results on classification tasks
[Table]
PTQ.
Table 4 shows the results of PTQ on GLUE tasks.
For 8-bit BERT models, although previous methods generally behave well, our methods can still achieve satisfying outcomes even on small datasets such as CoLA (4.49% upswings) and STS-B (1.33% upswings).
To fully exploit the limit, we try a more inspiring setting with weight and activation quantized to 6-bit.
It can be seen that ours is indeed close to FP value within 2.64% overall.
Meanwhile, we also compare with PEG [26] fairly by taking their quantization nodes.
To be noted, their per-embedding-group (PEG) quantization certainly brings extra computation overhead and might not be available on real deployment while ours brings favorable results and can enjoy lossless acceleration on hardware.
Besides, the experimental results on RoBERTa and BART consistently demonstrate our superiority whereas existing methods suffer from a non-negligible accuracy drop.
On average, ours achieves up to 8.64% and 11.79% better accuracy on RoBERT and BART.
To conclude, our proposed methods push the limit of 6-bit quantization to a new state of the art.
QAT.
In particular, we prove the compatibility of our methods on QAT.
Table 5 lists the results on
BERT, other see Sec.
D.4.
In a much harder setting (4-4-4 bit quantization), our outlier suppression framework wins near-floating-point performance with a reduction of 2.70% on average on 4-bit quantization.
Yielding a good initialization, ours obtain an acceptable accuracy drop (0.7% on
QQP, 1.7% on MNLI) without any distillation and data augmentation trick, versus 4.19% and 3.16% of LSQ+.
Furthermore, ours still enables performance improvements working with knowledge distillation, especially at 2-bit weight and embedding
[Table]
5.3.2 Results on question answering tasks
To demonstrate the wider applicability of our methods
[Table]
we evaluate them on SQuAD datasets.
When going down to 6-bit quantization, the performance of other methods drastically drops.
Ours still outperforms them by over 4.73% and 15.55% on BERT and RoBERTa on SQuAD v1.1.
Also, the boost can be 12.31% and 4.96% on RoBERTa and BART on SQuAD v2.0.
5.3.3 Results on summarization tasks
It is of high value to validate the effect of our methods on summarization tasks.
We choose classical datasets CNN/DailyMail and XSum and report the ROUGE 1/2/L score of BART.
Table 7 illustrates that our approaches also benefit the encoder-decoder models, and can bring a near-floating-point performance on 8-bit and about 4% enhancement on 6-bit
[Table]
[Table]
Conclusions and Discussions of Limitations
In this paper, we analyze the outlier phenomenon from the inducement and clipping impact on transformer language models.
Based on these, we establish an outlier suppression framework to suppress the outliers.
There also remain some open problems worthy of more in-depth investigations.
For example, it is valuable to systematically explore whether the conclusion in this paper benefits other fields such as computer vision.
Besides, as we supplement in the Appendix that the outliers occur not only in the fine-tuned (BERT) models but also in the pre-trained ones, it’s also meaningful to dive into the pre-training process for a better understanding.
Acknowledgment
We sincerely thank the anonymous reviewers for their serious reviews and valuable suggestions to make this better.
This work was supported in part by National Natural Science Foundation of China under Grant 62022009 and Grant and Beijing Nova Program of Science, and Technology under Grant Z191100001119050 and the Fundamental Research Funds for the Central Universities
[-INF, s 
-]