Towards Efﬁcient Post-training Quantization of Pre-trained Language Models
Abstract
Network quantization has gained increasing attention with the rapid growth of large pre-trained language models (PLMs).
However, most existing quantization methods for PLMs follow quantization-aware training (QAT) that requires end-toend training with full access to the entire dataset.
Therefore, they suffer from slow training, large memory overhead, and data accessibility issues.
In this paper, we study post-training quantization (PTQ) of PLMs, and propose module-wise quantization error minimization (MREM), an efﬁcient solution to mitigate these issues.
By partitioning the PLM into multiple modules, we minimize the reconstruction error incurred by quantization for each module.
In addition, we design a new model parallel training strategy such that each module can be trained locally on separate computing devices without waiting for preceding modules, which brings nearly the theoretical training speed-up (e.g., 4 ⇥ on 4 GPUs).
Experiments on GLUE and
SQuAD benchmarks show that our proposed PTQ solution not only performs close to QAT, but also enjoys signiﬁcant reductions in training time, memory overhead, and data consumption.
Introduction
Large pre-trained language models (PLMs) have achieved remarkable success in various natural language processing tasks [44, 12, 5].
However, the increasing size and computation overhead also make it prohibitive to deploy these PLMs on resource-constrained devices.
To obtain compact PLMs, various model compression methods have been proposed, such as pruning [32, 14, 50], knowledge distillation [40, 42, 24], weight-sharing [10, 25, 46, 22], dynamic computation with adaptive depth or width [18, 52, 59], and quantization [55, 41, 54, 56, 3, 38, 43].
Among these methods, network quantization enjoys the reduction of both model size and computation overhead without modifying the network architecture, and is thus extensively studied in PLM quantization [55, 41, 54, 56, 3, 38, 43].
However, most previous methods for PLM quantization methods follow quantization-aware training (QAT), which suffers from several issues.
Speciﬁcally,
QAT usually conducts end-to-end back-propagation training over the whole training set, which can be slow in training time, memory demanding and data consuming.
These issues can be somethimes prohibited for industrial PLMs.
Post-training quantization (PTQ), on the other hand, serves as an appealing alternative that is efﬁcient in training time, memory overhead and data consumption.
Instead of the full training set, PTQ can leverage only a small portion of training data to minimize the layer-wise reconstruction error incurred by quantization [34, 35, 33, 23].
This can be done by either calibrating the batch normalization statistics [34] or step sizes [35] in quantization functions.
The layer-wise objective also breaks down the end-to-end training, making the prob- lem more sample-efﬁcient [58] and memory-saving.
Nonetheless, we ﬁnd it non-trivial to directly apply previous PTQ methods for PLMs such as BERT [12], as the performance drops sharply.
In this paper, we aim at improving the performance of post-training quantization for PLM, while simultaneously maintaining its efﬁciency w.r.t training time, memory overhead and data accessibility.
First, we partition the PLM into multiple modules, and propose module-wise reconstruction error minimization (MREM).
Each module consists of multiple Transformer layers, which permits the sufﬁcient optimization than the previous layer-wise objective.
Meanwhile, the module size can be ﬂexibly adjusted for the proper trade-off between layer-wise correlation and memory overhead of computing devices.
While a similar blockwise objective is previously considered in [29], they require second-order Hessian matrices for optimization, which can be computationally prohibitive for large
PLMs.
Second, we design a new model parallel strategy to further accelerate the training process of MREM.
By allocating each partitioned module to the individual computing device, our design allows all modules to be trained jointly without synchronizing with adjacent partitions.
This brings nearly the theoretical speed-up (e.g., 4 ⇥ on 4 GPUs), superior to previous data parallel [9, 27] or model parallel [21, 36] techniques.
Third, we develop annealed teacher forcing for our parallel strategy.
We ﬁnd that the naive parallel training suffers from the propagation of reconstruction error, since each quantized module passes the error to its successors before it is fully optimized.
Inspired by [51], we use the full-precision module to provide clean signals to the next quantized module.
This breaks the reconstruction error propagation and further improves the performance of parallel strategy.
We summarize the contributions of this paper as follows (i)
We study the post-training quantization of PLMs, and propose module-wise reconstruction error minimization (MREM), a fast, memorysaving, and data-efﬁcient approach to improve the quantized PLMs.
(ii) We design a new model parallel strategy based on MREM to accelerate post-training quantization with theoretical speed-up for distributed training.
(iii)
The parallel MREM can be combined with annealed teacher forcing to alleviate the propagation of reconstruction error and boost the performance.
iv) Empirical results on the GLUE and SQuAD benchmarks demonstrate superiority of our PTQ solution over QAT.
For instance, Figure 1 shows that the BERT-large model trained by parallel MREM achieves 85 .
5% accuracy with only 4 K training samples.
Moreover, it consumes merely one-third of memory per
GPU and is more than 150 ⇥ faster than previous QAT training.
Motivation 2.1 Quantization Background
Network quantization replaces the original full-precision weight or activation x 2 R m ⇥ n with its lower-bit counterpart ˆ x .
Denoting s 2 R + as the step size, the b -bit symmetric uniform quantization function Q b ( · ) can be written as
[Equation]
where ⌦ ( b ) = {  2 b  1 + 1 , ..., 0 , ..., 2 b  1  1 } is the set of b -bit integers, and ⇧ ( · ) is the projection function that maps x /s to its closest integer.
To quantize Transformer-based PLMs, we follow the setting in previous works [55, 56, 3]: we quantize both the network weights and activations in each matrix multiplication.
We use symmetric uniform quantization for weights, embeddings, and activations, except activations after the self-attention and
GeLU function.
For these two activations, we adopt asymmetric quantization since their elements are mostly positive.
We skip the quantization for all layer-normalization layers, skip connections, biases
[Figure]
and the last classiﬁcation head due to little computation overhead or large performance drop.
We leave more details on the choice of quantization functions and their algorithmic details in Appendix A.
Post-training Quantization.
Recently popular PTQ methods [35, 47, 33, 23, 30] constructs an tiny unlabeled dataset (a.k.a.
calibration set)
˜ D from the original training data D .
These approaches usually aim at reconstruction error minimization (REM), i.e., minimizing the distance between the output of multiplication between the quantized and the full-precision counterpart as follows:
[Equation]
where w and a are full-precision weights and activations, ˆ w and ˆ a are their quantized representations with b w and b a bit-widths, and s denotes all step-sizes involved for quantization.
REM is usually conducted in a greedy manner.
It proceeds to the matrix multiplication only after the training of previous ones.
The simple objective in Equation (2) is lightweight to solve over ˜ D , and is also theoretically veriﬁed to be more sample-efﬁcient than end-to-end training [58].
While there are also other data-free PTQ variants [57, 34, 16, 54], REM-based solutions usually demonstrate better empirical results [33, 23].
Thus we mainly focus on the REM-based PTQ for PLMs in this paper.
2.2 Quantizing Pre-trained Language Models: QAT or PTQ?
To conceptually study the difference between QAT and REM-based PTQ over popular PLMs, we consider the following four dimensions, and summarize the results in Figure 2.
Training Time.
As QAT conducts training over the full training set D , it takes much more time than
PTQ over the calibtration set ˜ D .
Recent QAT methods [56, 3] further combine two-stage knowledge distillation [24], which can take nearly four times longer than FP, as shown in Figure 2(a).
Memory Overhead.
The increasing size of large PLMs makes it prohibited to conduct QAT on memory-limited devices.
From Figure 2(b), QAT [56] even consumes 8 .
3 GB more memory than FP when armed with knowledge distillation.
On the contrary, PTQ only caches intermediate results in
Equation (2), which can be fed into a single GTX 1080 Ti.
Data Accessibility.
Data accessibility can be often prohibited due to data security or privacy issues in the industry.
Unlike QAT that needs the full training set D , PTQ constructs the calibration data ˜
D ✓ D by sampling only 1 K ⇠ 4 K instances from D , as shown in Figure 2(c).
Performance.
QAT usually demonstrates superior quantization results than PTQ, and this holds true for PLMs.
From Figure 2(d), QAT remains steady across different bit-widths, while REM-based
PTQ drops sharply.
Thus the poor performance of PTQ is the main challenge in quantizing PLMs.
In summary, REM-based PTQ is superior to QAT with regard to training efﬁciency, memory overhead, and data accessibility.
Nevertheless, it is still often less preferred than QAT due to its severe performance drop especially for low quantization bit-width [55, 41, 56].
We partition both the full-precision model and quantized model into multiple modules and put these modules on different computing devices.
By sampling tensors from the input queue, each module can be trained locally without waiting for its predecessors.
Teacher forcing is applied to mitigate the issue of reconstruction error propagation on the quantized module.
Methodology
In this section, we aim at improving the performance of post-training quantization for PLMs, while preserving its merits of fast training, light memory overhead, and data consumption.
We extend the existing reconstruction error minimization from the layer-wise to the module-wise granularity to ﬁt
Transformer models.
Based on the module partition, we then design a new parallel training strategy that further speeds up the PTQ pipeline.
An overview of our solution can be found in Figure 3.
3.1 Module-wise Reconstruction Error Minimization
We propose a new PTQ solution called module-wise reconstruction error minimization (MREM) for PLMs.
Existing REM [33] solves Equation (2) for each matrix multiplication.
However, a standard transformer layer in PLMs consists of a Multi-Head Attention (MHA) and a Feed-Forward
Network (FFN), both of which contain a number of matrix multiplications that are coupled together.
Greedily tackling each matrix multiplication in REM can thus lead to suboptimal quantized networks.
Moreover, the insufﬁciently minimized reconstruction error shall propagate and enlarge along with transformer layers, and ﬁnally deteriorate the network output [6, 2].
Towards that end, the proposed module-wise reconstruction error minimization admits larger granularity by jointly optimizing all the coupled linear layers inside each module.
Speciﬁcally, given a
PLM with L transformer layers, embedding layers and the classiﬁcation head, we partition them into
N modules, where the n -th module include [ l n , l n +1 ) transformer layers with l n being the ﬁrst layer of this module 2 .
MREM aims at minimizing the joint reconstruction errors between all quantized
FFN output ˆ f l in the module from their full-precision counterpart f l as follows:
[Equation]
where w n and s n are all learnable parameters and quantization step sizes in the n -th module.
Similar to REM, MREM can be optimized sequentially: given previously trained modules, only parameters and quantization step sizes in the current module are updated.
Besides the grouped Transformer layers, we also minimize the MSE loss in the Transformer embedding and output logits respectively.
Note that the number of modules N can be adjusted depending on the memory constraint of computing resources.
When N = 1 , this reduces to intermediate-layer knowledge distillation [24], which can be memory-demanding when quantizing large PLMs on a single GPU.
3.2 Accelerated Parallel Training
Based on the proposed MREM, we propose a new model parallel strategy to further accelerate the training.
As shown in Figure 3, we put different modules on individual computing devices.
A set of input queues I =
{ I 1 , ..., I N  1 } is deployed between each pair of adjacent modules.
For the n -th module, the queue collects its output of the most recent t 0 steps, i.e., I tn =
{ f tl n , f t  1 l n , ..., f t  t 0 +1 l n } .
Meanwhile, the ( n + 1) -th module can always sample with replacement f l n ⇠ I tn from the queue without waiting for the n -th module.
Similar rules hold for the quantized module and their input queues ˆ I .
The design of the input queue resembles stale synchronous parallel [17] that caches the stale output locally so as to reduce the waiting time among workers, where t 0 is the stale threshold.
The training workﬂow is as follows.
Initially, every module is computed one after another in the ﬁrst t 0 step to ﬁll in the input queue, after which parallel training takes place.
Then the module samples input from the queue and calculates the loss ` ( n ) correspondingly for n = 1 , ..., N .
Meanwhile, the input queue is updated with the rule of ﬁrst-in-ﬁrst-out throughout the training.
In the backward pass, we constrain the gradients to propagate locally within each module, without affecting its predecessors.
Such a design can avoid the load imbalance issue from straggler modules, bringing nearly the theoretical N ⇥ speed-up.
The proposed parallel training strategy is superior to the conventional data parallel training in quantizing PTMs.
Meanwhile, it is also different from pipeline parallelism [21].
We leave the detailed comparisons with these parallelism techniques in Appendix C.
3.3 Annealed Teaching Forcing
Since all modules get optimized simultaneously instead of the sequential manner, the next module takes the output from the queue before its predecessor is fully optimized.
Therefore, the predecessor’s reconstruction error is propagated to the following modules before it is sufﬁciently minimized.
Inspired by teacher forcing [51] in training recurrent networks, the output f l n from the n -th fullprecision module naturally serves as the clean input to the ( n + 1) -th quantized module to substitute ˆ f l n .
Thus f l n stops the propagation of the accumulated error on the quantized module.
Nevertheless, such an approach breaks the connection to previous quantized modules and may suffer from forward inconsistency between training and inference [2] on the quantized model.
For the trade-off, we take the convex combination between the full-precision f l n and quantized ˆ f l n as follows:
[Equation]
where the hyperparameter  controls the strength of teacher forcing.
= 1 gives the full correction of reconstruction error but with forward inconsistency, while  = 0 reduces to the conventional setting that suffers from the propagated reconstruction error.
We adopt a linear decay strategy for  :  t = {+max(
 +} tT 0 , 0) , where T 0 is the preset maximum steps of the decay.
Intuitively, a large  is desired at the beginning when each module is rarely optimized.
Later, a small  is preferred to transit to normal training such that the forward inconsistency can be bridged.
The remaining T  T 0 steps stick to normal training so that each quantized module adapts to its own predecessors.
Finally, an overview of the proposed parallel module-wise reconstruction error minimization with annealed teacher forcing is shown in Algorithm 1 and Algorithm 2.
The Update ( · ) in Algorithmcan be any gradient update function such as AdamW [31] with learning rate ⌘ t .
the Quantized PLM
[Table] [Table] [Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Table] [Table]
Experiments
In this section, we empirically verify the proposed MREM for post-training quantization of PLMs.
We ﬁrst introduce the experimental setup in Section 4.1.
Then we present main results in Section 4.2, including comparisons with QAT and REM, and other existing quantization baselines.
In Section 4.4, we provide discussions on a variety of factors in our approach, such as the effect of teacher forcing, the number of model partitions, and calibration data size.
Our implementation is based on MindSpore [1].
4.1 Experimental Setup
Datasets and Metrics.
We evaluate post-training quantization on both the GLUE [45], and SQuAD benchmarks [39].
The size of calibration data is by default | ˜ D|
= 4 , 096 , by randomly sampling instances from the full training set.
As both RTE and MRPC tasks in the GLUE benchmark contain fewer than 4,096 samples, we use their full training set on these two tasks.
We leave the study of data size in Section 4.4.
Each experiment is repeated ten times with different calibration sets, and both the mean and standard deviations are reported.
We use the same evaluation metrics in [12, 56] for the development set of GLUE and SQuAD benchmarks.
For results in Section 4.2, we report accuracies on both the matched section and mis-matched sections of MNLI, and EM (exact match) and F1 score for SQuAD.
Additionally, we also report the training time (min), memory overhead (GB) as well as the size of the training set (K).
We also provide comparisons with other existing methods in Section 4.3, where we adopt Matthews correlation for CoLA, Spearman correlation for STS-B, and accuracy for the rest ones (i.e., RTE,
MRPC, SST-2, QQP, MNLI).
We also report the averaged performance on GLUE as an overview.
Implementation.
We use the standardly ﬁne-tuned BERT-base and BERT-large models 3 on downstream tasks for both quantization-aware training and post-training quantization.
We implement
MREM in both the sequential training (abbv.
MREM-S) in Section 3.1 and parallel training with teaching forcing (abbv.
MREM-P) in Section 3.3.
For each module, we train for 2 , 000 steps with an initial learning rate of 1e-4 on GLUE tasks, and 4 , 000 steps with an initial learning rate of 5eon SQuAD datasets.
The learning rate decays linearly as done in [24, 56].
By default, we partition the model into 4 modules on 4 NVIDIA-V100 GPUs.
For baselines, we mainly compare with QAT and REM, where the former measures how far is PTQ from QAT, and the latter studies the effect of module-wise granularity in PTQ training.
For a fair comparison of each method, we use the same quantization scheme for all methods, i.e., TWN [26] or LAQ [19] for 2-bit and 4-bit weight quantization, and LSQ [13] for all activation quantization.
Unlike QAT that picks the best model based on the development set results, MREM is only tested once after training, which does not require further access to the development set.
We leave more details of baseline implementation in
Appendix B.1.
The comparison with more published quantization baselines are left in Section 4.3.
4.2 Main Results: Comparison with QAT and REM
We ﬁrst compare MREM-S and MREM-P with QAT and REM over MNLI and SQuAD benchmarks.
We take BERT-base and BERT-large as backbone PLMs for quantization.
The results on MNLI and
SQuADv1.1 are summarized in Table 1 and Table 2 respectively, and results on SQuAD v2.0 are left to Table 6 in Appendix B.2.
We again summarize according to the four dimensions in Section 2.2.
Performance.
It can be found that our proposed MREM-S improves the performance of REM signiﬁcantly given the same training time, and is much closer to QAT.
For instance, according to in
Table 1, MREM-S with 4-bit weight quantization on BERT-base and BERT-large achieves accuracies of 83 .
5% ± 0 .
1 and 86 .
1% ± 0 .
1 on the matched section of MNLI, which is on average 10 . 2% " and 16 .
1% " better than REM, and only 1 . 1% # and 0 .
8% # inferior to QAT, respectively.
Moreover, with all modules trained in parallel, MREM-P is still close to or only slightly inferior to MREM-S.
From Table 2, MREM-P can even outperform MREM-S with the “W2-E2-A4” quantized BERT-large model on SQuAD 1.1, i.e., the EM score and F1 score are on average 0 .
4% " and 0 .
2% " respectively.
Training Time.
Our proposed MREM also enjoys signiﬁcantly less training time than QAT.
From
Table 1, MREM only takes 84 minutes to quantize BERT-large with 4 -bit weights, which is about 38 ⇥
[Table] [Table]
faster than QAT and 7 ⇥ faster than full-precision ﬁne-tuning.
Note that the full-precision ﬁne-tuning time is listed to conceptually compare how much time is further required for quantization.
Comparing with REM, MREM does not cache the output after every matrix multiplication, which admits more iterations given the same amount of time.
We discuss this further in Appendix B.3.
Moreover,
MREM-P is further 4 ⇥ faster than MREM-S, which achieves the theoretical linear speedup on
GPUs.
These together bring more than 150 ⇥ reduction of training time when compared with QAT.
Memory Overhead.
Our MREM-S and MREM-P take only around a third of the GPU memory by QAT, and a half of that by the full-precision ﬁne-tuning, e.g., 29 . 8 GB for QAT and 10 . 8 GB for
MREM-S on BERT-large.
Moreover, we also encounter memory overﬂow (more than 32GB memory of an NVIDIA V100 GPU) for QAT training on SQuAD due to longer sequence length, and we adopt gradient accumulation that inevitably doubles the training time (i.e., underlined ﬁgures (“ ”) in Table 2).
On the other hand, such issues can be easily mitigated in both REM and our proposed
MREM, both of which can be even fed into a single NVIDIA GTX 1080 Ti GPU.
We may further decrease the memory overhead of MREM to REM by increasing the number of modules, but this could harm the performance as discussed in Section 4.4.
Data Accessibility.
Both REM and our proposed MREM follow the common practice of PTQ, relying on only 4 , 096 randomly sampled instances on both MNLI and SQuAD, which is a tiny fraction of the original dataset.
We shall provide more discussion on the calibration size in Section 4.4.
4.3 Main Results: Comparison with Existing Methods
Next, we compare our MREM with existing state-of-the-art BERT quantization methods.
They include various QAT approaches such as Q-BERT [41], Quant-Noise [15], TernaryBERT [56], and
[Table] [Table]
[Table]
the PTQ baselines including GOBO [54], BRECQ [29], and QDrop [48].
The results are from the original papers.
Additionally, we also compare our MREM with REM.
From Table 3, both our proposed MREM-S and MREM-P outperform existing PTQ approaches in most cases.
For “W2-E2-A8” quantized models, MREM-S and MREM-P surpass GOBO by 11 .
7% " and 11 .
3% " on MNLI-m respectively.
Both BRECQ and QDrop lag behind our approach even with the conﬁguration “W4-E8-A4”.
For instance, the average scores on GLUE have the gaps of 9 . 3% and 8 . 0% between QDrop and our “W2-E2-A4” quantized MREM-S and MREM-P, respectively.
Our results are even close to QAT approaches in multiple entries.
For example, the “W4-E4-A8” quantized
MREM-S and MREM-P have the mean accuracies of 83 .
5% and 83 .
4% on MNLI respectively, both of which are on par with the “W2/4-E8-A8” quantized Q-BERT.
4.4 Discussions
In this section, we provide further discussions to better understand the proposed approach.
Unless speciﬁed otherwise, all experiments are built upon the BERT-base model over the MNLI dataset
Teacher Forcing.
We now study how teacher forcing beneﬁts MREM-P with different numbers of training steps, and results are listed in Table 4.
We ﬁnd that teacher forcing brings consistent improvement for both BERT-base and BERT-large, and the gain is more signiﬁcant with fewer training steps or lower quantization bit-width.
For example, it brings 3.4 % " and 2.8 % " on the “W2-E2-A4” quantized BERT-base and BERT-large respectively under 250 steps.
This matches our intuition that fewer training steps or higher compression ratio give larger reconstruction error, when the clean input from the full-precision module can beneﬁt more the quantized module.
As more training steps bring only marginal improvement, we by default set the training steps to 2 , 000 .
Additionally, we also plot training loss curves of the 2 -nd and 4 -th modules under 250 and 2, training steps in Figure 4, and more visualizations can be found in Figure 6 of Appendix B.4.
We ﬁnd that: 1) the loss curves with teacher forcing are apparently lower, especially when trained with fewer steps, which is consistent with Table 4; 2) the loss curves rise in the halfway during teacher forcing, since the modules need to adapt to quantized input with accumulated errors; 3) the loss curves of the 4 -th modules are lower than the 2 -nd ones, which matches the intuition that the late modules have
[Figure] [Figure]
more errors accumulated and thus beneﬁt more from teacher forcing.
We also try tuning the teacher forcing steps (i e., shaded area) within [20% , 80%] of total iterations, but observe no large difference in performance.
We thus choose 40% training steps for teacher forcing by default.
Number of Modules and Memory Overhead.
We verify the effect of model partition on the ﬁnal quantized performance, as well as their corresponding memory consumption.
According to
Figure 5(a) by varying the number of modules within { 1 , 2 , 3 , 4 , 6 } , it can be found that more model partitions give slightly lower performance, as layer-wise dependencies are less considered during reconstruction error minimization.
However, more partitions lead to less running memory, i.e., { 11 . 9 , 6 . 7 , 4 . 7 , 3 . 7 , 2 . 7 } GB for these partitions correspondingly.
As the decrease of memory diminishes with more partitions, we partition the model into 4 modules by default.
Size of Calibration Data.
The size of calibration data directly relates to the data accessibility issue in post-training quantization.
To learn its effects, we vary the calibration data size | ˜ D| within { 32 , 64 , 128 , 512 , 1024 , 2048 , 4096 , 8192 } , and list the results of REM, MREM-S and MREM-P.
From Figure 5(b), it can be found that while REM is ahead of MREM-S/P with fewer thantraining samples, the accuracy of REM rises slowly and saturates at around 60% afterwards.
We hypothesize that the simple training objective in REM can hardly hold more training instances for optimization.
MREM-S/P, on the other hand, can better exploit larger calibration data size, since the module-wise granularity admits higher ﬂexibility for the optimization.
As we ﬁnd the diminishing gain to increase the training size after 4 , 096 samples, we by default take 4 , 096 samples.
Reconstruction Error Propagation.
We visualize the propagation of reconstruction error for both “W2-E2-A8” and “W2-E2-A4” quantized BERT-base models in Figure 5(c) and Figure 5(d) respectively.
It can be observed that our MREM achieves both lower values and slower rising rates of the reconstruction error than REM across all layers, which veriﬁes the advantage of module-wise granularity to minimize the reconstruction error.
Interestingly, while the reconstruction error generally gets enlarged layer-wisely in the ﬁrst ten layers, it begins to decrease afterwards.
We speculate this is due to the classiﬁcation head that encourages concentrated hidden representations for the task.
Per-channel Quantization.
Per-channel Quantization (PCQ) is prevalent in the post-training quantization of convolution neural networks [35, 33, 23].
To learn its effect in PLMs, PCQ assigns different quantization step-sizes at each output dimension of the linear layer, which is also known as row-wise quantization in [56].
The PCQ results of REM and MREM are shown in Table 5.
It can be found that while PCQ improves REM by around 1 . 0% to 2 .
5% , the gain is very incremental on
MREM.
Our results are also similar to the ﬁndings in [56], where the row-wise quantization brings little improvement.
As PCQ requires to store more full-precision step sizes with minor improvement, we do not employ PCQ by default.
Related Work
Network quantization is a common approach to compress and accelerate deep neural networks [8, 26, 20, 19, 13, 28, 7].
For network quantization of pre-trained language models, previous efforts mostly adopt quantization-aware training [55, 41, 56, 3, 38].
For instance, Q8BERT [55] converts both parameters and activations with 8-bit representations with negligible task degradation.
Q-BERT [41] exploits the hessian matrix of loss curvature to determine the layer-wise quantization bit-width, achieving a higher compression rate.
TernaryBERT [56] proposes to ternarize BERT parameters with 2-bit representations, while BinaryBERT [3] further uses 1-bit to represent model parameters.
BiBERT [38] explores the fully-binarized BERT with 1-bit weights and activations.
Despite the success by these efforts, the heavy ﬁne-tuning of QAT makes it prohibitive given constraints on training time, memory size, and data accessibility.
Post-training quantization [37, 53, 11] is thus considered mitigate these issues from QAT.
One line of
PTQ research quantizes the network purely without using any training data, but removes outliers in the full-precision parameters [57, 34, 16, 54].
Methodologically, PTQ can be achieved by splitting an outlier neuron with a large magnitude into two parts [57], where the magnitude can be halved.
Alternatively, one can scale down outlier magnitude and multiply it back in subsequent layers, a.k.a weight equalization in [34].
Another solution is to treat the outliers and normal values in the distribution separately, by keeping two sets of quantization parameters [16, 54].
Another line in PTQ research [35, 47, 33, 23] minimizes the reconstruction error based on the calibration data.
Compared with training-free PTQ approaches, such an approach signiﬁcantly improves the performance of the quantized network, and we thus follow this line in this paper.
Recently, there are also concurrent efforts with us to apply PTQ for PTMs [49, 37, 53, 11].
In [49], a similar method is developed to minimize the quantization error via the calibration data.
Moreover,
PTQ is also explored for large generative PTMs (e.g. GPT-3 [5]) from 125M to 175B [37, 53, 11].
Conclusion
In this paper, we study post-training quantization for pre-trained language models.
We show that existing quantization-aware training solutions suffer from slow training, huge memory overhead, and data privacy issues when accessing the full training set.
To mitigate these issues, we propose module-wise reconstruction error minimization, an efﬁcient solution to quantize PLMs.
MREM can be conducted either sequentially or in parallel, where the parallel training can achieve the speedup close to the theoretical limit without apparent performance degradation.
Experimental results show that the proposed solution greatly improves the performance.
Meanwhile, it signiﬁcantly reduces the training time and memory overhead with only thousands of training instances.
We leave the broader impact and limitations of this work in Appendix D.
Acknowledgement and Disclosure of Funding
We gratefully acknowledge the support of MindSpore for this research, as well as the insightful suggestions from the anonymous reviewers.
The work described in this paper was partially supported by the National Key Research and Development Program of China (No. 2018AAA0100204), the
Research Grants Council of the Hong Kong Special Administrative Region, China (No. CUHK 2410021 of the Research Impact Fund, No. R5034-18; and No. CUHK 14210920 of the General
Research Fund).
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[Yes]
In the code document.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[No] Their license permits public use for research purpose.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] GLUE and SQuAD are widely used benchmark dataset in NLP research.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]