Towards Efﬁcient Post-training Quantization of Pre-trained Language Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Network quantization has gained increasing attention with the rapid growth oflarge pre-trained language models (PLMs).
However, most existing quantizationmethods for PLMs follow quantization-aware training (QAT) that requires end-toend training with full access to the entire dataset.
Therefore, they suffer from slowtraining, large memory overhead, and data accessibility issues.
In this paper, westudy post-training quantization (PTQ) of PLMs, and propose module-wise quantization error minimization (MREM), an efﬁcient solution to mitigate these issues.
By partitioning the PLM into multiple modules, we minimize the reconstructionerror incurred by quantization for each module.
In addition, we design a new modelparallel training strategy such that each module can be trained locally on separatecomputing devices without waiting for preceding modules, which brings nearly thetheoretical training speed-up (e.g., 4 ⇥ on 4 GPUs).
Experiments on GLUE and
SQuAD benchmarks show that our proposed PTQ solution not only performs closeto QAT, but also enjoys signiﬁcant reductions in training time, memory overhead,and data consumption.
Introduction
Large pre-trained language models (PLMs) have achieved remarkable success in various naturallanguage processing tasks [54, 11, 5].
However, the increasing size and computation overhead alsomake it prohibitive to deploy these PLMs on resource-constrained devices.
To obtain compact PLMs,various model compression methods have been proposed, such as pruning [38, 13], knowledgedistillation [48, 52, 26], weight-sharing [10, 27, 58, 23], dynamic computation with adaptive depth orwidth [19, 64, 75] and quantization [69, 49, 68, 72, 2, 43, 53].
Among these methods, network quantization enjoys the reduction of both model size and computation overhead without modifying the network architecture and is thus extensively studied in PLMquantization [69, 49, 68, 72, 2, 43, 53].
However, most previous methods for PLM quantizationmethods follow quantization-aware training (QAT), which suffers from several issues.
Speciﬁcally,
QAT usually conducts end-to-end back-propagation training over the whole training set, which canbe slow in training time, memory demanding and data consuming.
These issues can be somethimesprohibited for industrial PLMs.
Post-training quantization (PTQ), on the other hand, serves asan appealing alternative that is efﬁcient in training time, memory overhead and data consumption.
Instead of the full training set, PTQ can leverage only a small portion of training data to minimize the layer-wise reconstruction error incurred by quantization [40, 41, 39, 24].
This can bedone by either calibrating the batch normalization statistics [40] or step sizes [41] in quantizationfunctions.
The layer-wise objective also breaks down the end-to-end training, making the problem more sample-efﬁcient [74] and memory-saving.
Nonetheless, we ﬁnd it non-trivial to directlyapply previous PTQ methods for PLMs such as BERT [11], as the performance drops sharply
In this paper, we aim at improving the performanceof post-training quantization for PLM, while simultaneously maintaining its efﬁciency w.r.t training time,memory overhead and data accessibility.
First, wepartition the PLM into multiple modules, and propose module-wise reconstruction error minimization (MREM).
Each module consists of multiple Transformer layers, which permits the sufﬁcient optimizationthan the previous layer-wise objective.
Meanwhile, themodule size can be ﬂexibly adjusted for the propertrade-off between layer-wise correlation and memoryoverhead of computing devices.
While a similar blockwise objective is previously considered in [31], theyrequire second-order Hessian matrices for optimization, which can be computationally prohibitive for large
PLMs.
Second, we design a new model parallel strategy to further accelerate the training process of MREM.
By allocating each partitioned module to the individualcomputing device, our design allows all modules tobe trained jointly without synchronizing with adjacentpartitions.
This brings nearly the theoretical speed-up(e.g., 4 ⇥ on 4 GPUs), superior to previous data parallel [9, 29] or model parallel [22, 42] techniques.
Third, we develop annealed teacher forcing for our parallel strategy.
We ﬁnd that the naive paralleltraining suffers from the propagation of reconstruction error, since each quantized module passesthe error to its successors before it is fully optimized.
Inspired by [63], we use the full-precisionmodule to provide clean signals to the next quantized module.
This breaks the reconstruction errorpropagation and further improves the performance of parallel strategy.
We summarize the contributions of this paper as follows (i)
We study the post-training quantizationof PLMs, and propose module-wise reconstruction error minimization (MREM), a fast, memorysaving, and data-efﬁcient approach to improve the quantized PLMs.
(ii) We design a new modelparallel strategy based on MREM to accelerate post-training quantization with theoretical speed-upfor distributed training.
(iii)
The parallel MREM can be combined with annealed teacher forcing toalleviate the propagation of reconstruction error and boost the performance.
iv) Empirical resultson the GLUE and SQuAD benchmarks demonstrate superiority of our PTQ solution over QAT.
Forinstance, Figure 1 shows that the BERT-large model trained by parallel MREM achieves 85 .
5%accuracy with only 4 K training samples.
Moreover, it consumes merely one-third of memory per
GPU and is more than 150 ⇥ faster than previous QAT training.
Motivation2.1 Quantization Background
Network quantization replaces the original full-precision weight or activation x 2 R m ⇥ n with itslower-bit counterpart ˆ x .
Denoting s 2 R + as the step size, the b -bit symmetric uniform quantizationfunction Q b ( · ) can be written as
[Equation]
) where ⌦ ( b ) = {  2 b  1 + 1 , ..., 0 , ..., 2 b  1  1 } is the set of b -bit integers, and ⇧ ( · ) is the projectionfunction that maps x /s to its closest integer.
To quantize Transformer-based PLMs, we follow the setting in previous works [69, 72, 2]: we quantizeboth the network weights and activations in each matrix multiplication.
We use symmetric uniformquantization for weights, embeddings, and activations, except activations after the self-attention and
GeLU function.
For these two activations, we adopt asymmetric quantization since their elements aremostly positive.
We skip the quantization for all layer-normalization layers, skip connections, biasesand the last classiﬁcation head due to little computation overhead or large performance drop.
Post-training Quantization.
Recently popular PTQ methods [41, 59, 39, 24, 35] constructs antiny unlabeled dataset (a.k.a.
calibration set)
˜ D from the original training data D .
These approaches
[Figure]
usually aim at reconstruction error minimization (REM), i.e., minimizing the distance between theoutput of multiplication between the quantized and the full-precision counterpart as follows:
[Equation]
where w and a are full-precision weights and activations, ˆ w and ˆ a are their quantized representationswith b w and b a bit-widths, and s denotes all step-sizes involved for quantization.
REM is usuallyconducted in a greedy manner.
It proceeds to the matrix multiplication only after the training ofprevious ones.
The simple objective in Equation (2) is lightweight to solve over ˜ D , and is alsotheoretically veriﬁed to be more sample-efﬁcient than end-to-end training [74].
While there arealso other data-free PTQ variants [73, 40, 15, 68], REM-based solutions usually demonstrate betterempirical results [39, 24].
Thus we mainly focus on the REM-based PTQ for PLMs in this paper.
2.2 Quantizing Pre-trained Language Models: QAT or PTQ?
To conceptually study the difference between QAT and REM-based PTQ over popular PLMs, weconsider the following four dimensions, and summarize the results in Figure 2.
Training Time.
As QAT conducts training over the full training set D , it takes much more time than
PTQ over the calibtration set ˜ D .
Recent QAT methods [72, 2] further combine two-stage knowledgedistillation [26], which can take nearly four times longer than FP, as shown in Figure 2(a).
Memory Overhead.
The increasing size of large PLMs makes it prohibited to conduct QAT onmemory-limited devices.
From Figure 2(b), QAT [72] even consumes 8 .
3 GB more memory than FPwhen armed with knowledge distillation.
On the contrary, PTQ only caches intermediate results in
Equation (2), which can be fed into a single GTX 1080 Ti.
Data Accessibility.
Data accessibility can be often prohibited due to data security or privacy issuesin the industry.
Unlike QAT that needs the full training set D , PTQ constructs the calibration data˜ D ✓ D by sampling only 1 K ⇠ 4 K instances from D , as shown in Figure 2(c).
Performance.
QAT usually demonstrates superior quantization results than PTQ, and this holdstrue for PLMs.
From Figure 2(d), QAT remains steady across different bit-widths, while REM-based
PTQ drops sharply.
Thus the poor performance of PTQ is the main challenge in quantizing PLMs.
In summary, REM-based PTQ is superior to QAT with regard to training efﬁciency, memory overhead,and data accessibility.
Nevertheless, it is still often less preferred than QAT due to its severeperformance drop especially for low quantization bit-width [69, 49, 72].
Methodology
In this section, we aim at improving the performance of post-training quantization for PLMs, whilepreserving its merits of fast training, light memory overhead, and data consumption.
We extend theexisting reconstruction error minimization from the layer-wise to the module-wise granularity to ﬁt
Transformer models.
Based on the module partition, we then design a new parallel training strategythat further speeds up the PTQ pipeline.
An overview of our solution can be found in Figure 3.
We partition both the full-precision model and quantized model into multiple modules and put these modules on different computing devices.
By sampling tensors from the input queue, each module can be trained locally without waiting for its predecessors.
Teacher forcing is applied to mitigate the issue of reconstruction error propagation on the quantized module.
3.1 Module-wise Reconstruction Error Minimization
We propose a new PTQ solution called module-wise reconstruction error minimization (MREM)for PLMs.
Existing REM [39] solves Equation (2) for each matrix multiplication.
However, astandard transformer layer in PLMs consists of a Multi-Head Attention (MHA) and a Feed-Forward
Network (FFN), both of which contain a number of matrix multiplications that are coupled together.
Greedily tackling each matrix multiplication in REM can thus lead to suboptimal quantized networks.
Moreover, the insufﬁciently minimized reconstruction error shall propagate and enlarge along withtransformer layers, and ﬁnally deteriorate the network output [7, 1].
Towards that end, the proposed module-wise reconstruction error minimization admits larger granularity by jointly optimizing all the coupled linear layers inside each module.
Speciﬁcally, given a
PLM with L transformer layers, embedding layers and the classiﬁcation head, we partition them into
N modules, where the n -th module include [ l n , l n +1 ) transformer layers with l n being the ﬁrst layerof this module 1 .
MREM aims at minimizing the joint reconstruction errors between all quantized
FFN output ˆ f l in the module from their full-precision counterpart f l as follows:
[Equation]
where w n and s n are all learnable parameters and quantization step sizes in the n -th module.
Similarto REM, MREM can be optimized sequentially: given previously trained modules, only parametersand quantization step sizes in the current module are updated.
Besides the grouped Transformerlayers, we also minimize the MSE loss in the Transformer embedding and output logits respectively.
Note that the number of modules N can be adjusted depending on the memory constraint of computingresources.
When N = 1 , this reduces to intermediate-layer knowledge distillation [26], which can bememory-demanding when quantizing large PLMs on a single GPU.
3.2 Accelerated Parallel Training
Based on the proposed MREM, we propose a new model parallel strategy to further accelerate thetraining.
As shown in Figure 3, we put different modules on individual computing devices.
A set ofinput queues I =
{ I 1 , ..., I N  1 } is deployed between each pair of adjacent modules.
For the n -thmodule, the queue collects its output of the most recent t 0 steps, i.e., I tn =
{ f tl n , f t  1 l n , ..., f t  t 0 +1 l n } .
Meanwhile, the ( n + 1) -th module can always sample with replacement f l n ⇠ I tn from the queuewithout waiting for the n -th module.
Similar rules hold for the quantized module and their inputqueues ˆ I .
The design of the input queue resembles stale synchronous parallel [18] that caches thestale output locally so as to reduce the waiting time among workers, where t 0 is the stale threshold.
The training workﬂow is as follows.
Initially, every module is computed one after another in the ﬁrstt 0 step to ﬁll in the input queue, after which parallel training takes place.
Then the module samples input from the queue and calculates the loss ` ( n ) correspondingly for n = 1 , ..., N .
Meanwhile, theinput queue is updated with the rule of ﬁrst-in-ﬁrst-out throughout the training.
In the backwardpass, we constrain the gradients to propagate locally within each module, without affecting itspredecessors.
Such a design can avoid the load imbalance issue from straggler modules, bringingnearly the theoretical N ⇥ speed-up.
Note that our proposed parallel training is different from pipelineparallel [22].
We analyze their difference in section B.
3.3 Annealed Teaching Forcing
Since all modules get optimized simultaneously instead of the sequential manner, the next moduletakes the output from the queue before its predecessor is fully optimized.
Therefore, the predecessor’sreconstruction error is propagated to the following modules before it is sufﬁciently minimized.
Inspired by teacher forcing [63] in training recurrent networks, the output f l n from the n -th fullprecision module naturally serves as the clean input to the ( n + 1) -th quantized module to substituteˆ f l n .
Thus f l n stops the propagation of the accumulated error on the quantized module.
Nevertheless,such an approach breaks the connection to previous quantized modules and may suffer from forwardinconsistency between training and inference [1] on the quantized model.
For the trade-off, we takethe convex combination between the full-precision f l n and quantized ˆ f l n as follows:
[Equation]
[Equation]
) where the hyperparameter  controls the strength of teacher forcing.
= 1 gives the full correctionof reconstruction error but with forward inconsistency, while  = 0 reduces to the conventionalsetting that suffers from the propagated reconstruction error.
We adopt a linear decay strategy for  : t = {+max(
 +} tT 0 , 0) , where T 0 is the preset maximum steps of the decay.
Intuitively, a large  isdesired at the beginning when each module is rarely optimized.
Later, a small  is preferred to transitto normal training such that the forward inconsistency can be bridged.
The remaining T  T 0 stepsstick to normal training so that each quantized module adapts to its own predecessors.
Finally, an overview of the proposed parallel module-wise reconstruction error minimization withannealed teacher forcing is shown in Algorithm 1 and Algorithm 2.
The Update ( · ) in Algorithm 2can be any gradient update function such as AdamW [36] with learning rate ⌘ t .
[Table] [Table]
[Table]
[Table] [Table]
Experiments
In this section, we empirically verify the proposed MREM for post-training quantization of PLMs.
We ﬁrst introduce the experimental setup in Section 4.1.
Then we present main results in Section 4.2,including comparisons with QAT and REM, and other existing quantization baselines.
In Section 4.4,we provide discussions on a variety of factors in our approach, such as the effect of teacher forcing,the number of model partitions, and calibration data size.
Code will be released upon acceptance.
4.1 Experimental Setup
Datasets and Metrics.
We evaluate post-training quantization on both the GLUE [55], and SQuADbenchmarks [45].
The size of calibration data is by default | ˜ D|
= 4 , 096 , by randomly sampling
[Table] [Table]
instances from the full training set.
As both RTE and MRPC tasks in the GLUE benchmark containfewer than 4,096 samples, we use their full training set on these two tasks.
We leave the study of datasize in Section 4.4.
Each experiment is repeated ten times with different calibration sets, and both themean and standard deviations are reported.
We use the same evaluation metrics in [11, 72] for the development set of GLUE and SQuADbenchmarks.
For results in Section 4.2, we report accuracies on both the matched section andmis-matched sections of MNLI, and EM (exact match) and F1 score for SQuAD.
Additionally, wealso report the training time (min), memory overhead (GB) as well as the size of the training set (K).
We also provide comparisons with other existing methods in Section 4.3, where we adopt Matthewscorrelation for CoLA, Spearman correlation for STS-B, and accuracy for the rest ones (i.e., RTE,
MRPC, SST-2, QQP, MNLI).
We also report the averaged performance on GLUE as an overview.
Implementation.
We use the standardly ﬁne-tuned BERT-base and BERT-large models 2 on downstream tasks for both quantization-aware training and post-training quantization.
We implement MREM in both the sequential training (abbv.
MREM-S) in Section 3.1 and parallel training withteaching forcing (abbv.
MREM-P) in Section 3.3.
For each module, we train for 2 , 000 steps withan initial learning rate of 1e-4 on GLUE tasks, and 4 , 000 steps with an initial learning rate of 5e-5on SQuAD datasets.
The learning rate decays linearly as done in [26, 72].
By default, we partitionthe model into 4 modules on [-
-]4 NVIDIA-V100 GPU cards.
For baselines, we mainly compare with QAT and REM, where the former measures how far is PTQ from QAT, and the latter studies theeffect of module-wise granularity in PTQ training.
For a fair comparison of each method, we use thesame quantization scheme for all methods, i.e., TWN [28] or LAQ [20] for 2-bit and 4-bit weight
[Table] [Table]
[Table]
[Table]
quantization, and LSQ [12] for all activation quantization.
Unlike QAT that picks the best modelbased on the development set results, MREM is only tested once after training, which does notrequire further access to the development set.
We leave more details of baseline implementation in Appendix C.1.
The comparison with more published quantization baselines are left in Section 4.3.
4.2 Main Results: Comparison with QAT and REM
We ﬁrst compare MREM-S and MREM-P with QAT and REM over MNLI and SQuAD benchmarks.
We take BERT-base and BERT-large as backbone PLMs for quantization.
The results on MNLI and
SQuADv1.1 are summarized in Table 1 and Table 2 respectively, and results on SQuAD
v2.0 are leftto Table 6 in Appendix C.2.
We again summarize according to the four dimensions in Section 2.2.
Performance.
It can be found that our proposed MREM-S improves the performance of REMsigniﬁcantly given the same training time, and is much closer to QAT.
For instance, according to in
Table 1, MREM-S with 4-bit weight quantization on BERT-base and BERT-large achieves accuraciesof 83 .
5% ± 0 .
1 and 86 .
1% ± 0 .
1 on the matched section of MNLI, which is on average 10 . 2% " and 225 16 .
1% " better than REM, and only 1 . 1% # and 0 .
8% # inferior to QAT, respectively.
Moreover,with all modules trained in parallel, MREM-P is still close to or only slightly inferior to MREM-S.
From Table 2, MREM-P can even outperform MREM-S with the “W2-E2-A4” quantized BERT-largemodel on SQuAD 1.1, i.e., the EM score and F1 score are on average 0 .
4% " and 0 .
2% " respectively.
Training Time.
Our proposed MREM also enjoys signiﬁcantly less training time than QAT.
From
Table 1, MREM only takes 84 minutes to quantize BERT-large with 4 -bit weights, which is about 38 ⇥faster than QAT and 7 ⇥ faster than full-precision ﬁne-tuning.
Note that the full-precision ﬁne-tuningtime is listed to conceptually compare how much time is further required for quantization.
Comparingwith REM, MREM does not cache the output after every matrix multiplication, which admits moreiterations given the same amount of time.
We discuss this further in Appendix C.3.
Moreover, MREM-P is further 4 ⇥ faster than MREM-S, which achieves the theoretical linear speedup on {+
+} GPUs.
These together bring more than 150 ⇥ reduction of training time when compared with QAT.
Memory Overhead.
Our MREM-S and MREM-P take only around a third of the GPU memoryby QAT, and a half of that by the full-precision ﬁne-tuning, e.g., 29 . 8 GB for QAT and 10 .
8 GB for
MREM-S on BERT-large.
Moreover, we also encounter memory overﬂow (more than 32GB memoryof an NVIDIA V100 GPU card) for QAT training on SQuAD due to longer sequence length, and weadopt gradient accumulation that inevitably doubles the training time (i.e., underlined ﬁgures (“ ”) [Figure] [Figure] in Table 2).
On the other hand, such issues can be easily mitigated in both REM and our proposed
MREM, both of which can be even fed into a single NVIDIA GTX 1080 Ti GPU card.
We mayfurther decrease the memory overhead of MREM to REM by increasing the number of modules, butthis could harm the performance as discussed in Section 4.4.
Data Accessibility.
Both REM and our proposed MREM follow the common practice of PTQ,relying on only 4 , 096 randomly sampled instances on both MNLI and SQuAD, which is a tiny fractionof the original dataset.
We shall provide more discussion on the calibration size in Section 4.4.
4.3 Main Results: Comparison with Existing Methods
Next, we compare our MREM with existing state-of-the-art BERT quantization methods.
Theyinclude various QAT approaches such as Q-BERT [49], Quant-Noise [14], TernaryBERT [72], andthe PTQ baseline GOBO [68].
The results are from the original papers.
From Table 3, both our proposed MREM-S and MREM-P outperform GOBO (the existing PTQapproach) in most cases, and are even close to QAT approaches in multiple entries.
For example,the “W4-E4-A8” quantized MREM-S and MREM-P have the mean accuracies of 83 .
5% and 83 .
4%on MNLI respectively, both of which are on par with “W2/4-E8-A8” quantized Q-BERT.
For “W2E2-A8” quantized models, MREM-S and MREM-P surpass GOBO by 11.7 % " and 11.3 % " on MNLI-m respectively.
4.4 Discussions
In this section, we provide further discussions to better understand the proposed approach.
Unlessspeciﬁed otherwise, all experiments are built upon the BERT-base model over the MNLI dataset
Teacher Forcing.
We now study how teacher forcing beneﬁts MREM-P with different numbersof training steps, and results are listed in Table 4.
We ﬁnd that teacher forcing brings consistentimprovement for both BERT-base and BERT-large, and the gain is more signiﬁcant with fewertraining steps or lower quantization bit-width.
For example, it brings 3.4 % " and 2.8 % " on the“W2-E2-A4” quantized BERT-base and BERT-large respectively under 250 steps.
This matches our intuition that fewer training steps or higher compression ratio give larger reconstruction error, whenthe clean input from the full-precision module can beneﬁt more the quantized module.
As moretraining steps bring only marginal improvement, we by default set the training steps to 2 , 000 .
Additionally, we also plot training loss curves of the 2 -nd and 4 -th modules under 250 and 2,000training steps in Figure 4, and more visualizations can be found in Figure 6 of Appendix C.4.
We ﬁndthat: 1) the loss curves with teacher forcing are apparently lower, especially when trained with fewersteps, which is consistent with Table 4; 2) the loss curves rise in the halfway during teacher forcing,since the modules need to adapt to quantized input with accumulated errors; 3) the loss curves of the4 -th modules are lower than the 2 -nd ones, which matches the intuition that the late modules havemore errors accumulated and thus beneﬁt more from teacher forcing.
We also try tuning the teacherforcing steps (i.e., shaded area) within [20% , 80%] of total iterations, but observe no large differencein performance.
We thus choose 40% training steps for teacher forcing by default.
Number of Modules and Memory Overhead.
We verify the effect of model partition on theﬁnal quantized performance, as well as their corresponding memory consumption.
According to
Figure 5(a), by varying the number of modules within { 1 , 2 , 3 , 4 , 6 } , it can be found that moremodel partitions give slightly lower performance, as layer-wise dependencies are less consideredduring reconstruction error minimization.
However, more partitions lead to less running memory,i.e., { 11 . 9 , 6 . 7 , 4 . 7 , 3 . 7 , 2 . 7 } GB for these partitions correspondingly.
As the decrease of memorydiminishes with more partitions, we partition the model into 4 modules by default.
Size of Calibration Data.
The size of calibration data directly relates to the data accessibilityissue in post-training quantization.
To learn its effects, we vary the calibration data size | ˜ D| within{ 32 , 64 , 128 , 512 , 1024 , 2048 , 4096 , 8192 } , and list the results of REM, MREM-S and MREM-P.
From Figure 5(b), it can be found that while REM is ahead of MREM-S/P with fewer than 128training samples, the accuracy of REM rises slowly and saturates at around 60% afterwards.
Wehypothesize that the simple training objective in REM can hardly hold more training instances foroptimization.
MREM-S/P, on the other hand, can better exploit larger calibration data size, since themodule-wise granularity admits higher ﬂexibility for the optimization.
As we ﬁnd the diminishinggain to increase the training size after 4 , 096 samples, we by default take 4 , 096 samples.
Reconstruction Error Propagation.
We visualize the propagation of reconstruction error forboth “W2-E2-A8” and “W2-E2-A4” quantized BERT-base models in Figure 5(c) and Figure 5(d)respectively.
It can be observed that our MREM achieves both lower values and slower rising rates ofthe reconstruction error than REM across all layers, which veriﬁes the advantage of module-wisegranularity to minimize the reconstruction error.
Interestingly, while the reconstruction error generallygets enlarged layer-wisely in the ﬁrst ten layers, it begins to decrease afterwards.
We speculate this isdue to the classiﬁcation head that encourages concentrated hidden representations for the task.
Per-channel Quantization.
Per-channel Quantization (PCQ) is prevalent in the post-trainingquantization of convolution neural networks [41, 39, 24].
To learn its effect in PLMs, PCQ assignsdifferent quantization step-sizes at each output dimension of the linear layer, which is also known asrow-wise quantization in [72].
The PCQ results of REM and MREM are shown in Table 5.
It canbe found that while PCQ improves REM by around 1 . 0% to 2 .
5% , the gain is very incremental on
MREM.
Our results are also similar to the ﬁndings in [72], where the row-wise quantization bringslittle improvement.
As PCQ requires to store more full-precision step sizes with minor improvement,we do not employ PCQ by default.
Conclusion
In this paper, we study post-training quantization for pre-trained language models.
We show thatexisting quantization-aware training solutions suffer from slow training, huge memory overhead,and data privacy issues when accessing the full training set.
To mitigate these issues, we proposemodule-wise reconstruction error minimization, an efﬁcient solution to quantize PLMs.
MREM canbe conducted either sequentially or in parallel, where the parallel training can achieve the speedupclose to the theoretical limit without apparent performance degradation.
Experimental results showthat the proposed solution greatly improves the performance.
Meanwhile, it signiﬁcantly reduces thetraining time and memory overhead with only thousands of training instances.
We leave the broaderimpact and limitations of this work in Appendix D.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ??
• Did you include the license to the code and datasets?
[No] The code and the data are
proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes]
In the code document.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No] Their license permits public use for research purpose.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No] GLUE and SQuAD are widely used benchmark
dataset in NLP research.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]