GPTQ: A CCURATE P OST -T RAINING Q UANTIZATION FOR G ENERATIVE P RE - TRAINED T RANSFORMERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Generative Pre-trained Transformer (GPT) models have set themselves apart by breakthrough performance across complex language modelling tasks, but also by their extremely high computational costs.
Specifically, due to memory costs, even inference for large, highly-accurate GPT models may require multiple performant
GPUs to execute, which limits their usability.
While there is emerging work on relieving this pressure via model compression, the applicability and performance of existing compression techniques is limited by the scale and complexity of GPT models.
In this paper, we address this challenge, and propose GPTQ, a new oneshot weight quantization method based on approximate second-order information, that is both highly-accurate and highly-efficient.
Specifically, GPTQ can quantize GPT models with 175 billion parameters in approximately four GPU hours, reducing the bitwidth down to 3 or 4 bits per weight, with negligible accuracy degradation relative to the uncompressed baseline.
Our method more than doubles the compression gains relative to previously-proposed one-shot quantization methods, preserving accuracy, allowing us for the first time to execute anbillion-parameter model inside a single GPU.
We show experimentally that these improvements can be leveraged for end-to-end inference speedups over FP16, of around 2x when using high-end GPUs (NVIDIA A100) and 4x when using more cost-effective ones (NVIDIA A6000).
I NTRODUCTION
Pre-trained generative models from the Transformer (Vaswani et al., 2017) family, commonly known as GPT or OPT (Radford et al., 2019; Brown et al., 2020; Zhang et al., 2022), have shown breakthrough performance for complex language modelling tasks, leading to massive academic and practical interest.
One major obstacle to the usability of such models is their computational and storage cost, which ranks among the highest for known models.
For instance, the best-performing model variants, e.g. GPT3-175B, have in the order of 175 billion parameters and require tens-to-hundreds of GPU years to train (Zhang et al., 2022).
Even the simpler task of inferencing over a pre-trained model, which is our focus in this paper, is highly challenging: for instance, the parameters of GPT3175B occupy 326GB (counting in multiples of 1024) of memory when stored in a compact float format.
This exceeds the capacity of even the highest-end single GPUs, and thus inference must be performed using more complex and expensive setups, such as multi-GPU deployments.
Although a standard approach to eliminating these overheads is model compression , e.g. (Hoefler et al., 2021; Gholami et al., 2021), surprisingly little is known about compressing such models for inference.
One reason is that more complex methods for low-bitwidth quantization or model pruning usually require model retraining , which is extremely expensive for billion-parameter models.
Alternatively, post-training methods (Nagel et al., 2020; Wang et al., 2020; Hubara et al., 2020;
Nahshan et al., 2021), which compress the model in one shot, without retraining, would be very appealing.
Unfortunately, the more accurate variants of such methods (Li et al., 2021; Hubara et al., 2021; Frantar et al., 2022) are complex, and do not scale to billions of parameters (Yao et al., 2022).
To date, only basic variants of round-to-nearest quantization (Yao et al., 2022; Dettmers et al., 2022) have been applied at the scale of GPT-175B; while this works well for low compression targets, e.g., 8-bit weights, they fail to preserve accuracy at higher rates.
It therefore remains open whether one-shot post-training quantization to higher compression rates is generally-feasible.
Contribution.
In this paper, we present a new post-training quantization method, called GPTQ,which is efficient enough to execute on models with hundreds of billions of parameters in at most
[Figure]
a few hours, and precise enough to compress such models to 3 or 4 bits per parameter without significant loss of accuracy.
GPTQ is the first method to leverage approximate second-order (Hessian) information at the scale of > 100 billion parameter models, and can be implemented extremely efficiently on modern GPUs.
For illustration, GPTQ can quantize the largest publicly-available models,
OPT-175B and BLOOM-176B, in approximately four GPU hours, with minimal increase in perplexity, known to be a very stringent accuracy metric.
In addition, we develop an execution harness which allows us to execute the resulting compressed models efficiently for generative tasks.
Specifically, we are able to run the compressed OPT-175B model for the first time on a single NVIDIA
A100 GPU, or using only two more cost-effective NVIDIA A6000 GPUs.
We also implement bespoke GPU kernels which are able to leverage compression for faster memory transfer, resulting in speedups of ≈ 2 × when using A100 GPUs, and 4 × when using A6000 GPUs.
At a high level, our work tackles a standard problem in the low-precision quantization literature (Nagel et al., 2020; Li et al., 2021; Hubara et al., 2021; Yao et al., 2022; Frantar et al., 2022): for each layer, we wish to identify an assignment of standard-precision weights to quantization levels, which minimizes the discrepancy between the output of the original layer and that of the compressed one.
Usually, this is measured in ℓ 2 distance on a small amount of calibration data.
Once such accurate layer-wise weight assignments are found, the layers can be “stitched together” to reconstruct the quantized model.
This key layer-wise assignment problem is approached via increasingly-sophisticated solvers, using integer programming (Hubara et al., 2020) or second-order approximations (Li et al., 2021; Frantar et al., 2022).
Unfortunately, these approaches simply do not scale to large models from the GPT family (Yao et al., 2022).
More precisely, the GPTQ algorithm adopts a similar strategy to the recently-proposed Optimal
Brain Quantization (OBQ) solver (Frantar et al., 2022), which quantizes weights one-at-a-time , updating the remaining unquantized weights to minimize the layer’s output error.
Both the choice of the next weight to quantize and the weight update leverage second-order information, which is reasonable to work with given that the output error is quadratic.
Clearly, this approach is impractical at GPT scale: we estimate that quantizing GPT-175B using existing accurate approaches would take between 2 weeks and 6 months, even completely ignoring memory constraints.
In this context, the first key idea behind our approach is that there exists a way to quantize weights in large blocks that is orders of magnitude more efficient relative to the OBQ approach, but that can still leverage second-order information so that it stays highly accurate .
Concretely, central to
GPTQ is a new second-order quantization algorithm with per-layer runtime
O ( d 2 col · max { d row , d col } ) , where d row and d col are the layer matrix’s row and column dimensions, respectively.
Asymptotically, this is faster than OBQ by a factor of Θ( min { d row , d col } ) , which is often around three orders of magnitude in practice.
Yet, this algorithmic insight is not sufficient for a fast practical algorithm: we additionally address a number of barriers, including the low compute-to-memory ratio of the algorithm, and numerical stability issues when working with (inverse) matrices at this scale.
We implemented the resulting GPTQ algorithm in Pytorch (Paszke et al., 2019), and used it to quantize publicly-available models from the OPT (Zhang et al., 2022) and BLOOM (Laurenc¸on et al., 2022) model families, counting from 125M to 176B parameters.
Our results show that, both for standard perplexity tasks, e.g., WikiText2 (Merity et al., 2016) or C4 (Raffel et al., 2020) and for standard zero-shot tasks, e.g. LAMBADA (Paperno et al., 2016), our method is able to compress models down to 4 or even 3 bits per weight with negligible loss of accuracy.
Moreover, GPTQ compresses the largest models in approximately 4 GPU hours, and can execute on a single GPU.
As illustrated in Figure 1, relative to prior work, GPTQ is the first method to reliably compress LLMs to 4 bits or less, more than doubling compression at a similar level of accuracy relative to prior work, and allowing for the first time to fit an OPT-175B model for inference inside a single GPU.
To realize speedups in practice, we observe that our method particularly benefits generative inference, which is one of most interesting use-cases for GPT models.
Specifically, generative inference produces outputs in token-by-token fashion, in an autoregressive manner.
Thus, the method cannot benefit from batching, and so most of the execution reduces to matrix-vector products, which have low arithmetic intensity and are bottlenecked by the memory subsystem.
By quantizing the model, we can significantly reduce the number of necessary slow memory accesses while introducing only minor extra compute overhead for weight unpacking, leading to significant overall inference speedups.
We illustrate this technique by implementing a fast GPU kernel, which allows us to maintain weights in quantized format, dynamically “decoding” them as they are required at execution time.
We show that this approach can result in end-to-end speedups of 1.9-4x on modern GPUs.
To our knowledge, we are the first to show that extremely accurate language models with hundreds of billions of parameters can be quantized to 3-4 bits/component: prior post-training methods only remain accurate at 8 bits (Yao et al., 2022; Dettmers et al., 2022), while prior training-based techniques have only tackled models that are smaller by one to two orders of magnitude (Wu et al., 2022).
This high degree of compression may appear unsurprising, as these networks are overparametrized; yet, as we discuss in our detailed analysis of results, compression induces non-trivial tradeoffs between the accuracy of the language modeling (perplexity), bit-width, and the size of the original model.
We hope that our work will stimulate further research in this area, and can be a further step towards making these models available to a wider audience.
In terms of limitations, our method currently does not provide speedups for the actual multiplications, due to the lack of hardware support for mixed-precision operands (e.g. FP16 x INT4) on mainstream architectures.
Moreover, our current results do not include activation quantization, as they are not a significant bottleneck in our target scenarios; however, this can be supported using orthogonal techniques (Yao et al., 2022).
R ELATED W ORK
Quantization methods fall broadly into two categories: quantization during training, and posttraining methods.
The former quantize models during typically extensive retraining and/or finetuning, using some approximate differentiation mechanism for the rounding operation (Gholami et al., 2021; Nagel et al., 2021).
By contrast, post-training (“one-shot”)
methods quantize a pretrained model using modest resources, typically a few thousand data samples and a few hours of computation.
Post-training approaches are particularly interesting for huge models, for which full model training or even finetuning can be extremely resource intensive.
We thus focus on this scenario here.
Post-training Quantization.
Historically, post-training methods have focused on vision models: for instance, quantizing the ResNet50 model (He et al., 2016) is as a standard benchmark.
Usually, accurate methods operate by quantizing either individual layers, or small blocks of consecutive layers.
(See Section 3.1 for more details.)
The AdaRound method (Nagel et al., 2020) computes a datadependent rounding by annealing a penalty term, which encourages weights to move towards grid points corresponding to quantization levels.
BitSplit (Wang et al., 2020) constructs quantized values bit-by-bit using a squared error objective on the residual error, while AdaQuant (Hubara et al., 2021) performs optimization of the quantization levels based on straight-through estimation.
BRECQ
(Li et al., 2021) introduces Fisher information into the objective, and optimizes layers within a single residual block jointly.
Finally, Optimal Brain Quantization (OBQ) (Frantar et al., 2022) generalizes the classic Optimal Brain Surgeon (OBS) second-order weight pruning framework (Hassibi et al., 1993; Singh & Alistarh, 2020; Frantar et al., 2021) to apply to quantization.
OBQ quantizes weights one-by-one, in order of quantization error, adjusting the remaining weights.
While these approaches can produce good results for models up to ≈ 100 million parameters in a few GPU hours, they will not scale to networks orders of magnitude larger.
We compare against all these methods on the ResNet18/ResNet50 benchmark (see Table 1).
Although it is not our goal to compete with highlyaccurate methods on smaller models, GPTQ is in fact competitive in terms of accuracy, while being at least an order of magnitude faster (at this smaller model scale) than existing PTQ methods.
Large-model Quantization.
With the recent open-source releases of models like
BLOOM (Laurenc¸on et al., 2022) or OPT-175B (Zhang et al., 2022), researchers have started to develop affordable methods for compressing such giant networks for inference.
To our knowl- edge, all existing works—ZeroQuant (Yao et al., 2022), LLM.int8() (Dettmers et al., 2022), and nuQmm (Park et al., 2022)—while they carefully select quantization granularity, e.g., vector-wise, they ultimately just round weights to the nearest (RTN) quantization level to maintain acceptable runtimes for very large models.
ZeroQuant further proposes layer-wise knowledge distillation, similar to AdaQuant, but the largest model it can apply this approach to has only 1.3 billion parameters.
At this scale, ZeroQuant already takes ≈ 3 hours of compute: by our conservative estimate, ZeroQuant would require at least 500 hours to quantize OPT-175B. LLM.int8() observes that activation outliers in a few feature dimensions break the quantization of larger models, and proposes to fix this problem by keeping those dimensions in higher precision.
Lastly, nuQmm develops efficient GPU kernels for a binary-coding based quantization scheme, which can, at low bit-width, bring significant speedups over full precision, when executing single layers in isolation.
Relative to this line of work, we show that a significantly more complex and accurate quantizer can be implemented efficiently at large model scale.
Specifically, GPTQ more than doubles the amount of compression relative to these prior techniques, at similar accuracy, as it is the first method to achieve accurate quantization to 3-4 bits per weight, which brings significant practical gains.
B ACKGROUND & M ETHODS 3. L AYER -W ISE Q UANTIZATION
At a high level, our method follows the structure of state-of-the-art post-training quantization methods (Nagel et al., 2020; Wang et al., 2020; Hubara et al., 2021; Frantar et al., 2022), by performing quantization layer-by-layer, solving a corresponding reconstruction problem for each layer.
Concretely, let W ℓ be the weights corresponding to a linear layer ℓ and let X ℓ denote the layer input corresponding to a small set of m data points running through the network.
Then, the objective is to find a matrix of quantized weights (cid:99) W which minimizes the squared error, relative to the full precision layer output.
Formally, this can be restated as
[Equation]
Further, similar to (Nagel et al., 2020; Li et al., 2021; Frantar et al., 2022), we assume that the quantization grid for (cid:99) W is fixed before the process, and that individual weights can move freely as in (Hubara et al., 2021; Frantar et al., 2022).
3. O PTIMAL B RAIN Q UANTIZATION (OBQ)
Our approach builds on the recently-proposed Optimal Brain Quanization (OBQ) method (Frantar et al., 2022) for solving the layer-wise quantization problem defined above, to which we perform a series of major modifications, which allow it to scale to large language models, providing more than three orders of magnitude computational speedup.
To aid understanding, we first briefly summarize the original OBQ method.
The OBQ method starts from the observation that Equation (1) can be written as the sum of the squared errors, over each row of W .
Then, OBQ handles each row w independently, quantizing one weight at a time while always updating all not-yet-quantized weights, in order to compensate for the error incurred by quantizing a single weight.
Since the corresponding objective is a quadratic, whose Hessian is H F = 2 X F X ⊤ F , where F denotes the set of remaining full-precision weights, the greedy-optimal weight to quantize next, which we denote by w q , and the corresponding optimal update of all weights in F , denoted by δ F , are given by the following formulas, where quant ( w ) rounds w to the nearest value on the quantization grid:
[Equation]
OBQ quantizes weights iteratively using these two equations, until all the weights of w are quantized.
This is done efficiently, avoiding expensive full recomputations of H − 1 , by removing the q th row and column of H , which is necessary after quantizing w q , directly in the inverse via one step of
Gaussian elimination.
Namely, the updated inverse is given by the formula
[Equation]
This method comes with a vectorized implementation, handling multiple rows of W in parallel.
Eventually, the algorithm can achieve reasonable runtimes on medium-sized models: for instance, it can fully quantize the ResNet-50 model (25M parameters) in ≈ 1 hour on a single GPU, which is roughly in line with other post-training methods achieving state-of-the-art accuracy (Frantar et al., 2022).
However, the fact that OBQ’s runtime for a d row × d col matrix W has cubic input dependency
O ( d row · d 3 col ) means that applying it to models with billions of parameters is impossible.
3. T HE GPTQ A LGORITHM
Step 1: Initial Insight.
As explained in the previous section, OBQ quantizes weights in greedy order, i.e. it always picks the weight which currently incurs the least additional quantization error.
Interestingly, we find that, while this quite natural strategy does indeed seem to perform very well, its improvement over quantizing the weights in arbitrary order is generally small, in particular on large, heavily-parametrized layers.
Most likely, this is because the slightly lower number of quantized weights with large individual error is balanced out by those weights being quantized towards the end of the process, when only few other unquantized weights that can be adjusted for compensation remain.
As we will now discuss, the simple insight that any fixed order may perform well , especially on large models, has interesting ramifications.
[Figure]
The quantization procedure is applied recursively
The original OBQ method quantizes rows of W independently, in a specific order defined by the corresponding errors.
By contrast, we will aim to quantize the weights of all rows in the same order , and will show that this typically yields results with a final squared error that is similar to the original solutions.
As a consequence, the set of unquantized weights F and similarly H − 1 F is always the same for all rows.
(Please see Figure 2 for an illustration.)
In more detail, the latter is due to the fact that H F depends only on the layer inputs X F , which are the same for all rows, and not on any weights.
Therefore, we have to perform the update of H − 1 F given by
Equation (3) only d col times, once per column, rather than d row · d col times, once per weight.
This reduces the overall runtime from O ( d row · d 3 col ) to O ( max { d row · d 2 col , d 3 col } ) , i.e., by a factor of min { d row , d col } .
For larger models, this difference consists of several orders of magnitude.
However, before this algorithm can actually be applied to very large models in practice, two additional major problems need to be addressed.
Step 2: Achieving Fast Runtime.
First, a direct implementation of the scheme described previously will not be fast in practice, because the algorithm has a relatively low compute-to-memory-access ratio.
For example, Equation (3) needs to update all elements of a potentially huge matrix using just a few FLOPs for each entry.
Such operations cannot properly utilize the massive compute capabilities of modern GPUs, and will be bottlenecked by the significantly lower memory bandwidth.
Fortunately, this problem can be resolved by the following observation: The final rounding decisions for column i are only affected by updates performed on this very column, and so updates to later columns are irrelevant at this point in the process.
This makes it possible to “batch” updates together, thus achieving much better GPU utilization.
Concretely, we apply the algorithm to B =columns at a time, keeping updates contained to those columns and the corresponding B × B block of H − 1 .
(Please see Figure 2 for an illustration.)
Only once a block has been fully processed, we perform global updates of the entire H − 1 and W matrices using the multi-weight versions of Equations (2) and (3) given below, with Q denoting a set of indices, and H − 1 − Q denoting the inverse matrix with the corresponding rows and columns removed:
[Equation]
Although this strategy does not reduce the theoretical amount of compute, it effectively addresses the memory-throughput bottleneck.
This provides an order of magnitude speedup for very large models in practice, making it a critical component of our algorithm.
Step 3: Numerical Stability.
The final technical issue we have to address is given by numerical inaccuracies, which can become a major problem at the scale of existing models, especially when combined with the block updates discussed in the previous step.
Specifically, it can occur that the matrix H − 1 F becomes indefinite, which we notice can cause the algorithm to aggressively update the remaining weights in incorrect directions, resulting in an arbitrarily-bad quantization of the corresponding layer.
In practice, we observed that the probability of this happening increases with model size: concretely, it almost certainly occurs for at least a few layers on models that are larger than a few billion parameters.
The main issue appears to be the repeated applications of Equation (5), which accumulate various numerical errors, especially through the additional matrix inversion.
For smaller models, applying dampening, that is adding a small constant λ (we always choose 1% of the average diagonal value) to the diagonal elements of H appears to be sufficient to avoid numerical issues.
However, larger models require a more robust and general approach.
To address this, we begin by noting that the only information required from H − 1 F q , where F q denotes the set of unquantized weights when quantizing weight q , is row q , or more precisely, the elements in this row starting with the diagonal.
The consequence is that we could precompute all of these rows using a more numerically-stable method without any significant increase in memory consumption.
Indeed, the row removal via (3) for our symmetric H − 1 essentially corresponds to taking a Cholesky decomposition, except for the minor difference that the latter divides row q by (cid:113)
[ H − 1 F q ] qq .
Hence, we can leverage state-of-the-art Cholesky kernels to compute all information we will need from H −upfront.
In combination with mild dampening, the resulting method is robust enough to execute on huge models without issues.
As a bonus, using a well-optimized Cholesky kernel also yields further speedup.
We detail all small changes necessary for the Cholesky version of the algorithm next.
The Full Algorithm.
Finally, we present the full pseudocode for our algorithm, including the optimizations discussed above.
We provide an efficient Pytorch implementation in the Supplementary.
Algorithm 1 Quantize W given inverse Hessian H − 1 = (2 XX ⊤ + λ I ) − 1 and blocksize B .
Q ← 0 d row ×
d col // quantized output E ← 0 d row × B // block quantization errors
H − 1 ← Cholesky ( H − 1 ) //
Hessian inverse information for i = 0 , B, 2 B, . . .
do for j = i, . .
, i + B − 1 do Q : ,j ← quant ( W : ,j ) // quantize column
E : ,j − i ← ( W : ,j − Q : ,j ) /
[ H − 1 ] jj // quantization error
W : ,j :( i + B ) ← E : ,j − i · H − 1 j,j :( i + B ) // update weights in block end for W : , ( i + B ): ← E · H − 1 i :( i + B ) , ( i + B ): // update all remaining weights end for
E XPERIMENTAL V ALIDATION
Overview.
We begin our experiments by validating the accuracy of GPTQ relative to other accuratebut-expensive quantizers, on smaller models, for which these methods provide reasonable runtimes.
Next, we examine GPTQ’s runtime scaling for very large models.
Then, we present 3- and 4-bit quantization results for the entire BLOOM and OPT model families, evaluated via perplexity on challenging language generation tasks.
To complement this perplexity analysis, we also evaluate the resulting quantized models on a series of standard zero-shot tasks.
Finally, we focus on the two largest (and interesting) openly-available models, Bloom-176B and OPT-175B, where we perform a detailed evaluation on several tasks.
For these models, we also present practical improvements, namely reducing the number of GPUs required for inference as well as end-to-end speedups for generative tasks.
Setup.
We implemented GPTQ in PyTorch (Paszke et al., 2019) and worked with the HuggingFace integrations of the BLOOM (Laurenc¸on et al., 2022) and OPT (Zhang et al., 2022) model families.
We quantized all models (including the 175 billion parameter variants) using a single NVIDIA A
GPU with 80GB of memory.
Our entire GPTQ calibration data consists of 128 random 2048 token segments from the C4 dataset (Raffel et al., 2020), i.e., excerpts from randomly crawled websites, which represents generic text data.
We emphasize that this means that GPTQ does not see any taskspecific data, and our results thus remain truly “zero-shot”.
We perform standard uniform per-row asymmetric quantization on the min-max grid, similar to Dettmers et al.
To ensure that the entire compression procedure can be performed with significantly less GPU memory than what would be required to run the full precision model, some care must be taken.
Specifically, we always load one Transformer block, consisting of 6 layers, at a time into GPU memory and then accumulate the layer-Hessians and perform quantization.
Finally, the current block inputs are sent through the fully quantized block again to produce the new inputs for the quantization of the next block.
Hence, the quantization process operates not on the layer inputs in the full precision model but on the actual layer inputs in the already partially quantized one.
We find that this brings noticeable improvements at negligible extra cost.
Baselines.
Our primary baseline, denoted by RTN, consists of rounding all weights to the nearest quantized value on exactly the same asymmetric per-row grid that is also used for GPTQ, meaning that it corresponds precisely to the state-of-the-art weight quantization of LLM.int8().
This is currently the method of choice in all works on quantization of very large language models (Dettmers et al., 2022; Yao et al., 2022; Park et al., 2022): its runtime scales well to networks with many billions of parameters, as it simply performs direct rounding.
As we will also discuss in detail, more accurate methods, such as AdaRound (Nagel et al., 2020) or BRECQ (Li et al., 2021), are currently far too slow for models with many billions of parameters, the main focus of this work.
Nevertheless, we also show that GPTQ is competitive with such methods for small models, while scaling to huge ones like OPT-175B.
Quantizing Small Models.
As a first ablation study, we compare GPTQ’s performance relative to state-of-the-art post-training quantization (PTQ) methods, on ResNet18 and ResNet50, which are standard PTQ benchmarks, in the same setup as (Frantar et al., 2022).
As can be seen in Table 1,
GPTQ performs on par at 4-bit, and slightly worse than the most accurate methods at 3-bit.
At the same time, it significantly outperforms AdaQuant, the fastest amongst prior PTQ methods.
Further, we compare against the full greedy OBQ method on two smaller language models: BERT-base (Devlin et al., 2019) and OPT-125M.
The results are shown in Appendix Table 6.
At 4 bits, both methods perform similarly, and for 3 bits, GPTQ surprisingly performs slightly better.
We suspect that this is because some of the additional heuristics used by OBQ, such as early outlier rounding, might require careful adjustments for optimal performance on non-vision models.
Overall, GPTQ appears to be competitive with state-of-the-art post-training methods for smaller models, while taking only < 1 minute rather than ≈ 1 hour.
This enables scaling to much larger models.
[Table] [Table]
Runtime.
Figure 3 compares GPTQ’s runtime for fully quantizing OPT models of varying size on a single NVIDIA A100 GPU, with runtime estimates for other post-training quantization approaches (runtime numbers for all models are provided in Appendix A.2).
Concretely, we compare with methods based on the straight-through estimator (STE)
(Yao et al., 2022; Hubara et al., 2021).
We estimate their runtime by linearly extrapolating the results of the most efficient implementation, ZeroQuant-LKD (Yao et al., 2022), taking ≈ 1 hour per 350 million parameters.
Further, we estimate the runtime of rounding-based methods (Nagel et al., 2020; Li et al., 2021), which are significantly more accurate, but typically perform 10 × more SGD steps.
Hence, we optimistically approximate their runtime as 10 × the estimate for STE methods.
While GPTQ can quantizebillion parameter models in ≈ 4 hours, this would take current STE methods about 3 weeks, and current adaptive rounding methods would need half a year.
Our estimates are generous, as they do not consider factors such as memory constraints, and assume that the number of SGD steps is constant w.r.t.
model size, whereas some methods, e.g., Yao et al.
(2022), increase them linearly.
Thus,
GPTQ is the first highly-accurate post-training method that can scale to extremely large models.
Language Generation.
We begin our large-scale study by compressing the entire OPT and BLOOM model families to 3- and 4-bit.
We then evaluate those models on several language tasks including
WikiText2 (Merity et al., 2016)
(see Figure 1 and Appendix A.5),
Penn Treebank
(PTB) (Marcus et al., 1994)
(see Tables 2 and 3) and C4 (Raffel et al., 2020) (see Appendix A.5).
We focus on these perplexity-based tasks, as they are known to be particularly sensitive to model quantization (Yao et al., 2022).
On OPT models, GPTQ clearly outperforms RTN, by significant margins.
For example,
GPTQ loses 0.13 perplexity at 4-bit on the 175B model, while RTN drops 4.53 points, performing worse than the nearly 50 × smaller full-precision 2.7B model.
At 3-bit, RTN collapses completely, while GPTQ can still maintain reasonable perplexity, in particular for larger models.
BLOOM shows a similar pattern: the gaps between methods are however usually a bit smaller, indicating that this model family might be easier to quantize.
One interesting trend (see also Figure 1) is that larger models generally (with the exception of OPT-66B) appear easier to quantize.
This is good news for practical applications, as these are the cases where compression is also the most necessary.
[Table] [Table]
175 Billion Parameter Models.
We now examine BLOOM-176B and OPT-175B, the largest dense openly-available models.
Table 4 summarizes results across Wikitext-2, PTB, C4.
We observe that, at 4 bits, GPTQ models reach only < 0 .
2 lower perplexity than the full-precision versions, with a large gap to RTN results on OPT-175B. At 3-bit, RTN collapses, while GPTQ is still able to maintain good performance on most tasks, losing only 0 .
3 − 0 .
5 points for more than 5 × compression.
We note that GPTQ’s accuracy can be further improved via finer-granularity grouping (Park et al., 2022): with group-size 1024, GPTQ’s OPT-175B 3-bit WikiText2 PPL further improves from 8.68 to 8.45.
[Table]
Practical Speedups.
Finally, we focus on practical applications.
As an interesting use-case, we focus on the OPT-175B model: quantized to 3 bits, this model takes approximately 63GB of memory, including the embeddings and the output layer
[Table]
which are kept in full FP16 precision.
Additionally, storing the complete history of keys and values for all layers, a common optimization for generation tasks, consumes another ≈ 9 GB for the maximum of 2048 tokens.
Hence, we can actually fit the entire quantized model into a single 80GB A100 GPU
[Table]
which can be executed by dynamically dequantizing layers as they are required during inference.
(The model would not fully fit using bits.)
For reference, standard FP16 execution
requires
5x80GB GPUs, and
the
state-of-the-art 8bit
LLM.int
() quantizer
(Dettmers et al., 2022)
requires
3 such GPUs.
We focus on language generation, one of the most appealing applications of these models, with the goal of latency reduction.
Unlike LLM.int8(), which reduces memory costs but has the same runtime as the FP16 baseline, we show that our quantized models can achieve significant speedups for this application.
For language generation, the model processes and outputs one token at-a-time, which for OPT-175B can easily take a few 100s of milliseconds per token.
Increasing the speed at which the user receives generated results is challenging, as compute is dominated by matrix-vector products.
Unlike matrix-matrix products, these are primarily limited by memory bandwidth.
We address this problem by developing a quantized-matrix full-precision-vector product kernel which performs a matrix vector product by dynamically dequantizing weights when needed.
Most notably, this does not require any activation quantization.
While dequantization consumes extra compute, the kernel has to access a lot less memory, leading to significant speedups, as shown in Table 5.
We note that almost all of the speedup is due to our kernels, as communication costs are negligible.
[Table]
For example, using our kernels, the 3-bit OPT-175B model obtained via GPTQ running on a single
A100 is over 1 . 9 × faster than the FP16 version (running on 5 GPUs) in terms of average time per token.
More accessible GPUs, such as the NVIDIA A6000, have much lower memory bandwidth, so this strategy is even more effective: executing the 3-bit OPT-175B model on 2x A6000 GPUs reduces latency from 589 milliseconds for FP16 inference (on 8 GPUs) to 132 milliseconds, a 4 . 46 × reduction.
This is only ≈ 10% slower than 3-bit inference on the more expensive A100 GPU.
[Figure]
Zero-Shot Tasks.
While our focus is on language generation, we also evaluate the performance of quantized models on some popular zero-shot tasks, namely LAMBADA (Paperno et al., 2016), ARC (Easy and Challenge)
(Boratko et al., 2018) and PIQA (Tata & Patel, 2003).
Figure 4 visualizes model performance on LAMBADA (and see also “Lam” results in Table 4).
We observe similar behavior as before: the outliers are that 1) quantization appears “easier” across the whole spectrum of models at 4-bit, where even RTN performs relatively well, and 2) at 3-bit, RTN breaks down, while GPTQ still provides good accuracy.
We provide additional results in Appendix A.6.
S UMMARY AND L IMITATIONS
We have presented GPTQ, an approximate second-order method for quantizing truly large language models.
GPTQ can accurately compress some of the largest publicly-available models down toand 4 bits, which leads to significant usability improvements, and to end-to-end speedups, at low accuracy loss.
We hope that our method will make these models accessible to more researchers and practitioners.
At the same time, we emphasize some significant limitations: On the technical side, our method obtains speedups from reduced memory movement, and does not lead to computational reductions.
In addition, our study focuses on generative tasks, and does not consider activation quantization.
These are natural directions for future work, and we believe this can be achieved with carefully-designed GPU kernels and existing techniques (Yao et al., 2022; Wu et al., 2022).
E THICS S TATEMENT
Our work introduces a general method for compressing large language models (LLMs) via quantization, with little-to-no accuracy loss in terms of standard accuracy metrics such as perplexity.
Our method is task-agnostic, as it only uses a tiny amount of randomly-chosen data for calibration.
We therefore do not foresee any significant ethical implications arising directly from the technical details of our method.
However, one possible consideration is that our study focused on “leading accuracy” metrics that are standard in the literature, such as perplexity, which is essentially standard in the literature (Dettmers et al., 2022; Yao et al., 2022).
We believe a thorough study of the impact of compression upon secondary measures, and in particular bias effects (Bender et al., 2021) is warranted, and may be rendered easier through our work.
At the same time, our work makes inference on extremely large language models more accessible, for better or for worse.
We believe that, in time, such tools will become much easier to use and deploy, making the need to understand their power and limitations even more stringent.
R EPRODUCIBILITY S TATEMENT
In the Supplementary Materials, we provide code to reproduce our “Language Generation” experiments.
Concretely, this includes:
• Compressing all models from the OPT and BLOOM model families to 3 or 4 bits.
• Evaluating perplexity of the quantized models.
• Our 3-bit CUDA kernel together with a small benchmarking script for individual matrixvector products.
• A README file providing sample commands and information on how to run all scripts.
Code for the ZeroShot experiments and for running inference on compressed models will be released at a later date.
A A PPENDIX A. A DDITIONAL C OMPARISON WITH OBQ
We now provide an additional comparison between GPTQ and OBQ on BERT-base/SQuAD Rajpurkar et al
(2016) and OPT-125M/WikiText2, which is one of the largest models to which OBQ can be reasonably applied.
[Table] [Table]
[Table]
A. GPTQ R UNTIMES
Tables 7 and 8 provide runtime numbers for GPTQ on all models.
A. A DDITIONAL T RICKS & 2- BIT Q UANTIZATION
While our experiments in the main paper have focused exclusively on vanilla row-wise quantization
[Table]
we now show that GPTQ is also compatible with more advanced quantization tricks, leading to further improvements.
Specifically
[Table]
we investigate standard grouping (Alistarh et al., 2017; Park et al., 2022),
i
[Table]
e. applying independent quantization to groups of G consecutive weights.
This interacts very well with GPTQ
[Table]
as the group parameters can actually be determined during the quantization process of each layer, always using the most current updated weights.
The last row of Table(marked 3G) shows 3-bit results with group-size 1024 for BLOOM and OPT-175B. At the cost of only < 0 .
05 bits extra storage per weight, this reduces the accuracy drops by another 0 .
2 − 0 .points, bringing 3-bit even closer to FP16 performance.
[Table] [Table]
While we find grouping at 4-bit and further reducing the group-size at 3-bit to bring only rather minor gains, grouping actually makes it possible to achieve reasonable performance for extreme 2-bit quantization.
Table 10 shows results on WikiText2 when quantizing the biggest models to 2-bit with varying group-sizes.
Even at group-size 1024, the perplexity drops only by about 3 .points, with the accuracy loss quickly decreasing for smaller group-sizes to only ≈ 1 at group-size 64, corresponding to 2.5 bit.
Interestingly, we find that the performance of G = 32 is only 0 .
2 − 0 .points worse than vanilla 3-bit quantization, at the same memory consumption, which might also be interesting for practical kernel implementations.
In summary, we think that these results are a very encouraging first step towards pushing highly-accurate one-shot compression of very large language models even lower than 3-bit.
A. E XPERIMENT D ETAILS
This section provides additional details about our experiment setup, in particular regarding the model evaluation and the
A.4. E VALUATION
For language generation experiments, we calculate the perplexity, in standard fashion like Radford et al.
(2019), as follows:
First, the entire validation set is concatenated using two linebreaks as separators and encoded using the default HuggingFace tokenizer of each model.
Next, the sequence is split into non-overlapping segments of width 2048, the full context size of our models.
These are sent through the model to collect the log-probabilities corresponding to the next token each.
Their exponentiated average is the final perplexity we report.
For zero-shot tasks we follow the EleutherAI evaluation harness 2 in terms of data preprocessing and final score calculation.
We note that we evaluate all individual samples separately and thus do not apply any padding.
A.4. T IMING E XPERIMENT S ETUP
Our timing experiments are performed following the standard HuggingFace/accelerate 3 setup also used by the recent work LLM.int8() (Dettmers et al., 2022).
In this setting, the model is split by distributing chunks of consecutive layers across GPUs.
Importantly, in this setup the communication costs are minimal, < 5% of the total runtime even when working with 8 GPUs.
This means almost all of the reported speedups are due to our quantized-matrix full-precision vector product kernels.
We emphasize that the only difference between the FP16 baseline and our quantized models are the kernels used to perform the underlying matrix-vector products.
This means all overheads due to HuggingFace, attention or non-quantized operations like residuals or LayerNorms are exactly the same.
Consequently, our quantized models should benefit from more advanced distribution strategies (Zheng et al., 2022) or more efficient attention kernels (Dao et al., 2022) just as much as our baseline.
In general, our kernels target generative inference in the low batch-size setting (for simplicity, we consider only batchsize 1) where the underlying (close to) matrix-vector products are memorybound.
For non-generative and large-batch applications, operations may be compute- rather than memory-bound and our kernels thus not directly applicable.
Instead, one could simply decompress the matrix before performing the corresponding matrix-matrix calculations: this takes ≈ 1.5ms on an A100 and ≈ 3ms on an A6000 compared to 76ms/365ms for the subsequent OPT-175B FC2 layer computation with batchsize 16 × 1024 tokens.
Hence, for such applications our methods significantly reduce the required number of GPUs at very little computational overhead.
This is similar to recent work (Dettmers et al., 2022), but we achieve a 2 .
5 × higher compression rate.
A. A DDITIONAL L ANGUAGE G ENERATION R ESULTS
Tables 11, 12, 13 and 14 show additional results for language generation tasks.
[Table] [Table]
[Table] [Table]
A. A DDITIONAL Z ERO S HOT R ESULTS
This section contains additional results for zero-shot tasks.
[Table]
[Table] [Table]
175B
[Table]
RTN 59.
GPTQ 60.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
1B
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
7B 6.7B
[Table]
30B 66B 175B
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
GPTQ 39.
RTN 30.
GPTQ 36.
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]